Haystack Pipelines for Search Applications

Haystack’s distinguishing decision is that connections between steps are declared explicitly and validated when the pipeline is built. That sounds like ceremony and is in fact the feature: a mismatch between what one component produces and what the next expects is an error at construction time rather than a confusing failure forty seconds into a run over ten thousand documents.

A component is a typed function object

A component declares its inputs, declares the type of its outputs, and implements a run method returning a dictionary of named outputs. That is the whole contract, and it is what makes the framework’s built-in components and yours interchangeable.

@component
class TruncateToBudget:
    """Drop retrieved documents until the total fits a token budget."""

    def __init__(self, max_tokens: int = 3000):
        self.max_tokens = max_tokens

    @component.output_types(documents=List[Document])
    def run(self, documents: List[Document]):
        kept, used = [], 0
        for doc in documents:
            cost = estimate_tokens(doc.content)
            if used + cost > self.max_tokens:
                break
            kept.append(doc)
            used += cost
        return {"documents": kept}

Writing your own is deliberately cheap, and that is the right expectation to have. The framework supplies the graph and a library of components; the parts specific to your problem — a budget rule, a domain filter, a scoring tweak — are twenty lines each and slot in beside the built-in ones with no special status.

Explicit wiring, checked before it runs

A pipeline is built by adding named components and connecting one component’s named output to another’s named input. The connection is checked for type compatibility as it is made.

pipe = Pipeline()

pipe.add_component("embedder",  TextEmbedder(model=EMBED_MODEL))
pipe.add_component("retriever", EmbeddingRetriever(document_store=store))
pipe.add_component("truncate",  TruncateToBudget(max_tokens=3000))
pipe.add_component("prompt",    PromptBuilder(template=TEMPLATE))
pipe.add_component("llm",       Generator(model=CHAT_MODEL))

pipe.connect("embedder.embedding",  "retriever.query_embedding")
pipe.connect("retriever.documents", "truncate.documents")
pipe.connect("truncate.documents",  "prompt.documents")
pipe.connect("prompt.prompt",       "llm.prompt")

result = pipe.run({
    "embedder": {"text": question},
    "prompt":   {"question": question},
})

Two things in that snippet are worth dwelling on. Inputs are supplied per component by name at run time, which is how a value like the user’s question reaches two components at different depths without being threaded through everything in between. And the connection strings name outputs and inputs explicitly, so a rename fails loudly instead of producing an empty result.

The comparison with chain-style libraries is exactly here. A pipe operator gives you brevity and defers every mismatch to run time. An explicit connection graph costs four extra lines and tells you at construction that the retriever emits documents while the next component was expecting strings. On a pipeline of eight components over a large corpus, that trade is not close.

Building a retrieval pipeline

Two pipelines, not one. This is the structural point most tutorials skip and the one that decides whether the project stays coherent.

1. An indexing pipeline. Converters to text, a cleaner, a splitter, an embedder, a writer to the document store. It runs as a job, not in a request, and it is where the expensive decisions live — chunk size, what metadata is attached, what the document identity is.
2. A query pipeline. Embed the query, retrieve, optionally rerank, filter, build the prompt, generate. It runs per request and should be fast.
3. One shared configuration between them. Both must agree on the embedding model, or you have embedded documents with one model and queries with another, and retrieval quietly returns noise. Define the model in one place both import. This is the single most common way to break a retrieval system without any error appearing.
4. Serialise the pipeline definition. Pipelines can be written out to and loaded from a serialised form, which means the production topology can be a reviewed file rather than whatever the code happened to construct.

If you also want keyword matching alongside vector similarity, that is a second retriever and a joining component rather than a different architecture — the reasoning for wanting both is in semantic versus keyword search, and adding a reranker afterwards is another component in the same chain.

Evaluation as part of the same pipeline

Because evaluators are ordinary components, evaluation is a pipeline built from the same pieces as production rather than a separate script that drifts. That is the practice worth adopting on day one, and it costs an hour if you do it before there is anything to migrate.

• Evaluate retrieval separately from generation. Two different failures with two different fixes. Recall at k over a set of questions with known relevant documents tells you whether the right chunk was even available; nothing about the generator matters if it was not.
• Keep a small labelled set in the repository. Fifty to a hundred questions with the document that answers each. This is the artefact, not the framework — it survives every rewrite and it is the thing you cannot generate on demand.
• Run it in continuous integration on every change to the query pipeline. Chunk size, k, the prompt and the model are all changes that can improve one metric and destroy another, and without a run you will notice weeks later.
• Record the numbers over time. A single evaluation score is nearly meaningless; a series across commits is how you tell an improvement from noise. Sample sizes and what counts as a real difference are covered in evaluation statistics.

Branching, loops and where it gets awkward

Pipelines support conditional routing — a component that sends its output down one of several connections based on a condition — and cycles, so a self-correcting retrieval loop is expressible. Both work. Both are also where the explicitness that pays off in a linear pipeline starts to cost, because the wiring for a branch with a join is more verbose than the equivalent conditional in ordinary code.

The honest guidance: if your flow is a search-shaped pipeline with one or two branches, Haystack is a good fit and the validation is worth it. If it is a genuine agent — an open-ended loop where the model picks the next action every turn — you are describing a state machine in a pipeline vocabulary, and a graph library or a plain loop expresses it better.

What Haystack is and is not for

It is for search and retrieval applications where the flow is known in advance, where the components are heterogeneous — converters, retrievers, rerankers, generators — and where the team values the topology being explicit enough to review. In that setting the extra four lines per connection buy real safety.

It is not a lightweight wrapper for a single model call. One prompt and one generation does not need a pipeline; it needs a function. Adopting the framework for that produces the worst version of every framework criticism at once: more concepts than code, and an abstraction between you and the prompt for no return.