LlamaIndex for Document-Heavy Apps
10 min read · updated August 4, 2026
LlamaIndex is two libraries wearing one name. The ingestion half — readers, node parsers, transformations, metadata, incremental updates — is genuinely good and worth adopting. The query half is a convenience layer with strong defaults that most applications outgrow. Knowing which half you are in tells you what to keep.
The shape of the library
The data flow has been stable through several major versions, and it is the thing to hold on to when the class names move:
source files -> Reader loads bytes into Documents -> Node parser splits Documents into Nodes (chunks + metadata) -> Transformations extract metadata, embed -> Vector store persists nodes and vectors -> Retriever nodes for a query -> Postprocessors rerank, filter by score, expand windows -> Response synthesiser builds prompts, calls the model, assembles an answer
Everything the library does sits somewhere on that line. When something behaves unexpectedly, the useful question is always “which stage?”, and the answer is nearly always the node parser or the response synthesiser.
Ingestion is the part worth having
The five-line quickstart hides why. What you are actually buying is not the loop over a folder — you can write that — it is the accumulated handling of formats that are awkward in specific ways: PDFs where the text layer is out of order, HTML where the content is 8% of the bytes, spreadsheets where a row is the semantic unit, slide decks where the notes matter more than the slide.
# The quickstart. Fine for a prototype and misleading about what
# a real ingestion pipeline needs.
documents = SimpleDirectoryReader("./data").load_data()
index = VectorStoreIndex.from_documents(documents)
engine = index.as_query_engine()
print(engine.query("What is the refund policy?"))The second thing worth having is incremental ingestion. A pipeline that hashes each document and skips unchanged ones turns a forty-minute re-index into a thirty-second one, and it is the difference between re-indexing on every deploy and re-indexing quarterly because it hurts. Get this working before your corpus is large enough to make it urgent; see index freshness for the wider problem.
Nodes, metadata and the ID that matters
A node is a chunk plus metadata plus relationships to its neighbours and its parent document. Three things about nodes decide whether the system is maintainable a year in.
- Metadata is a filter and a prompt input at once. Fields on a node can be used to filter retrieval, and by default some of them are also injected into the text the model sees. That second behaviour surprises people: a verbose metadata field silently eats context on every retrieved chunk. Control which fields are visible to the model and which are for filtering only — the library has explicit settings for this, and they are worth finding before you wonder where your context window went.
- Node identity determines whether updates work. If node identifiers are random per run, every re-ingestion duplicates your corpus. Derive them from something stable — source path plus chunk index, or a hash of the content — so re-ingesting a changed document replaces its nodes instead of adding a second copy.
- Relationships enable the trick worth knowing. Nodes know their neighbours, which lets you retrieve on small precise chunks and then expand to the surrounding window before answering. Small chunks retrieve better; large chunks answer better; this is how you get both, and it is the single highest-value feature in the library.
The query side, and its defaults
A query engine bundles a retriever, some postprocessors and a response synthesiser. It works immediately, which is its virtue and its problem: the defaults are opinions, and they are invisible.
The default retrieval count is small — a handful of chunks. The default synthesiser mode stuffs those chunks into one prompt, but other modes exist that call the model once per chunk and combine the answers, and if you switch to one of those without noticing, your cost per query multiplies by the number of chunks. That is the single most expensive default in the library, and it is one line of configuration.
The prompts are templates you can read and replace. Do read them. A large share of “the model ignored my instruction” reports in document-QA systems are the framework’s own template overriding what the user thought they were sending, and the fix is to substitute your own — which also makes the prompt something you can version like the rest of your code.
Where the abstractions stop helping
Four reliable signals that you have reached the end of the useful part.
| Signal | Description |
|---|---|
| You need hybrid retrieval you control | Combining vector similarity with keyword search and a rank fusion you can tune. The library has hybrid support, but once you are tuning fusion weights per query type you are writing retrieval logic anyway, and doing it against your store's own API is clearer. |
| Multi-tenancy is a hard requirement | Per-tenant filtering must be enforced somewhere that cannot be forgotten. Relying on every call site to pass the right metadata filter is one missed call from a cross-tenant leak. That belongs in your own data access layer, not in a query engine's arguments. |
| You are fighting the response synthesiser | When you are subclassing it, or post-processing its output to undo something it did, stop. Retrieval returns a list of chunks; building a prompt from a list is ten lines you fully understand. |
| Evaluation needs the intermediate values | Measuring retrieval separately from generation means you need the retrieved set, its scores and the exact prompt, per query, as data. If getting them requires callbacks and introspection, the abstraction is now costing you the thing that matters most. |
None of those means abandoning the library. All four are reasons to keep ingestion and write the query path yourself — which is a two-hundred-line file, not a rewrite. If you cannot evaluate retrieval independently of generation, read RAG evaluation first, because that is the capability being traded away.
How to structure a project that lasts
- Put ingestion in its own module with a CLI. It should be runnable as a job, not only from a notebook, and it should report how many documents it saw, skipped and updated.
- Own your node identifiers and your metadata schema. Write them down as a typed structure in your own code. This is the contract between ingestion and retrieval, and it is what makes either side replaceable. See document storage schema.
- Use a real vector store from the start. The in-memory default is for the tutorial. Moving later means re-embedding everything, which costs real money once the corpus is large.
- Wrap retrieval in one function of your own. Signature: query in, list of chunks with scores and sources out. Whether the body calls a LlamaIndex retriever or your store directly becomes an implementation detail on the day you want to change it.
- Build the prompt where you can see it. Even if you keep the query engine, log the final prompt. Every debugging session in a document-QA system starts with what the model was actually shown.