AI models have a context window: the amount of text they can read and reason about in one go. For years, this window was small enough that developers had to be clever about what they put in. Now that Gemini offers up to one million tokens in its API and Claude supports 200,000 token contexts, the question has changed. You can often just put everything in. But should you?
Long context, RAG, and memory are three different strategies for giving models the information they need. Choosing the right one depends on how much data you have, how often it changes, how much accuracy matters, and how sensitive your costs are.
Long context: just put it all in
Long context is the simplest approach. You take a document, a conversation history, a knowledge base, or any other source of information and include it directly in the prompt. The model reads all of it before responding.
This works well when the document is small enough to fit, when you need the model to consider every part of it, and when the task happens infrequently enough that cost is not a driving constraint. Gemini's one-million-token context window and Anthropic's 200K window make it practical for documents that would previously have required chunking.
- Best for: single document analysis, contract review, long transcript summarization.
- Strength: the model sees everything and cannot miss relevant content.
- Weakness: expensive at scale, slower latency, cost grows linearly with document length.
- Avoid if: you are processing hundreds of documents per hour or cost is a primary constraint.
RAG: retrieve then generate
Retrieval-augmented generation splits the process in two. First, it embeds your documents into a vector database. When a user query arrives, it retrieves the most semantically relevant chunks and includes only those in the prompt. The model generates its response based on the retrieved context.
RAG is more scalable than long context because the retrieval step is fast and the prompt only includes relevant sections rather than the whole corpus. But it introduces a new failure point: retrieval quality. If the retrieval step misses the most relevant chunk, the model generates a response without the information it needs, often without flagging the gap.
- Best for: large knowledge bases, support bots, code search, internal documentation search.
- Strength: scales to millions of documents without ballooning token cost on every query.
- Weakness: retrieval can fail on ambiguous or indirect queries.
- Avoid if: the task requires synthesizing across the entire corpus rather than finding specific facts.
Memory: storing facts between sessions
Memory is different from both long context and RAG. It is about persisting information about users, preferences, projects, or context across separate conversations or sessions. Instead of reading a long document, the model recalls facts it has stored about this specific user or workflow.
Memory is most relevant in AI assistants, co-pilots, and products where the same user interacts over time and expects continuity. It is less relevant for document search or one-off analysis tasks.
- Best for: AI assistants, personalized products, ongoing project co-pilots.
- Strength: reduces repetition, makes the AI feel like it knows you.
- Weakness: requires careful management of what gets stored and when it expires.
- Avoid if: the use case is stateless (each query is independent) or privacy requires no persistence.
Vector databases: the infrastructure behind RAG
RAG requires a place to store and retrieve document embeddings. Pinecone is a managed vector database with strong tooling for production use. Weaviate and Qdrant are open-source options with self-hosting flexibility. Chroma is commonly used for local development. The choice of vector database usually matters less than the quality of the chunking and embedding strategy.
How to choose
| Situation | Best approach | Key reason |
|---|---|---|
| Analyze one long document | Long context | Model sees everything without retrieval gaps. |
| Search across 1,000 documents | RAG | Scales without sending everything in every prompt. |
| Personalized assistant | Memory | Continuity across sessions matters. |
| High-volume document processing | RAG with caching | Cost control at scale. |
| Regulatory review of a contract | Long context | Missing a clause could be critical. |
Long context is simple but expensive at scale. RAG is scalable but can miss things. Memory is powerful but requires discipline about what to store.
Dera's recommendation
Start with long context for prototyping and low-volume tasks where simplicity beats cost. Move to RAG when document scale or query volume makes token cost prohibitive. Add memory when your product has returning users who benefit from continuity. The approaches are not mutually exclusive and many production AI products use combinations of all three.