Tokens versus characters, context-window budgeting, truncation and compaction strategies, prompt caching, and how long context degrades.
3 questions in Generative AI & Large Language Models.
No. Cost and latency scale with what you send, accuracy over a long context is uneven rather than flat, and a corpus has no upper bound while a window does. Long context did change the job - it absorbs whole documents and forgives loose recall - but it replaces chunking, not retrieval.
Write down requests per day, input tokens per request and expected output tokens, multiply by the provider's per-million rates, then test each assumption. Output tokens cost several times input, retries and agent loops multiply the input, and a cached prefix rewrites the arithmetic.
Subtract the answer reservation and the fixed prompt first, give retrieval a token allowance rather than a passage count, and let conversation history have what remains under an explicit eviction policy - because history is the only term that grows every turn while everything else stays flat.