Prompting and structured output, context and tokens, adaptation, evaluation, inference economics, injection defence, and running LLM features in production.
21 published across 7 topics.
Generative AI and LLM fundamentals57 short answers on one page, for revising rather than studying.Prompt design that survives a model upgrade, schema-constrained output, few-shot choices, and getting parseable results without a retry loop.
Constrain the decoder to a schema instead of asking for JSON in prose, reserve enough output tokens that a complete document fits, validate at the boundary, and allow one repair retry that quotes the exact violation back.
Pin the model version, keep the prompt to stable intent and a schema, and record every model-specific workaround as dated, deletable debt. The durable artefacts are the output schema and the eval set - they are what tells you whether the new model is better before you ship it.
The system prompt holds what is identical on every request - role, policy, output contract, tool schemas, examples. Anything varying by user or request, retrieved documents included, belongs in the turns below, to keep the cached prefix stable and untrusted text away from system authority.
Tokens versus characters, context-window budgeting, truncation and compaction strategies, prompt caching, and how long context degrades.
No. Cost and latency scale with what you send, accuracy over a long context is uneven rather than flat, and a corpus has no upper bound while a window does. Long context did change the job - it absorbs whole documents and forgives loose recall - but it replaces chunking, not retrieval.
Subtract the answer reservation and the fixed prompt first, give retrieval a token allowance rather than a passage count, and let conversation history have what remains under an explicit eviction policy - because history is the only term that grows every turn while everything else stays flat.
Write down requests per day, input tokens per request and expected output tokens, multiply by the provider's per-million rates, then test each assumption. Output tokens cost several times input, retries and agent loops multiply the input, and a cached prefix rewrites the arithmetic.
When to fine-tune rather than prompt or retrieve, LoRA and parameter-efficient methods, dataset curation, and evaluating whether it helped.
Compare it against a genuinely strong prompted baseline on a held-out set drawn from real traffic, using the task metric your product cares about, and run a broad regression set alongside to catch capability lost elsewhere. Falling training loss is evidence that training worked, not that the product improved.
Ask what is missing. If the answer changes when your documents change, you need retrieval; if the model knows enough but will not behave, format or specialise the way you need, you fine-tune. Prompting comes first either way because it costs an afternoon and tells you which of the other two you actually need.
LoRA freezes the base weights and trains a small low-rank update added to chosen projections, so you train a fraction of the parameters and can swap adapters per use case. It reshapes behaviour and format readily, installs facts poorly, and rank is the capacity dial with overfitting beyond it.
Building an offline eval set, LLM-as-judge and its failure modes, regression suites for prompts, and measuring hallucination honestly.
A judge is reliable for pairwise comparison against a concrete rubric and unreliable as an absolute scorer. It carries position, verbosity and self-preference biases and agrees with whatever the prompt implies, so validate it against human labels before trusting any number it gives you.
Harvest cases from real inputs and past incidents rather than inventing them, record for every case why it exists and what failure it guards, separate deterministic assertions from graded quality, and freeze a regression slice you never tune against.
Report two rates separately: ungroundedness, meaning claims the provided sources do not support, and factual error, meaning claims wrong about the world. Measure per claim, publish the abstention rate beside them, and name the query set that produced the numbers.
Token economics, streaming and time-to-first-token, batching and KV cache, quantisation trade-offs, and routing between models.
Split the measurement into time-to-first-token and inter-token latency, because they come from different phases. The first is driven by prompt length and queueing and is fixed with shorter prompts, prefix caching and streaming; the second is the decode loop, fixed only by generating fewer tokens.
Decide from the task. For grounded work over your own corpus, quality is set by what reaches the context, so retrieval dominates model size and a smaller model usually suffices. Keep the larger model for long multi-step reasoning, and treat routing as a second system to evaluate and monitor.
Batching amortises the memory traffic of reading model weights across many requests, so throughput rises far faster than per-step time does. Continuous batching lets requests join and leave each iteration, but the gain is paid for in tail latency, which is why interactive and bulk traffic want separate pools.
Indirect prompt injection, data exfiltration through tools, sandboxing an agent's capabilities, PII handling, and what a guardrail cannot do.
Decide what may cross the provider boundary and redact before it does, checking the provider's retention and training-use terms rather than assuming them. Then treat the debugging trace as a second copy of user data, with its own retention, access control and deletion obligations.
Indirect prompt injection is when instructions hidden in content the model reads are followed as if they came from you. Instructions and data share one token sequence, so there is no complete fix and the defence is architectural: least-privilege tools, filtered output, and human approval for side effects.
Assume the agent's context can be manipulated and control the exits instead. Scope each tool to the least data it needs, resolve the caller's identity in the executor rather than from a model-supplied argument, treat generated arguments as hostile input, and allow-list egress destinations.
Versioning prompts as artefacts, canarying a model change, tracing and cost observability, and surviving a provider deprecation.
Inventory everywhere the model is coupled to behaviour, then use the eval set as your migration harness, shadow the candidate on live traffic to compare without user exposure, and re-measure cost and latency because both shift. Use the migration to delete instructions that only compensated for the old model.
Treat the prompt as a versioned artefact under review, run your eval set in CI against a pinned model version, canary on a slice of traffic with the version recorded per request, and keep rollback a config change rather than a deploy. Pinning is what makes a regression attributable to your edit.
Trace every request with prompt and model versions, token counts and cost, then add sampled groundedness grading, refusal and retry rates, validation failures and implicit user signals. Alert on countable rates, because an averaged quality score moves too little and too late to page anyone.