Generative AI & Large Language Models

What this area actually covers

This area is the engineering of product features whose behaviour is produced by a language model rather than by code you wrote. You define what the feature should do, you construct the input that will make a model do it, you constrain the output enough that the rest of your system can consume it, you measure whether the whole thing is any good, and you run it inside a latency budget and a cost budget that somebody will hold you to. The model itself is a dependency, usually one you did not train and cannot inspect.

That framing is deliberate, because it locates the difficulty correctly. Almost nobody working in this area is training a foundation model, and the small number who are belong to a different discipline with different prerequisites. What the majority do is substantially closer to integration engineering with an unusual dependency: a dependency that is probabilistic, whose behaviour changes when the provider updates it, whose failures are fluent and plausible rather than loud, and which will do whatever the text flowing through it tells it to do. Every hard problem in this section descends from one of those four properties.

The boundary with adjacent areas is worth drawing now, because the vocabulary overlaps and the interviews do not. Retrieval — how you find the right documents to put in front of the model — is a large enough engineering surface that Preptima keeps it separate in RAG, Retrieval & Agents, and in a real system it is usually where most of your effort goes. Training models from data you own, choosing between gradient boosting and a neural network, and reasoning about leakage and class imbalance belong to Machine Learning. The infrastructure for versioning, deploying, monitoring and paying for models is MLOps & ML Platform, and while the LLMOps subsection here covers what is specific to language models, the general discipline lives there.

There is also a housekeeping point about this site. The older Data & AI Engineering section still carries an LLMs & RAG subsection alongside MLOps, ML Fundamentals and Statistics & Experimentation from earlier authoring waves. Those pages have not been removed and are still worth reading; they are simply a single-subsection treatment of material that has since been given four sections of its own. If you find the same topic in both places, the version here is the more detailed one and the older one is the compressed introduction.

Three things get wrongly bundled into this area. Prompt engineering as a standalone job is one — prompting is a technique inside the work, not the work, and a role defined by it alone has not survived contact with hiring. Data science is another: the instinct to treat an LLM feature as a modelling problem with a train and test split leads people to skip the part where a product decision has to be made about what "correct" means. And model research is the third; reading papers about architectures is interesting and is almost entirely orthogonal to whether your summarisation feature returns parseable JSON under load.

The seven areas underneath

The seven subsections follow the order in which the problems actually arrive. You start by getting the model to do the thing at all, which is prompting and structured output. You immediately hit the window, which is context and tokenisation. You wonder whether to adapt the model, which is fine-tuning. Then the three that separate a demo from a product: evaluation, cost and latency, and safety. LLMOps is last because it is what you need once the feature exists and has to keep existing.

Prompting & Structured Output

This covers writing instructions that keep working after a model upgrade, choosing few-shot examples that generalise rather than teaching the model your examples, constraining output to a schema so the caller can parse it, and the difference between a prompt that usually works and one that works on the tail of your input distribution. It exists as its own subsection because it is where every reader's intuition is strongest and least reliable: everybody has written a prompt, and comparatively few have written one that holds up when the input is a document nobody anticipated. What you will find here is the mechanics — schema-constrained decoding, why a retry loop around a parse failure is a symptom rather than a fix, and how to write an instruction that fails loudly instead of guessing.

Context & Tokenisation

Tokens rather than characters or words, how a context window is actually spent, what truncation and compaction strategies cost you, prompt caching and what invalidates it, and the well-documented degradation of retrieval accuracy as relevant information sits further from the edges of a long context. This is separate because tokens are simultaneously the billing unit, the latency driver and the correctness constraint, and a reader who has not internalised that treats the context window as free storage. Expect material on budgeting a prompt as an allocation problem: system instructions, retrieved material, conversation history and expected output all competing for one window.

Fine-tuning & Adaptation

When adaptation beats prompting or retrieval, what parameter-efficient methods such as low-rank adaptation change and what they do not, how to curate a dataset that is small enough to build and representative enough to help, and — the part people skip — how to tell afterwards whether it worked. It is its own area because it is the most commonly proposed and least commonly justified intervention in the field. Fine-tuning reliably teaches format, tone and a narrow task shape; it is a poor way to add knowledge, which is retrieval's job, and a poor first response to a quality problem you have not yet measured. The interview version of this is usually a question about which of the three levers you would reach for and why.

Evaluation & Guardrails

Building an offline evaluation set from real inputs, choosing between programmatic checks, human review and a model acting as judge, understanding how a judge model inherits its own biases, running prompts through a regression suite the way you would run code through tests, and measuring hallucination in a way that survives scrutiny. This is the most important subsection on the page and it is the one candidates are least prepared for. It exists separately because evaluation is not a phase at the end; it is the instrument you need before any other change is meaningful, and a team without it is tuning by anecdote.

Inference, Cost & Latency

Token economics per request and per user action, streaming and why time-to-first-token is the number users feel, batching and the key-value cache, quantisation and what accuracy it trades for throughput, and routing between a cheap model and an expensive one based on request difficulty. Separate because in this area cost and latency are design constraints rather than optimisations, in a way that is unusual even for engineers used to performance work: a design that triples quality and quadruples cost per request may be an unshippable design, and knowing that before you build it is the skill.

Safety & Prompt Injection

Indirect prompt injection through content the model reads, exfiltration of data through a tool the model can call, sandboxing an agent's capabilities, handling personal data in prompts and logs, and the honest limits of a guardrail. This is its own subsection because it is the area where the correct answer is most often uncomfortable: there is no known complete defence against prompt injection, and the engineering response is to reduce what a compromised model can do rather than to hope it cannot be compromised. The material is architectural — capability limits, human confirmation on consequential actions, separating trusted instructions from untrusted content — rather than a list of filters.

LLMOps & Deployment

Treating a prompt as a versioned artefact with a change history and an owner, canarying a model or prompt change behind an evaluation gate, tracing a request through its stages so you can attribute a bad answer, cost observability at the level of a feature rather than an invoice, and surviving a provider deprecating the model you built on. Last for a reason: these are the problems of a feature that already exists and is now load-bearing. It is separate from the general MLOps section because the artefacts differ — you are versioning text and configuration rather than weights, and your dependency is somebody else's roadmap.

Where it sits in a real system

An LLM feature is a stage in a request path, and drawing that path is the fastest way to see where the engineering actually lives. A user action arrives at your service. You assemble a prompt from static instructions, whatever conversation state you keep, and whatever material you retrieved. You call a provider. You parse and validate what comes back, because a downstream system needs a shape rather than prose. Then you either act on it, or you refuse, and you record enough about the whole exchange to answer "why did it say that" a week later.

flowchart TD
    A[User action] --> B[Assemble prompt<br/>instructions plus retrieved material]
    B --> C[Provider call<br/>tokens in, tokens out]
    C --> D[Validate against schema]
    D -->|parses| E[Act on the result]
    D -->|fails| F[Repair or refuse]
    E --> G[Trace, cost and eval capture]
    F --> G
    G --> H[Offline eval set grows]
    H --> B

The edge worth studying is the one from capture back into the prompt. Everything upstream of it is a demo; the loop that turns real traffic into an evaluation set, and the evaluation set into a justified prompt change, is the part that makes it a product. The Validate branch matters too: a team that has no defined behaviour for the failing branch has a feature that occasionally returns prose into a code path expecting fields.

Two integration realities shape designs more than any prompt technique. The first is that the provider is a shared, rate-limited, occasionally slow external dependency, which means every resilience pattern you know applies here — timeouts, retries with jitter, circuit breaking, a degraded mode that does something useful without the model. The second is that responses are slow enough that synchronous request-response is often the wrong shape: streaming for anything a human waits on, and a job with a status for anything longer.

The vocabulary you will hit on day one

Interviews in this area lean on a small vocabulary that is used loosely in public writing and precisely by practitioners. Getting these straight is cheap and the difference is audible.

The last row is the one to hold onto. In a conventional system you defend against injection by separating code from data — parameterised queries exist precisely for that. A language model has one channel, and everything in it is potentially instruction. That single architectural fact is why this section treats safety as design rather than validation.

What an LLM feature costs, worked through

Cost questions get answered with hand-waving because people reason about a single call rather than a feature. Work it as a unit economics problem instead, carrying the units and stating every assumption, exactly as you would size a system design.

Assumptions, all stated and all arguable
  System instructions                        800 tokens
  Retrieved material, 6 chunks                2,400 tokens
  Conversation history carried                1,000 tokens
  Generated answer                              500 tokens
  Calls per user action                            2   one draft, one check

Per action
  Input tokens   = 800 + 2,400 + 1,000        = 4,200
  With 2 calls   = 4,200 x 2                  = 8,400 input tokens
  Output tokens  = 500 x 2                    = 1,000 output tokens

What this tells you before you know any price
  Input dominates output roughly 8 to 1, so the retrieval budget is the cost lever
  Caching the 800-token instruction block helps far less than cutting 6 chunks to 3
  The second call doubles cost and adds a full round trip to perceived latency
  A 10x traffic increase is a 10x bill - there is no economy of scale on tokens

No price appears in that calculation and it is still the more useful analysis, because prices change monthly and the ratios do not. The interview-grade observation is the third line: a verification call is the most commonly added and least commonly costed quality improvement in the field, and being able to say what it costs in both money and time-to-first-token is what distinguishes an engineer from an enthusiast.

Who does this work

The most common title is some variant of AI or LLM application engineer, and the most common actual path into it is a backend engineer who was handed an AI feature. That person spends their week doing recognisable backend work with an unusual dependency: building the prompt assembly layer, wiring retrieval, handling provider timeouts, defining what happens when the output does not parse, and arguing with product about what the feature should refuse to do. Their hardest week is the one where quality is "sometimes bad" and they have no evaluation set with which to find out why.

Machine learning engineers move across from modelling work and bring genuinely transferable instincts — held-out sets, metric discipline, suspicion of a number that looks too good — and one recurring blind spot, which is expecting the model to be the thing they tune. Applied research engineers sit further towards the model, doing adaptation and evaluation methodology, and are a much smaller population than job-board noise suggests. AI platform engineers build the shared substrate: gateways, prompt registries, evaluation harnesses, cost attribution, and the guardrail layer everyone else consumes. Alongside all of them, a technical product manager typically owns the definition of correct behaviour, which is not a formality — in this area the specification of "good output" is a genuine product decision and engineers cannot make it alone.

The row people underrate is the last. A great deal of apparent model failure is an unstated specification: nobody decided whether an uncertain answer should be given with a caveat or withheld, so the model decides, differently each time.

Demand, adoption and how that is changing

Demand is very high, and the reason is specific rather than atmospheric: a large number of organisations have committed publicly to shipping AI features and comparatively few have people who have shipped one and kept it working. That gap is what is being hired for. Note the shape of it, because it determines what interviews ask. The scarce skill is not access to models, which is a credit card, nor prompting, which is quickly learned. It is the ability to say whether a feature is good enough to ship, to demonstrate that a change improved it, and to run the thing inside a budget.

Three shifts are visible and worth stating carefully. The first is that the interview has moved from capability to evidence: two years ago describing a retrieval-augmented architecture was a differentiator, and now the differentiating answer is how you knew it worked. The second is consolidation of the tooling layer — the early proliferation of frameworks has thinned, and teams increasingly build directly against provider APIs with a thin internal abstraction, which means framework fluency has depreciated faster than the underlying concepts. The third is that cost discipline arrived: the first wave of features was funded as experiment, and the current wave is expected to justify its unit cost, which has made the inference and latency material a hiring topic rather than an afterthought.

The honest counterweights matter. A meaningful share of demand is for one specific thing — a retrieval-based assistant over internal documents — and a candidate whose experience is only that will find the market narrower than the headline suggests. Capabilities that were projects last year become a provider feature this year, so any expertise that consists of working around a model limitation has a short half-life. And the people being hired above entry level are being hired for judgement about whether to use a model at all, which is not acquirable by tutorial. What holds its value is evaluation, cost reasoning, failure handling and security thinking, none of which are model-specific.

What makes it hard

Evaluation is the bottleneck, and it is a bottleneck of a kind most engineers have not met. In ordinary software you assert an expected value; here the correct output is one of many acceptable outputs, the difference between two prompt versions is a distribution shift rather than a pass or a fail, and the only way to know whether last week's change helped is to have built a measurement instrument first. Teams that skip this do not notice they have skipped it, because the feature demos well. They discover it when a model upgrade quietly changes behaviour and nobody can say by how much.

Nondeterminism breaks the habits that make you productive elsewhere. You cannot debug by reproducing, because the same input may not produce the same output. You cannot fix a bug with confidence, because you can only observe that a case now passes, not that a class of cases does. Regression is a statistical claim. Engineers who insist on treating this as a bug to be eliminated — pinning every knob and hoping for reproducibility — spend a lot of effort for a weak guarantee; the productive response is to accept variance and measure distributions.

Cost and latency are constraints, not optimisations. A design that improves quality by adding a second model call has doubled the bill and added a round trip that a waiting user feels. Retrieval that fills the window improves grounding and increases both cost and the chance that relevant material is ignored. Almost every quality lever in this area has a cost or latency counterweight, which makes the work unusually trade-off shaped and is why it interviews well alongside system design.

Failures are fluent. A conventional bug announces itself with an exception or a wrong number; an LLM failure is a confident, well-formed, plausible falsehood that no type system will catch and that a casual reviewer will accept. This changes what you must build: grounding requirements, citation checks, uncertainty handling, and a defined refusal behaviour, all of which are product decisions before they are engineering ones.

Prompt injection has no complete fix, and pretending otherwise is the actual risk. Because instructions and data share one channel, any untrusted text that reaches the model — a web page it reads, a document it summarises, an email it processes — may carry instructions. The consequences scale with capability: a model that can only produce text can be made to produce wrong text, while a model that can call tools can be made to act. The engineering response is architectural. Limit what the model can do, require human confirmation for consequential actions, treat all retrieved content as hostile, and assume that any data reachable by the model is data an attacker may be able to exfiltrate. An interviewer asking about injection is usually testing whether you reach for a filter or for a capability boundary, and the second answer is the stronger one.

The dependency is somebody else's product. The model you built and evaluated against will be deprecated, updated, re-priced or rate-limited on a schedule you do not control. That is an unusual amount of exposure to place at the centre of a feature, and it is why prompt versioning, a maintained evaluation suite and a tested fallback path are not process overhead but the mechanisms by which you survive a Tuesday announcement.

Why study it

Study it if you build products and want to be the person who can tell whether an AI feature is real. That skill is currently scarce and unusually legible in an interview: a candidate who can describe an evaluation set they built, a change it caught, and the cost per action of the feature they shipped is answering a question most applicants answer with enthusiasm. Study it if you like problems where the difficulty is in the specification — deciding what correct means, and what the system should do when it does not know — because that is genuinely most of the work.

It also transfers better than it looks. Evaluation discipline, unit-cost reasoning, and designing for a dependency that fails plausibly rather than loudly are all skills that improve ordinary engineering. Several of the strongest people in this area came to it from performance work or from platform engineering rather than from machine learning.

Be honest about who should not prioritise it. If you want to work on models themselves — architectures, training, the research end — this is not that discipline and the route there runs through mathematics and through Machine Learning rather than through prompting. If your target roles are in classical modelling, forecasting or recommendation, the fashionable material here will not help you and the machine learning section will. If most of your intended value is retrieval over an organisation's own documents, read RAG, Retrieval & Agents first, because the model is the least interesting component in that system. And if you have never operated a production service, the failure-handling and cost material will read as abstract, because it is the residue of specific incidents.

Your first hour

Build an evaluation set. Not a feature, not a framework comparison — an evaluation set, because it is the artefact that everything else depends on and the one nobody has.

Take any task you can define in a sentence: extract the invoice number, due date and total from a supplier email; classify a support message into one of five categories; summarise a meeting transcript into decisions and owners. Collect twenty real inputs, including at least five that are awkward — the truncated one, the one in another language, the one where the answer genuinely is not present. For each, write down the acceptable output, and where several outputs would be acceptable write down the property that makes an output acceptable rather than a single string.

Then write two prompts that differ in exactly one way — one asks for JSON in prose, the other constrains the output to a schema, say — run both across all twenty inputs, and record the results in a table.

Case  Input                          Prompt A         Prompt B         Note
1     Standard invoice email         pass             pass
2     Invoice as a scanned image      fail no text     fail no text     Retrieval problem, not prompt
3     Two invoices in one email       wrong - merged   pass             A found one total
4     Total present, no due date      hallucinated     refused          B correct: absence is an answer
5     Amount in another currency      pass             pass             Neither stated the currency

Passes: A 12/20, B 16/20
The four cases B wins are all absence or ambiguity, not formatting

The artefact is that table, and its most valuable column is the last. Two numbers tell you which prompt is better; the notes tell you why, which is the only thing that tells you what to change next. Case 4 in that sketch is the whole discipline in one line: the improvement was not better extraction but a defined behaviour for "the answer is not there".

If you have a second hour, add the cost column. Record input and output tokens for each case, total them per prompt, and work out the cost per hundred actions. You will now have done, at small scale, the exact analysis a senior interviewer is probing for — a quality claim with evidence and a price attached.

What this is not

It is not prompt engineering. Prompting is one technique among several, it is the fastest part of the work to learn, and a role defined by it has not proved durable. Candidates who prepare only prompting arrive fluent in the least differentiated skill in the section.

It is not machine learning, and conflating them costs people interviews in both directions. Nothing here involves choosing a model family from your data, reasoning about bias and variance, or worrying about leakage between training and test sets. Those are real skills, they are examined in a different loop, and they live in Machine Learning.

It is not retrieval, although in practice the two arrive together. If your feature answers questions over a corpus, most of your quality problem is which chunks reached the model, and swapping the model will not fix a retrieval failure. Diagnosing which stage failed is covered in RAG, Retrieval & Agents.

It is not model research. Understanding attention and transformer architecture is worth an afternoon for the vocabulary and is nearly orthogonal to shipping a feature. The people who work on architectures are a small, specialised population with a different entry route.

And it is not a field where the model is the interesting part. The interesting parts are what you put in front of it, what you refuse to let it do, how you know it is working, and what it costs. A candidate whose experience is a list of models they have called has described a dependency, not a system.

The differentiating answer in an LLM interview is not which model you used. It is how you knew your change made the feature better, and what that improvement cost per request.