Generative AI and LLM fundamentals

Predicting one token at a time. The model takes the tokens so far, runs them through a stack of transformer layers, and produces a probability distribution over its entire vocabulary for what comes next; a sampler picks one token from that distribution, appends it, and the whole forward pass runs again. There is no plan and no draft being revised. Everything that looks like reasoning is the consequence of each token being conditioned on all the tokens before it, including the ones the model itself just produced.

That single fact explains most behaviour candidates find surprising. A model cannot know how its sentence will end when it starts, it cannot revise a claim it has already emitted without saying so out loud, and asking it to think before answering genuinely changes the answer, because the intermediate tokens become part of the condition for the final ones.

Temperature rescales the distribution before sampling. Low values sharpen it so the highest-probability token dominates; high values flatten it so unlikely tokens get a real chance. It does not make the model more or less capable, and it does not make it more or less truthful — a confidently wrong distribution stays wrong at temperature zero, it simply becomes reliably wrong instead of variably so.

The practical reading is that temperature is a determinism and diversity control, not a quality control. Extraction, classification and anything you intend to parse want it near zero. Drafting and ideation, where you will look at several attempts and pick one, want it higher. Turning it down because the model is hallucinating is the wrong lever, and the fact that it sometimes appears to work is worth understanding: it only ever suppresses the tail.

Because temperature zero removes sampling randomness, not every source of nondeterminism. Floating-point addition is not associative, so the order in which a GPU reduces partial sums changes the last bits of a logit, and two logits that differ in the last bits can swap rank. Batching makes this concrete: your request is executed alongside whatever else arrived at the same moment, and the kernel chosen and the reduction order can differ between runs.

Beyond arithmetic, the provider's model is not frozen unless you pin a version, and mixture-of-experts routing adds its own batch-dependent behaviour. So treat temperature zero as "as repeatable as this stack gets", and if a test needs an exact string, assert on the parsed structure rather than on the bytes.

A logprob is the log probability the model assigned to a token it emitted, and where a provider exposes them they give you the one honest confidence signal available without a second call. The classic use is classification: ask for a single-token label and read the probability mass on each candidate, which lets you set an abstention threshold rather than accepting whatever came out.

They are also a cheap detector for the places a model was unsure — a sudden dip across a span of a generated answer is a reasonable trigger for review. What they are not is a truth signal. The model can be confidently wrong, so a high probability tells you the model was unsurprised by its own output, not that the output corresponds to anything real.

Four stages, and knowing which one you are waiting on is most of latency work.

Loading diagram…

Look at where the batch appears. Tokenisation and routing are cheap and roughly constant. The prefill stage processes your whole prompt in parallel and builds the attention cache for it, and its cost scales with prompt length — this is the part that grows when you paste ten pages of context. Then decode runs one token at a time, each step reusing the cache, so output tokens arrive at a fairly steady rate that depends on how many other requests share the device.

The consequence is that a long prompt delays the first token, while a long answer extends the tail. They are separate problems with separate fixes: trimming context or caching a shared prefix helps time to first token, and only asking for less output, or streaming so the reader starts sooner, helps the tail. Candidates who report one number for "latency" usually cannot say which of the two they measured.

Because it collapses every text task into one objective. An encoder-decoder split suits translation, where input and output are distinct sequences, but it needs task-specific heads and paired data. A decoder-only stack trained purely to predict the next token learns from any text at all, and then classification, extraction, summarisation and dialogue are all expressible as "continue this". That uniformity is what made scale pay: the same architecture and the same objective absorb more data and more parameters without redesign.

The cost is that the model has no separate representation of the input. Your instructions and the document you pasted occupy the same sequence, which is the structural reason prompt injection is hard to fix rather than a bug in any one product.

A base model is the raw next-token predictor. Give it a question and a plausible continuation is a list of similar questions, because that is what documents containing questions tend to look like. An instruction-tuned model has been further trained on instruction and response pairs, usually followed by preference tuning, so that the highest-probability continuation of a request is an attempt to satisfy it.

The distinction matters in interviews because it separates capability from behaviour. Base models are the better starting point for further fine-tuning on a narrow task, since you are not fighting an existing response style. Anything user-facing wants the tuned variant, and the tuning is also where refusal behaviour and format habits come from.