Walk me through what happens between the source you write and the instructions the CPU runs, and where does a JIT fit into that?

The pipeline from text to instructions

A compiler front end reads your source as characters and turns it into structure. Lexical analysis groups characters into tokens, so total += 1 becomes an identifier, a compound-assignment operator and a literal. Parsing checks those tokens against the grammar and builds a syntax tree; this is where a missing brace is reported. Semantic analysis walks that tree with a symbol table, resolving names to declarations and checking types, which is where "cannot convert String to int" comes from. Everything up to here is language-specific and knows nothing about a CPU.

The middle end lowers the tree to an intermediate representation — a simpler, more explicit form, usually with control flow made into basic blocks and expressions broken down into single operations. Optimisation happens on the IR because it is easier to reason about than either the source or the target: constant folding, dead code elimination, common subexpression elimination, inlining, loop transformations. The reason an IR exists at all is combinatorial. With n languages and m targets you would otherwise need n × m compilers; with a shared IR you need n front ends and m back ends, which is why LLVM is used by compilers for languages whose authors never coordinated.

The back end emits code for a specific architecture, and this is where register allocation and instruction selection happen. Then two steps that are not the compiler and are commonly attributed to it. The linker combines object files and resolves symbols across them, which is why an undefined-reference error looks different from a compile error and arrives later. The loader, at run time, maps the executable into memory, and for a dynamically linked program resolves the remaining symbols against shared libraries, which is why a program that compiled and linked fine can still fail on startup for a missing .so or DLL.

Compilation, interpretation, and what sits between

Ahead-of-time compilation does all of the above before you ship, producing machine code for a known target. C, C++, Rust and Go work this way. You get no startup cost and no runtime compilation overhead, and you pay for it by having to decide everything statically — the compiler must generate code that is correct for every input the program might receive, because it will never see the actual ones.

A pure interpreter walks a representation of the program and executes it directly, deciding at each step what to do. It starts instantly and is portable, and it is slow, because the dispatch work is repeated on every execution of the same instruction rather than done once.

Most managed languages sit in between and it is worth being precise about the split. javac compiles Java source to bytecode, a portable instruction set for an abstract stack machine, and that is the end of the ahead-of-time story — bytecode is not machine code and the JVM will not execute it directly on hardware. At run time the JVM loads and verifies each class, then begins interpreting the bytecode while counting how often methods and loop back-edges execute. CPython does something structurally similar: source is compiled to bytecode, cached in a .pyc, and executed by an evaluation loop, with no native code generated in the default configuration.

What a JIT knows that an ahead-of-time compiler does not

A just-in-time compiler compiles bytecode to machine code while the program is running, and its advantage is not that it compiles later but that it compiles with the profile in hand. It knows which branch was taken 99% of the time, which method was hot, which types actually flowed through a call site, and what the loop bounds really were. A static compiler can only guess at all of that.

The most valuable use of that knowledge is inlining virtual calls. In payment.charge(amount) where charge is an interface method, a static compiler must generally emit an indirect call, because any implementation could be there. A JIT can observe that in this program only one implementation has ever appeared, inline its body directly, and then optimise across the boundary it has just erased — which usually matters more than the call overhead saved. The speculation is guarded: the compiled code checks that the receiver's type is still the expected one.

When a guard fails, the runtime deoptimises. It discards the compiled code, transfers execution back into the interpreter at the equivalent point with the same state, and recompiles later with the wider set of types. This is the mechanism behind performance behaviour that otherwise looks inexplicable: a service that is fast in staging and slower in production because production loads a second implementation of an interface, or a method that was fast for an hour and then regressed after an unusual input arrived.

HotSpot runs this as tiered compilation, with a fast baseline compiler (C1) producing code quickly and lightly optimised, and an aggressive optimising compiler (C2) taking over for the genuinely hot methods. V8 has the same shape: the Ignition interpreter executes bytecode while TurboFan optimises hot functions, with inline caches and hidden classes providing the type feedback, and a shape change to an object it had specialised for triggers the same deoptimisation. CPython has moved partway in this direction — 3.11 introduced a specialising adaptive interpreter that rewrites bytecode into type-specialised forms at run time, and 3.13 shipped an experimental just-in-time compiler that has to be enabled when the interpreter is built.

Why this explains real behaviour

Three things you will have observed become obvious once the pipeline is clear.

Warm-up. A freshly deployed JVM or Node service is genuinely slower for its first traffic, because the hot paths are still interpreted and the profile does not yet exist. That is why load balancers benefit from ramping traffic to new instances, and why the first few requests after a deploy show latency that never appears again.

Startup versus peak. Ahead-of-time compilation of Java through a native image starts in milliseconds and needs no warm-up, which is why it is attractive for short-lived functions and CLI tools. It also gives up profile-guided speculation and must assume a closed world, so long-running throughput can plateau below what a JIT reaches, and reflection-heavy code needs explicit configuration. Neither strategy is simply faster; they optimise different points on the same curve.

Benchmarks that lie. Timing a loop on a JIT runtime without warming it measures the interpreter, and a microbenchmark whose result is unused can be deleted entirely by dead code elimination — so the method appears to run in almost no time because it did not run. That whole class of error is why harnesses such as JMH exist, and why a benchmark number quoted without a warm-up phase should be assumed wrong.

A JIT is not a compiler that runs late; it is a compiler that gets to see the data, and every surprising thing about managed-runtime performance follows from that trade.