Why does a dropped frame matter far more in VR than in a flat-screen game, and what does the compositor do to hide it?
6 min readUpdated 2026-08-15
0
Answer
Short answer
A headset draws to eyes that are still moving, so a late frame is presented in the wrong place for where the head now is; the world appears to slip, the vestibular and visual senses disagree, and the result is discomfort rather than a visual blemish.
Keep frame timing explicit in the answer because that is the concept the interviewer is actually trying to test.
Was this answer helpful?
Likely follow-ups
→How would you decide whether to render at half rate and let reprojection fill in, versus cutting resolution to hit full rate?
→What visual artefacts appear when reprojection has to account for translation as well as rotation, and why?
→How does late latching of the head pose reduce motion-to-photon latency without changing the render cost?
→If your app hits its frame budget on average but misses one frame in fifty, what do you instrument to find the cause?
Start with arithmetic, because the whole discipline follows from it. Current headsets commonly run somewhere between 72 and 120Hz. At 90Hz a frame must be finished every 1000/90 milliseconds, so roughly 11.1ms; at 72Hz roughly 13.9ms; at 120Hz roughly 8.3ms. That budget is not just your draw calls. It has to contain simulation, animation, culling, the GPU work for two eye views, and whatever the runtime's compositor needs afterwards to distort and present the image. The share genuinely available to your own rendering is meaningfully less than the headline figure, which is why a VR title targets a frame cost that would look conservative on a monitor.
A flat-screen game with the same budget has a softer failure. Miss it and the image on the monitor holds for one extra refresh. The player sees a hitch, judders through a pan, and carries on. Nothing about their body contradicts what they see, because their head was not the camera.
Why the headset case is different in kind
In a headset your head is the camera, and the camera is being tracked continuously. Rendering a frame takes time, so the runtime cannot render against where the head is now; it renders against a prediction of where the head will be when that frame is actually lit up on the panel. When the frame arrives on schedule the prediction is close and the world sits still relative to the room.
When the frame is late, the frame that is presented was rendered for a pose that is now stale. Your head has kept turning; the image has not. The world appears to swim or slide in the direction of the turn, then snap back when a fresh frame lands. That is the crucial difference: on a monitor a late frame costs you smoothness, and in a headset it costs you world-lock. A virtual world that does not stay bolted to the room reads to the brain as the room moving, and the room is not something the brain is willing to be wrong about.
The discomfort follows from the disagreement. Your inner ear reports a head rotation of a certain magnitude. Your eyes report a smaller one, because the image lagged. Sustained conflict between those two channels is the classic recipe for the nausea, cold sweat and headache that people call simulation sickness. This is why frame timing in spatial computing is a human-factors requirement and not a quality setting, and why the interviewer cares that you say so.
Consistency beats average
The corollary that separates a considered answer from a rehearsed one is that mean frame rate is close to the wrong metric. An application that holds a steady 72Hz is comfortable. An application that averages 90Hz by alternating between 6ms and 18ms frames is not, because every long frame produces a slip. You are optimising the worst frames and the variance, not the mean, and that changes what you do: you attack spikes from garbage collection, shader compilation on first use, texture streaming and asset loading before you attack the steady-state cost of your material shaders.
It also changes how you read a profiler. A frame-time histogram or a plot of every frame's duration is diagnostic; an average frames-per-second counter actively hides the thing you need to see.
The reprojection safety net
Because misses are inevitable, the runtime keeps a mechanism to soften them. If no new frame has been submitted in time, the compositor takes the most recent completed frame, samples a fresh head pose, and re-warps that image to match the newer orientation before presenting it. Rotational reprojection of this kind is cheap and remarkably effective, because rotation of a distant scene is close to a 2D transform of the already-rendered image. Some runtimes go further and extrapolate translation and object motion as well.
Ideal frame at 90Hz, budget 11.1ms
0.0 simulate + submit -> compositor -> photons at ~11ms (pose was predicted for ~11ms: correct)
Late frame
0.0 app still rendering at 11.1ms deadline
compositor has nothing new
re-warps the previous frame against a fresh pose -> photons on time
cost: correct head orientation, stale world state
What reprojection cannot do is invent information the old frame never contained. Rotate far enough and you expose the edges of the rendered image. Translate the head sideways and the geometry behind a foreground object should be revealed, but that geometry was never drawn, so you get smearing or a halo around moving objects and around your own hands. Extrapolated motion overshoots when an object changes direction. All of this is much less unpleasant than a slipping world, which is exactly why the mechanism exists.
Loading diagram…
The edge worth looking at is the join: both branches converge on the same warp step, so from the panel's point of view a dropped frame and a delivered frame are indistinguishable in timing. They differ only in how old the world state behind the warp is.
Treating reprojection as headroom is the mistake
The answer that reads as adequate stops at "the runtime reprojects, so a missed frame is handled". The answer that reads as experienced says why you must not budget for it. Reprojection corrects head orientation and nothing else. Everything driven by simulation rather than by pose keeps running at your real rate: animation, your own hand and controller models, projectiles, other people's avatars. A reprojected frame therefore presents a correctly oriented world in which moving things stutter and hands lag the physical hands holding the controllers. Users report that hand lag as a loss of presence long before they can name what changed.
There is a legitimate version of leaning on it, and knowing the difference is the point. Deliberately rendering at half the display rate with motion-aware reprojection filling every other frame is a real technique, chosen up front so the content is authored for it and the cadence is stable. That is different from missing frames unpredictably and calling the safety net a plan, because an intermittent miss gives you the artefacts and the variance.
What this changes about how you work
Practically, this pushes the whole pipeline towards predictability. You want fixed costs rather than cheap-on-average ones, so you prefer a fixed-cost lighting approach to one that spikes with the number of visible lights. You warm shaders and stream assets before the user can reach the area that needs them. You keep a hard per-frame draw-call and CPU budget rather than a target average. And you measure with the headset on your head, because a slip is easier to feel than to see in a chart.
Comfort is a function of the worst frames, not the average ones, so the metric that matters is frame-time variance and the artefact you are preventing is the world coming unbolted from the room.