AR/VR and spatial computing fundamentals

VR names a display arrangement: the user's whole visual field is synthesised. Spatial computing names the broader problem of computing that understands where things are — the room's geometry, the user's hands, the position of a virtual object relative to a real table — regardless of whether the display is opaque, transparent or a phone screen. The distinction matters in an interview because the hard parts are shared. Tracking, registration of virtual content against real space, latency budgets and the maths of projection are the same whether the headset blocks the world or blends with it. Only the display and the compositing differ, so a candidate who can only talk about opaque VR has learned the narrower half.

Treat them as a scale of how much of the real world survives. VR replaces it entirely, so you control every pixel and every lighting decision, and you own the user's comfort completely. AR overlays content on the real world, typically without the content knowing much about that world, so registration accuracy and legibility against arbitrary backgrounds become the problems. MR is AR where the system has a model of the environment, so virtual objects can be occluded by real furniture, rest on real surfaces and be lit plausibly. The engineering consequence is that MR needs scene understanding and depth sensing, which AR can skip, and neither can rely on the total control VR enjoys.

Three degrees of freedom tracks orientation only: yaw, pitch and roll. Six adds translation in x, y and z. A three-degrees-of-freedom system can tell you the user turned their head but not that they leaned in, so a virtual object stays glued to a fixed distance from the face however the body moves. That is tolerable for a 360 video and unacceptable for anything the user might inspect, because leaning towards something and having it follow you is exactly the cue the brain reads as the world being fake. It is also a comfort issue: parallax from small involuntary head movements is a constant confirmation that the world is real, and removing it produces the same class of sensory disagreement as latency.

One line of setup: motion-to-photon latency is the interval between your head physically moving and the light representing that new viewpoint reaching your eye.

Contributors, in order:
  sensor sample + fusion        pose becomes available
  application frame            simulate, cull, submit
  GPU render                   two eye views
  compositor                   distortion, chromatic correction, present
  panel scanout + persistence  photons

At 90Hz, one refresh interval = 1000/90 = 11.1ms
A pipeline holding two frames in flight therefore spends
  >= 2 x 11.1 = 22.2ms  before the panel even starts

The reason to derive it rather than memorise a number is that the arithmetic shows you where the time actually goes, and it is not mostly in your shaders. Sensor fusion and scanout are fixed costs you do not control. The application frame and the GPU render are yours. The compositor's share is the runtime's.

The figure usually quoted as the comfort target is on the order of twenty milliseconds, and the useful observation is that a naive pipeline with two frames of buffering exceeds it before any work is done. That is why every part of the stack is built to cheat the arithmetic rather than to fit inside it: the pose used to render is a prediction of where the head will be at scanout, not where it was at sample time, and the compositor re-warps the finished frame against a newer pose immediately before presenting it.

The detail candidates leave out is that prediction converts a latency problem into an accuracy problem. Predict well and the effective latency is near zero even though the pipeline is slow. Predict badly — because the head changed direction, or because your frame time is variable so the prediction interval is unknown — and you get overshoot, which the user experiences as the world wobbling.

World-locked content stays at a fixed position in the room as the user moves and turns, which is the default expectation for anything meant to be an object. The alternatives are head-locked, meaning fixed relative to the viewer's face, and body-locked, meaning it follows the body's position but not the head's rotation. Head-locking is comfortable for nothing except very brief overlays, because it removes all parallax and the content becomes a smear the user cannot look away from. It is also the reason a naive port of a flat-screen heads-up display fails immediately: a full-screen UI pinned to the face is uncomfortable within seconds, and the fix is to place the UI in the world or on the wrist.

Presence is the involuntary sense that you are somewhere, rather than looking at a picture of somewhere. It is not a graphics-quality property; low-fidelity scenes achieve it routinely. It is produced by consistency — the world staying put when you move, your hands appearing where your hands are, objects behaving as their size implies. It breaks on contradiction: a lagged hand, an object that intersects your body, a scale that is subtly wrong, a frame that arrives late enough to let the world slip. Because presence rests on consistency rather than richness, the engineering priority in a headset is inverted relative to a flat game, and stability outranks detail.

One line of setup: two ways to let a user see the real world while wearing a display.

Axis                          Video passthrough            Optical see-through
----------------------------  ---------------------------  --------------------------
How the world arrives         cameras, drawn as a texture  directly, through a
                                                           transparent combiner
Latency on the real world     non-zero, the room itself    zero, light takes the
                              is a delayed image           short path
Occlusion of virtual by real  possible, both are pixels    hard, you cannot
                              you own                      subtract light
True black                    achievable                   not achievable, the
                                                           content is additive
Failure if compute stalls     the room freezes             the room is unaffected
Colour and dynamic range      limited by the sensor        the real thing
Field of view on the world    whatever the cameras and     whatever the combiner
                              the render target cover      passes, often narrower

The row that decides most architectural arguments is latency on the real world. In passthrough, the physical room is itself a rendered image on your latency budget, so a stall does not merely freeze the virtual content, it freezes reality — which is a safety consideration and not just a comfort one. Optical see-through cannot fail that way.

The row that decides most visual arguments is true black. An additive combiner can only add light, so a virtual object can never be darker than the wall behind it. Dark UI, shadows and solid occlusion are unavailable, and designs that assume them look transparent and unfinished. Passthrough owns every pixel and therefore can composite properly, at the cost of rendering the room.

The detail candidates miss is that occlusion of virtual content by real geometry needs a depth model of the room in both cases, and only passthrough can act on it convincingly.

An anchor is a persistent reference frame attached to a recognised feature of the real environment, so a virtual object placed on a specific table is still on that table in the next session or for a second user. It is needed because the tracking system's own coordinate origin is arbitrary and drifts, established wherever tracking initialised. Storing a world position relative to that origin means storing a number that quietly becomes wrong. An anchor pushes the correction problem into the platform: the runtime re-recognises the feature and adjusts the anchor's pose, and your object moves with it. The consequence for application code is that anchor poses can change between frames, so treating one as a constant is a bug.

One line of setup: the transform chain, written as the multiplication that actually happens.

object/model space   vertex as authored
  x model matrix     -> world space        object placed in the scene
  x view matrix      -> view/eye space     camera at origin, looking down -Z
  x projection       -> clip space         homogeneous, w carries depth
  / w                -> NDC                perspective divide; cube or box
  x viewport         -> screen space       pixels, plus a depth value

clip = P * V * M * vec4(position, 1.0)

Each step exists because the previous one cannot express what the next needs. Model space lets an artist author a chair once and place it two hundred times. World space is where physics, culling and lighting positions are meaningful. View space simplifies the projection by putting the camera at the origin, so the projection matrix does not have to encode the camera's placement.

Clip space is the one candidates fumble. It is deliberately before the divide, because clipping against the frustum is done with linear inequalities on the homogeneous coordinates, and doing it after the divide breaks for vertices behind the camera where w is negative. The divide by w is what makes perspective happen at all: it scales x and y down in proportion to distance.

For a headset, the chain matters because only V and P differ between the eyes. M is shared, and so is everything computed in world space. That single fact is the basis of every stereo optimisation, and it is why an interviewer asking about the matrix chain is often really asking about multiview.

The detail worth volunteering is that the handedness and the NDC depth range differ between APIs, which is the usual cause of a scene rendering inside out or clipped away when porting.

Because translation is not a linear operation on three-dimensional vectors and therefore cannot be written as a 3x3 matrix, while rotation and scale can. Adding a fourth coordinate makes translation expressible as a 4x4 matrix multiply, which means the whole transform chain composes by multiplication and a vertex can be transformed by one matrix rather than a matrix and an addition. The fourth component also carries the perspective divide: setting w from the view-space depth and dividing at the end produces perspective foreshortening for free. The practical convention that falls out is that a position has w equal to 1 and a direction has w equal to 0, so a direction is unaffected by translation.

Euler angles suffer gimbal lock: at certain orientations two of the three axes align and one degree of freedom is lost, so the representation cannot express a rotation the object can physically make. They also interpolate badly, because the result depends on the order the three rotations are applied, and they have no natural notion of the shortest path between two orientations. A quaternion is four numbers representing a rotation about an arbitrary axis, has no singularities, composes by multiplication, and interpolates smoothly along the shortest arc. The cost is that it is unreadable to humans, so tools still expose Euler angles at the edges and convert. Storage also drifts: repeated multiplication accumulates error, so quaternions need periodic renormalisation.

They are two orthogonal questions that get conflated. The first is memory layout: whether consecutive floats in the array are a row or a column. The second is mathematical convention: whether you write vectors as columns and multiply on the left, or as rows and multiply on the right, which reverses the order in which transforms compose. A library can be row-major in memory and still use column-vector convention. The bug appears when data crosses a boundary — an engine to a shading language, or a maths library to a graphics API — and one side transposes while the other does not. The symptom is a transform that works for pure rotation and breaks the moment translation is involved, because translation lives in the row or column that got swapped.

One line of setup: apply the model matrix to a normal under non-uniform scale and see what happens.

Model matrix scales x by 2, leaves y and z:
  S = [2 0 0]
      [0 1 0]
      [0 0 1]

Surface at 45 degrees: normal n = (0.707, 0.707, 0)
Naive transform:  S * n = (1.414, 0.707, 0)  normalised -> (0.894, 0.447, 0)
That is a 26.6 degree normal.

But the surface itself, stretched 2x in x, now sits at 26.6 degrees
from the x axis, so its true normal is 63.4 degrees from x
i.e. (0.447, 0.894, 0) - the naive result, with the components swapped.

Correct matrix for normals: transpose of the inverse of the upper 3x3.

The reason is that a normal is not a direction along the surface, it is perpendicular to it, and perpendicularity is not preserved by non-uniform scaling. Stretching a surface in x makes it flatter relative to x, which tilts its normal the opposite way to how a tangent vector tilts. The inverse transpose is precisely the matrix that transforms a vector so that its dot product with every transformed tangent stays zero.

Two consequences follow. For a matrix containing only rotation and uniform scale, the inverse transpose equals the original up to a scale factor, so the naive version works and this whole issue hides. That is why the bug ships: it appears only when an artist scales an object unevenly, and it manifests as lighting that is subtly wrong on that one object rather than as an obvious break.

The detail candidates omit is that translation must be dropped, which is why it is the upper 3x3 and not the full 4x4 — a normal is a direction and has no position.

It maps view-space depth into the normalised range the depth buffer stores, but not linearly: the division by w compresses distant depths into a narrow band of values while near depths get most of the range. The consequence is that depth precision is concentrated near the near plane, and the effect scales with the ratio of far to near rather than their difference. So pulling the near plane from 0.1 to 1.0 buys far more precision than pulling the far plane in from 10000 to 5000, which is the practical lesson. The classic symptom of getting it wrong is z-fighting on distant coplanar surfaces. A reversed-Z depth buffer with a floating-point format redistributes the precision far more evenly and is the standard modern fix.

One line of setup: the calculation behind every gaze pointer and every surface placement.

Ray:    p(t) = o + t*d          o = origin, d = normalised direction
Plane:  dot(n, x) = k           n = unit normal, k = plane offset

Substitute:  dot(n, o + t*d) = k
             dot(n,o) + t*dot(n,d) = k
             t = (k - dot(n,o)) / dot(n,d)

Guard: dot(n,d) near 0  -> ray parallel to plane, no usable hit
       t < 0            -> plane is behind the ray origin, reject
Hit point: o + t*d

Three lines of algebra, and both guards are where real bugs live. The parallel case divides by something approaching zero, producing an enormous t and a "hit" thousands of metres away that then fails a distance test in a confusing manner. The negative-t case is a hit behind the user, which for a gaze ray means selecting something behind their head.

This matters in spatial computing because it is the primitive underneath the interactions users notice most. A raycast from the eye or the controller through the scene is how selection works, and its numerical robustness is felt directly as a pointer that jitters or that grabs the wrong thing near a grazing angle.

The detail worth adding is that a real implementation clamps the ray to a maximum length and takes the nearest hit among candidates, and that testing against a plane is only the first step — a plane is infinite, so a hit must then be tested against the bounds of the actual surface.

A conventional projection assumes the view direction passes through the centre of the image, giving a frustum symmetric about the axis. In a headset each eye looks through its own lens, and the eye is not centred on the region of the display it sees; the field of view extends further towards the outside of the face than towards the nose. The projection therefore needs independent left, right, top and bottom extents rather than a single field-of-view angle. Code that builds a projection from one FOV value and an aspect ratio cannot express this, and forcing it produces a mismatch between the two eyes that the user perceives as strain. The runtime supplies the per-eye extents; your job is not to discard them.

The brain infers distance and size partly from binocular disparity, so the separation you render with is a scale control. Render with a wider separation than the user's actual interpupillary distance and every disparity grows, which the brain interprets as a world that is smaller and nearer — the miniaturised look familiar from hyperstereo photography. Render narrower and the world inflates. This is why interpupillary distance is a per-user setting rather than a constant, and why a title that hard-codes an average will feel subtly wrong to people at either end of the distribution. It is also a design lever: deliberately altering the separation is how a scene is made to feel gigantic, at the cost of the disparity no longer matching the user's eyes.

One line of setup: the fixed order that every rasterising GPU implements, however it is named.

flowchart TD
  A[Vertex fetch<br/>and vertex shader] --> B[Primitive assembly<br/>and clipping]
  B --> C[Perspective divide<br/>and viewport map]
  C --> D[Rasterise to<br/>fragments]
  D --> E[Early depth test]
  E --> F[Fragment shader]
  F --> G[Blend and<br/>write to target]

The interesting edge is between the early depth test and the fragment shader, because that is the only place where work can be discarded before it is paid for. A fragment that fails the depth test there costs nothing. A fragment that reaches the shader and is then overwritten costs everything.

That single boundary explains most of the practical advice in real-time rendering. Sorting opaque geometry roughly front to back lets early depth reject later fragments. Writing to the depth value in a fragment shader, or using alpha-to-coverage or discard, can defeat early depth entirely on some hardware and turn a cheap scene expensive. A depth pre-pass exists purely to populate the depth buffer so that the expensive shading pass benefits from perfect rejection.

The stages before rasterisation scale with vertex count and the stages after scale with pixel count, which is the first thing to establish when a frame is too slow. A dense mesh far away costs vertex work and almost no fragment work; a full-screen transparent effect is the opposite.

The detail candidates omit is that clipping happens before the divide, on homogeneous coordinates, for the reason given earlier: a vertex behind the camera has negative w and cannot be clipped correctly after dividing by it.

It solves visibility for opaque geometry independently of draw order: each fragment records its depth, and a later fragment is kept only if it is nearer, so objects can be submitted in any order and still occlude correctly. What it does not solve is anything requiring more than one surface's contribution to a pixel. Transparency needs the surfaces behind the near one, which the depth buffer has already discarded. Order-independent transparency, refraction and volumetric effects all exist because of that limitation. It also stores a single depth per pixel, so it cannot represent a partially covered edge, which is why antialiasing is a separate mechanism rather than a consequence of depth.

Because blending is not commutative. Compositing a translucent surface over what is already in the buffer depends on what is already in the buffer, so the result changes with order, whereas an opaque write simply replaces. The standard approach is to render opaque geometry first with depth writes on, then transparent geometry sorted back to front with depth writes off but depth testing on. It fails in two well-known ways: intersecting transparent surfaces cannot be ordered correctly at all, because the correct order differs per pixel; and per-object sorting by centre distance mis-orders large or concave objects. In a headset this is more visible than on a monitor because incorrect blending produces disparity errors between eyes, so the artefact reads as wrong depth rather than as wrong colour.

Forward rendering shades each fragment as it rasterises, so its cost scales with geometry times lights and it suffers from overdraw, but it supports arbitrary materials, handles transparency naturally and needs little bandwidth. Deferred rendering writes surface properties into a g-buffer and shades once per pixel afterwards, decoupling light count from geometry, at the cost of several full-screen render targets and their bandwidth, plus awkward transparency and constrained material variety. On mobile-class hardware in a headset the bandwidth cost is usually decisive, so forward rendering with a small fixed light budget, often with per-tile or per-cluster light assignment, is the common choice. The reasoning to state is that bandwidth, not arithmetic, is the scarce resource.

Overdraw is shading a pixel more than once in a frame because a nearer surface is drawn later and overwrites it, so the earlier shading was wasted. It is measured with a debug view that colours each pixel by how many times it was written, and it is one of the few performance problems with a genuinely intuitive visualisation. The usual culprits are large transparent quads, particle systems, and dense foliage — anything where many surfaces cover the same pixels and depth testing cannot reject them because they are blended. Fixes are to sort opaque draws front to back, reduce particle overlap and size, and use a depth pre-pass where shading is expensive. It matters more in a headset because two eye views multiply the cost.

A draw call is one command telling the GPU to render a batch of primitives with a given pipeline state. It matters because the cost is mostly on the CPU, in validating state and constructing commands, and it is paid per call regardless of how many triangles the call covers. Ten thousand triangles in one call is dramatically cheaper than ten calls of a thousand. That makes draw-call count a CPU budget rather than a GPU one, and on a standalone headset with a modest CPU it is often the binding constraint. The mitigations are batching by material, instancing for repeated geometry, and reducing state changes; multiview removes the doubling that stereo would otherwise impose.

One line of setup: the memory movement a pass implies, which is invisible in the API.

Tile-based GPU, one render pass:
  load    framebuffer tile from main memory into on-chip tile memory
  render  all draws for that tile, entirely on chip
  store   resolved tile back to main memory

Cost model:
  work inside a pass  -> on-chip, fast, low power
  each extra pass     -> a full store + a full load of the render target

Full-screen post-processing chain, 3 effects, naive:
  3 extra passes = 3 stores + 3 loads of a full-resolution, two-eye target

The API presents a render pass as bookkeeping, so it is easy to add one without noticing. On a tile-based architecture it is the single most expensive thing you can add, because it forces the entire render target through main memory and back.

Two rules follow. First, merge post-processing into as few passes as possible, ideally folding effects into the final pass of the main render rather than chaining separate full-screen passes. Second, declare correctly whether a pass needs the previous contents: if you are going to overwrite everything, saying so lets the driver skip the load entirely, and if you will not read the depth buffer afterwards, saying so lets it skip the store.

The reason this dominates specifically in a headset is that main-memory bandwidth is also the power budget, and power is heat, and heat is throttling. A bandwidth-heavy frame does not merely run slower, it degrades over the session.

The detail candidates omit is that multisample antialiasing is comparatively cheap here for the same architectural reason: the resolve happens inside tile memory, so the extra samples never travel to main memory.

The lens between the panel and the eye distorts the image, so the runtime's compositor pre-warps the rendered frame by the inverse distortion, and typically corrects chromatic aberration per colour channel at the same time. It happens in the compositor rather than in the application for two reasons. It is device-specific — the correction belongs to the hardware, not the content — and it must be applied to the freshest available image, after any reprojection, immediately before scanout. A consequence worth knowing is that the pre-warp compresses the outer region of the rendered image, so pixels near the edge of your render target contribute fewer pixels on the panel than pixels near the centre. That non-uniformity is the optical justification for fixed foveated rendering.

A vertex shader runs once per vertex and computes position and whatever per-vertex data the rasteriser will interpolate. A fragment shader runs once per covered fragment and computes colour. The decision is arithmetic: a mesh may have a few thousand vertices and cover a million fragments, so anything that can be computed per vertex and interpolated is hundreds of times cheaper there. The limit is that interpolation is linear, so a quantity that varies non-linearly across a triangle — a normalised vector, a specular term, anything with a division — is wrong if computed per vertex and interpolated, and must move to the fragment stage. That trade between cost and correctness is most of practical shader authoring.

One line of setup: what a divergent if actually executes.

// 32 or 64 lanes execute in lockstep, depending on the vendor.
if (material == METAL) {
    colour = expensiveMetal();   // lanes where false are masked off, not skipped
} else {
    colour = cheapDiffuse();     // then the other lanes are masked off
}
// Cost when a group contains both: metal + diffuse, not one of them.

A GPU does not execute one fragment at a time. It groups fragments — commonly 32 or 64 depending on the vendor — into a unit that shares a single instruction pointer. Every lane executes the same instruction; lanes for which the branch condition is false are masked so their results are discarded.

The consequence is that a branch is free when every lane in the group agrees and costs the sum of both sides when they do not. So a branch on a uniform, or on something spatially coherent like which region of the screen a fragment is in, is genuinely cheap. A branch on a per-pixel texture lookup, where neighbouring pixels disagree, is close to worst case.

This reframes the optimisation. You do not avoid branches; you make them coherent. Sorting draws so that one material's fragments are contiguous, or splitting one branching shader into two shader variants selected per draw, converts a divergent branch into a uniform one.

The detail candidates omit is that the same logic applies to loops with data-dependent trip counts: the group runs for the longest iteration count any lane needs.

A texture fetch has latency, and the GPU hides latency by having other work in flight. If the coordinates of a fetch are known early — interpolated from the vertex stage — the hardware can begin prefetching well before the result is needed. If the coordinates are computed by the shader from the result of a previous fetch, the second fetch cannot start until the first returns, so the latency is exposed rather than hidden. Chains of these serialise. That is why techniques built on indirection, such as sampling a normal map and then using it to index an environment lookup, cost more than the instruction count suggests, and why mobile-class GPUs with less latency-hiding capacity suffer most.

Compute-bound means the arithmetic units are the limit, so reducing instruction count helps. Bandwidth-bound means moving data between main memory and the GPU is the limit, so reducing instructions helps nothing and reducing the size and number of memory accesses helps everything. The distinction decides which optimisations are worth attempting, and getting it wrong is how a week disappears. On mobile-class hardware in a headset, bandwidth is usually the constraint, so the effective levers are smaller texture formats, aggressive mipmapping, fewer and smaller render targets, fewer render passes, and lower resolution — not shorter shaders. Bandwidth is also the dominant power cost, so a bandwidth win is simultaneously a thermal win.

Shaders are compiled for the specific GPU, and if that happens the first time a material becomes visible the compile blocks the frame, producing a stall of tens or hundreds of milliseconds. On a monitor that is a noticeable hitch; in a headset it is a dropped frame with the world slipping, which is far worse. The fix is to compile everything up front rather than lazily: enumerate the shader variants the content actually uses, warm them during loading, and cache the compiled result on disk so subsequent sessions skip the cost. The trap is combinatorial variant explosion — every optional feature multiplying the set — so keeping the variant count deliberately small is part of the solution rather than a separate concern.

One line of setup: the arithmetic that determines whether it is worth enabling.

Frame area normalised to 1.0. Shading rates as fragment-shader
invocations per pixel:

  1x1  = 1.00   full rate
  2x2  = 0.25   one invocation per 4 pixels
  4x4  = 0.0625 one per 16 pixels

Suppose:  centre 20% at 1x1, mid 30% at 2x2, outer 50% at 4x4
  cost = 0.20*1.00 + 0.30*0.25 + 0.50*0.0625
       = 0.200 + 0.075 + 0.031
       = 0.306   -> about 31% of the original fragment work

The mechanism is that the hardware decouples the rasterised pixel grid from the rate at which the fragment shader is invoked, driven by a per-region rate map. One invocation then colours a block of pixels.

The arithmetic above is why it is attractive and also why the headline number misleads. The saving applies to fragment-shader invocations and nothing else. Vertex processing is unchanged, draw-call count is unchanged, the depth buffer is still full resolution, and the render target is still full size, so bandwidth on the target is barely reduced. A frame that is vertex-bound or bandwidth-bound sees almost none of that 69% reduction.

It is the mechanism underneath both fixed and gaze-tracked foveation, which are policies for choosing the rate map rather than separate features.

The detail candidates omit is what the coarse regions cost visually: undersampling hurts high-frequency content most, so specular highlights, thin geometry and fine texture detail degrade long before flat surfaces do, and a hard boundary between two rates is itself visible. The falloff has to be gradual.

A mipmap chain is a set of progressively halved versions of a texture, and the hardware selects a level based on how densely the texture is sampled across the screen. Without it, a texture minified into a small screen area is point-sampled at widely spaced texels, so which texels are hit changes as the camera moves and the surface shimmers — spatial aliasing turned into temporal noise. Mipmaps also improve cache behaviour dramatically, because a distant surface reads a small texture instead of scattering reads across a large one, which is a bandwidth win as much as a quality one. In a headset shimmer is particularly objectionable, because the head never holds perfectly still and any aliasing becomes continuous motion.

One line of setup: the arithmetic every decision in a headset title descends from.

Refresh   Interval          Realistic app share
72Hz      1000/72 = 13.9ms  leave ~2-3ms for compositor -> ~11ms
80Hz      1000/80 = 12.5ms                              -> ~10ms
90Hz      1000/90 = 11.1ms                              -> ~8-9ms
120Hz     1000/120 = 8.3ms                              -> ~6ms

Inside the app share, both eyes must be rendered.
At 90Hz with ~9ms available and a 1ms CPU submission cost:
  ~8ms of GPU time for two eye views

Two things fall out immediately. First, the number you can actually spend is smaller than the refresh interval, because the compositor's distortion and present work occupies the tail of every interval and is not yours. Treating the full interval as your budget guarantees you miss.

Second, the increment between refresh rates is brutal at the top. Going from 72Hz to 90Hz removes roughly a fifth of your time; going to 120Hz removes nearly half of a 90Hz budget. Committing to a higher refresh rate is a content decision, not a settings toggle, because the art and the lighting have to fit.

The reason the budget is a hard ceiling rather than a target is comfort. A flat game that misses shows a hitch; a headset that misses shows the world slipping relative to the room, which produces sensory conflict and nausea. There is no graceful degradation available at the level of "slightly lower frame rate".

The detail candidates omit is that the budget must be met on a warm device, not a cold one. A frame time measured in the first minute is not the frame time the user will experience ten minutes in.

A standalone headset is a battery-powered computer strapped to a face, with a small thermal envelope and no room for a large cooler. Sustained load raises temperature until the system reduces clocks to stay within limits, so a frame that fits the budget when cold stops fitting after several minutes. This is why sustained-performance measurement matters more than peak, and why bandwidth reduction is doubly valuable: memory traffic is a large share of power, so cutting it lowers heat as well as time. It also changes what "it runs at 90Hz" means as a claim. The honest version is "it holds 90Hz after twenty minutes of representative play on a device that has been running that whole time".

It reduces the shading rate towards the edges of each eye's rendered image on a static map, without knowing where the user is looking. It is defensible because the optics are themselves non-uniform: the lens resolves the centre best, and the compositor's distortion pre-warp compresses the outer region of the render target, so peripheral pixels contribute less to the final image than central ones. The saving is therefore partly free rather than purely perceptual. Because it cannot follow the gaze, the full-rate region must stay large enough to cover wherever the eye might be, so the saving is modest compared with a gaze-tracked version. Its virtues are that it needs no extra hardware, no calibration and no per-user confidence handling.

One line of setup: the shader change, and then the honest accounting.

#extension GL_OVR_multiview2 : require
layout(num_views = 2) in;

uniform mat4 viewProj[2];   // left eye, right eye
in vec3 position;

void main() {
    // The only per-eye branch: pick this view's matrix.
    gl_Position = viewProj[gl_ViewID_OVR] * vec4(position, 1.0);
}

You bind a two-layer texture array, issue one draw call, and the driver runs the vertex shader once per view with the view index exposed to the shader. The application submits a mono-shaped frame and receives a stereo result.

What that removes is the second scene traversal and the second set of draw calls, so the CPU cost of drawing approaches that of a single view. On a standalone headset, where CPU submission is frequently the binding constraint, this is the single largest structural win available.

What it does not remove is per-vertex transform work, which still runs for both views, and fragment shading, which still fills two images. If a frame is fragment-bound, multiview will read as having done nothing.

Adoption is a shader audit rather than a flag. Every shader reading a view matrix, a camera position or a screen-space coordinate must obtain it through the view index. Missing one produces both eyes rendered from the left eye's viewpoint for that object, which looks correct in a screenshot and, in the headset, places the object at the wrong distance with no stereo depth. That is the defect that reaches users, because the review process usually looks at a single-eye capture.

Because the image is magnified by a lens close to the eye, so an aliased edge subtends a larger angle than the same edge on a monitor, and because the head is never still, so aliasing turns into continuous crawling rather than a static jagged edge. Motion makes temporal aliasing the dominant complaint, and stereo compounds it because the two eyes alias differently and the disagreement itself is fatiguing. Multisample antialiasing is the usual choice on tile-based mobile GPUs specifically because the resolve happens in on-chip tile memory and therefore costs comparatively little bandwidth. Post-process approaches that reconstruct across frames need care, because their history buffers are per-eye and their artefacts appear as depth errors.

Render scale is a multiplier on the eye-buffer resolution, so 0.8 renders at 64% of the pixels and 1.2 at 144%. It is the cleanest single lever for fragment cost, because it scales the dominant per-pixel work almost linearly and needs no content changes. Choosing it is an empirical trade against the visible loss of detail on text and thin geometry, which are the first things to suffer. The important practice is to make it dynamic: measure frame time and adjust the scale within a bounded range, so the system holds its refresh rate by giving up sharpness rather than by dropping frames. Sharpness degrading gracefully is far better than frame timing degrading at all.

In the real world your eyes converge on an object and simultaneously focus at that distance, and the two cues agree. Most headsets present images on a panel at one fixed optical distance while varying the convergence angle to place objects at different apparent depths, so the eyes converge for a near object while focusing at the fixed panel distance. The cues disagree, and sustained disagreement causes eye strain and fatigue. The practical consequence is a comfort zone: content the user must examine closely should be placed within a moderate range of distances rather than very near the face, and text in particular should not be put at arm's length or closer. It is a hardware limitation, so the mitigation is content design.

If the application misses its deadline, the compositor takes the last completed frame, samples a fresh head pose, and re-warps the image to match the newer orientation before presenting. This keeps the world locked to the room, which is the artefact that actually causes discomfort. What it cannot do is update anything not derived from head pose: animation, hand and controller models, projectiles and other users continue at the application's real rate, so they stutter and hands lag the physical hands. Rotation-only warping also exposes the edges of the rendered image when the head turns far, and any variant extrapolating translation must invent geometry that was never drawn, producing smearing and haloes around moving objects.

Cameras on the headset observe the surroundings, identify visual features, and compute the headset's pose relative to them, fusing that with inertial measurements. It replaced external base stations because it needs no room setup, which is what made standalone headsets practical. Its failure modes all come from the visual half. It needs light, so a dark room degrades it. It needs texture, so blank walls and plain floors give it nothing to lock onto. It is confused by large moving surfaces, mirrors and repetitive patterns, and by a moving vehicle where the visible environment is static but the inertial sensors report acceleration. Occluding the cameras with your own hands or a blanket also removes the input entirely.

Inertial sensors measure acceleration and angular rate, so position must be obtained by integrating twice, and any small bias in the measurement accumulates without bound. Left alone, a purely inertial estimate of position becomes useless within seconds. Drift is that accumulation, and the correction is the visual half of the system: recognising features whose positions are already known pins the estimate back to the world. The division of labour is that inertial data provides high-rate, low-latency motion between camera frames, while vision provides the absolute correction at a lower rate. When vision is unavailable the system falls back to orientation only or to a degraded estimate, which is why a headset in the dark drifts visibly.

One line of setup: the two points at which a pose is consumed, and why they differ.

sequenceDiagram
  participant T as Tracker
  participant A as App
  participant G as GPU
  participant C as Compositor
  T->>A: predicted pose for scanout
  A->>G: submit both eye views
  T->>C: fresher pose, sampled late
  C->>C: re-warp using fresher pose
  C->>T: present

The gap to look at is between the first arrow and the third. The application commits to a pose early because it needs one before it can render, and by the time the frame is finished that pose is several milliseconds old.

Late latching is the practice of pushing that commitment as far towards the end as possible. In its simplest form the compositor samples a newer pose and re-warps the finished frame, which is the mechanism shown above and is what makes a slow pipeline feel responsive. A stronger form has the application write the pose into a buffer the GPU reads at the last moment, so the rendered geometry itself uses a fresher pose rather than being corrected afterwards.

Both are attacking the same arithmetic. Motion-to-photon latency is dominated by how old the pose baked into the presented image is, not by how long the pipeline took, so reducing pose age is worth far more than shaving a millisecond off render time.

The detail candidates omit is that the same argument applies to gaze for foveation and to controller poses for hand rendering. A late-latched head pose combined with a frame-old controller pose produces hands that lag a world that does not, which users notice immediately.

A controller reports a pose and unambiguous button state, and it works in the dark, behind your back, and while your fingers are relaxed. Camera-based hand tracking infers pose from images, so it fails when the hands leave the camera field of view, when one hand occludes the other, when fingers occlude each other in a fist or a pinch, in poor lighting, and at the extremes of reach. It also has no reliable equivalent of a button press: a pinch is a gesture the system must classify, so it has both false positives and false negatives, and it gives the user no tactile confirmation. Designs therefore avoid precise or safety-critical actions on hand tracking and provide large targets and visible confirmation states.

The system infers gaze direction from images of the eye, and the mapping from what the camera sees to where the user is looking depends on the individual: interpupillary distance, eye shape, corneal curvature, and exactly how the headset is sitting on that face at that moment. Calibration fits that mapping by having the user look at known targets. It degrades when the headset shifts during a session, and quality varies with glasses, contact lenses, eye makeup and unusual eyelid geometry. The consequence for a renderer relying on gaze is that it must consume a confidence value alongside the gaze direction and degrade gracefully when confidence falls, rather than assuming the signal is always valid.

Discomfort comes from the eyes reporting motion the inner ear does not, so the comfortable schemes are the ones that avoid unmatched visual acceleration. Teleportation is the most comfortable because there is no continuous motion to disagree with. Smooth locomotion is the least, and the aggravating factors are acceleration and deceleration rather than constant speed, strafing, and above all rotating the view for the user rather than letting them turn their head. Snap turning in fixed increments avoids the rotational case. Mitigations that help include a static reference frame in view, such as a cockpit, and narrowing the field of view during motion to reduce peripheral motion signal. None of this substitutes for offering the user the choice.

Direct manipulation is the more natural interaction and should be used for anything within arm's reach, because the user's proprioception does most of the work. Beyond reach it stops functioning: small angular errors in tracking translate into large positional errors at distance, so a virtual hand becomes imprecise exactly where precision is needed, and the user cannot reach the target anyway. A ray extends the interaction without extending the arm, and its errors are angular rather than positional, which matches how the tracking error behaves. The cost is that a ray amplifies hand tremor with distance, so real implementations smooth the ray's direction and often snap to targets.

Because comfort depends on the worst frames and the variance between them, not the mean. An application holding a steady 72Hz is comfortable; one averaging 90Hz by alternating 6ms and 18ms frames is not, because every long frame presents a stale pose and the world slips. An average also actively conceals the pattern you need, since a single 40ms spike per second barely moves it. The correct instruments are a per-frame time series, a histogram, and high percentiles — the 99th and the maximum — plus a count of missed deadlines. Changing the metric changes the work: you go after spikes from shader compilation, asset streaming and garbage collection before steady-state shader cost.

One line of setup: the question that has to be answered before any optimisation is chosen.

flowchart TD
  A[Measure CPU frame time<br/>and GPU frame time] --> B{Which is larger?}
  B -->|CPU| C[Reduce draw calls<br/>batching, instancing, multiview]
  B -->|GPU| D{Vertex or fragment?}
  D -->|Vertex| E[Fewer vertices<br/>LODs, simpler meshes]
  D -->|Fragment| F[Lower render scale<br/>foveation, cheaper materials]
  B -->|Neither, both idle| G[Look for stalls<br/>and synchronisation]

The branch worth noticing is the last one. If neither processor is busy for the whole frame and you are still missing the budget, no amount of making work cheaper will help, because the frame is spent waiting — on a synchronisation point, a readback of GPU results to the CPU, or an asset load.

The reason to force this decision explicitly is that the two remedies have almost no overlap. Reducing draw calls does nothing for a fragment-bound frame, and lowering resolution does nothing for a CPU-bound one. A team that optimises without answering this first spends weeks producing measurements that do not move.

The subdivision under GPU-bound matters just as much. A test that scales cleanly is to change render scale: if frame time moves roughly in proportion, you are fragment-bound; if it barely moves, you are vertex- or bandwidth-bound and the pixels were not the problem.

The detail candidates omit is that CPU and GPU work overlap by design, so the frame time is roughly the larger of the two rather than their sum, and a small CPU reduction on a GPU-bound frame is invisible.

The GPU runs asynchronously, so wall-clock timing on the CPU around a draw call measures submission, not execution. A timestamp query asks the GPU to record when it reached a given point in its own command stream, and the difference between two such records gives the GPU time for the work between them. That is how a frame is attributed to shadow rendering, the main pass and post-processing rather than treated as one number. The results arrive a frame or more later, so a profiler reads last frame's timings, which is fine for analysis and a trap for anything adaptive. Worse, forcing the CPU to wait for them destroys the overlap between CPU and GPU and changes what you are measuring.

One line of setup: two builds with the same average, presented as a distribution.

Budget at 90Hz = 11.1ms. 600 frames each.

              Build A            Build B
mean          10.8ms             10.8ms
p50           10.7ms             8.2ms
p95           11.0ms             15.4ms
p99           11.2ms             24.1ms
max           12.0ms             41.0ms
over budget   2 frames           47 frames

Both builds report the same average, and an on-screen counter would show both hovering near 92 frames per second. Only one of them is shippable.

Build A is tight against its budget with almost no spread, so it is close to the edge but predictable. Its risk is that any regression, or a warm device, pushes the whole distribution over — the fix is headroom.

Build B has a much faster typical frame and a long tail. Forty-seven misses in six hundred frames is roughly one slip every second and a half, which is enough to make a user uncomfortable within minutes. Its problem is not cost, it is variance, and lowering resolution would not touch it.

The two need opposite work. A needs everything to get cheaper. B needs the tail found: a spike profile of the worst frames, checking for shader compilation on first sight of a material, texture streaming, garbage collection or a physics step that occasionally does much more work.

The detail candidates omit is that the "over budget" count is the metric closest to what the user feels, and it is the one no frames-per-second counter can show you.

From per-frame allocation in managed code: temporary vectors and matrices, string formatting for debug output or UI, closures and iterators allocated inside update loops, and collections rebuilt each frame rather than reused. Each individually is trivial; together they fill a heap generation and eventually trigger a collection that can exceed the frame budget outright. Because it depends on accumulated allocation, the spike appears at unpredictable intervals, which is precisely the variance pattern that hurts comfort. The remedies are to allocate nothing in steady state, pooling and reusing buffers, avoiding string work outside debug builds, and preferring value types and pre-sized collections. The habit to build is watching allocation rate as a first-class metric alongside frame time.

The count or rate of frames that missed the deadline, over a window, on a warm device. It is the number closest to what a user experiences, it is unambiguous, and it does not average away the thing that matters. Frames per second hides tails, mean frame time hides tails, and both invite arguing about a target rather than about misses. A miss rate also stays meaningful across refresh rates, because the deadline moves with the device. In practice you would show it alongside a high percentile of frame time so you can tell a build that is uniformly close to the edge from one with an occasional catastrophic spike, since those need different work.

Accept that some defects have no numeric signature you will find first, and build the loop that surfaces them anyway. Record the full per-frame time series during a headset session with markers you can trigger by hand, so a tester who feels a slip can flag the moment and you can go to those frames afterwards. Log deadline misses with enough attribution to say which pass overran. Keep captures per eye rather than a single screenshot, because whole classes of stereo bug are invisible in one view. And test on a warm retail device with a representative scene, because the defects that survive to users are the ones that only appear after twenty minutes.

Both test the pipeline and the maths. The addition is that the user's body is part of the system, so every performance question has a human-factors answer behind it and a candidate who treats frame timing as a quality setting has missed the point of the role. Expect to be pushed on why a budget is a hard ceiling, on what a dropped frame does to a person, on the asymmetry between the two eye views, and on mobile-class constraints such as bandwidth and thermals that a desktop graphics role can largely ignore. The other addition is tracking: you are expected to know where the pose comes from, how stale it is, and what happens when it is wrong.

Say so once, briefly, and then reason from the constraints, because the constraints are public and the specifics are not. You can talk accurately about frame budgets derived from a stated refresh rate, about the split between application and compositor, about what reprojection can and cannot fix, about stereo sharing and about bandwidth on tile-based hardware without naming a device. Interviewers are generally testing whether you understand why the platform is shaped the way it is, and a candidate who reasons from mechanism travels between platforms. Guessing at a specific device's internal latency or panel specification is the failure mode: it is checkable, and being confidently wrong about their hardware is worse than not knowing it.

Something small and rendered, with a performance constraint attached, because that combination reveals more than either alone. A typical shape is a scene or effect to implement and then a requirement to hold a frame budget on a specified device, or an existing scene that misses its budget and a request to find out why and fix it. What is being read is your method: whether you measured before changing anything, whether you identified the binding constraint, whether your changes are justified by numbers you can show, and whether you can say what you gave up. A submission that is fast with no explanation of why scores below a slower one with a clear diagnosis.

One line of setup: what one missed deadline is worth, carried through from a stated premise.

Refresh 90Hz  ->  deadline = 1000/90 = 11.1ms

A frame that takes 20ms misses one deadline, so the pose baked
into the presented image is about 20ms old rather than about 11ms.

Premise: the head is yawing at 100 degrees per second
  pose error = 0.100 deg/ms x 20ms = 2.0 degrees

Premise: 100 degrees of horizontal field across a 2000-pixel eye buffer
  20 pixels per degree  ->  2.0 degrees = about 40 pixels of slip

What the user is shown:
  frame n    on time   world stable
  frame n+1  late      world slides ~40 px against the room
  frame n+2  on time   world snaps back to where the room says it is

The two premises are stated rather than measured, and that is the point of writing it out: whatever numbers you substitute, the slip scales linearly with how stale the pose is, so a doubling of frame time doubles the visible error. Nothing about the scene's content appears in the arithmetic at all.

A strong answer starts with what a late frame does. The renderer commits to a predicted head pose, so a frame that arrives late is presented for a pose that has expired; the head kept turning and the image did not, so the world slides relative to the room and then snaps back. The inner ear reports one magnitude of rotation and the eyes report a smaller one, and sustained disagreement between those channels produces nausea, sweating and headache. That is a bodily response, not a visual complaint, which is why there is no acceptable slightly-lower frame rate the way there is an acceptable slightly-lower texture resolution.

The answer then draws the consequences, and this is where candidates separate. Because misses cause the harm, the metric is deadline misses and frame-time variance rather than average frame rate. Because the variance is what hurts, the first targets are spikes — shader compilation on first sight, asset streaming, garbage collection — before steady-state shader cost. Because you cannot fail gracefully on timing, you build a lever that fails gracefully on something else instead, which is dynamic render scale trading sharpness for schedule. And because reprojection only corrects head orientation, it is a safety net rather than headroom, since a reprojected frame still shows lagging hands and stuttering animation.

A weak answer substitutes a target number for the mechanism: it names a refresh rate, asserts that dropping below it is bad, and offers a list of generic optimisations. It usually also treats reprojection as making misses acceptable, which inverts the relationship, and it reaches for average frame rate as the measure — the one metric that hides the frames doing the damage.