AR/VR and spatial computing fundamentals

VR names a display arrangement: the user's whole visual field is synthesised. Spatial computing names the broader problem of computing that understands where things are — the room's geometry, the user's hands, the position of a virtual object relative to a real table — regardless of whether the display is opaque, transparent or a phone screen. The distinction matters in an interview because the hard parts are shared. Tracking, registration of virtual content against real space, latency budgets and the maths of projection are the same whether the headset blocks the world or blends with it. Only the display and the compositing differ, so a candidate who can only talk about opaque VR has learned the narrower half.

Treat them as a scale of how much of the real world survives. VR replaces it entirely, so you control every pixel and every lighting decision, and you own the user's comfort completely. AR overlays content on the real world, typically without the content knowing much about that world, so registration accuracy and legibility against arbitrary backgrounds become the problems. MR is AR where the system has a model of the environment, so virtual objects can be occluded by real furniture, rest on real surfaces and be lit plausibly. The engineering consequence is that MR needs scene understanding and depth sensing, which AR can skip, and neither can rely on the total control VR enjoys.

Three degrees of freedom tracks orientation only: yaw, pitch and roll. Six adds translation in x, y and z. A three-degrees-of-freedom system can tell you the user turned their head but not that they leaned in, so a virtual object stays glued to a fixed distance from the face however the body moves. That is tolerable for a 360 video and unacceptable for anything the user might inspect, because leaning towards something and having it follow you is exactly the cue the brain reads as the world being fake. It is also a comfort issue: parallax from small involuntary head movements is a constant confirmation that the world is real, and removing it produces the same class of sensory disagreement as latency.

One line of setup: motion-to-photon latency is the interval between your head physically moving and the light representing that new viewpoint reaching your eye.

Contributors, in order:
  sensor sample + fusion        pose becomes available
  application frame            simulate, cull, submit
  GPU render                   two eye views
  compositor                   distortion, chromatic correction, present
  panel scanout + persistence  photons

At 90Hz, one refresh interval = 1000/90 = 11.1ms
A pipeline holding two frames in flight therefore spends
  >= 2 x 11.1 = 22.2ms  before the panel even starts

The reason to derive it rather than memorise a number is that the arithmetic shows you where the time actually goes, and it is not mostly in your shaders. Sensor fusion and scanout are fixed costs you do not control. The application frame and the GPU render are yours. The compositor's share is the runtime's.

The figure usually quoted as the comfort target is on the order of twenty milliseconds, and the useful observation is that a naive pipeline with two frames of buffering exceeds it before any work is done. That is why every part of the stack is built to cheat the arithmetic rather than to fit inside it: the pose used to render is a prediction of where the head will be at scanout, not where it was at sample time, and the compositor re-warps the finished frame against a newer pose immediately before presenting it.

The detail candidates leave out is that prediction converts a latency problem into an accuracy problem. Predict well and the effective latency is near zero even though the pipeline is slow. Predict badly — because the head changed direction, or because your frame time is variable so the prediction interval is unknown — and you get overshoot, which the user experiences as the world wobbling.

World-locked content stays at a fixed position in the room as the user moves and turns, which is the default expectation for anything meant to be an object. The alternatives are head-locked, meaning fixed relative to the viewer's face, and body-locked, meaning it follows the body's position but not the head's rotation. Head-locking is comfortable for nothing except very brief overlays, because it removes all parallax and the content becomes a smear the user cannot look away from. It is also the reason a naive port of a flat-screen heads-up display fails immediately: a full-screen UI pinned to the face is uncomfortable within seconds, and the fix is to place the UI in the world or on the wrist.

Presence is the involuntary sense that you are somewhere, rather than looking at a picture of somewhere. It is not a graphics-quality property; low-fidelity scenes achieve it routinely. It is produced by consistency — the world staying put when you move, your hands appearing where your hands are, objects behaving as their size implies. It breaks on contradiction: a lagged hand, an object that intersects your body, a scale that is subtly wrong, a frame that arrives late enough to let the world slip. Because presence rests on consistency rather than richness, the engineering priority in a headset is inverted relative to a flat game, and stability outranks detail.

One line of setup: two ways to let a user see the real world while wearing a display.

Axis                          Video passthrough            Optical see-through
----------------------------  ---------------------------  --------------------------
How the world arrives         cameras, drawn as a texture  directly, through a
                                                           transparent combiner
Latency on the real world     non-zero, the room itself    zero, light takes the
                              is a delayed image           short path
Occlusion of virtual by real  possible, both are pixels    hard, you cannot
                              you own                      subtract light
True black                    achievable                   not achievable, the
                                                           content is additive
Failure if compute stalls     the room freezes             the room is unaffected
Colour and dynamic range      limited by the sensor        the real thing
Field of view on the world    whatever the cameras and     whatever the combiner
                              the render target cover      passes, often narrower

The row that decides most architectural arguments is latency on the real world. In passthrough, the physical room is itself a rendered image on your latency budget, so a stall does not merely freeze the virtual content, it freezes reality — which is a safety consideration and not just a comfort one. Optical see-through cannot fail that way.

The row that decides most visual arguments is true black. An additive combiner can only add light, so a virtual object can never be darker than the wall behind it. Dark UI, shadows and solid occlusion are unavailable, and designs that assume them look transparent and unfinished. Passthrough owns every pixel and therefore can composite properly, at the cost of rendering the room.

The detail candidates miss is that occlusion of virtual content by real geometry needs a depth model of the room in both cases, and only passthrough can act on it convincingly.

An anchor is a persistent reference frame attached to a recognised feature of the real environment, so a virtual object placed on a specific table is still on that table in the next session or for a second user. It is needed because the tracking system's own coordinate origin is arbitrary and drifts, established wherever tracking initialised. Storing a world position relative to that origin means storing a number that quietly becomes wrong. An anchor pushes the correction problem into the platform: the runtime re-recognises the feature and adjusts the anchor's pose, and your object moves with it. The consequence for application code is that anchor poses can change between frames, so treating one as a constant is a bug.