AR/VR and Spatial Computing

What this area actually covers

Spatial computing is the engineering of software that knows where things are. That sounds abstract until you notice what it forces: if a program is going to place a virtual object on a real table and keep it there while a person walks around it, the program needs a model of the room, a continuous estimate of where the person's head is, a way to turn three-dimensional geometry into two images, and a schedule strict enough that the object never appears to slide. Those four obligations are the area. Everything specific — a headset, a phone doing augmented reality, a handheld display, a pair of glasses — is a different arrangement of the same four.

The concrete work divides into three layers that interview questions move between freely. There is the maths: coordinate spaces, the transform chain a vertex travels from the model an artist authored to a pixel on a panel, rotations and why they are stored as quaternions, ray intersection because that is what a gaze pointer is. There is the pipeline: how a graphics processor turns triangles into fragments, what each stage costs, which stage is your bottleneck today and which optimisation therefore does nothing. And there is the platform: two eye views instead of one, a lens that distorts, a compositor you do not own, a battery-powered device with a thermal ceiling, and a human nervous system that reacts badly to being lied to about motion.

The boundary that matters most is with general real-time graphics, because they overlap by perhaps seventy per cent and the remaining thirty is what the interview is about. A desktop game renderer and a headset renderer share the maths, the pipeline, the shading models and the profiling method. They differ in that a desktop game which drops to fifty frames per second has a performance problem, while a headset which drops a frame has a physiological one. That single asymmetry propagates outwards into everything: which metrics you keep, what you are allowed to do lazily, how you spend a saved millisecond, and why the platform runs a process between your renderer and the display that will correct your frame behind your back.

Things people wrongly bundle in are worth naming, because they will otherwise eat your preparation time. Three-dimensional content authoring is a different discipline — modelling, rigging, texturing and lighting are what technical artists do, and although you will work with them daily, no one will ask you to do it. Computer vision research is adjacent rather than included: the algorithms that turn camera images into a pose are a specialism with its own hiring pipeline, and most spatial computing engineers consume a pose rather than compute one. Interaction design is a genuine skill and a genuine job, and it is not this one, although you are expected to know why a full-screen menu pinned to the face is uncomfortable. And "the metaverse" is a product proposition, not an engineering area; it went through a hype cycle and largely out again, while the engineering underneath carried on being used for training simulators, surgical planning, industrial visualisation and games.

What rendering pipelines covers

This section currently has one subsection, and rather than being a limitation that is close to an honest description of how the discipline is examined. Almost every technical question in a spatial computing interview arrives through the pipeline, because the pipeline is where the maths becomes visible and where the platform's constraints become arithmetic you can do out loud.

Rendering Pipelines covers the fixed order every rasterising graphics processor implements — vertex work, primitive assembly and clipping, the perspective divide, rasterisation into fragments, the depth test, fragment shading, blending — and then what changes when that pipeline has to produce two images of the same scene, sixty to a hundred and twenty times a second, on a device you can hold in one hand.

Three themes recur inside it, and they are worth knowing before you open a single question. The first is that the pipeline has one place where work can be thrown away before it is paid for, which is the depth test that precedes fragment shading, and most practical advice in real-time rendering is downstream of that one fact. Sorting opaque geometry roughly front to back, adding a depth pre-pass, avoiding shaders that write depth themselves — all of it is about keeping that rejection working.

The second is that stereo is not a doubling. The model matrix that places an object in the world is shared between the eyes; only the view and projection matrices differ. That means the scene traversal, the culling decisions, the animation and the lighting positions can be computed once, which is why the platform offers a mechanism for issuing one set of draw commands that produces two images. Fragment shading, by contrast, genuinely doubles, because there are twice as many pixels. A candidate who can say which half of the cost stereo actually duplicates is demonstrating that they understand the transform chain rather than reciting it.

The third is that the budget is arithmetic, and the arithmetic is short enough to do in an interview. Current headsets commonly run somewhere between seventy-two and a hundred and twenty hertz, and from a stated refresh rate everything else follows:

Refresh rate  ->  interval available for one frame
  72Hz            1000/72  = 13.9ms
  90Hz            1000/90  = 11.1ms
 120Hz            1000/120 =  8.3ms

The application does not get all of it. The compositor's distortion
correction and present work occupies the tail of every interval.

At 90Hz, allowing the compositor a couple of milliseconds:
  ~9ms for the application
  minus ~1ms of command submission on the CPU
  = ~8ms of GPU time to render two eye views

Derive it that way rather than memorising a figure, because the derivation is what the interviewer is listening for and because it makes the shape of the trade obvious. Moving from seventy-two hertz to ninety removes about a fifth of your time. Moving to a hundred and twenty removes nearly half of a ninety hertz budget. Committing to a higher refresh rate is therefore an art and lighting decision, not a settings toggle, and that is the kind of statement that separates someone who has shipped from someone who has read.

Open the subsection expecting questions in three shapes. Mechanism questions ask you to walk a stage and name its cost. Diagnostic questions hand you a frame that misses its budget and watch how you narrow it down. And judgement questions ask what you would give up: resolution, lighting quality, or the headroom that protects you from a warm device twenty minutes into a session.

Where it sits in a real system

A spatial application is a loop with a hard deadline, wrapped around a pose estimate it does not produce. Follow one frame from the sensors to the photons.

Cameras and inertial sensors feed a tracking subsystem, usually part of the platform runtime rather than your code, which fuses them into a pose — where the head is and which way it is facing. The runtime hands your application not the current pose but a predicted one, an estimate of where the head will be at the moment the display actually lights up. Your application uses that pose to build a view for each eye, decide what is visible, run animation and physics, and submit draw commands. The graphics processor renders two images. Then the compositor takes over: it samples a fresher pose, re-warps your finished images to match it, applies the inverse of the lens distortion and a per-channel chromatic correction, and presents the result to the panel.

flowchart TD
  A[Cameras and IMU] --> B[Tracking and sensor fusion]
  B --> C[Predicted pose for scanout]
  C --> D[Application frame<br/>cull, animate, submit]
  D --> E[GPU renders two eye views]
  E --> F[Compositor<br/>reproject and distort]
  B --> F
  F --> G[Panel scanout<br/>photons reach the eye]

The edge worth staring at is the second arrow into the compositor. Tracking feeds the compositor directly, bypassing your application entirely, and that is not a detail — it is the architectural fact that makes the whole platform work. Your frame is corrected after you have finished with it, using information you never saw.

Two consequences follow, and both come up constantly. The first is that motion-to-photon latency is dominated not by how long your pipeline takes but by how old the pose baked into the presented image is. Prediction and late correction attack pose age directly, which is why they are worth more than shaving a millisecond off your render time. The second is that prediction converts a latency problem into an accuracy problem. Predict well and a slow pipeline feels responsive. Predict badly — because the head changed direction, or because your frame times are variable so the prediction interval is unknown — and the world overshoots and wobbles. Stable frame timing is therefore not only a comfort requirement in its own right; it is an input to the prediction that protects you.

The handshake between application and runtime is worth seeing as a sequence, because the ordering explains why certain APIs look strange.

sequenceDiagram
  participant R as Runtime
  participant A as Application
  participant G as GPU
  participant C as Compositor
  A->>R: wait for the next frame slot
  R->>A: predicted pose and per-eye projection
  A->>G: submit both eye views
  A->>C: hand over the finished layer
  R->>C: fresher pose sampled late
  C->>A: present and release

The interesting part is the first exchange. The application asks the runtime when to begin, rather than rendering as fast as it can and letting the display sort it out. The runtime is pacing you, because it knows when scanout happens and you do not, and because a frame started at the wrong moment is stale by the time it lands however quickly it renders. Anyone who has written a game loop that spins as fast as the hardware allows has to unlearn it here.

Around that loop sits the rest of a real product. Content pipelines bake meshes, textures and lighting into forms the device can load quickly. Scene understanding supplies planes, meshes and anchors so virtual objects can rest on real surfaces and be occluded by real furniture. Interaction code turns controller poses, hand poses and gaze into selections. Networking, if the application is shared, must reconcile poses between people. And an analytics layer records frame times, deadline misses and thermal state, because the defects that reach users are the ones that only appear on a warm device in a real room.

Who does this work

The roles split along a line that job adverts often blur, so it is worth drawing plainly.

A rendering engineer's day is mostly measurement. You take a capture, attribute the frame to its passes, form a hypothesis about which resource is the constraint, and test it with a change small enough that the result means something. The satisfaction and the difficulty are both in that loop being short: you can see the result immediately, and you can also fool yourself immediately by measuring a cold device or a single eye.

A technical artist's day is the same problem approached from the other side, and the pairing is the most productive relationship in the discipline. The engineer knows that a full-screen transparent effect is costing four milliseconds; the artist knows which of the three ways to remove it will still read as fog. Teams where these two people talk daily ship; teams where the engineer sends a document about polygon budgets do not.

Distinguish all of these from the people who specify the work. Producers and product managers decide which platforms and which refresh rate a title targets, which is a commitment with enormous engineering consequences, and they usually make it before anyone has profiled anything. Part of a senior engineer's job in this area is to make that commitment an informed one, early, in arithmetic the non-specialists can follow. The frame budget calculation above is the single most useful thing you can put in front of a room.

Demand, adoption and how that is changing

Be honest with yourself about this one, because it is the field where enthusiasm most reliably outruns hiring.

Demand here is genuinely niche, and it has been strongly cyclical in a way that very few engineering specialisms are. The area has been through several waves of investment and retrenchment, each driven by a belief that consumer head-worn computing was about to become mainstream, and each followed by a contraction when it did not on the expected timetable. That pattern is the defining feature of the market, and it has two practical implications. Postings cluster: a small number of platform holders and studios hire in bursts rather than a broad market hiring steadily. And the specialism does not transfer as cleanly as its practitioners expect, because a team hiring for a headset title wants someone who has held a frame budget on mobile-class hardware, which is a narrower claim than "graphics engineer".

What keeps the demand real rather than notional is that the underlying work has non-consumer customers who do not care about the hype cycle at all. Training and simulation, where the alternative to a headset is an expensive physical rig. Surgical and medical planning. Industrial design review and factory visualisation. Defence. Location-based entertainment. These buyers are less numerous than a consumer market would be and considerably more durable, and they are where a large share of the steady employment sits. Someone deciding whether to invest in this area should look at that list rather than at consumer device announcements.

The technical direction is clearer than the commercial one. Standardisation has consolidated: rather than one proprietary interface per device, applications increasingly target a common runtime interface, which lowers the cost of shipping to several devices and correspondingly lowers the value of device-specific expertise. Passthrough-based mixed reality has become the dominant mode for head-worn hardware, which pulls camera pipelines, scene understanding and colour processing into what used to be a purely synthetic rendering problem. Gaze tracking, where present, is being used mainly as a rendering optimisation rather than as an interaction channel. And the boundary with mobile graphics has thinned, because the constraints are the same constraints: tile-based architectures, bandwidth as the scarce resource, thermal limits over a session.

There is also an honest observation about absorption. Some of what was distinctively spatial computing work five years ago is now handled inside game engines by default, which means fewer people write it and more people configure it. The work that resists absorption is the work that requires knowing why the engine does what it does: diagnosing a frame that the defaults do not fit, extending the renderer, or building for hardware the engine does not target well. That is where the roles are, and it is a smaller set than the field's visibility suggests.

What makes it hard

The difficulty is not the volume of material, and it is not the maths — the maths is a fortnight of work for anyone comfortable with matrices. Three things make it hard, and only the first is obvious.

The deadline is absolute, and there is no graceful degradation on the axis that matters. Every other kind of software can trade latency for throughput or quality for speed at runtime and merely feel worse. Here, missing a deadline produces a specific bodily response in a specific person, so the schedule cannot be traded against anything. What you can do instead is build a lever that degrades something else on your behalf, which is why dynamic resolution scaling exists: it gives up sharpness continuously so that timing never has to give at all.

flowchart TD
  A[Application misses its deadline] --> B[Compositor reprojects<br/>the last finished frame]
  B --> C[World stays locked<br/>to the room]
  B --> D[Hands and controllers<br/>lag the real ones]
  B --> E[Animation and other users<br/>stutter]
  B --> F[Edges of the image<br/>exposed on a large turn]
  C --> G[Comfort mostly preserved]
  D --> H[Visibly wrong, but tolerable]

Follow the branch out of the reprojection box rather than the box itself. The safety net catches exactly one thing — the world's apparent position relative to the room — and nothing else. Treating it as headroom is a common and expensive mistake, because a title that habitually misses and relies on reprojection produces lagging hands, stuttering animation and smearing on moving objects while its frame counter looks acceptable.

The second difficulty is that the important defects are invisible in the artefacts you normally review. A screenshot is one eye. A screen recording is one eye at the wrong frame rate. An average frame rate hides exactly the frames that hurt, because a single long frame per second barely moves a mean. So a whole class of bug — a shader that ignores the eye index and renders both views from the left eye's position, a mis-sorted transparent surface that reads as wrong depth, a hand that lags a stable world — passes review and reaches users. Building the instruments that make these visible, per-eye captures and per-frame time series and deadline miss counts, is a real part of the job and is not something a general graphics background teaches you.

The third is thermal, and it is the one people forget until a launch. A standalone headset is a battery-powered computer strapped to a face with no room for a large cooler. Sustained load raises the temperature until clocks are reduced, so a frame that fits comfortably in the first minute may not fit in the twentieth. This inverts the usual optimisation instinct in a useful way: memory traffic is a large share of the power budget, so reducing bandwidth buys you both time and thermal headroom, whereas shortening a shader that was never the constraint buys neither. It also changes what a performance claim means. "It runs at ninety hertz" is only meaningful as "it holds ninety hertz after twenty minutes of representative use on a device that has been running the whole time".

Underneath all three is a quieter difficulty: the feedback signal is partly subjective. Some defects have no numeric signature you will find first, and you find them by putting the headset on. Experience is genuinely not substitutable here, because knowing that a particular wobble means prediction overshoot rather than a dropped frame is pattern recognition built from having felt both.

Why study it

Study it if the intersection is what appeals: real-time systems with a hard deadline, graphics, and a human being in the loop whose physiology is part of the specification. There are not many places in software where all three apply at once, and people who like this work tend to like it a great deal.

Study it also for the transferable core, which is larger than it looks. The discipline teaches performance engineering in its most honest form. You learn to find the binding constraint before changing anything, to distinguish a compute-bound frame from a bandwidth-bound one and to know which optimisations are therefore pointless, to instrument by percentile and deadline miss rather than by average, and to treat variance as the enemy rather than mean cost. Those habits transfer directly into game development, mobile graphics, embedded and signal-processing work, and any latency-sensitive backend where tail behaviour decides whether a service is acceptable. Engineers who have held a frame budget on mobile-class hardware are noticeably better at capacity arguments in unrelated domains.

Do not study it, however, if the goal is the shortest route to a well-paid engineering job, because it is not that route. The market is small, cyclical and concentrated, and a candidate who has spent a year on headset rendering has narrowed their options relative to one who spent it on backend or data engineering. If graphics is the actual attraction, general real-time rendering serves the same interest with a larger employer pool, and games and mobile graphics will teach you most of the same pipeline. And if the attraction is the product vision rather than the engineering, interaction design or product work in the same industry may fit better than the renderer.

There is one strong pragmatic case worth stating. If you already work in real-time graphics or mobile performance, the incremental cost of becoming competent here is small — the stereo asymmetry, the compositor's role, the tracking pipeline and the comfort constraints are a few weekends of study on top of what you have — and it gives you access to a hiring pool that is small but also thinly contested. Being one of a few hundred credible candidates for a narrow role is not obviously worse than being one of many thousands for a broad one.

Your first hour

Do not start by installing an engine, because you will spend the hour on the editor rather than on the subject. Start with the arithmetic and one small artefact, and finish with a number you derived yourself.

Spend the first fifteen minutes writing out the frame budget for three refresh rates by hand, exactly as it appears earlier on this page. Then extend it: pick a refresh rate, assume the application gets the interval minus two milliseconds for the compositor, and work out how much time one eye view gets if fragment shading dominates. Then ask what happens to that number when the device warms up and clocks drop by a fifth. You now have the argument that most spatial computing decisions descend from, in your own handwriting.

Spend the next twenty minutes on the transform chain, on paper. Write the multiplication that takes a vertex from the model an artist authored to a position on screen, naming each space and what it exists for. Then take one small step beyond recitation: mark which matrices in that chain differ between the left and right eye and which are shared. That single distinction is the foundation of every stereo optimisation, and it is the answer to a question you will be asked.

Spend the remaining time building something you can look at, and the lowest-friction route is the browser. A WebXR-capable page can be served from a local static server and opened on a headset over the network, or run in an emulated device in a desktop browser if you do not have hardware to hand. The exercise that teaches most is deliberately small: render one cube that stays in one place in the room while you move around it, then log per-frame timing to the console and look at the distribution rather than the average. If you have a headset, the more instructive version is to make the cube head-locked instead of world-locked and wear it for sixty seconds, because the discomfort you feel is the whole subject arriving through your inner ear rather than through a document.

If a browser is not available to you, an equivalent first artefact is a frame capture. Take any real-time application you already have, capture one frame in a graphics debugger, and attribute its time to passes. The skill being built is the same: attributing time before changing anything.

Read the fundamentals sheet for this section afterwards rather than first. It is written to be skimmed once the vocabulary has something to attach to, and an hour of doing the arithmetic yourself makes it read as confirmation instead of as a list.

What this is not

It is not three-dimensional content creation. Modelling, texturing, rigging and lighting are a different craft with different tools, and although the constraints you impose shape what artists can make, you will not be asked to make it.

It is not computer vision research. The algorithms that turn camera frames into a pose — feature detection, mapping, relocalisation, sensor fusion — are a specialism with its own literature and its own hiring pipeline. Spatial computing engineers are expected to know what the tracking system provides, how stale it is, how it fails and what to do when it does. They are generally not expected to write it.

It is not the same as game development, though the overlap is large and many people move between them. A game is a design and content problem with a rendering problem inside it; this area is the rendering and platform problem with the design owned by someone else. The skills that dominate here — budget arithmetic, capture analysis, stereo correctness, thermal behaviour — are a subset of game development that has been made unusually strict.

It is not "the metaverse", and it is worth separating the engineering from the product narrative deliberately, because the narrative's collapse caused capable engineers to write off a field that was never dependent on it. Shared persistent virtual worlds are a product bet. Rendering two views of a scene on a deadline is a technical discipline with paying customers in training, medicine, industry and entertainment regardless of how that bet resolves.

And it is not a field where knowing a particular device's specification is the skill. Named hardware details go stale within a product cycle and are checkable in seconds, so asserting one you are not sure of is the worst trade available in an interview. Reasoning from mechanism — what the compositor must do, why bandwidth dominates on tile-based hardware, what a late frame does to a person — travels between devices and between years, which is exactly why interviewers ask for it.

A dropped frame in a headset is not a performance regression, it is something a person feels in their body; every other rule in this area is downstream of that.