Customers need to upload two-gigabyte files to your API over a flaky connection. What does that endpoint look like, and what breaks if you just accept a POST body?
6 min readUpdated 2026-08-15Target archetype: Big Tech, Enterprise Captive, Product Startup
0
Answer
Short answer
A single large POST fails on the last percent and starts again, and it drags every proxy body limit and idle timeout into your critical path. Either hand out a scoped credential so bytes go straight to storage, or model the upload as a resource with a committed offset the client can resume from.
This is an HTTP file upload design problem. Good HTTP upload design uses resumable chunks, idempotent part commits, checksums, and a completion step that can be retried. Weak file upload designs stream the whole body through one request and make the user start again after a network drop. Strong file upload handling treats the upload as a session with recoverable state.
Was this answer helpful?
Likely follow-ups
→The client's connection drops at 1.8 GB. What exactly does the next request look like?
→Two clients start uploads for the same target resource at the same time. What happens?
→Where does virus scanning or transcoding fit, and what does the client see while it runs?
→Half your uploads are abandoned midway. What cleans them up, and how do you avoid deleting one that is still in progress?
Two gigabytes over a domestic connection is minutes of transfer, and the plain design commits everything to one request that must survive all of it. The first cost is that there is no resume point. A dropped connection at 1.8 GB does not leave 1.8 GB of progress anywhere useful, so the client starts from zero, and on a flaky link the probability of completing grows worse the larger the file gets. Users experience this as an endpoint that works for small files and is broken for large ones.
The second cost is that you have enlisted every intermediary into the critical path. A reverse proxy or gateway has a maximum request body size, and exceeding it returns a rejection after the client has already sent much of the data. Some proxies buffer the entire body before forwarding it, which turns your upload into a disk write on a shared component and adds the whole transfer time to the latency your application observes. Idle timeouts are the other half of this: they usually measure time with no bytes flowing rather than total duration, which is why a slow but steady upload survives and a client that stalls for thirty seconds mid-file does not.
The third cost is inside your process. If the handler reads the body into memory before doing anything, concurrent uploads multiply directly into heap, and a handful of them is an out-of-memory kill. If it streams to disk instead, you have bounded memory and introduced a local file whose cleanup you now own. Either way a worker is occupied for minutes doing nothing but copying bytes, which is the least valuable use of a thread in your service.
There are two designs that solve this properly, and they differ in who touches the bytes.
Get the bytes out of your request path
The strongest answer for most systems is that the file never transits your application. Your API authorises an upload and returns a short-lived, narrowly scoped credential for object storage, and the client sends the bytes there directly. What matters is the scoping: the credential should be limited to a single key you chose, a maximum size, an expiry measured in minutes, and where the storage supports it a required content type, so that possession of the credential does not become permission to write anything anywhere.
Loading diagram…
The awkward edge is the notification arrow. It can be delayed or lost, so the completion path cannot be the only way an upload becomes visible: you also need a reconciliation that lists objects for pending uploads and finishes the ones storage already holds. A design that assumes the event always arrives will lose a small, permanent fraction of uploads and will look fine in testing.
The other consequence is that you have not seen the file, so anything you wanted to validate about it has to happen afterwards rather than as a rejection. That is usually acceptable, and it means the record you create for the upload needs a state that says "bytes received but not yet accepted", which the client can poll and which your own code must never treat as usable. Storage-side lifecycle rules should expire incomplete multipart uploads and orphaned objects, because otherwise you are paying to store the abandoned half of every failed attempt indefinitely.
When the upload must come through you, model it as a resource
Some uploads have to pass through your service, whether because the storage is on your own infrastructure or because policy forbids handing out credentials. Then the fix is to stop thinking of the upload as a request and start thinking of it as a resource with state.
The client creates an upload with a small POST carrying metadata: intended size, content type, target, and a checksum if it can compute one. You reply with a created resource and its location. The client then sends the content in bounded parts against that resource, each part identified by its byte offset, and each part acknowledged only once it is durably stored. After any interruption the client asks the upload resource what its committed offset is and continues from there, which is the whole point of the shape: the server owns the answer to "how much do you have", because it is the only party that knows what was persisted rather than merely sent.
Two details separate a working protocol from a plausible one. Sending the same part twice must be harmless, so a part write is defined by its offset and length rather than by an implicit cursor, and re-sending a part that is already committed is a success rather than a duplication. And the offsets need to be carried in something you define. Range in HTTP is specified for retrieving parts of a representation, not for placing them, so a resumable upload protocol carries its own offset field rather than reusing range semantics for a direction they were not written for.
Finish with verification. The client's declared total and its checksum are checked when the last part lands, and the upload only transitions to complete if both agree with what you stored. Without that step, a truncated transfer that happened to end on a part boundary is indistinguishable from a finished one.
The HTTP details that are easy to get wrong
Expect: 100-continue earns its place here. It lets a client offer the headers first and wait for an interim response before sending the body, so a rejection for size, media type or authorisation arrives before two gigabytes have crossed the network rather than after. Support is uneven across intermediaries, which is a reason to design for it rather than depend on it, but a candidate who names it has thought about the wasted-transfer case.
Where the size is not known in advance, chunked transfer coding is what allows a body without a Content-Length. Be honest that this removes your ability to reject the request up front on size, which means the limit has to be enforced as you read, by counting bytes and aborting when the count exceeds what the upload declared. A limit that exists only as a check on a header is not a limit.
On status codes, the useful discipline is that each one tells the client what to do next. A created upload resource returns a location the client will use. A payload that exceeds the configured maximum is rejected as too large rather than as a generic bad request, because the client can act on that. An unsupported media type is its own answer. And if the file is accepted but not yet processed, say so with an accepted status and a state the client can poll, rather than returning success for work that has not happened.
What you must not trust
The filename, the declared content type and the declared size all come from the client and all can be wrong, deliberately. Derive the type from the content where it matters, store under a key you generated rather than a name the client chose, and treat the declared size as a hint you verify. Compressed formats need a bound on what they expand to, not only on what they arrived as. None of this is specific to HTTP, but an upload endpoint is where these mistakes are most expensive, because the artefact persists and something downstream will open it.