The same multiprocessing code runs fine on your Linux box and breaks on a colleague's Mac. What is different?
The start method. Under fork the child inherits the parent's whole memory image; under spawn, the default on macOS since Python 3.8 and the only option on Windows, it starts a fresh interpreter, re-imports your module and unpickles the arguments, so anything implicit stops working.
What the interviewer is scoring
- Does the candidate name the start method rather than attributing the difference to the operating system
- Whether the main-module guard is explained as a requirement of re-importing rather than as a convention
- That the picklability constraint on the callable and its arguments is stated
- Whether forking a process that already holds locks or threads is identified as unsafe
- Does the answer say how to make the code behave identically on both machines
Answer
There is no single way to create a process
multiprocessing supports several start methods, and which one you get depends on the platform and the Python version rather than on anything in your code. spawn is the only option on Windows, and it became the default on macOS in Python 3.8 because forking is unsafe there in the presence of system libraries that use threads. On Linux, fork was the default for a long time, and CPython has been steadily moving away from it for the same underlying reason: Python 3.12 emits a DeprecationWarning when os.fork() is called from a process that already has more than one thread. Because the default is a moving target, the reliable habit is to ask rather than assume — multiprocessing.get_start_method() tells you what you are getting — and to pin the method explicitly for anything that has to behave the same everywhere.
The two methods produce genuinely different children, and almost every cross-platform bug in this area is a piece of code that quietly depended on which one it had.
What fork gives you, for free and dangerously
fork clones the calling process. The child begins with a copy of the parent's entire memory image, which means every module already imported, every global variable already set, every open file descriptor, and every object the target function closes over. Nothing needs to be pickled and nothing needs to be re-imported, so worker startup is fast and code that reads a module-level connection pool or a preloaded dataframe simply works.
The danger is that it copies state that only made sense in the parent. Only the calling thread survives in the child, so any lock held by another thread at the instant of the fork is inherited in the locked state with no thread left alive to release it, and the child deadlocks the first time it touches that lock. Logging handlers, connection pools and anything with an internal mutex are the usual culprits, which is why the symptom is so often a worker that hangs rather than one that raises. Inherited descriptors are the second hazard: a database connection or socket used from both parent and child is two processes interleaving bytes on one stream, and it corrupts the protocol rather than failing cleanly.
What spawn demands of your code
spawn starts a fresh interpreter with nothing inherited. To hand it work, multiprocessing pickles the callable and its arguments, and the child re-imports the module the callable is defined in so that it can resolve it.
Both of those become requirements. The target and every argument must be picklable, which rules out lambdas, locally defined functions, open file handles, database connections, sockets and most objects wrapping a native resource. And because the child imports your main module, any code at module top level runs again in every worker, which is why the guard is mandatory rather than stylistic:
import multiprocessing as mp
def work(chunk):
return sum(chunk)
if __name__ == "__main__":
# Under spawn each child re-imports this module. Without the guard,
# that import would create another pool, recursively.
ctx = mp.get_context("spawn") # same behaviour on every platform
with ctx.Pool(4) as pool:
print(pool.map(work, [range(1000)] * 8))
The wider consequence is that anything the workers need must be passed rather than assumed. A model loaded into a global, a configuration object mutated at startup, an environment variable set in code after import — all present under fork and all absent under spawn. This is the specific failure behind "it works on my machine": the Linux developer never had to pass the thing, so the code never learned how.
forkserver, and picking one deliberately
forkserver is the compromise. At first use, Python starts a single-threaded server process, and each new worker is forked from that rather than from your main process. Startup is still cheap because it is a fork, but the thing being forked is clean, so the inherited-lock and inherited-thread hazards go away. The cost is that the child does not inherit your parent's state either, so it has the same picklability and re-import constraints as spawn.
The practical recommendation is to choose explicitly and write to the strictest method you might run under. Obtain a context with mp.get_context("spawn") and create pools from it rather than calling mp.Pool directly, or if you use concurrent.futures, pass that context as mp_context to ProcessPoolExecutor. If you would rather set a process-wide default, set_start_method can be called only once per process and belongs inside the main-module guard, before anything creates a pool.
Writing to spawn semantics costs a little explicitness: pass configuration as arguments, load large read-only data through an initialiser that runs once per worker rather than relying on inheritance, and keep connections out of the arguments entirely by opening them inside the worker. That code then runs identically under all three methods, which is worth more than the fork-inherited convenience it gives up.
The platform difference is a start-method difference.
forkinherits everything, including locks it cannot release;spawninherits nothing and re-imports your module, so it exposes every implicit dependency. Choose the method explicitly and write to the stricter one.
Likely follow-ups
- Why does forking a process that already has threads risk a deadlock in the child?
- What does forkserver give you that neither fork nor spawn does?
- How would you share a large read-only array with workers without pickling a copy per worker?
- Why must set_start_method be called only once, and where should that call live?
Related questions
- An operation in this code takes two locks and it deadlocks every few weeks. How would you design the locking so that cannot happen?hardAlso on deadlock6 min
- What does the GIL actually prevent, and how do you decide between threads, processes, and asyncio?mediumAlso on multiprocessing4 min
- Why does calling .Result or .Wait() on an async method deadlock, and how do you fix it properly?hardAlso on deadlock5 min
- You read a four gigabyte CSV into pandas and the process needs far more memory than that. Where did it go?mediumSame kind of round: concept4 min
- A list of a million small objects uses far more memory than the data inside them. Where is it going?mediumSame kind of round: concept5 min
- A promise rejects and nothing is awaiting it. What does Node do?hardSame kind of round: concept4 min
- Implement a FIFO queue using only two stacks, then tell me what a dequeue costs.mediumSame kind of round: coding4 min
- You kick off background work from a controller action and it throws once the response has gone out. What is happening?mediumSame kind of round: coding5 min