@@ -19,7 +19,7 @@ offers both local and remote concurrency, effectively side-stepping the
1919:term:`Global Interpreter Lock <global interpreter lock>` by using
2020subprocesses instead of threads. Due
2121to this, the :mod:`multiprocessing` module allows the programmer to fully
22-leverage multiple processors on a given machine. It runs on both Unix and
22+leverage multiple processors on a given machine. It runs on both POSIX and
2323Windows.
24242525The :mod:`multiprocessing` module also introduces APIs which do not have
@@ -99,11 +99,11 @@ necessary, see :ref:`multiprocessing-programming`.
9999100100101101102+.. _multiprocessing-start-methods:
103+102104Contexts and start methods
103105~~~~~~~~~~~~~~~~~~~~~~~~~~
104106105-.. _multiprocessing-start-methods:
106-107107Depending on the platform, :mod:`multiprocessing` supports three ways
108108to start a process. These *start methods* are
109109@@ -115,7 +115,7 @@ to start a process. These *start methods* are
115115 will not be inherited. Starting a process using this method is
116116 rather slow compared to using *fork* or *forkserver*.
117117118- Available on Unix and Windows. The default on Windows and macOS.
118+ Available on POSIX and Windows platforms. The default on Windows and macOS.
119119120120 *fork*
121121 The parent process uses :func:`os.fork` to fork the Python
@@ -124,32 +124,39 @@ to start a process. These *start methods* are
124124 inherited by the child process. Note that safely forking a
125125 multithreaded process is problematic.
126126127- Available on Unix only. The default on Unix.
127+ Available on POSIX systems. Currently the default on POSIX except macOS.
128128129129 *forkserver*
130130 When the program starts and selects the *forkserver* start method,
131- a server process is started. From then on, whenever a new process
131+ a server process is spawned. From then on, whenever a new process
132132 is needed, the parent process connects to the server and requests
133- that it fork a new process. The fork server process is single
134- threaded so it is safe for it to use :func:`os.fork`. No
135- unnecessary resources are inherited.
133+ that it fork a new process. The fork server process is single threaded
134+ unless system libraries or preloaded imports spawn threads as a
135+ side-effect so it is generally safe for it to use :func:`os.fork`.
136+ No unnecessary resources are inherited.
136137137- Available on Unix platforms which support passing file descriptors
138- over Unix pipes.
138+ Available on POSIX platforms which support passing file descriptors
139+ over Unix pipes such as Linux.
140+141+.. versionchanged:: 3.12
142+ Implicit use of the *fork* start method as the default now raises a
143+:exc:`DeprecationWarning`. Code that requires it should explicitly
144+ specify *fork* via :func:`get_context` or :func:`set_start_method`.
145+ The default will change away from *fork* in 3.14.
139146140147.. versionchanged:: 3.8
141148142149 On macOS, the *spawn* start method is now the default. The *fork* start
143150 method should be considered unsafe as it can lead to crashes of the
144- subprocess. See :issue:`33725`.
151+ subprocess as macOS system libraries may start threads. See :issue:`33725`.
145152146153.. versionchanged:: 3.4
147- *spawn* added on all Unix platforms, and *forkserver* added for
148- some Unix platforms.
154+ *spawn* added on all POSIX platforms, and *forkserver* added for
155+ some POSIX platforms.
149156 Child processes no longer inherit all of the parents inheritable
150157 handles on Windows.
151158152-On Unix using the *spawn* or *forkserver* start methods will also
159+On POSIX using the *spawn* or *forkserver* start methods will also
153160start a *resource tracker* process which tracks the unlinked named
154161system resources (such as named semaphores or
155162:class:`~multiprocessing.shared_memory.SharedMemory` objects) created
@@ -211,10 +218,10 @@ library user.
211218212219.. warning::
213220214- The ``'spawn'`` and ``'forkserver'`` start methods cannot currently
221+ The ``'spawn'`` and ``'forkserver'`` start methods generally cannot
215222 be used with "frozen" executables (i.e., binaries produced by
216- packages like **PyInstaller** and **cx_Freeze**) on Unix.
217- The ``'fork'`` start method does work.
223+ packages like **PyInstaller** and **cx_Freeze**) on POSIX systems.
224+ The ``'fork'`` start method may work if code does not use threads.
218225219226220227Exchanging objects between processes
@@ -629,14 +636,14 @@ The :mod:`multiprocessing` package mostly replicates the API of the
629636 calling :meth:`join()` is simpler.
630637631638 On Windows, this is an OS handle usable with the ``WaitForSingleObject``
632- and ``WaitForMultipleObjects`` family of API calls. On Unix, this is
639+ and ``WaitForMultipleObjects`` family of API calls. On POSIX, this is
633640 a file descriptor usable with primitives from the :mod:`select` module.
634641635642 .. versionadded:: 3.3
636643637644 .. method:: terminate()
638645639- Terminate the process. On Unix this is done using the ``SIGTERM`` signal;
646+ Terminate the process. On POSIX this is done using the ``SIGTERM`` signal;
640647 on Windows :c:func:`TerminateProcess` is used. Note that exit handlers and
641648 finally clauses, etc., will not be executed.
642649@@ -653,7 +660,7 @@ The :mod:`multiprocessing` package mostly replicates the API of the
653660654661 .. method:: kill()
655662656- Same as :meth:`terminate()` but using the ``SIGKILL`` signal on Unix.
663+ Same as :meth:`terminate()` but using the ``SIGKILL`` signal on POSIX.
657664658665 .. versionadded:: 3.7
659666@@ -676,16 +683,17 @@ The :mod:`multiprocessing` package mostly replicates the API of the
676683 .. doctest::
677684678685 >>> import multiprocessing, time, signal
679- >>> p = multiprocessing.Process(target=time.sleep, args=(1000,))
686+ >>> mp_context = multiprocessing.get_context('spawn')
687+ >>> p = mp_context.Process(target=time.sleep, args=(1000,))
680688 >>> print(p, p.is_alive())
681-<Process ... initial> False
689+<...Process ... initial> False
682690 >>> p.start()
683691 >>> print(p, p.is_alive())
684-<Process ... started> True
692+<...Process ... started> True
685693 >>> p.terminate()
686694 >>> time.sleep(0.1)
687695 >>> print(p, p.is_alive())
688-<Process ... stopped exitcode=-SIGTERM> False
696+<...Process ... stopped exitcode=-SIGTERM> False
689697 >>> p.exitcode ==-signal.SIGTERM
690698True
691699@@ -815,7 +823,7 @@ For an example of the usage of queues for interprocess communication see
815823 Return the approximate size of the queue. Because of
816824 multithreading/multiprocessing semantics, this number is not reliable.
817825818- Note that this may raise :exc:`NotImplementedError` on Unix platforms like
826+ Note that this may raise :exc:`NotImplementedError` on platforms like
819827 macOS where ``sem_getvalue()`` is not implemented.
820828821829 .. method:: empty()
@@ -1034,9 +1042,8 @@ Miscellaneous
1034104210351043 Returns a list of the supported start methods, the first of which
10361044 is the default. The possible start methods are ``'fork'``,
1037- ``'spawn'`` and ``'forkserver'``. On Windows only ``'spawn'`` is
1038- available. On Unix ``'fork'`` and ``'spawn'`` are always
1039- supported, with ``'fork'`` being the default.
1045+ ``'spawn'`` and ``'forkserver'``. Not all platforms support all
1046+ methods. See :ref:`multiprocessing-start-methods`.
1040104710411048 .. versionadded:: 3.4
10421049@@ -1048,7 +1055,7 @@ Miscellaneous
10481055 If *method* is ``None`` then the default context is returned.
10491056 Otherwise *method* should be ``'fork'``, ``'spawn'``,
10501057 ``'forkserver'``. :exc:`ValueError` is raised if the specified
1051- start method is not available.
1058+ start method is not available. See :ref:`multiprocessing-start-methods`.
1052105910531060 .. versionadded:: 3.4
10541061@@ -1062,8 +1069,7 @@ Miscellaneous
10621069 is true then ``None`` is returned.
1063107010641071 The return value can be ``'fork'``, ``'spawn'``, ``'forkserver'``
1065- or ``None``. ``'fork'`` is the default on Unix, while ``'spawn'`` is
1066- the default on Windows and macOS.
1072+ or ``None``. See :ref:`multiprocessing-start-methods`.
1067107310681074.. versionchanged:: 3.8
10691075@@ -1084,11 +1090,26 @@ Miscellaneous
10841090 before they can create child processes.
1085109110861092 .. versionchanged:: 3.4
1087- Now supported on Unix when the ``'spawn'`` start method is used.
1093+ Now supported on POSIX when the ``'spawn'`` start method is used.
1088109410891095 .. versionchanged:: 3.11
10901096 Accepts a :term:`path-like object`.
109110971098+.. function:: set_forkserver_preload(module_names)
1099+1100+ Set a list of module names for the forkserver main process to attempt to
1101+ import so that their already imported state is inherited by forked
1102+ processes. Any :exc:`ImportError` when doing so is silently ignored.
1103+ This can be used as a performance enhancement to avoid repeated work
1104+ in every process.
1105+1106+ For this to work, it must be called before the forkserver process has been
1107+ launched (before creating a :class:`Pool` or starting a :class:`Process`).
1108+1109+ Only meaningful when using the ``'forkserver'`` start method.
1110+1111+ .. versionadded:: 3.4
1112+10921113.. function:: set_start_method(method, force=False)
1093111410941115 Set the method which should be used to start child processes.
@@ -1102,6 +1123,8 @@ Miscellaneous
11021123 protected inside the ``if __name__ == '__main__'`` clause of the
11031124 main module.
110411251126+ See :ref:`multiprocessing-start-methods`.
1127+11051128 .. versionadded:: 3.4
1106112911071130.. note::
@@ -1906,7 +1929,8 @@ their parent process exits. The manager classes are defined in the
1906192919071930 .. doctest::
190819311909- >>> manager = multiprocessing.Manager()
1932+ >>> mp_context = multiprocessing.get_context('spawn')
1933+ >>> manager = mp_context.Manager()
19101934 >>> Global = manager.Namespace()
19111935 >>> Global.x =10
19121936 >>> Global.y ='hello'
@@ -2018,8 +2042,8 @@ the proxy). In this way, a proxy can be used just like its referent can:
2018204220192043.. doctest::
202020442021- >>> from multiprocessing import Manager
2022- >>> manager = Manager()
2045+ >>> mp_context = multiprocessing.get_context('spawn')
2046+ >>> manager =mp_context.Manager()
20232047 >>> l = manager.list([i*i for i inrange(10)])
20242048 >>> print(l)
20252049[0, 1, 4, 9, 16, 25, 36, 49, 64, 81]
@@ -2520,7 +2544,7 @@ multiple connections at the same time.
25202544 *timeout* is ``None`` then it will block for an unlimited period.
25212545 A negative timeout is equivalent to a zero timeout.
252225462523- For both Unix and Windows, an object can appear in *object_list* if
2547+ For both POSIX and Windows, an object can appear in *object_list* if
25242548 it is
2525254925262550 * a readable :class:`~multiprocessing.connection.Connection` object;
@@ -2531,7 +2555,7 @@ multiple connections at the same time.
25312555 A connection or socket object is ready when there is data available
25322556 to be read from it, or the other end has been closed.
253325572534- **Unix**: ``wait(object_list, timeout)`` almost equivalent
2558+ **POSIX**: ``wait(object_list, timeout)`` almost equivalent
25352559 ``select.select(object_list, [], [], timeout)``. The difference is
25362560 that, if :func:`select.select` is interrupted by a signal, it can
25372561 raise :exc:`OSError` with an error number of ``EINTR``, whereas
@@ -2803,7 +2827,7 @@ Thread safety of proxies
2803282728042828Joining zombie processes
280528292806- On Unix when a process finishes but has not been joined it becomes a zombie.
2830+ On POSIX when a process finishes but has not been joined it becomes a zombie.
28072831 There should never be very many because each time a new process starts (or
28082832:func:`~multiprocessing.active_children` is called) all completed processes
28092833 which have not yet been joined will be joined. Also calling a finished
@@ -2866,7 +2890,7 @@ Joining processes that use queues
2866289028672891Explicitly pass resources to child processes
286828922869- On Unix using the *fork* start method, a child process can make
2893+ On POSIX using the *fork* start method, a child process can make
28702894 use of a shared resource created in a parent process using a
28712895 global resource. However, it is better to pass the object as an
28722896 argument to the constructor for the child process.