gh-134939: Add the concurrent.interpreters Module (gh-133958) · python/cpython@6214373

GitHub

@@ -83,6 +83,7 @@ and improvements in user-friendliness and correctness.

8383.. PEP-sized items next.

84848585* :ref:`PEP 649 and 749: deferred evaluation of annotations <whatsnew314-pep649>`

86+* :ref:`PEP 734: Multiple Interpreters in the Stdlib <whatsnew314-pep734>`

8687* :ref:`PEP 741: Python Configuration C API <whatsnew314-pep741>`

8788* :ref:`PEP 750: Template strings <whatsnew314-pep750>`

8889* :ref:`PEP 758: Allow except and except* expressions without parentheses <whatsnew314-pep758>`

@@ -123,6 +124,101 @@ of Python. See :ref:`below <whatsnew314-refcount>` for details.

123124New features

124125============

125126127+.. _whatsnew314-pep734:

128+129+PEP 734: Multiple Interpreters in the Stdlib

130+--------------------------------------------

131+132+The CPython runtime supports running multiple copies of Python in the

133+same process simultaneously and has done so for over 20 years.

134+Each of these separate copies is called an "interpreter".

135+However, the feature had been available only through the C-API.

136+137+That limitation is removed in the 3.14 release,

138+with the new :mod:`concurrent.interpreters` module.

139+140+There are at least two notable reasons why using multiple interpreters

141+is worth considering:

142+143+* they support a new (to Python), human-friendly concurrency model

144+* true multi-core parallelism

145+146+For some use cases, concurrency in software enables efficiency and

147+can simplify software, at a high level. At the same time, implementing

148+and maintaining all but the simplest concurrency is often a struggle

149+for the human brain. That especially applies to plain threads

150+(for example, :mod:`threading`), where all memory is shared between all threads.

151+152+With multiple isolated interpreters, you can take advantage of a class

153+of concurrency models, like CSP or the actor model, that have found

154+success in other programming languages, like Smalltalk, Erlang,

155+Haskell, and Go. Think of multiple interpreters like threads

156+but with opt-in sharing.

157+158+Regarding multi-core parallelism: as of the 3.12 release, interpreters

159+are now sufficiently isolated from one another to be used in parallel.

160+(See :pep:`684`.) This unlocks a variety of CPU-intensive use cases

161+for Python that were limited by the :term:`GIL`.

162+163+Using multiple interpreters is similar in many ways to

164+:mod:`multiprocessing`, in that they both provide isolated logical

165+"processes" that can run in parallel, with no sharing by default.

166+However, when using multiple interpreters, an application will use

167+fewer system resources and will operate more efficiently (since it

168+stays within the same process). Think of multiple interpreters as

169+having the isolation of processes with the efficiency of threads.

170+171+.. XXX Add an example or two.

172+.. XXX Link to the not-yet-added HOWTO doc.

173+174+While the feature has been around for decades, multiple interpreters

175+have not been used widely, due to low awareness and the lack of a stdlib

176+module. Consequently, they currently have several notable limitations,

177+which will improve significantly now that the feature is finally

178+going mainstream.

179+180+Current limitations:

181+182+* starting each interpreter has not been optimized yet

183+* each interpreter uses more memory than necessary

184+ (we will be working next on extensive internal sharing between

185+ interpreters)

186+* there aren't many options *yet* for truly sharing objects or other

187+ data between interpreters (other than :type:`memoryview`)

188+* many extension modules on PyPI are not compatible with multiple

189+ interpreters yet (stdlib extension modules *are* compatible)

190+* the approach to writing applications that use multiple isolated

191+ interpreters is mostly unfamiliar to Python users, for now

192+193+The impact of these limitations will depend on future CPython

194+improvements, how interpreters are used, and what the community solves

195+through PyPI packages. Depending on the use case, the limitations may

196+not have much impact, so try it out!

197+198+Furthermore, future CPython releases will reduce or eliminate overhead

199+and provide utilities that are less appropriate on PyPI. In the

200+meantime, most of the limitations can also be addressed through

201+extension modules, meaning PyPI packages can fill any gap for 3.14, and

202+even back to 3.12 where interpreters were finally properly isolated and

203+stopped sharing the :term:`GIL`. Likewise, we expect to slowly see

204+libraries on PyPI for high-level abstractions on top of interpreters.

205+206+Regarding extension modules, work is in progress to update some PyPI

207+projects, as well as tools like Cython, pybind11, nanobind, and PyO3.

208+The steps for isolating an extension module are found at

209+:ref:`isolating-extensions-howto`. Isolating a module has a lot of

210+overlap with what is required to support

211+:ref:`free-threading <whatsnew314-free-threaded-cpython>`,

212+so the ongoing work in the community in that area will help accelerate

213+support for multiple interpreters.

214+215+Also added in 3.14: :ref:`concurrent.futures.InterpreterPoolExecutor

216+<whatsnew314-concurrent-futures-interp-pool>`.

217+218+.. seealso::

219+:pep:`734`.

220+221+126222.. _whatsnew314-pep750:

127223128224PEP 750: Template strings

@@ -1109,6 +1205,8 @@ calendar

11091205concurrent.futures

11101206------------------

111112071208+.. _whatsnew314-concurrent-futures-interp-pool:

1209+11121210* Add :class:`~concurrent.futures.InterpreterPoolExecutor`,

11131211 which exposes "subinterpreters" (multiple Python interpreters in the

11141212 same process) to Python code. This is separate from the proposed API