In this section, we explain how to configure and compile the CPython project with a C
. We will not explain how to write a C extension module and prefer to give you some links where you can read good documentation:
https://docs.python.org/dev/c-api/
https://docs.python.org/dev/extending/
https://pythonextensionpatterns.readthedocs.io/en/latest/
Some modules in the standard library, such as
or
, have identical implementations in C and Python; the C implementation, when available, is expected to improve performance (such extension modules are commonly referred to as accelerator modules).
Other modules mainly implemented in Python may import a C helper extension providing implementation details (for instance, the
module uses the internal _csv module defined in
).
Classifying extension modules
Extension modules can be classified into two categories:
A built-in extension module is a module built and shipped with the Python interpreter. A built-in module is statically linked into the interpreter, thereby lacking a __file__ attribute.
Built-in modules are built with the Py_BUILD_CORE_BUILTIN macro defined.
A shared (or dynamic) extension module is built as a shared library (.so or .dll file) and is dynamically linked into the interpreter.
In particular, the module’s __file__ attribute contains the path to the .so or .dll file.
Shared modules are built with the Py_BUILD_CORE_MODULE macro defined. Using the Py_BUILD_CORE_BUILTIN macro instead causes an
when importing the module.
Note
Informally, built-in extension modules can be regarded as required while shared extension modules are optional in the sense that they might be supplied, overridden or disabled externally.
Usually, accelerator modules are built as shared extension modules, especially if they already have a pure Python implementation.
According to
, new extension modules MUST provide a working and tested pure Python implementation, unless a special dispensation from the
is given.
Adding an extension module to CPython
Assume that the standard library contains a pure Python module foo with the following foo.greet() function:
Lib/foo.py
defgreet():return"Hello World!"Instead of using the Python implementation of foo.greet(), we want to use its corresponding C extension implementation exposed in the _foo module. Ideally, we want to modify Lib/foo.py as follows:
Lib/foo.py
try:# use the C implementation if possiblefrom_fooimportgreetexceptImportError:# fallback to the pure Python implementationdefgreet():return"Hello World!"Note
Accelerator modules should never be imported directly. The convention is to mark them as private implementation details with the underscore prefix (namely, _foo in this example).
In order to incorporate the accelerator module, we need to determine:
where to update the CPython project tree with the extension module source code,
which files to modify to configure and compile the CPython project, and
which Makefile rules to invoke at the end.
Updating the CPython project tree
Usually, accelerator modules are added in the
directory of the CPython project. If more than one file is needed for the extension module, it is more convenient to create a sub-directory in
.
In the simplest example where the extension module consists of one file, it may be placed in
as Modules/_foomodule.c. For a non-trivial example of the extension module _foo, we consider the following working tree:
— the extension module implementation.
— the extension helpers declarations.
— the extension helpers implementations.
By convention, the source file containing the extension module implementation is called <NAME>module.c, where <NAME> is the name of the module that will be later imported (in our case _foo). In addition, the directory containing the implementation should also be named similarly.
Modules/_foo/helper.h
#ifndef _FOO_HELPER_H#define _FOO_HELPER_H#include"Python.h"typedefstruct{/* ... */}foomodule_state;staticinlinefoomodule_state*get_foomodule_state(PyObject*module){void*state=PyModule_GetState(module);assert(state!=NULL);return(foomodule_state*)state;}/* Helper used in Modules/_foo/_foomodule.c * but implemented in Modules/_foo/helper.c. */externPyObject*_Py_greet_fast(void);#endif // _FOO_HELPER_HTip
Functions or data that do not need to be shared across different C source files should be declared static to avoid exporting their symbols from libpython.
If symbols need to be exported, their names must start with Py or _Py. This can be verified by makesmelly. For more details, please refer to the section on
.
Modules/_foo/helper.c
#include"_foomodule.h"PyObject*_Py_greet_fast(void){returnPyUnicode_FromString("Hello World!");}Modules/_foo/_foomodule.c
#include"helper.h"#include"clinic/_foomodule.c.h"/* Functions for the extension module's state */staticintfoomodule_exec(PyObject*module){// imports, static attributes, exported classes, etcreturn0;}staticintfoomodule_traverse(PyObject*m,visitprocvisit,void*arg){foomodule_state*st=get_foomodule_state(m);// call Py_VISIT() on the state attributesreturn0;}staticintfoomodule_clear(PyObject*m){foomodule_state*st=get_foomodule_state(m);// call Py_CLEAR() on the state attributesreturn0;}staticvoidfoomodule_free(void*m){(void)foomodule_clear((PyObject*)m);}/* Implementation of publicly exported functions. *//*[clinic input]module foo[clinic start generated code]*//*[clinic end generated code: output=... input=...]*//*[clinic input]foo.greet -> object[clinic start generated code]*/staticPyObject*foo_greet_impl(PyObject*module)/*[clinic end generated code: output=... input=...]*/{return_Py_greet_fast();}/* Exported module's data */staticPyMethodDeffoomodule_methods[]={// macro in 'clinic/_foomodule.c.h' after running 'make clinic'FOO_GREET_METHODDEF{NULL,NULL}};staticstructPyModuleDef_Slotfoomodule_slots[]={// 'foomodule_exec' may be NULL if the state is trivial{Py_mod_exec,foomodule_exec},{Py_mod_multiple_interpreters,Py_MOD_PER_INTERPRETER_GIL_SUPPORTED},{Py_mod_gil,Py_MOD_GIL_NOT_USED},{0,NULL},};staticstructPyModuleDeffoomodule={PyModuleDef_HEAD_INIT,.m_name="_foo",.m_doc="some doc",// or NULL if not needed.m_size=sizeof(foomodule_state),.m_methods=foomodule_methods,.m_slots=foomodule_slots,.m_traverse=foomodule_traverse,// or NULL if the state is trivial.m_clear=foomodule_clear,// or NULL if the state is trivial.m_free=foomodule_free,// or NULL if the state is trivial};PyMODINIT_FUNCPyInit__foo(void){returnPyModuleDef_Init(&foomodule);}Tip
Recall that the PyInit_<NAME> function must be suffixed by the module name <NAME> used in import statements (here _foo), and which usually coincides with
.
Other identifiers such as those used in
inputs do not have such naming requirements.
Configuring the CPython project
Now that we have added our extension module to the CPython source tree, we need to update some configuration files in order to compile the CPython project on different platforms.
Updating Modules/Setup.{bootstrap,stdlib}.in
Depending on whether the extension module is required to get a functioning interpreter or not, we update
or
. In the former case, the extension module is necessarily built as a built-in extension module.
For built-in extension modules, update
by adding the following line after the *static* marker:
*static* ... _foo _foo/_foomodule.c _foo/helper.c ... The syntax is <NAME><SOURCES> where <NAME> is the name of the module used in
statements and <SOURCES> is the list of space-separated source files.
For other extension modules, update
by adding the following line after the *@MODULE_BUILDTYPE@* marker but before the *shared* marker:
*@MODULE_BUILDTYPE@* ... @MODULE__FOO_TRUE@_foo _foo/_foomodule.c _foo/helper.c ... *shared* The @MODULE_<NAME_UPPER>_TRUE@<NAME> marker expects <NAME_UPPER> to be the upper-cased form of <NAME>, where <NAME> has the same meaning as before (in our case, <NAME_UPPER> and <NAME> are _FOO and _foo respectively). The marker is followed by the list of source files.
If the extension module must be built as a shared module, put the @MODULE__FOO_TRUE@_foo line after the *shared* marker:
... *shared* ... @MODULE__FOO_TRUE@_foo _foo/_foomodule.c _foo/helper.c Updating
Locate the SRCDIRS variable and add the following line:
AC_SUBST([SRCDIRS]) SRCDIRS="\ ... Modules/_foo \ ..." Note
This step is only needed when adding new source directories to the CPython project.
Find the section containing PY_STDLIB_MOD and PY_STDLIB_MOD_SIMPLE usages and add the following line:
dnl always enabled extension modules ... PY_STDLIB_MOD_SIMPLE([_foo], [-I\$(srcdir)/Modules/_foo], []) ... The PY_STDLIB_MOD_SIMPLE macro takes as arguments:
the module name <NAME> used in
statements,
the compiler flags (CFLAGS), and
the linker flags (LDFLAGS).
If the extension module may not be enabled or supported depending on the host configuration, use the PY_STDLIB_MOD macro instead, which takes as arguments:
the module name <NAME> used in
statements,
a boolean indicating whether the extension is enabled or not,
a boolean indicating whether the extension is supported or not,
the compiler flags (CFLAGS), and
the linker flags (LDFLAGS).
For instance, enabling the _foo extension on Linux platforms, but only providing support for 32-bit architecture, is achieved as follows:
PY_STDLIB_MOD([_foo], [test "$ac_sys_system" = "Linux"], [test "$ARCH_RUN_32BIT" = "true"], [-I\$(srcdir)/Modules/_foo], []) More generally, the host’s configuration status of the extension is determined as follows:
Enabled
Supported
Status
true
true
yes
true
false
missing
false
true or false
disabled
The extension status is n/a if the extension is marked unavailable by the PY_STDLIB_MOD_SET_NA macro. To mark an extension as unavailable, find the usages of PY_STDLIB_MOD_SET_NA in
and add the following line:
dnl Modules that are not available on some platforms AS_CASE([$ac_sys_system], ... [PLATFORM_NAME], [PY_STDLIB_MOD_SET_NA([_foo])], ... )
Tip
Consider reading the comments and configurations for existing modules in
for guidance on adding new external build dependencies for extension modules that need them.
Updating
If needed, add the following line to the section for module dependencies:
########################################################################## # Module dependencies and platform-specific files ... MODULE__FOO_DEPS=$(srcdir)/Modules/_foo/helper.h ... The MODULE_<NAME_UPPER>_DEPS variable follows the same naming requirements as the @MODULE_<NAME_UPPER>_TRUE@<NAME> marker.
Updating MSVC project files
We describe the minimal steps for compiling on Windows using MSVC.
Update
:
...// add the entry point prototypeexternPyObject*PyInit__foo(void);...// update the entry points tablestruct_inittab_PyImport_Inittab[]={...{"_foo",PyInit__foo},...{0,0}};...Each item in _PyImport_Inittab consists of the module name to import, here _foo, with the corresponding PyInit_* function correctly suffixed.
Update
:
<!-- group with header files ..\Modules\<MODULE>.h --><ItemGroup>... <ClIncludeInclude="..\Modules\_foo\helper.h"/>... </ItemGroup><!-- group with source files ..\Modules\<MODULE>.c --><ItemGroup>... <ClCompileInclude="..\Modules\_foo\_foomodule.c"/><ClCompileInclude="..\Modules\_foo\helper.c"/>... </ItemGroup>
Update
PCbuild/pythoncore.vcxproj.filters
:
<!-- group with header files ..\Modules\<MODULE>.h --><ItemGroup>... <ClIncludeInclude="..\Modules\_foo\helper.h"><Filter>Modules\_foo</Filter></ClInclude>... </ItemGroup><!-- group with source files ..\Modules\<MODULE>.c --><ItemGroup>... <ClCompileInclude="..\Modules\_foo\_foomodule.c"><Filter>Modules\_foo</Filter></ClCompile><ClCompileInclude="..\Modules\_foo\helper.c"><Filter>Modules\_foo</Filter></ClCompile>... <ItemGroup>
Tip
Header files use <ClInclude> tags, whereas source files use <ClCompile> tags.
Compiling the CPython project
Now that the configuration is in place, it remains to compile the project:
makeregen-configure ./configure makeregen-all makeregen-stdlib-module-names make Tip
Use make-jN to speed up compilation by utilizing as many CPU cores as possible, where N is as many CPU cores you want to spare (and have memory for). Be careful using make-j with no argument, as this puts no limit on the number of jobs, and compilation can sometimes use up a lot of memory (like when building with LTO).
makeregen-configure updates the
script.
The
script must be generated using a specific version of autoconf. To that end, the
Tools/build/regen-configure.sh
script which the regen-configure rule is based on either requires Docker or Podman, the latter being assumed by default.
Tip
We recommend installing
instead of Docker since the former does not require a background service and avoids creating files owned by the root user in some cases.
makeregen-all is responsible for regenerating header files and invoking other scripts, such as
. Execute this rule if you do not know which files should be updated.
makeregen-stdlib-module-names updates the standard module names, making _foo discoverable and importable via import_foo.
The final make step is generally not needed since the previous make invocations may completely rebuild the project, but it could be needed in some specific cases.
Troubleshooting
This section addresses common issues that you may face when following this example of adding an extension module.
No rule to make target regen-configure
This usually happens after running makedistclean (which removes the Makefile). The solution is to regenerate the
script as follows:
./configure# for creating the 'Makefile' file makeregen-configure# for updating the 'configure' script ./configure# for updating the 'Makefile' fileIf missing, the
script can be regenerated by executing
Tools/build/regen-configure.sh
:
./Tools/build/regen-configure.sh# create an up-to-date 'configure' ./configure# create an up-to-date 'Makefile'makeregen-configure and missing permissions with Docker
If Docker complains about missing permissions, this Stack Overflow post could be useful in solving the issue:
How to fix docker: permission denied
. Alternatively, you may try using
.
Tips
In this section, we give some tips for improving the quality of extension modules meant to be included in the standard library.
Restricting to the Limited API
In order for non-CPython implementations to benefit from new extension modules, it is recommended to use the
. Instead of exposing the entire Stable ABI, define the
macro before the #include"Python.h" directive:
Using the 3.13 Limited API.
#include"pyconfig.h"// Py_GIL_DISABLED#ifndef Py_GIL_DISABLED# define Py_LIMITED_API 0x030d0000#endif#include"Python.h"This makes the extension module non-CPython implementation-friendly by removing the dependencies to CPython internals.