Inspecting C Extension Modules

[gobater]

I have developed a python extension module in "C" (https://docs.python.org/3/c-api/index.html) and I am facing several issues when trying to generate some documentation with mkdocsstrings-python and griffe.

Several issues:
1. Constructor
The C API does not allow to specify a docstring for the __init__ slot (constructor) and therefore they always have a default signature:

def __init__(self, *args, **kwargs)

I try to overcome that limitation using a stub (and triggering griffe with '-B') for my module but somehow griffe does not produce the output I expect.

The stub file contains following definition

class CLASS_NAME:
    def __init__(self, filename: Union[str, PurePath]) -> None:
        """CLASS_NAME(filename:str)
        --

        Constructor
        """
        ...

see here some command line DEBUG output

DEBUG Loading path C:\XXX_core.cp310-win_amd64.pyd
DEBUG Loading path C:\XXX_core.pyi
DEBUG Trying to merge C:\XXX_core.cp310-win_amd64.pyd and C:\XXX_core.pyi

The output JSON, for the constructor provides the following information:

"__init__": {
"decorators": [],
"docstring": {
  "endlineno": null,
  "lineno": null,
  "parsed": [
	{
	  "kind": "text",
	  "value": "Initialize self.  See help(type(self)) for accurate signature."
	}
  ],
  "value": "Initialize self.  See help(type(self)) for accurate signature."
},
"filepath": "C:\\XXX\\_core.cp310-win_amd64.pyd",
"kind": "function",
"labels": [
  "method descriptor"
],
"members": {},
"name": "__init__",
"parameters": [
  {
	"annotation": null,
	"default": null,
	"kind": "positional-only",
	"name": "self"
  },
  {
	"annotation": null,
	"default": null,
	"kind": "variadic positional",
	"name": "args"
  },
  {
	"annotation": null,
	"default": null,
	"kind": "variadic keyword",
	"name": "kwargs"
  }
],
"path": "PACKAGE_NAME._core.CLASS_NAME.__init__",
"relative_filepath": "PACKAGE_NAME\\_core.cp310-win_amd64.pyd",
"relative_package_filepath": "PACKAGE_NAME\\_core.cp310-win_amd64.pyd",
"returns": null
},

Searching for the word "Constructor" in the JSON output of griffe produces 0 results.

How can I overcome this problem?

2. Properties / Attributes
In C extension modules, the structure "PyMemberDef" (https://docs.python.org/3/c-api/structures.html#c.PyMemberDef) can be used to describe attributes (read-write or read-only). See details of CPython: https://docs.python.org/3/library/inspect.html#inspect.ismemberdescriptor

My C code contains the following:

static PyMemberDef MyClass_members[] = {
    {"size", Py_T_LONG_LONG, offsetof(PyMyClass, size), Py_READONLY, PyDoc_STR("int\n--\n\nSize of the file in bytes. READ-ONLY") },
    {NULL}  /* Sentinel */
};

JSON output

  "size": {
	"docstring": {
	  "endlineno": null,
	  "lineno": null,
	  "parsed": [
		{
		  "kind": "text",
		  "value": "int\n--\n\nSize of the file in bytes. READ-ONLY"
		}
	  ],
	  "value": "int\n--\n\nSize of the file in bytes. READ-ONLY"
	},
	"filepath": "C:\\XXX\\_core.cp310-win_amd64.pyd",
	"kind": "attribute",
	"labels": [
	  "class"
	],
	"members": {},
	"name": "size",
	"path": "PACKAGE_NAME._core.CLASS_NAME.size",
	"relative_filepath": "PACKAGE_NAME\\_core.cp310-win_amd64.pyd",
	"relative_package_filepath": "PACKAGE_NAME\\_core.cp310-win_amd64.pyd",
	"value": "<member 'size' of 'PACKAGE_NAME._core.CLASS_NAME' objects>"
  },

The output rendered with mkdocstrings-python

As you can see, value shows repr() of the object which looks weird.

As an alternative, I've tried to provide an empty (NULL) docstring in the C extension. In that case, griffe takes the docstring from the stub, in two different forms, as an attribute or as a property, but still tries to show the repr of the object...

Similar occurs when using PyGetSetDef (https://docs.python.org/3/c-api/structures.html#c.PyGetSetDef). See details of CPython: https://docs.python.org/3/library/inspect.html#inspect.isgetsetdescriptor

static PyGetSetDef MyClass_attributes[] = {
    {"timespan", (getter) MyClass_timespan, NULL, PyDoc_STR("float, float\n--\n\nTimespan of the messages within the     file", NULL},
    {NULL}  /* Sentinel */
};

Note that here, the type of the attribute is read from the stub Tuple[float,float]but also the repr is shown...

Am I doing something wrong? Is it possible to overcome any of the two problems described above? I could try to provide a 'simple' C module for testing if needed.

I guess all this might be related due to the usage of "-x" (dynamic inspection). I guess using "-X" could be an alternative, but then I will be forced to document everything in the stub and maintaining in parallel the docstrings in the C extension module

[pawamoy]

Hi @gobater, thanks for the detailed report.

Looking at Griffe's code, I can already tell you that a stubs docstring is only used if the actual object does not have a docstring itself, explaining why setting docstrings to NULL in your C code helps. Are you able to set constructors' docstrings to NULL too, so that the one from stubs is used? We can consider always using stubs docstrings, but I'll have to evaluate whether it could be a breaking change.

About attribute values being shown as their representation (repr(value)), yes, that is expected. In this case it looks like even if you don't assign something to the attribute, it has a special "compiled attribute" value of some kind? (I'm totally unfamiliar with C extensions for Python). Apart from overriding mkdocstrings-python's templates, I'm not sure how we could prevent that. Surely you cannot set a __repr__ method on a tuple[float, float] 🤔

[gobater]

Hi @pawamoy ,

Unfortunately it's not possible to set/remove the docstring of the constructor in CPython. The constructor and some other dunder methods always have a weird meaning-less default docstring :(

About repr, I can definitely customize it in CPython, but in my case the attributes have standard types (int, str, float, ...), so I guess that in those cases value shall be set to None

Another issue I am facing is the "labels" field. When inspecting, because the parent is a class, the attributes get the label "class" instead of "instance-attribute" which shall be correct in my case.

I've seen where all this in the code happens, so I will try to hack something and provide a PR.

I guess it could make sense to customize the priority of docstrings for c-extensions and stubs... since as mentioned before, there is no chance to do it in C for the constructor

[pawamoy]

Thanks for the additional info! It's great if you can send a PR to improve our dynamic analysis and handling of stubs 😄 At first glance I don't see any issue with giving precedence to docstrings in stubs, so we'll likely do that.

[pawamoy]

Yep I checked the history and the current code is there from the beginning, so there's no particular reason why we don't give precedence to docstrings in stubs.