Add iree.build to readthedocs, fix a few warnings, refresh readme. (i…

…ree-org#19236)

Progress on iree-org#19019 and
iree-org#5477.

We have API documentation for the Python APIs hosted at
https://iree-python-api.readthedocs.io/, but it has been pretty
neglected, to the point where that URL is buried at
https://iree.dev/reference/bindings/python/#usage. I'm not actually sure
if any users have found that page (readthedocs does provide traffic data
analytics, but we don't have enough views for that to register).

This makes a few incremental improvements while I learn my way around
Sphinx and what we have already for doc configuration, docstrings in
source files, packages, etc.

* Updated documentation for building the Sphinx website locally and
recommended developers use `sphinx-autobuild`
* Added an initial page for the `iree.build` package
* Used code blocks in more places consistently, and closed some that
were opened without closing ` characters
* Fixed a few other warnings from Sphinx (wrong indentation, section
delimiters not using enough characters, etc.)
* Worked around a weird style issue with some custom css
* Bumped the copyright year (as we do for https://iree.dev)

compiler/bindings/python/iree/build/main.py

@@ -37,6 +37,7 @@ def iree_build_main(
     add a stanza at the end:
 
     .. code-block:: python
+
         from iree.build import *
 
         if __name__ == "__main__":

compiler/bindings/python/iree/compiler/api/__init__.py

@@ -7,7 +7,7 @@
 """Compiler invocation API.
 
 This package defines Python API wrappers around the IREE compiler C embedding
-API. Refer to the C documentation in bindings/c/iree/compiler/embedding_api.h
+API. Refer to the C documentation in `bindings/c/iree/compiler/embedding_api.h`
 for the most up to date information.
 
 The objects in the C API are represented in Python as classes:
@@ -18,8 +18,8 @@
 * `iree_compiler_source_t`: `Source` class.
 * `iree_compiler_output_t`: `Output` class.
 
-In MLIR parlance, the `Session` wraps an MLIRContext with a set of flags for
-configuring the compiler and context setup. `Invocation` wraps a Module in
+In MLIR parlance, the `Session` wraps an `MLIRContext` with a set of flags for
+configuring the compiler and context setup. `Invocation` wraps a `Module` in
 the process of being compiled.
 """
 

compiler/bindings/python/iree/compiler/tools/__init__.py

@@ -7,7 +7,7 @@
 R'''
 This module contains Python wrappers for various IREE command-line tools.
 
-This top-level API provides access to the `iree-compiler` tool, which compiles
+This top-level API provides access to the `iree-compile` tool, which compiles
 MLIR ASM via IREE's compiler to a supported output format (i.e. VM FlatBuffer, C
 source code, etc).
 

compiler/bindings/python/iree/compiler/tools/debugging.py

@@ -12,7 +12,7 @@
   a reproducer will be output at the listed path.
 * `extra_args=[...]` - Passes extra arguments to the compiler. Useful for \
   various standard features of MLIR based compilers like
-  `-mlir-print-ir-after-all`.
+  `--mlir-print-ir-after-all`.
 
 
 In addition, the core compiler and frontend compiler APIs have a unified

compiler/bindings/python/iree/compiler/tools/tf.py

@@ -4,7 +4,7 @@
 # Licensed under the Apache License v2.0 with LLVM Exceptions.
 # See https://llvm.org/LICENSE.txt for license information.
 # SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
-"""Imports TensorFlow artifacts via the `iree-import-tf tool."""
+"""Imports TensorFlow artifacts via the `iree-import-tf` tool."""
 
 from __future__ import annotations
 from dataclasses import dataclass, field

docs/api_docs/python/README.md

@@ -2,29 +2,60 @@
 
 Documentation for the Python API is built with Sphinx under this directory.
 When new versions are released, the documentation is generated and published
-to the [Read the Docs](https://readthedocs.org/projects/iree-python-api/)
+to the [Read the Docs](https://app.readthedocs.org/projects/iree-python-api/)
 project and is served at
 [readthedocs.io](https://iree-python-api.readthedocs.io/en/latest/).
 
-## Building Docs
+## Building the API documentation locally
 
-### Install IREE binaries
+### Setup virtual environment with requirements
 
-Either install python packages or, from the build directory:
+```shell
+python -m venv .venv
+source .venv/bin/activate
+python -m pip install -r requirements.txt
+```
+
+### (Optional) Using locally built IREE packages
+
+You can build the site using either released Python packages or local packages,
+whichever appear first on the `PYTHONPATH` environment variable. The
+`requirements.txt` file used in the previous step downloads the latest
+pre-release (nightly) `iree-base-compiler` and `iree-base-runtime` packages.
+
+To use local packages, such as when changing docstrings and wanting to see how
+they appear in the generated documentation, follow the instructions for building
+the Python bindings from source at
+<https://iree.dev/building-from-source/getting-started/#using-the-python-bindings>.
+
+In particular, after building with `-DIREE_BUILD_PYTHON_BINDINGS=ON`, you will
+need to extend your `PYTHONPATH` to include the relevant build directories. The
+the generated `.env` files can help with this:
+
+```shell
+source ../../../iree-build/.env && export PYTHONPATH
+```
+
+### Build docs
 
 ```shell
-export PYTHONPATH=$PWD/bindings/python:$PWD/compiler-api/python_package
+sphinx-build -b html . _build
 ```
 
-### Install dependencies
+### Serve locally locally with autoreload
 
 ```shell
-python -m pip install -r requirements.txt
+sphinx-autobuild . _build
 ```
 
+Then open http://127.0.0.1:8000 as instructed by the logs and make changes to
+the files in this directory as needed to update the documentation.
 
-### Build docs
+### Clean to show all warnings
+
+A clean rebuild will show all warnings again:
 
 ```shell
+make clean
 sphinx-build -b html . _build
 ```

docs/api_docs/python/_static/custom.css

@@ -0,0 +1,7 @@
+/*
+Fix for horizontal stacking weirdness in the RTD theme with Python properties:
+https://github.com/readthedocs/sphinx_rtd_theme/issues/1301
+*/
+.py.property {
+  display: block !important;
+}

docs/api_docs/python/compiler/api.rst

@@ -4,6 +4,9 @@ In-Process Compiler API
 IREE provides access to its MLIR-based compiler via a dedicated set of APIs
 presented here.
 
+~~~~
+
 .. automodule:: iree.compiler.api
   :imported-members:
   :members:
+  :undoc-members:

docs/api_docs/python/compiler/build.rst

@@ -0,0 +1,9 @@
+Build API
+=========
+
+.. automodule:: iree.build
+  :members:
+  :imported-members:
+  :undoc-members:
+
+.. TODO: organize into sections (core executor, arguments, built in actions, etc.)

docs/api_docs/python/compiler/index.rst

@@ -7,6 +7,7 @@ Compiler API
 
    api.rst
    tools.rst
+   build.rst
+   mlir.rst
    iree_input_dialect.rst
    mlir_dialects.rst
-   mlir.rst

docs/api_docs/python/compiler/tools.rst

@@ -2,12 +2,12 @@ Invoking Command Line Tools
 ===========================
 
 As with many compilers, IREE's compiler consists of many command line tools,
-some of which are designed for compiler devs and are only accessible via source
-builds. User level tools are distributed via the Python packages and are also
-accessible via dedicated Python APIs, documented here.
+some of which are designed for compiler developers and are only accessible via
+source builds. User level tools are distributed via the Python packages and are
+also accessible via dedicated Python APIs, documented here.
 
 Core Compiler (`iree-compile`)
------------------------
+------------------------------
 
 .. automodule:: iree.compiler.tools
   :members: compile_file, compile_str

docs/api_docs/python/conf.py

@@ -17,8 +17,8 @@
 # -- Project information -----------------------------------------------------
 
 project = "IREE Python API"
-copyright = "2021, IREE Authors"
-author = "IREE Authors"
+copyright = "2024, The IREE Authors"
+author = "The IREE Authors"
 
 # The full version, including alpha/beta/rc tags
 release = "snapshot"
@@ -42,7 +42,7 @@
 # List of patterns, relative to source directory, that match files and
 # directories to ignore when looking for source files.
 # This pattern also affects html_static_path and html_extra_path.
-exclude_patterns = ["_build", "Thumbs.db", ".DS_Store"]
+exclude_patterns = ["_build", "Thumbs.db", ".DS_Store", "*.venv", "README.md"]
 
 intersphinx_mapping = {"python": ("https://docs.python.org/3", None)}
 
@@ -60,6 +60,7 @@
 # relative to this directory. They are copied after the builtin static files,
 # so a file named "default.css" will overwrite the builtin "default.css".
 html_static_path = ["_static"]
+html_css_files = ["custom.css"]
 
 # -- Markdown from docstrings ------------------------------------------------
 

docs/api_docs/python/requirements.txt

@@ -1,16 +1,19 @@
-sphinx==4.2.0
-myst-parser==0.15.2  # For markdown
-sphinx_rtd_theme==1.0.0
+# Sphinx documentation generator.
 commonmark==0.9.1
-enum_tools==0.10.0
-sphinx_toolbox==2.16.1
+enum-tools==0.10.0
+myst-parser==0.15.2  # For markdown
+sphinx==4.2.0
+sphinx-rtd-theme==1.0.0
+sphinx-autobuild
+sphinx-toolbox==2.16.1
 sphinxcontrib-applehelp==1.0.4
 sphinxcontrib-devhelp==1.0.2
 sphinxcontrib-htmlhelp==2.0.1
 sphinxcontrib-serializinghtml==1.1.5
 sphinxcontrib-qthelp==1.0.3
 
-# IREE Python API
--f https://iree.dev/pip-release-links.html
+# IREE Python API (using the latest pre-release / nightly packages).
+--find-links https://iree.dev/pip-release-links.html
+--pre
 iree-base-compiler
 iree-base-runtime

runtime/bindings/python/iree/runtime/array_interop.py

@@ -42,23 +42,25 @@ class DeviceArray(numpy.lib.mixins.NDArrayOperatorsMixin):
     """An IREE device array.
 
     Device arrays can be in one of two states:
-      1. Host accessible: The array will be backed by host accessible memory
-         and can have the usual things done with it that one expects to be
-         able to do with an ndarray.
-      2. Device resident: The array is just a handle to a device resident
-         Buffer (and BufferView wrapper). Metadata about the array are accessible
-         (shape and dtype) but anything that touches the data cannot be accessed
-         in this state.
+
+    1. Host accessible: The array will be backed by host accessible memory
+       and can have the usual things done with it that one expects to be
+       able to do with an ndarray.
+    2. Device resident: The array is just a handle to a device resident
+       Buffer (and BufferView wrapper). Metadata about the array are accessible
+       (shape and dtype) but anything that touches the data cannot be accessed
+       in this state.
 
     How a device array comes into existence controls how it can transition
     between these states:
-      * A user can create a DeviceArray explicitly with a device allocator.
-        Such an array will not be implicitly convertible to host accessible,
-        although accessors exist to do so.
-      * When created by the platform with a synchronization policy, then
-        implicit transfer back to the host will trigger appropriate waits and
-        be performed automatically (this is the common case for function return
-        values if not otherwise configured, as an example).
+
+    * A user can create a DeviceArray explicitly with a device allocator.
+      Such an array will not be implicitly convertible to host accessible,
+      although accessors exist to do so.
+    * When created by the platform with a synchronization policy, then
+      implicit transfer back to the host will trigger appropriate waits and
+      be performed automatically (this is the common case for function return
+      values if not otherwise configured, as an example).
     """
 
     def __init__(