Add VM design doc. (iree-org#17518)

This was copied from a [Discord
discussion](https://discord.com/channels/689900678990135345/689900680009482386/1241425926709973103),
reformatted, and then slightly modified for grammar and style.

docs/website/docs/developers/design-docs/vm.md

@@ -0,0 +1,62 @@
+# Virtual machine (VM)
+
+## Overview
+
+The VM is an abstract machine that defines a type system of primitive and
+reference objects, module machinery, and a fairly involved mechanic for calls
+and dynamically binding extern funcs to their defs.
+
+It comes with a bytecode module type, which implements the module interface and
+exposes a CFG based instruction set (sans pointers, as there are some security
+and portability to device scheduler goals in play) with an infinite number of
+registers. A VMFB, which the compiler produces by default is a serialization of
+this. The bytecode module has an interpreter for it.
+
+## VM modules
+
+There are multiple module types provided by default in the runtime:
+
+* HAL for interacting with devices
+* io_params for external data
+* check for testing
+
+There are also Python bindings to create a module object dynamically and define
+exports in Python. Since the module interface is just natively defined in C,
+IREE also lets you load a .so dynamically which exports a function to create a
+new named module (this is what emitc produces for example).
+
+When the tools are creating an `iree_vm_context` for execution, this is
+primarily about instantiating modules and adding them. Each module resolved its
+imports from the exports of the priors. In practice, an io_params module is
+added to access parameter providers, a hal module for devices, one bytecode
+module for each vmfb listed, and a native .so module for any .so listed.
+
+That's all it is at the runtime. There's a lot of machinery on the compiler
+side for producing these modules and their interplay. The lowest level there to
+get a feel for what the compiler can emit, either a bytecode or a C based
+export, look at the vm dialect.
+
+## Lowering from the VM to C
+
+There has been talk for years of having a direct lowering of VM to LLVM without
+going through C. While desirable in theory, it's just never become a priority...
+The C based export is what embedded folks want (because you never want to pin
+your system toolchain to a random LLVM like that). And the bytecode side has
+never been the bottleneck for others. It's also an annoying, maintenance prone
+bit of code to write and just never got done.
+
+The high "it just works" quotient on the bytecode side has probably helped drive
+that too. "vmfb" has become a bit synonymous with "IREE" and teams using it
+think that is the main thing. But it is just one serialization of a VM module
+defining a program... But it has the best tools and debugging experience.
+
+## Call interface
+
+The VM call interface is modeled as a coroutine, and the bytecode interpreter
+supports multi task suspend/resume on a single thread. This is used for a lot
+of things (i.e. multi batch, async submissive, interfacing to io fabric, etc).
+Most of what people think of as "async" in the context of device interactions
+comes down to this and the cooperation of the hal module which provides device
+based synchronization and scheduling primitives.
+
+The way it is structured, the VM was not designed to block, but it can suspend.

docs/website/docs/reference/bindings/c-api.md

@@ -411,6 +411,9 @@ dense math into HAL command buffers that are dispatched to hardware backends.
       }
     ```
 
+For more detailed information about the design of the VM, see
+[this design doc](../../developers/design-docs/vm.md).
+
 #### HAL
 
 <!-- TODO(scotttodd): command buffer construction -> dispatch diagram -->

docs/website/mkdocs.yml

@@ -206,6 +206,7 @@ nav:
           - "developers/design-docs/cuda-hal-driver.md"
           - "developers/design-docs/hip-hal-driver.md"
           - "developers/design-docs/metal-hal-driver.md"
+          - "developers/design-docs/vm.md"
       - "Other topics":
           - "developers/usage-best-practices.md"
           - "developers/vulkan-environment-setup.md"