-
Notifications
You must be signed in to change notification settings - Fork 73
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Update webR documentation for building R packages and mounting filesy…
…stem images (#316) * Update documentation for pkg building and mounting * Add mounting example * Minor tweaks
- Loading branch information
1 parent
6fd7fac
commit 415b937
Showing
8 changed files
with
251 additions
and
17 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,28 @@ | ||
| --- | ||
| title: "Building R Packages" | ||
| format: html | ||
| toc: true | ||
| --- | ||
|
|
||
| Many R packages contain C, C++ or Fortran code that must be compiled^[Prerequisite system libraries must also be compiled for WebAssembly. The webR build process optionally includes building a collection of system libraries for Wasm.] for WebAssembly (Wasm) before they can be loaded in webR. While the [webR binary repository](packages.qmd#downloading-packages-from-a-webr-binary-repository) provides pre-compiled Wasm binaries^[Other R package repositories providing WebAssembly binaries are also available, such as [R-universe](https://r-universe.dev/).], custom R packages must be independently compiled from source. | ||
|
|
||
| ::: callout-note | ||
| It is not possible to directly install packages from source in webR. This is likely to remain the case in the future, as such a process would depend on an entire R development toolchain running in the browser. Loading pre-compiled WebAssembly binaries is the only supported way to install packages in webR. | ||
| ::: | ||
|
|
||
| ## Compiling R packages using GitHub Actions | ||
|
|
||
| GitHub Actions can be used to remotely build R packages for Wasm by making use of the workflows provided by [r‑wasm/actions](https://github.com/r-wasm/actions). This is a convenient method that does not require the setup of a local WebAssembly compiler toolchain or webR development environment. | ||
|
|
||
| The workflow can also be configured to automatically upload the resulting R package binaries to GitHub Pages, ready to be used with webR's [`install()`](packages.qmd#downloading-packages-from-a-webr-binary-repository) or [`mount()`](packages.qmd#mounting-an-r-library-filesystem-image) functions. | ||
|
|
||
| Further details can be found in the documentation and `examples` directory of [r‑wasm/actions](https://github.com/r-wasm/actions). | ||
|
|
||
| ## Compiling R packages locally | ||
|
|
||
| To locally build R packages for webR we recommend using the [rwasm](https://r-wasm.github.io/rwasm/) package in a native R installation. The package automates the use of R's [`Makevars`](https://cran.r-project.org/doc/manuals/r-devel/R-exts.html#Using-Makevars) mechanism^[While it is possible to manually build R packages using R's `Makevars` mechanism alone, the process can be quite involved and error-prone depending on the specific R package involved.] to cross-compile R packages for Wasm, by setting up the host environment so that the [Emscripten](https://emscripten.org/index.html) compiler toolchain is used for compilation and applying a selection of configuration overrides for certain packages. | ||
|
|
||
| The [rwasm](https://r-wasm.github.io/rwasm/) package can also be used to deploy the resulting R package binaries as a CRAN-like repository, or as an Emscripten filesystem image containing an R package library. The resulting collection of R packages can then be made available through static file hosting^[e.g. GitHub Pages, Netlify, AWS S3, etc.]. | ||
|
|
||
| Running the [rwasm](https://r-wasm.github.io/rwasm/) package requires pre-setup of a configured webR development environment, either by running the package under a [pre-prepared webR Docker container](https://github.com/r-wasm/webr/pkgs/container/webr), or by configuring a local webR development build. See the [Getting Started](https://r-wasm.github.io/rwasm/articles/rwasm.html) section of the [rwasm](https://r-wasm.github.io/rwasm/) package documentation for further details. | ||
|
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,135 @@ | ||
| --- | ||
| title: "Mounting Filesystem Data" | ||
| format: html | ||
| toc: true | ||
| --- | ||
|
|
||
| ## The virtual filesystem | ||
|
|
||
| The [Emscripten filesystem API](https://emscripten.org/docs/api_reference/Filesystem-API.html) provides a Unix-like virtual filesystem for the WebAssembly (Wasm) R process running in webR. This virtual filesystem has the ability to [mount](https://emscripten.org/docs/api_reference/Filesystem-API.html#FS.mount) filesystem images or host directories so that the associated file and directory data is accessible to the Wasm R process. | ||
|
|
||
| Mounting images and directories in this way gives the Wasm R process access to arbitrary external data, potentially including datasets, scripts, or R packages [pre-compiled for WebAssembly](building.qmd). | ||
|
|
||
| Emscripten's API provides several types of virtual filesystem, but for technical reasons^[Currently, webR blocks in the JavaScript worker thread while it waits for R input to be evaluated. This blocking means that Emscripten filesystems that depend on asynchronous browser APIs, such as [`IDBFS`](https://emscripten.org/docs/api_reference/Filesystem-API.html#filesystem-api-idbfs), do not work.] only the following filesystems are available for use with webR. | ||
|
|
||
| | Filesystem | Description | Web Browser | Node.js | | ||
| |------|-----|------|------| | ||
| | `WORKERFS` | Mount filesystem images. | ✅ | ✅ | | ||
| | `NODEFS` | Mount existing host directories. | ❌ | ✅ | | ||
|
|
||
| ## Emscripten filesystem images | ||
|
|
||
| The [`file_packager`](https://emscripten.org/docs/porting/files/packaging_files.html#packaging-using-the-file-packager-tool) tool, provided by Emscripten, takes in a directory structure as input and produces webR compatible filesystem images as output. The [`file_packager`](https://emscripten.org/docs/porting/files/packaging_files.html#packaging-using-the-file-packager-tool) tool may be invoked from R using the [rwasm](https://r-wasm.github.io/rwasm/) R package: | ||
|
|
||
| ```{r eval=FALSE} | ||
| > rwasm::file_packager("./input", out_dir = ".", out_name = "output") | ||
| ``` | ||
|
|
||
| It can also be invoked directly using its CLI^[See the [`file_packager`](https://emscripten.org/docs/porting/files/packaging_files.html#packaging-using-the-file-packager-tool) Emscripten documentation for details. ], if you prefer: | ||
|
|
||
| ```bash | ||
| $ file_packager output.data --preload ./input@/ \ | ||
| --separate-metadata --js-output=output.js | ||
| ``` | ||
|
|
||
| In the above examples, the files in the directory `./input` are packaged and an output filesystem image is created^[When using the `file_packager` CLI, a third file named `output.js` will also be created. If you only plan to mount the image using webR, this file may be discarded.] consisting of a data file, `output.data`, and a metadata file, `output.js.metadata`. | ||
|
|
||
| To prepare for mounting the filesystem image with webR, ensure that both files have the same basename (in this example, `output`) and are deployed to static file hosting^[e.g. GitHub Pages, Netlify, AWS S3, etc.]. The resulting URLs for the two files should differ only by the file extension. | ||
|
|
||
|
|
||
| ## Mount a filesystem image from URL | ||
|
|
||
| By default, the [`webr::mount()`](api/r.qmd#mount) function downloads and mounts a filesystem image from a URL source, using the `WORKERFS` filesystem type. | ||
|
|
||
| ```{r eval=FALSE} | ||
| webr::mount( | ||
| mountpoint = "/data", | ||
| source = "https://example.com/output.data" | ||
| ) | ||
| ``` | ||
|
|
||
| A URL for the filesystem image `.data` file should be provided as the source argument, and the image will be mounted in the virtual filesystem under the path given by the `mountpoint` argument. If the `mountpoint` directory does not exist, it will be created prior to mounting. | ||
|
|
||
| ### JavaScript API | ||
|
|
||
| WebR's JavaScript API includes the [`WebR.FS.mount()`](api/js/classes/WebR.WebR.md#fs) function, a thin wrapper around Emscripten's own [`FS.mount()`](https://emscripten.org/docs/api_reference/Filesystem-API.html#FS.mount). The JavaScript API provides more flexibility but requires a little more set up, including creating the `mountpoint` directory if it does not already exist. | ||
|
|
||
| The filesystem type should be provided as a `string`, with the `options` argument a JavaScript object of type [`FSMountOptions`](api/js/modules/WebR.md#fsmountoptions). The filesystem image data should be provided as a JavaScript `Blob` and the metadata as a JavaScript object deserialised from the underlying JSON content. | ||
|
|
||
| ::: {.panel-tabset} | ||
| ## JavaScript | ||
|
|
||
| ``` javascript | ||
| // Create mountpoint | ||
| await webR.FS.mkdir('/data') | ||
|
|
||
| // Download image data | ||
| const data = await fetch('https://example.com/output.data'); | ||
| const metadata = await fetch('https://example.com/output.js.metadata'); | ||
|
|
||
| // Mount image data | ||
| const options = { | ||
| packages: [{ | ||
| blob: await data.blob(), | ||
| metadata: await metadata.json(), | ||
| }], | ||
| } | ||
| await webR.FS.mount("WORKERFS", options, '/data'); | ||
| ``` | ||
|
|
||
| ## TypeScript | ||
|
|
||
| ``` typescript | ||
| import { FSMountOptions } from 'webr'; | ||
|
|
||
| // Create mountpoint | ||
| await webR.FS.mkdir('/data') | ||
|
|
||
| // Download image data | ||
| const data = await fetch('https://example.com/output.data'); | ||
| const metadata = await fetch('https://example.com/output.js.metadata'); | ||
|
|
||
| // Mount image data | ||
| const options: FSMountOptions = { | ||
| packages: [{ | ||
| blob: await data.blob(), | ||
| metadata: await metadata.json(), | ||
| }], | ||
| } | ||
| await webR.FS.mount("WORKERFS", options, '/data'); | ||
| ``` | ||
|
|
||
| ::: | ||
|
|
||
| See the [Emscripten `FS.mount()` documentation](https://emscripten.org/docs/api_reference/Filesystem-API.html#FS.mount) for further details about the structure of the `options` argument. | ||
|
|
||
| ## Mount an existing host directory | ||
|
|
||
| ::: callout-warning | ||
| `NODEFS` is only available when running webR under Node.js. | ||
| ::: | ||
|
|
||
| The `NODEFS` filesystem type maps directories that exist on the host machine so that they are accessible in the WebAssembly process. | ||
|
|
||
| To mount the directory `./extra` on the virtual filesystem at `/data`, use either the JavaScript or R mount API with the filesystem type set to `"NODEFS"`. | ||
|
|
||
| ::: {.panel-tabset} | ||
| ## JavaScript | ||
|
|
||
| ``` javascript | ||
| await webR.FS.mkdir('/data') | ||
| await webR.FS.mount('NODEFS', { root: './extra' }, '/data'); | ||
| ``` | ||
|
|
||
| ## R | ||
| ```{r eval=FALSE} | ||
| webr::mount( | ||
| mountpoint = "/data", | ||
| source = "./extra", | ||
| type = "NODEFS" | ||
| ) | ||
| ``` | ||
|
|
||
|
|
||
| ::: | ||
|
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,13 @@ | ||
| <html> | ||
| <head> | ||
| <title>WebR Mounting Example</title> | ||
| </head> | ||
| <body> | ||
| <div> | ||
| <h1>WebR Mounting Example</h1> | ||
| <p id="loading">Loading webR, please wait...</p> | ||
| <pre><code id="out"></code></pre> | ||
| </div> | ||
| <script type="module" src="./mount.js"></script> | ||
| </body> | ||
| </html> |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,30 @@ | ||
| import { WebR } from 'https://webr.r-wasm.org/latest/webr.mjs'; | ||
| const webR = new WebR(); | ||
|
|
||
| // Remove the loading message once webR is ready | ||
| await webR.init(); | ||
| document.getElementById('loading').remove(); | ||
|
|
||
| // Download a filesystem image | ||
| await webR.FS.mkdir('/data') | ||
| const data = await fetch('./output.data'); | ||
| const metadata = await fetch('./output.js.metadata'); | ||
|
|
||
| // Mount the filesystem image data | ||
| const options = { | ||
| packages: [{ | ||
| blob: await data.blob(), | ||
| metadata: await metadata.json(), | ||
| }], | ||
| } | ||
| await webR.FS.mount("WORKERFS", options, '/data'); | ||
|
|
||
| // Read the contents of a file from the filesystem image | ||
| const result = await webR.evalR('readLines("/data/abc.txt")'); | ||
| try { | ||
| let output = await result.toArray(); | ||
| let text = output.join('\n'); | ||
| document.getElementById('out').innerText += `Contents of the filesystem image file at /data/abc.txt:\n${text}\n\n`; | ||
| } finally { | ||
| webR.destroy(result); | ||
| } |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1 @@ | ||
| abcd1234 |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1 @@ | ||
| {"files":[{"filename":"/abc.txt","start":0,"end":9}],"remote_package_size":9} |