diff --git a/README.md b/README.md index 4e8074e..12b304b 100644 --- a/README.md +++ b/README.md @@ -10,14 +10,22 @@ - [Background](#background) - [Usage](#usage) - [Install](#install) - - [Postgres Setup](#postgres-setup) - - [Supabase](#supabase) + - [Postgres setup](#postgres-setup) + - [Self-hosted](#self-hosted) + - [Amazon RDS](#amazon-rds) + - [Supabase](#supabase) - [Create a vault](#create-a-vault) - [Start replicating a database](#start-replicating-a-database) - - [Write a Parquet file](#write-a-parquet-file) - - [Listing Vaults](#listing-vaults) - - [Listing Events](#listing-events) - - [Retrieving](#retrieving) + - [Write files](#write-files) + - [Listing vaults](#listing-vaults) + - [Listing events](#listing-events) + - [Retrieving data](#retrieving-data) + - [HTTP APIs](#http-apis) + - [Create a vault](#create-a-vault-1) + - [Write files](#write-files-1) + - [Listing vaults](#listing-vaults-1) + - [List events](#list-events) + - [Retrieving data](#retrieving-data-1) - [Development](#development) - [Running](#running) - [Run tests](#run-tests) @@ -48,7 +56,9 @@ cd basin-cli make install ``` -### Postgres Setup +### Postgres setup + +You can either write files directly to the network, _or_ you can replicate one or more tables from Postgres database. The replication process requires a few configuration steps before you can create a vault and start streaming data. #### Self-hosted @@ -89,7 +99,7 @@ make install - You'll probably need to reboot the instance for changes to take effect (be careful!) - After reboot, check if logical replication is enabled -### Supabase +#### Supabase - Log into the [Supabase](https://supabase.io/) dashboard and go to your project, or create a new one. - Check if logical replication is enabled. This should be the default setting, so you shouldn't have to change anything. You can do this in the `SQL Editor` section on the left hand side of the Supabase dashboard by running `SHOW wal_level;` query, which should log `logical`. @@ -107,20 +117,24 @@ vaults account create [FILENAME] A new private key will be written to `FILENAME`. -The name of a vault contains a `namespace` (e.g. `my_company`) and the name of an existing database relation (e.g. `my_table`), separated by a period (`.`). Use `vaults create` to create a new vault. See `vaults create --help` for more info. +The name of a vault contains a `namespace` (e.g. `my_company`) and an identifier (e.g., `my_data`), separated by a period (`.`). Use `vaults create` to create a new vault. See `vaults create --help` for more info. ```bash -vaults create --account [WALLET_ADDRESS] namespace.relation_name +vaults create --account [WALLET_ADDRESS] [namespace.identifier] ``` -🚧 Vaults currently only replicates `INSERT` statements, which means that it only replicates append-only data (e.g., log-style data). Row updates and deletes will be ignored. 🚧 +To crete a vault with a Time-to-Live (TTL) cache period (in minutes), use the `--cache` flag: + +```bash +vaults create --account [WALLET_ADDRESS] --cache 10 [namespace.identifier] +``` ### Start replicating a database Use `vaults stream` to start a daemon that will continuously push changes to the underlying table/view to the network. See `vaults stream --help` for more info. ```bash -vaults stream --dburi [DB_URI] --tables t1,t2 --private-key [PRIVATE_KEY] namespace.relation_name +vaults stream --dburi [DB_URI] --tables t1,t2 --private-key [PRIVATE_KEY] [namespace.identifier] ``` The `--dburi` should follow this format: @@ -129,29 +143,31 @@ The `--dburi` should follow this format: postgresql://[USER]:[PASSWORD]@[HOST]:[PORT]/[DATABASE] ``` -### Write a Parquet file +> 🚧 Vaults currently only replicates `INSERT` statements, which means that it only replicates append-only data (e.g., log-style data). Row updates and deletes will be ignored. 🚧 -Before writing a Parquet file, you need to [Create a vault](#create-a-vault), if not already created. Then, use `vaults write` to write a Parquet file. +### Write files + +Before writing a file, you need to [Create a vault](#create-a-vault), if not already created. Then, use `vaults write` to write a Parquet file. ```bash -vaults write --vault [namespace.relation_name] --private-key [PRIVATE_KEY] filepath +vaults write --vault [namespace.identifier] --private-key [PRIVATE_KEY] filepath ``` You can attach a timestamp to that file write, e.g. ```bash -vaults write --vault [namespace.relation_name] --private-key [PRIVATE_KEY] --timestamp 1699984703 filepath +vaults write --vault [namespace.identifier] --private-key [PRIVATE_KEY] --timestamp 1699984703 filepath # or use data format -vaults write --vault [namespace.relation_name] --private-key [PRIVATE_KEY] --timestamp 2006-01-02 filepath +vaults write --vault [namespace.identifier] --private-key [PRIVATE_KEY] --timestamp 2006-01-02 filepath # or use RFC3339 format -vaults write --vault [namespace.relation_name] --private-key [PRIVATE_KEY] --timestamp 2006-01-02T15:04:05Z07:00 filepath +vaults write --vault [namespace.identifier] --private-key [PRIVATE_KEY] --timestamp 2006-01-02T15:04:05Z07:00 filepath ``` If a timestamp is not provided, the CLI will assume the timestamp is the current client epoch in UTC. -### Listing Vaults +### Listing vaults You can list the vaults from an account by running: @@ -159,7 +175,7 @@ You can list the vaults from an account by running: vaults list --account [ETH_ADDRESS] ``` -### Listing Events +### Listing events You can list events of a given vault by running: @@ -176,7 +192,7 @@ vaults events --vault demotest.data --before 2023-11-09T19:38:23-03:00 vaults events --vault demotest.data --after 2023-11-09 ``` -### Retrieving +### Retrieving data You can retrieve a file from a vault by running: @@ -187,9 +203,107 @@ vaults retrieve bafybeifr5njnrw67yyb2h2t7k6ukm3pml4fgphsxeurqcmgmeb7omc2vlq You can specify the file where the retrieved content will be save: ```bash -vaults retrieve --output filename bafybeifr5njnrw67yyb2h2t7k6ukm3pml4fgphsxeurqcmgmeb7omc2vlq +vaults retrieve --output [FILENAME] bafybeifr5njnrw67yyb2h2t7k6ukm3pml4fgphsxeurqcmgmeb7omc2vlq +``` + +### HTTP APIs + +Instead of using the CLI, you can use the HTTP APIs directly. All requests use the following base URL: + +- `https://basin.tableland.xyz` + +#### Create a vault + +`POST /vaults/{vault_id}` + +- Note: the `vault_id` must contain a namespace and identifier, separated by a `.`—such as `test_vault.data`. + +Params: + +- `account` (required) +- `cache` (optional) + +**Examples** + +**Without cache** + +```bash +curl --data 'account=0x78C61e68f9f985C43e36dD5ced3f5a24aD0c503e' \ +'https://basin.tableland.xyz/vaults/test_vault.data' +``` + +**With cache (in minutes)** + +```bash +curl --data 'account=0x78C61e68f9f985C43e36dD5ced3f5a24aD0c503e&cache=10' \ +'https://basin.tableland.xyz/vaults/test_vault.data' +``` + +#### Write files + +`POST /vaults/{vault_id}/events` + +Headers: + +- `filename`: The name to store the file as. + +Params: + +- `signature` (required): hex encoded keccak hash of the event +- `timestamp` (optional): unix timestamp + +**Examples** + +```bash +curl -H "filename: data.parquet" --data-binary "@data.parquet" \ +'https://basin.tableland.xyz/vaults/test_vault.data/event?timestamp=1708987192&signature=a4cb49a595988e2a3b20e6ee468d50a8d3c3cb01a278754c07efda3a89a7e60527545deb512204b034100d6d6b9d169a2d22f5e6286c9c0272e8dc920981941a00' +``` + +Note about implementation: + +- In this example, the `timestamp` and `signature` are query parameters. +- You **must sign the event before sending**. For example, you can use CLI's `sign` command to get the signature string needed in the request above: + ``` + vaults sign --private-key 0x1234abcd /path/to/file + ``` + +#### Listing vaults + +`GET /vaults?account={address}` + +**Examples** + +```bash +curl 'https://basin.tableland.xyz/vaults?account=0x78C61e68f9f985C43e36dD5ced3f5a24aD0c503e' +``` + +#### List events + +`GET /vaults/{vault_id}/events` + +It supports the `limit`, `offset`, `before` and `after` as optional params. + +**Examples** + +```bash +curl 'https://basin.tableland.xyz/vaults/cache_long.test/events' +``` + +- Note: the default value for `limit` is `10`, so be sure to request more events or filter with `before` or `after`, in case there are more than 10 events in a vault. + +#### Retrieving data + +`GET /events/{event_id}` + +**Examples** + +```bash +curl 'https://basin.tableland.xyz/events/bafkreibq2xwn6xejhdemvoqqjzvbns5agqob4f2zhegutshbzpz6zyemv4' -o data.parquet ``` +> Currently, this endpoint can **only download data from the cache**. +> +> If the cache has expired (or was never set), nothing will be downloaded via the HTTP API. Only the CLI `retrieve` command is able to retrieve a non-cached file directly from IPFS/Filecoin. We’re working to implement ways to make the HTTP API as seamless as the CLI flow. ## Development @@ -205,7 +319,7 @@ PORT=8888 ./scripts/server.sh ./scripts/run.sh account create pk.out # Start replicating -./scripts/run.sh vaults stream --dburi [DB_URI] --tables t1,t2 --private-key [PRIVATE_KEY] namespace.relation_name +./scripts/run.sh vaults stream --dburi [DB_URI] --tables t1,t2 --private-key [PRIVATE_KEY] namespace.identifier ``` ### Run tests @@ -225,4 +339,4 @@ Small note: If editing the README, please conform to the ## License -MIT AND Apache-2.0, © 2021-2023 Tableland Network Contributors +MIT AND Apache-2.0, © 2021-2024 Tableland Network Contributors