Merge pull request #43 from tablelandnetwork/dtb/readme-http-api

README.md

@@ -10,14 +10,22 @@
 - [Background](#background)
 - [Usage](#usage)
   - [Install](#install)
-  - [Postgres Setup](#postgres-setup)
-  - [Supabase](#supabase)
+  - [Postgres setup](#postgres-setup)
+    - [Self-hosted](#self-hosted)
+    - [Amazon RDS](#amazon-rds)
+    - [Supabase](#supabase)
   - [Create a vault](#create-a-vault)
   - [Start replicating a database](#start-replicating-a-database)
-  - [Write a Parquet file](#write-a-parquet-file)
-  - [Listing Vaults](#listing-vaults)
-  - [Listing Events](#listing-events)
-  - [Retrieving](#retrieving)
+  - [Write files](#write-files)
+  - [Listing vaults](#listing-vaults)
+  - [Listing events](#listing-events)
+  - [Retrieving data](#retrieving-data)
+  - [HTTP APIs](#http-apis)
+    - [Create a vault](#create-a-vault-1)
+    - [Write files](#write-files-1)
+    - [Listing vaults](#listing-vaults-1)
+    - [List events](#list-events)
+    - [Retrieving data](#retrieving-data-1)
 - [Development](#development)
   - [Running](#running)
   - [Run tests](#run-tests)
@@ -48,7 +56,9 @@ cd basin-cli
 make install
 ```
 
-### Postgres Setup
+### Postgres setup
+
+You can either write files directly to the network, _or_ you can replicate one or more tables from Postgres database. The replication process requires a few configuration steps before you can create a vault and start streaming data.
 
 #### Self-hosted
 
@@ -89,7 +99,7 @@ make install
     - You'll probably need to reboot the instance for changes to take effect (be careful!)
 - After reboot, check if logical replication is enabled
 
-### Supabase
+#### Supabase
 
 - Log into the [Supabase](https://supabase.io/) dashboard and go to your project, or create a new one.
 - Check if logical replication is enabled. This should be the default setting, so you shouldn't have to change anything. You can do this in the `SQL Editor` section on the left hand side of the Supabase dashboard by running `SHOW wal_level;` query, which should log `logical`.
@@ -107,20 +117,24 @@ vaults account create [FILENAME]
 
 A new private key will be written to `FILENAME`.
 
-The name of a vault contains a `namespace` (e.g. `my_company`) and the name of an existing database relation (e.g. `my_table`), separated by a period (`.`). Use `vaults create` to create a new vault. See `vaults create --help` for more info.
+The name of a vault contains a `namespace` (e.g. `my_company`) and an identifier (e.g., `my_data`), separated by a period (`.`). Use `vaults create` to create a new vault. See `vaults create --help` for more info.
 
 ```bash
-vaults create  --account [WALLET_ADDRESS] namespace.relation_name
+vaults create --account [WALLET_ADDRESS] [namespace.identifier]
 ```
 
-🚧 Vaults currently only replicates `INSERT` statements, which means that it only replicates append-only data (e.g., log-style data). Row updates and deletes will be ignored. 🚧
+To crete a vault with a Time-to-Live (TTL) cache period (in minutes), use the `--cache` flag:
+
+```bash
+vaults create --account [WALLET_ADDRESS] --cache 10 [namespace.identifier]
+```
 
 ### Start replicating a database
 
 Use `vaults stream` to start a daemon that will continuously push changes to the underlying table/view to the network. See `vaults stream --help` for more info.
 
 ```bash
-vaults stream --dburi [DB_URI] --tables t1,t2 --private-key [PRIVATE_KEY] namespace.relation_name
+vaults stream --dburi [DB_URI] --tables t1,t2 --private-key [PRIVATE_KEY] [namespace.identifier]
 ```
 
 The `--dburi` should follow this format:
@@ -129,37 +143,39 @@ The `--dburi` should follow this format:
 postgresql://[USER]:[PASSWORD]@[HOST]:[PORT]/[DATABASE]
 ```
 
-### Write a Parquet file
+> 🚧 Vaults currently only replicates `INSERT` statements, which means that it only replicates append-only data (e.g., log-style data). Row updates and deletes will be ignored. 🚧
 
-Before writing a Parquet file, you need to [Create a vault](#create-a-vault), if not already created. Then, use `vaults write` to write a Parquet file.
+### Write files
+
+Before writing a file, you need to [Create a vault](#create-a-vault), if not already created. Then, use `vaults write` to write a Parquet file.
 
 ```bash
-vaults write --vault [namespace.relation_name] --private-key [PRIVATE_KEY] filepath
+vaults write --vault [namespace.identifier] --private-key [PRIVATE_KEY] filepath
 ```
 
 You can attach a timestamp to that file write, e.g.
 
 ```bash
-vaults write --vault [namespace.relation_name] --private-key [PRIVATE_KEY] --timestamp 1699984703 filepath
+vaults write --vault [namespace.identifier] --private-key [PRIVATE_KEY] --timestamp 1699984703 filepath
 
 # or use data format
-vaults write --vault [namespace.relation_name] --private-key [PRIVATE_KEY] --timestamp 2006-01-02 filepath
+vaults write --vault [namespace.identifier] --private-key [PRIVATE_KEY] --timestamp 2006-01-02 filepath
 
 # or use RFC3339 format
-vaults write --vault [namespace.relation_name] --private-key [PRIVATE_KEY] --timestamp 2006-01-02T15:04:05Z07:00 filepath
+vaults write --vault [namespace.identifier] --private-key [PRIVATE_KEY] --timestamp 2006-01-02T15:04:05Z07:00 filepath
 ```
 
 If a timestamp is not provided, the CLI will assume the timestamp is the current client epoch in UTC.
 
-### Listing Vaults
+### Listing vaults
 
 You can list the vaults from an account by running:
 
 ```bash
 vaults list --account [ETH_ADDRESS]
 ```
 
-### Listing Events
+### Listing events
 
 You can list events of a given vault by running:
 
@@ -176,7 +192,7 @@ vaults events --vault demotest.data --before 2023-11-09T19:38:23-03:00
 vaults events --vault demotest.data --after 2023-11-09
 ```
 
-### Retrieving
+### Retrieving data
 
 You can retrieve a file from a vault by running:
 
@@ -187,9 +203,107 @@ vaults retrieve bafybeifr5njnrw67yyb2h2t7k6ukm3pml4fgphsxeurqcmgmeb7omc2vlq
 You can specify the file where the retrieved content will be save:
 
 ```bash
-vaults retrieve --output filename bafybeifr5njnrw67yyb2h2t7k6ukm3pml4fgphsxeurqcmgmeb7omc2vlq
+vaults retrieve --output [FILENAME] bafybeifr5njnrw67yyb2h2t7k6ukm3pml4fgphsxeurqcmgmeb7omc2vlq
+```
+
+### HTTP APIs
+
+Instead of using the CLI, you can use the HTTP APIs directly. All requests use the following base URL:
+
+- `https://basin.tableland.xyz`
+
+#### Create a vault
+
+`POST /vaults/{vault_id}`
+
+- Note: the `vault_id` must contain a namespace and identifier, separated by a `.`—such as `test_vault.data`.
+
+Params:
+
+- `account` (required)
+- `cache` (optional)
+
+**Examples**
+
+**Without cache**
+
+```bash
+curl --data 'account=0x78C61e68f9f985C43e36dD5ced3f5a24aD0c503e' \
+'https://basin.tableland.xyz/vaults/test_vault.data'
+```
+
+**With cache (in minutes)**
+
+```bash
+curl --data 'account=0x78C61e68f9f985C43e36dD5ced3f5a24aD0c503e&cache=10' \
+'https://basin.tableland.xyz/vaults/test_vault.data'
+```
+
+#### Write files
+
+`POST /vaults/{vault_id}/events`
+
+Headers:
+
+- `filename`: The name to store the file as.
+
+Params:
+
+- `signature` (required): hex encoded keccak hash of the event
+- `timestamp` (optional): unix timestamp
+
+**Examples**
+
+```bash
+curl -H "filename: data.parquet" --data-binary "@data.parquet" \
+'https://basin.tableland.xyz/vaults/test_vault.data/event?timestamp=1708987192&signature=a4cb49a595988e2a3b20e6ee468d50a8d3c3cb01a278754c07efda3a89a7e60527545deb512204b034100d6d6b9d169a2d22f5e6286c9c0272e8dc920981941a00'
+```
+
+Note about implementation:
+
+- In this example, the `timestamp` and `signature` are query parameters.
+- You **must sign the event before sending**. For example, you can use CLI's `sign` command to get the signature string needed in the request above:
+  ```
+  vaults sign --private-key 0x1234abcd /path/to/file
+  ```
+
+#### Listing vaults
+
+`GET /vaults?account={address}`
+
+**Examples**
+
+```bash
+curl 'https://basin.tableland.xyz/vaults?account=0x78C61e68f9f985C43e36dD5ced3f5a24aD0c503e'
+```
+
+#### List events
+
+`GET /vaults/{vault_id}/events`
+
+It supports the `limit`, `offset`, `before` and `after` as optional params.
+
+**Examples**
+
+```bash
+curl 'https://basin.tableland.xyz/vaults/cache_long.test/events'
+```
+
+- Note: the default value for `limit` is `10`, so be sure to request more events or filter with `before` or `after`, in case there are more than 10 events in a vault.
+
+#### Retrieving data
+
+`GET /events/{event_id}`
+
+**Examples**
+
+```bash
+curl 'https://basin.tableland.xyz/events/bafkreibq2xwn6xejhdemvoqqjzvbns5agqob4f2zhegutshbzpz6zyemv4' -o data.parquet
 ```
 
+> Currently, this endpoint can **only download data from the cache**.
+>
+> If the cache has expired (or was never set), nothing will be downloaded via the HTTP API. Only the CLI `retrieve` command is able to retrieve a non-cached file directly from IPFS/Filecoin. We’re working to implement ways to make the HTTP API as seamless as the CLI flow.
 
 ## Development
 
@@ -205,7 +319,7 @@ PORT=8888 ./scripts/server.sh
 ./scripts/run.sh account create pk.out
 
 # Start replicating
-./scripts/run.sh vaults stream --dburi [DB_URI] --tables t1,t2 --private-key [PRIVATE_KEY] namespace.relation_name
+./scripts/run.sh vaults stream --dburi [DB_URI] --tables t1,t2 --private-key [PRIVATE_KEY] namespace.identifier
 ```
 
 ### Run tests
@@ -225,4 +339,4 @@ Small note: If editing the README, please conform to the
 
 ## License
 
-MIT AND Apache-2.0, © 2021-2023 Tableland Network Contributors
+MIT AND Apache-2.0, © 2021-2024 Tableland Network Contributors