-
Notifications
You must be signed in to change notification settings - Fork 1
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
docs: update readme with http api #43
Merged
Merged
Changes from all commits
Commits
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -10,14 +10,22 @@ | |
- [Background](#background) | ||
- [Usage](#usage) | ||
- [Install](#install) | ||
- [Postgres Setup](#postgres-setup) | ||
- [Supabase](#supabase) | ||
- [Postgres setup](#postgres-setup) | ||
- [Self-hosted](#self-hosted) | ||
- [Amazon RDS](#amazon-rds) | ||
- [Supabase](#supabase) | ||
- [Create a vault](#create-a-vault) | ||
- [Start replicating a database](#start-replicating-a-database) | ||
- [Write a Parquet file](#write-a-parquet-file) | ||
- [Listing Vaults](#listing-vaults) | ||
- [Listing Events](#listing-events) | ||
- [Retrieving](#retrieving) | ||
- [Write files](#write-files) | ||
- [Listing vaults](#listing-vaults) | ||
- [Listing events](#listing-events) | ||
- [Retrieving data](#retrieving-data) | ||
- [HTTP APIs](#http-apis) | ||
- [Create a vault](#create-a-vault-1) | ||
- [Write files](#write-files-1) | ||
- [Listing vaults](#listing-vaults-1) | ||
- [List events](#list-events) | ||
- [Retrieving data](#retrieving-data-1) | ||
- [Development](#development) | ||
- [Running](#running) | ||
- [Run tests](#run-tests) | ||
|
@@ -48,7 +56,9 @@ cd basin-cli | |
make install | ||
``` | ||
|
||
### Postgres Setup | ||
### Postgres setup | ||
|
||
You can either write files directly to the network, _or_ you can replicate one or more tables from Postgres database. The replication process requires a few configuration steps before you can create a vault and start streaming data. | ||
|
||
#### Self-hosted | ||
|
||
|
@@ -89,7 +99,7 @@ make install | |
- You'll probably need to reboot the instance for changes to take effect (be careful!) | ||
- After reboot, check if logical replication is enabled | ||
|
||
### Supabase | ||
#### Supabase | ||
|
||
- Log into the [Supabase](https://supabase.io/) dashboard and go to your project, or create a new one. | ||
- Check if logical replication is enabled. This should be the default setting, so you shouldn't have to change anything. You can do this in the `SQL Editor` section on the left hand side of the Supabase dashboard by running `SHOW wal_level;` query, which should log `logical`. | ||
|
@@ -107,20 +117,24 @@ vaults account create [FILENAME] | |
|
||
A new private key will be written to `FILENAME`. | ||
|
||
The name of a vault contains a `namespace` (e.g. `my_company`) and the name of an existing database relation (e.g. `my_table`), separated by a period (`.`). Use `vaults create` to create a new vault. See `vaults create --help` for more info. | ||
The name of a vault contains a `namespace` (e.g. `my_company`) and an identifier (e.g., `my_data`), separated by a period (`.`). Use `vaults create` to create a new vault. See `vaults create --help` for more info. | ||
|
||
```bash | ||
vaults create --account [WALLET_ADDRESS] namespace.relation_name | ||
vaults create --account [WALLET_ADDRESS] [namespace.identifier] | ||
``` | ||
|
||
🚧 Vaults currently only replicates `INSERT` statements, which means that it only replicates append-only data (e.g., log-style data). Row updates and deletes will be ignored. 🚧 | ||
To crete a vault with a Time-to-Live (TTL) cache period (in minutes), use the `--cache` flag: | ||
|
||
```bash | ||
vaults create --account [WALLET_ADDRESS] --cache 10 [namespace.identifier] | ||
``` | ||
|
||
### Start replicating a database | ||
|
||
Use `vaults stream` to start a daemon that will continuously push changes to the underlying table/view to the network. See `vaults stream --help` for more info. | ||
|
||
```bash | ||
vaults stream --dburi [DB_URI] --tables t1,t2 --private-key [PRIVATE_KEY] namespace.relation_name | ||
vaults stream --dburi [DB_URI] --tables t1,t2 --private-key [PRIVATE_KEY] [namespace.identifier] | ||
``` | ||
|
||
The `--dburi` should follow this format: | ||
|
@@ -129,37 +143,39 @@ The `--dburi` should follow this format: | |
postgresql://[USER]:[PASSWORD]@[HOST]:[PORT]/[DATABASE] | ||
``` | ||
|
||
### Write a Parquet file | ||
> 🚧 Vaults currently only replicates `INSERT` statements, which means that it only replicates append-only data (e.g., log-style data). Row updates and deletes will be ignored. 🚧 | ||
|
||
Before writing a Parquet file, you need to [Create a vault](#create-a-vault), if not already created. Then, use `vaults write` to write a Parquet file. | ||
### Write files | ||
|
||
Before writing a file, you need to [Create a vault](#create-a-vault), if not already created. Then, use `vaults write` to write a Parquet file. | ||
|
||
```bash | ||
vaults write --vault [namespace.relation_name] --private-key [PRIVATE_KEY] filepath | ||
vaults write --vault [namespace.identifier] --private-key [PRIVATE_KEY] filepath | ||
``` | ||
|
||
You can attach a timestamp to that file write, e.g. | ||
|
||
```bash | ||
vaults write --vault [namespace.relation_name] --private-key [PRIVATE_KEY] --timestamp 1699984703 filepath | ||
vaults write --vault [namespace.identifier] --private-key [PRIVATE_KEY] --timestamp 1699984703 filepath | ||
|
||
# or use data format | ||
vaults write --vault [namespace.relation_name] --private-key [PRIVATE_KEY] --timestamp 2006-01-02 filepath | ||
vaults write --vault [namespace.identifier] --private-key [PRIVATE_KEY] --timestamp 2006-01-02 filepath | ||
|
||
# or use RFC3339 format | ||
vaults write --vault [namespace.relation_name] --private-key [PRIVATE_KEY] --timestamp 2006-01-02T15:04:05Z07:00 filepath | ||
vaults write --vault [namespace.identifier] --private-key [PRIVATE_KEY] --timestamp 2006-01-02T15:04:05Z07:00 filepath | ||
``` | ||
|
||
If a timestamp is not provided, the CLI will assume the timestamp is the current client epoch in UTC. | ||
|
||
### Listing Vaults | ||
### Listing vaults | ||
|
||
You can list the vaults from an account by running: | ||
|
||
```bash | ||
vaults list --account [ETH_ADDRESS] | ||
``` | ||
|
||
### Listing Events | ||
### Listing events | ||
|
||
You can list events of a given vault by running: | ||
|
||
|
@@ -176,7 +192,7 @@ vaults events --vault demotest.data --before 2023-11-09T19:38:23-03:00 | |
vaults events --vault demotest.data --after 2023-11-09 | ||
``` | ||
|
||
### Retrieving | ||
### Retrieving data | ||
|
||
You can retrieve a file from a vault by running: | ||
|
||
|
@@ -187,9 +203,107 @@ vaults retrieve bafybeifr5njnrw67yyb2h2t7k6ukm3pml4fgphsxeurqcmgmeb7omc2vlq | |
You can specify the file where the retrieved content will be save: | ||
|
||
```bash | ||
vaults retrieve --output filename bafybeifr5njnrw67yyb2h2t7k6ukm3pml4fgphsxeurqcmgmeb7omc2vlq | ||
vaults retrieve --output [FILENAME] bafybeifr5njnrw67yyb2h2t7k6ukm3pml4fgphsxeurqcmgmeb7omc2vlq | ||
``` | ||
|
||
### HTTP APIs | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. added docs for HTTP APIs (I've been sharing a Notion doc, so i figured it'd best if it's just public) |
||
|
||
Instead of using the CLI, you can use the HTTP APIs directly. All requests use the following base URL: | ||
|
||
- `https://basin.tableland.xyz` | ||
|
||
#### Create a vault | ||
|
||
`POST /vaults/{vault_id}` | ||
|
||
- Note: the `vault_id` must contain a namespace and identifier, separated by a `.`—such as `test_vault.data`. | ||
|
||
Params: | ||
|
||
- `account` (required) | ||
- `cache` (optional) | ||
|
||
**Examples** | ||
|
||
**Without cache** | ||
|
||
```bash | ||
curl --data 'account=0x78C61e68f9f985C43e36dD5ced3f5a24aD0c503e' \ | ||
'https://basin.tableland.xyz/vaults/test_vault.data' | ||
``` | ||
|
||
**With cache (in minutes)** | ||
|
||
```bash | ||
curl --data 'account=0x78C61e68f9f985C43e36dD5ced3f5a24aD0c503e&cache=10' \ | ||
'https://basin.tableland.xyz/vaults/test_vault.data' | ||
``` | ||
|
||
#### Write files | ||
|
||
`POST /vaults/{vault_id}/events` | ||
|
||
Headers: | ||
|
||
- `filename`: The name to store the file as. | ||
|
||
Params: | ||
|
||
- `signature` (required): hex encoded keccak hash of the event | ||
- `timestamp` (optional): unix timestamp | ||
|
||
**Examples** | ||
|
||
```bash | ||
curl -H "filename: data.parquet" --data-binary "@data.parquet" \ | ||
'https://basin.tableland.xyz/vaults/test_vault.data/event?timestamp=1708987192&signature=a4cb49a595988e2a3b20e6ee468d50a8d3c3cb01a278754c07efda3a89a7e60527545deb512204b034100d6d6b9d169a2d22f5e6286c9c0272e8dc920981941a00' | ||
``` | ||
|
||
Note about implementation: | ||
|
||
- In this example, the `timestamp` and `signature` are query parameters. | ||
- You **must sign the event before sending**. For example, you can use CLI's `sign` command to get the signature string needed in the request above: | ||
``` | ||
vaults sign --private-key 0x1234abcd /path/to/file | ||
``` | ||
|
||
#### Listing vaults | ||
|
||
`GET /vaults?account={address}` | ||
|
||
**Examples** | ||
|
||
```bash | ||
curl 'https://basin.tableland.xyz/vaults?account=0x78C61e68f9f985C43e36dD5ced3f5a24aD0c503e' | ||
``` | ||
|
||
#### List events | ||
|
||
`GET /vaults/{vault_id}/events` | ||
|
||
It supports the `limit`, `offset`, `before` and `after` as optional params. | ||
|
||
**Examples** | ||
|
||
```bash | ||
curl 'https://basin.tableland.xyz/vaults/cache_long.test/events' | ||
``` | ||
|
||
- Note: the default value for `limit` is `10`, so be sure to request more events or filter with `before` or `after`, in case there are more than 10 events in a vault. | ||
|
||
#### Retrieving data | ||
|
||
`GET /events/{event_id}` | ||
|
||
**Examples** | ||
|
||
```bash | ||
curl 'https://basin.tableland.xyz/events/bafkreibq2xwn6xejhdemvoqqjzvbns5agqob4f2zhegutshbzpz6zyemv4' -o data.parquet | ||
``` | ||
|
||
> Currently, this endpoint can **only download data from the cache**. | ||
> | ||
> If the cache has expired (or was never set), nothing will be downloaded via the HTTP API. Only the CLI `retrieve` command is able to retrieve a non-cached file directly from IPFS/Filecoin. We’re working to implement ways to make the HTTP API as seamless as the CLI flow. | ||
|
||
## Development | ||
|
||
|
@@ -205,7 +319,7 @@ PORT=8888 ./scripts/server.sh | |
./scripts/run.sh account create pk.out | ||
|
||
# Start replicating | ||
./scripts/run.sh vaults stream --dburi [DB_URI] --tables t1,t2 --private-key [PRIVATE_KEY] namespace.relation_name | ||
./scripts/run.sh vaults stream --dburi [DB_URI] --tables t1,t2 --private-key [PRIVATE_KEY] namespace.identifier | ||
``` | ||
|
||
### Run tests | ||
|
@@ -225,4 +339,4 @@ Small note: If editing the README, please conform to the | |
|
||
## License | ||
|
||
MIT AND Apache-2.0, © 2021-2023 Tableland Network Contributors | ||
MIT AND Apache-2.0, © 2021-2024 Tableland Network Contributors |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
idk if the
namespace.identifier
is the right way to define it, but i figured the "relation" nomenclature should be dropped.