Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

docs: update readme with http api #43

Merged
merged 1 commit into from
Mar 28, 2024
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
162 changes: 138 additions & 24 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,14 +10,22 @@
- [Background](#background)
- [Usage](#usage)
- [Install](#install)
- [Postgres Setup](#postgres-setup)
- [Supabase](#supabase)
- [Postgres setup](#postgres-setup)
- [Self-hosted](#self-hosted)
- [Amazon RDS](#amazon-rds)
- [Supabase](#supabase)
- [Create a vault](#create-a-vault)
- [Start replicating a database](#start-replicating-a-database)
- [Write a Parquet file](#write-a-parquet-file)
- [Listing Vaults](#listing-vaults)
- [Listing Events](#listing-events)
- [Retrieving](#retrieving)
- [Write files](#write-files)
- [Listing vaults](#listing-vaults)
- [Listing events](#listing-events)
- [Retrieving data](#retrieving-data)
- [HTTP APIs](#http-apis)
- [Create a vault](#create-a-vault-1)
- [Write files](#write-files-1)
- [Listing vaults](#listing-vaults-1)
- [List events](#list-events)
- [Retrieving data](#retrieving-data-1)
- [Development](#development)
- [Running](#running)
- [Run tests](#run-tests)
Expand Down Expand Up @@ -48,7 +56,9 @@ cd basin-cli
make install
```

### Postgres Setup
### Postgres setup

You can either write files directly to the network, _or_ you can replicate one or more tables from Postgres database. The replication process requires a few configuration steps before you can create a vault and start streaming data.

#### Self-hosted

Expand Down Expand Up @@ -89,7 +99,7 @@ make install
- You'll probably need to reboot the instance for changes to take effect (be careful!)
- After reboot, check if logical replication is enabled

### Supabase
#### Supabase

- Log into the [Supabase](https://supabase.io/) dashboard and go to your project, or create a new one.
- Check if logical replication is enabled. This should be the default setting, so you shouldn't have to change anything. You can do this in the `SQL Editor` section on the left hand side of the Supabase dashboard by running `SHOW wal_level;` query, which should log `logical`.
Expand All @@ -107,20 +117,24 @@ vaults account create [FILENAME]

A new private key will be written to `FILENAME`.

The name of a vault contains a `namespace` (e.g. `my_company`) and the name of an existing database relation (e.g. `my_table`), separated by a period (`.`). Use `vaults create` to create a new vault. See `vaults create --help` for more info.
The name of a vault contains a `namespace` (e.g. `my_company`) and an identifier (e.g., `my_data`), separated by a period (`.`). Use `vaults create` to create a new vault. See `vaults create --help` for more info.
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

idk if the namespace.identifier is the right way to define it, but i figured the "relation" nomenclature should be dropped.


```bash
vaults create --account [WALLET_ADDRESS] namespace.relation_name
vaults create --account [WALLET_ADDRESS] [namespace.identifier]
```

🚧 Vaults currently only replicates `INSERT` statements, which means that it only replicates append-only data (e.g., log-style data). Row updates and deletes will be ignored. 🚧
To crete a vault with a Time-to-Live (TTL) cache period (in minutes), use the `--cache` flag:

```bash
vaults create --account [WALLET_ADDRESS] --cache 10 [namespace.identifier]
```

### Start replicating a database

Use `vaults stream` to start a daemon that will continuously push changes to the underlying table/view to the network. See `vaults stream --help` for more info.

```bash
vaults stream --dburi [DB_URI] --tables t1,t2 --private-key [PRIVATE_KEY] namespace.relation_name
vaults stream --dburi [DB_URI] --tables t1,t2 --private-key [PRIVATE_KEY] [namespace.identifier]
```

The `--dburi` should follow this format:
Expand All @@ -129,37 +143,39 @@ The `--dburi` should follow this format:
postgresql://[USER]:[PASSWORD]@[HOST]:[PORT]/[DATABASE]
```

### Write a Parquet file
> 🚧 Vaults currently only replicates `INSERT` statements, which means that it only replicates append-only data (e.g., log-style data). Row updates and deletes will be ignored. 🚧

Before writing a Parquet file, you need to [Create a vault](#create-a-vault), if not already created. Then, use `vaults write` to write a Parquet file.
### Write files

Before writing a file, you need to [Create a vault](#create-a-vault), if not already created. Then, use `vaults write` to write a Parquet file.

```bash
vaults write --vault [namespace.relation_name] --private-key [PRIVATE_KEY] filepath
vaults write --vault [namespace.identifier] --private-key [PRIVATE_KEY] filepath
```

You can attach a timestamp to that file write, e.g.

```bash
vaults write --vault [namespace.relation_name] --private-key [PRIVATE_KEY] --timestamp 1699984703 filepath
vaults write --vault [namespace.identifier] --private-key [PRIVATE_KEY] --timestamp 1699984703 filepath

# or use data format
vaults write --vault [namespace.relation_name] --private-key [PRIVATE_KEY] --timestamp 2006-01-02 filepath
vaults write --vault [namespace.identifier] --private-key [PRIVATE_KEY] --timestamp 2006-01-02 filepath

# or use RFC3339 format
vaults write --vault [namespace.relation_name] --private-key [PRIVATE_KEY] --timestamp 2006-01-02T15:04:05Z07:00 filepath
vaults write --vault [namespace.identifier] --private-key [PRIVATE_KEY] --timestamp 2006-01-02T15:04:05Z07:00 filepath
```

If a timestamp is not provided, the CLI will assume the timestamp is the current client epoch in UTC.

### Listing Vaults
### Listing vaults

You can list the vaults from an account by running:

```bash
vaults list --account [ETH_ADDRESS]
```

### Listing Events
### Listing events

You can list events of a given vault by running:

Expand All @@ -176,7 +192,7 @@ vaults events --vault demotest.data --before 2023-11-09T19:38:23-03:00
vaults events --vault demotest.data --after 2023-11-09
```

### Retrieving
### Retrieving data

You can retrieve a file from a vault by running:

Expand All @@ -187,9 +203,107 @@ vaults retrieve bafybeifr5njnrw67yyb2h2t7k6ukm3pml4fgphsxeurqcmgmeb7omc2vlq
You can specify the file where the retrieved content will be save:

```bash
vaults retrieve --output filename bafybeifr5njnrw67yyb2h2t7k6ukm3pml4fgphsxeurqcmgmeb7omc2vlq
vaults retrieve --output [FILENAME] bafybeifr5njnrw67yyb2h2t7k6ukm3pml4fgphsxeurqcmgmeb7omc2vlq
```

### HTTP APIs
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

added docs for HTTP APIs (I've been sharing a Notion doc, so i figured it'd best if it's just public)


Instead of using the CLI, you can use the HTTP APIs directly. All requests use the following base URL:

- `https://basin.tableland.xyz`

#### Create a vault

`POST /vaults/{vault_id}`

- Note: the `vault_id` must contain a namespace and identifier, separated by a `.`—such as `test_vault.data`.

Params:

- `account` (required)
- `cache` (optional)

**Examples**

**Without cache**

```bash
curl --data 'account=0x78C61e68f9f985C43e36dD5ced3f5a24aD0c503e' \
'https://basin.tableland.xyz/vaults/test_vault.data'
```

**With cache (in minutes)**

```bash
curl --data 'account=0x78C61e68f9f985C43e36dD5ced3f5a24aD0c503e&cache=10' \
'https://basin.tableland.xyz/vaults/test_vault.data'
```

#### Write files

`POST /vaults/{vault_id}/events`

Headers:

- `filename`: The name to store the file as.

Params:

- `signature` (required): hex encoded keccak hash of the event
- `timestamp` (optional): unix timestamp

**Examples**

```bash
curl -H "filename: data.parquet" --data-binary "@data.parquet" \
'https://basin.tableland.xyz/vaults/test_vault.data/event?timestamp=1708987192&signature=a4cb49a595988e2a3b20e6ee468d50a8d3c3cb01a278754c07efda3a89a7e60527545deb512204b034100d6d6b9d169a2d22f5e6286c9c0272e8dc920981941a00'
```

Note about implementation:

- In this example, the `timestamp` and `signature` are query parameters.
- You **must sign the event before sending**. For example, you can use CLI's `sign` command to get the signature string needed in the request above:
```
vaults sign --private-key 0x1234abcd /path/to/file
```

#### Listing vaults

`GET /vaults?account={address}`

**Examples**

```bash
curl 'https://basin.tableland.xyz/vaults?account=0x78C61e68f9f985C43e36dD5ced3f5a24aD0c503e'
```

#### List events

`GET /vaults/{vault_id}/events`

It supports the `limit`, `offset`, `before` and `after` as optional params.

**Examples**

```bash
curl 'https://basin.tableland.xyz/vaults/cache_long.test/events'
```

- Note: the default value for `limit` is `10`, so be sure to request more events or filter with `before` or `after`, in case there are more than 10 events in a vault.

#### Retrieving data

`GET /events/{event_id}`

**Examples**

```bash
curl 'https://basin.tableland.xyz/events/bafkreibq2xwn6xejhdemvoqqjzvbns5agqob4f2zhegutshbzpz6zyemv4' -o data.parquet
```

> Currently, this endpoint can **only download data from the cache**.
>
> If the cache has expired (or was never set), nothing will be downloaded via the HTTP API. Only the CLI `retrieve` command is able to retrieve a non-cached file directly from IPFS/Filecoin. We’re working to implement ways to make the HTTP API as seamless as the CLI flow.

## Development

Expand All @@ -205,7 +319,7 @@ PORT=8888 ./scripts/server.sh
./scripts/run.sh account create pk.out

# Start replicating
./scripts/run.sh vaults stream --dburi [DB_URI] --tables t1,t2 --private-key [PRIVATE_KEY] namespace.relation_name
./scripts/run.sh vaults stream --dburi [DB_URI] --tables t1,t2 --private-key [PRIVATE_KEY] namespace.identifier
```

### Run tests
Expand All @@ -225,4 +339,4 @@ Small note: If editing the README, please conform to the

## License

MIT AND Apache-2.0, © 2021-2023 Tableland Network Contributors
MIT AND Apache-2.0, © 2021-2024 Tableland Network Contributors
Loading