Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve CLI help, add versioning, & list/retrieve enhancements #30

Merged
merged 12 commits into from
Dec 19, 2023
Merged

Improve CLI help, add versioning, & list/retrieve enhancements #30

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
764e83e
feat: version flag & GH tag/version action
dtbuchholz Dec 17, 2023
945ea7d
feat: add cmd aliases, descriptive error strings, & docs improvements
dtbuchholz Dec 17, 2023
a7b0ce6
chore: add install to makefile
dtbuchholz Dec 17, 2023
a1cf149
feat: add version command, list cmd json format, more docs
dtbuchholz Dec 18, 2023
f02b0ab
docs: update readme
dtbuchholz Dec 18, 2023
c270b94
fix: alter gh version action logic, version logging
dtbuchholz Dec 18, 2023
dea664e
chore: update version to v0.0.6
dtbuchholz Dec 18, 2023
b7a45aa
feat: add vault alias flag '-v'
dtbuchholz Dec 18, 2023
d71ceac
feat: retrieve to custom output dir or stdout
dtbuchholz Dec 18, 2023
571935a
chore: linting fixes; capitalization, new lines, colon usage in doc s…
dtbuchholz Dec 18, 2023
91d733a
docs: fix incorrect vaults install link
dtbuchholz Dec 19, 2023
a670d23
chore: alter GH version workflow
dtbuchholz Dec 19, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
25 changes: 25 additions & 0 deletions .github/workflows/update-version.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
name: Update version file
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nice! ideally, the binaries workflow would depend on a successful run of this one. not sure how easy it is to implement that, i think it's fine the way it is

Copy link
Contributor Author

@dtbuchholz dtbuchholz Dec 19, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@brunocalza maybe this would work? https://docs.github.com/en/actions/using-workflows/events-that-trigger-workflows#workflow_run

name: Generate binaries
on:
  workflow_run:
    workflows: ["Update version file"]
    types:
      - completed

i have the update-version workflow to only work on main, though, so would that pose an issue if the generated binaries workflow was dependent on it?

but yeah, maybe we can figure that out in a separate PR, if needed.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hum, I missed this was triggered when a push happens to main. Ideally, it would be triggered when a release is created, right? and the binaries would be run after that. because if you create version.json when a push happens, the next binary release will have the wrong version

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@brunocalza ah, i see. when the binaries are generated, it looks like that's where the tag gets created, which is what i fetch to get the version in the CLI.

  • publishing
    • scm releases
      • creating or updating release                 tag=v0.0.6 repo=tablelandnetwork/basin-cli
      • release updated                              name=v0.0.6 release-id=133993550 request-id=3BC0:2FB0:182D22B:1919694:657B5041
      • uploading to release                         file=dist/basin-darwin-amd64.tar.gz name=basin-darwin-amd64.tar.gz
      • uploading to release                         file=dist/basin-linux-arm64.tar.gz name=basin-linux-arm64.tar.gz
      • uploading to release                         file=dist/basin-linux-amd64.tar.gz name=basin-linux-amd64.tar.gz
      • uploading to release                         file=dist/basin-darwin-arm64.tar.gz name=basin-darwin-arm64.tar.gz
      • published                                    url=https://github.com/tablelandnetwork/basin-cli/releases/tag/v0.0.6
      • took: 4s

hm, so maybe my approach won't work because i'd need my GH action to run after that tag is created but before the binaries are generated?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The tag is not being created by the workflow. The tag is created by Github itself when a release is created.

I think you need to switch to

on:
  release:
    types:
      - created

and somehow get access to the tag being created inside the workflow.

You can test that, by drafting new releases targeting your branch.

If this is taking too much of your time. Feel free to merge it and I can figure that out for you.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@brunocalza yeah, if you could take a look, that might be best! i dont want to mess up anything there. should i just merge this into your branch now?

i did alter my version workflow to take your trigger into account—this seems like it should work, but i havent tested it:

name: Update version file

on:
  release:
    types:
      - created

jobs:
  update_version:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
        with:
          fetch-depth: 0

      - name: Update version.json
        run: |
          echo "{
            \"version\": \"${{ github.event.release.tag_name }}\"
          }" > version.json

      - name: Commit changes
        uses: stefanzweifel/git-auto-commit-action@v4
        with:
          commit_message: "chore: update version to ${{ github.event.release.tag_name }}"

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍 feel free to merge it


on:
release:
types:
- created

jobs:
update_version:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
with:
fetch-depth: 0

- name: Update version.json
run: |
echo "{
\"version\": \"${{ github.event.release.tag_name }}\"
}" > version.json

- name: Commit changes
uses: stefanzweifel/git-auto-commit-action@v4
with:
commit_message: "chore: update version to ${{ github.event.release.tag_name }}"
5 changes: 5 additions & 0 deletions Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,11 @@ build:
go build -o vaults cmd/vaults/*
.PHONY: build

# Install
install:
go install ./cmd/vaults
.PHONY: install

# Test
test:
go test ./... -short -race -timeout 1m
Expand Down
106 changes: 65 additions & 41 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,43 +3,54 @@
[![License](https://img.shields.io/github/license/tablelandnetwork/basin-cli.svg)](./LICENSE)
[![standard-readme compliant](https://img.shields.io/badge/standard--readme-OK-green.svg)](https://github.com/RichardLitt/standard-readme)

> Continuously publish data from your database to the Tableland network.
> Continuously publish data from your database or file uploads to the Tableland Vaults network.

# Table of Contents
## Table of Contents

- [Install](#install)
- [Postgres Setup](#postgres-setup)
- [Self-hosted](#self-hosted)
- [Amazon RDS](#amazon-rds)
- [Background](#background)
- [Usage](#usage)
- [Install](#install)
- [Postgres Setup](#postgres-setup)
- [Supabase](#supabase)
- [Create a vault](#create-a-vault)
- [Start replicating a database](#start-replicating-a-database)
- [Write a Parquet file](#write-a-parquet-file)
- [Listing Vaults](#listing-vaults)
- [Listing Events](#listing-events)
- [Running](#running)
- [Run tests](#run-tests)
- [Retrieving](#retrieving)
- [Create a vault](#create-a-vault)
- [Start replicating a database](#start-replicating-a-database)
- [Write a Parquet file](#write-a-parquet-file)
- [Listing Vaults](#listing-vaults)
- [Listing Events](#listing-events)
- [Retrieving](#retrieving)
- [Development](#development)
- [Running](#running)
- [Run tests](#run-tests)
- [Contributing](#contributing)
- [License](#license)

## Background

Textile Vaults is a secure and verifiable open data platform. The Vaults CLI is a tool that allows you to continuously replicate a table or view from your database to the network (currently, only PostgreSQL is supported). Or, you can directly upload files to the vault (currently, parquet is only supported)

# Background
> 🚧 Vaults is currently not in a production-ready state. Any data that is pushed to the network may be subject to deletion. 🚧

Textile Vaults is a secure and verifiable open data platform. The Vaults CLI is a tool that allows you to continuously replicate a table or view from your database to the network. Currently, only PostgreSQL is supported.
## Usage

> 🚧 Vaults is currently not in a production-ready state. Any data that is pushed to the network may be subject to deletion. 🚧
### Install

# Usage
You can either install the CLI from the remote source:

```bash
go install github.com/tablelandnetwork/basin-cli/cmd/vaults@latest
```

## Install
Or clone from source and run the Makefile `install` command:

```bash
git clone https://github.com/tablelandnetwork/basin-cli.git
cd basin-cli
go install ./cmd/vaults
make install
```

## Postgres Setup
### Postgres Setup

### Self-hosted
#### Self-hosted

- Make sure you have access to a superuser role. For example, you can create a new role such as `CREATE ROLE vaults WITH PASSWORD NULL LOGIN SUPERUSER;`.
- Check that your Postgres installation has the [wal2json](https://github.com/eulerto/wal2json) plugin installed.
Expand All @@ -53,7 +64,7 @@ go install ./cmd/vaults

- Restart the database in order for the new `wal_level` to take effect (be careful!).

### Amazon RDS
#### Amazon RDS

- Make sure you have a user with the `rds_superuser` role, and use `psql` to connect to your database.

Expand Down Expand Up @@ -88,35 +99,35 @@ go install ./cmd/vaults
postgresql://postgres:[PASSWORD]@db.[PROJECT_ID].supabase.co:5432/postgres
```

## Create a vault
### Create a vault

_Vaults_ define the place you push data into.

Vaults uses public key authentication, so you will need an Ethereum style (ECDSA, secp256k1) wallet to create a new vault. You can use an existing wallet or set up a new one with `vaults wallet create`. Your private key is only used locally for signing.

```bash
vaults wallet create [FILENAME]
vaults account create [FILENAME]
```

A new private key will be written to `FILENAME`.

The name of a vault contains a `namespace` (e.g. `my_company`) and the name of an existing database relation (e.g. `my_table`), separated by a period (`.`). Use `vaults create` to create a new vault. See `vaults create --help` for more info.

```bash
vaults create --dburi [DBURI] --account [WALLET_ADDRESS] namespace.relation_name
vaults create --dburi [DBURI] --account [WALLET_ADDRESS] namespace.relation_name
```

🚧 Vaults currently only replicates `INSERT` statements, which means that it only replicates append-only data (e.g., log-style data). Row updates and deletes will be ignored. 🚧

## Start replicating a database
### Start replicating a database

Use `vaults stream` to start a daemon that will continuously push changes to the underlying table/view to the network. See `vaults stream --help` for more info.

```bash
vaults stream --private-key [PRIVATE_KEY] namespace.relation_name
```

## Write a Parquet file
### Write a Parquet file

Before writing a Parquet file, you need to [Create a vault](#create-a-vault), if not already created. You can omit the `--dburi` flag, in this case.

Expand All @@ -126,7 +137,7 @@ Then, use `vaults write` to write a Parquet file.
vaults write --vault [namespace.relation_name] --private-key [PRIVATE_KEY] filepath
```

You can attach a timestamp to that file write, e.g.
You can attach a timestamp to that file write, e.g.

```bash
vaults write --vault [namespace.relation_name] --private-key [PRIVATE_KEY] --timestamp 1699984703 filepath
Expand All @@ -140,15 +151,15 @@ vaults write --vault [namespace.relation_name] --private-key [PRIVATE_KEY] --tim

If a timestamp is not provided, the CLI will assume the timestamp is the current client epoch in UTC.

## Listing Vaults
### Listing Vaults

You can list the vaults from an account by running:

```bash
vaults list --account [ETH_ADDRESS]
```

## Listing Events
### Listing Events

You can list events of a given vault by running:

Expand All @@ -162,48 +173,61 @@ Events command accept `--before`,`--after` , and `--at` flags to filter events b
# examples
vaults events --vault demotest.data --at 1699569502
vaults events --vault demotest.data --before 2023-11-09T19:38:23-03:00
vaults events --vault demotest.data --after 2023-11-09
vaults events --vault demotest.data --after 2023-11-09
```

## Retrieving
### Retrieving

You can retrieve a file from a vault by running:

```bash
vaults retrieve bafybeifr5njnrw67yyb2h2t7k6ukm3pml4fgphsxeurqcmgmeb7omc2vlq
```

# Development
You can also specify where to save the file:

## Running
```bash
vaults retrieve --output /path/to/dir bafybeifr5njnrw67yyb2h2t7k6ukm3pml4fgphsxeurqcmgmeb7omc2vlq
```

Or stream the file to stdout the `-` value (note: the short form `-o` is for `--output`), and then pipe it to something like [`car extract`](https://github.com/ipld/go-car) to unpack the CAR file's contents:

```bash
vaults retrieve -o - bafybeifr5njnrw67yyb2h2t7k6ukm3pml4fgphsxeurqcmgmeb7omc2vlq | car extract
```

## Development

### Running

You can make use of the scripts inside `scripts` to facilitate running the CLI locally without building.

```bash
# Starting the Provider Server
PORT=8888 ./scripts/server.sh

# Create a wallet
./scripts/run.sh wallet create pk.out
# Create an account
./scripts/run.sh account create pk.out

# Start replicating
./scripts/run.sh vaults stream --private-key [PRIVATE_KEY] namespace.relation_name
```

## Run tests
### Run tests

```bash
make test
```

Note: One of the tests requires Docker Engine to be running.


# Contributing
## Contributing

PRs accepted.

Small note: If editing the README, please conform to the
[standard-readme](https://github.com/RichardLitt/standard-readme) specification.

# License
## License

MIT AND Apache-2.0, © 2021-2023 Tableland Network Contributors
Loading
Loading