This repository has been archived by the owner on Oct 11, 2024. It is now read-only.
-
Notifications
You must be signed in to change notification settings - Fork 113
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
- Loading branch information
Showing
22 changed files
with
584 additions
and
488 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,4 +1,4 @@ | ||
| # Automatically collapse generated files in GitHub. | ||
| ethereum/wrappers/*.go linguist-generated=true | ||
| docs/json-rpc-clients/typescript/*.md linguist-generated=true | ||
| docs/graphql-clients/typescript/*.md linguist-generated=true | ||
|
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,56 +1,98 @@ | ||
| # How to keep an external database in-sync with a Mesh node | ||
|
|
||
| This guide will walk you through syncing an external database with a Mesh node so that the external database's state mirrors that of the Mesh node (and vice-versa). Whenever new orders are discovered or added to Mesh, they are inserted into the database. If an order is filled, cancelled, or has its fillability changed it is updated or removed from the database. We are assuming that your database is storing both the order itself and the remaining fillable amount (i.e., its `fillableTakerAssetAmount`). | ||
| This guide will walk you through syncing your own database with a Mesh node so that your database's | ||
| state mirrors that of the Mesh node (and vice-versa). Whenever new orders are discovered or added to | ||
| Mesh, they should be inserted into your database. If an order is filled, cancelled, or has its fillability | ||
| changed, you should update or remove it from your database. We are assuming that your database is storing | ||
| both the order itself and associated metadata (e.g., `fillableTakerAssetAmount`). | ||
|
|
||
| ## High-level architecture | ||
| This guide fairly advanced, and is specifically written for developers who _need_ to sync their own | ||
| database with Mesh (e.g. in order to join Mesh orders with application-specific data). Note that with | ||
| the introduction of the [GraphQL API](graphql_api.md), most developers do not need to worry about database | ||
| syncing. Instead, you can think of Mesh itself as the database for orders and query it directly. If you | ||
| aren't sure if this guide is for you, try using the [GraphQL API](graphql_api.md) first. | ||
|
|
||
| <img src="./mesh_db_sync_diagram.png" /> | ||
| ## Standalone or Browser nodes | ||
|
|
||
| Mesh is a stand-alone daemon that will be running independently from the rest of your infrastructure. Your backend will communicate with the Mesh node over [WebSockets](https://en.wikipedia.org/wiki/WebSocket) using [JSON-RPC](https://www.jsonrpc.org/). This is a bi-directional connection allowing both your server to notify Mesh of new orders, and Mesh to notify your server of order updates. | ||
| This guide is written with both standalone and browser nodes in mind and the process for syncing | ||
| your database is very similar. | ||
|
|
||
| If you are running a standalone node, you will interact with it via the [GraphQL API](graphql_api.md). | ||
|
|
||
| If you are running Mesh directly in the browser via the `@0x/mesh-browser` or `@0x/mesh-browser-lite` | ||
| packages, you will interact with it using the [TypeScript/JavaScript API](). <!-- FIXME(albrow) put the correct link here --> | ||
|
|
||
| ## Initial sync | ||
|
|
||
| When first connecting the DB and Mesh node, we first need to make sure both have the same orders and order-relevant state stored. We do this with the following steps: | ||
| When first connecting the DB and Mesh node, we need to make sure both have the same orders and | ||
| order-relevant state stored. We do this with the following steps: | ||
|
|
||
| #### 1. Subscribe to Mesh | ||
|
|
||
| Subscribe to the Mesh node's `orders` subscription over a WS connection. This can be done using our [golang](https://godoc.org/github.com/0xProject/0x-mesh/rpc) or [Typescript/Javascript](json_rpc_clients/typescript/README.md) clients or any other JSON-RPC WebSocket client. Whenever you receive an order event from this subscription, make the appropriate updates to your DB. Each order event has an associated [OrderEventEndState](https://godoc.org/github.com/0xProject/0x-mesh/zeroex#pkg-constants). | ||
| Subscribe to the Mesh node's `orderEvents` subscription over a WS connection. This can be done using | ||
| our [Typescript/Javascript](graphql_clients/typescript/README.md) GraphQL client or any other GraphQL | ||
| client which supports subscriptions. | ||
|
|
||
| | End state | DB operation | | ||
| | ------------------------------------------ | ------------ | | ||
| | ADDED | Insert | | ||
| | FILLED | Update | | ||
| | FULLY_FILLED, EXPIRED, CANCELLED, UNFUNDED | Remove | | ||
| | FILLABILITY_INCREASED | Upsert | | ||
| If you are using the `@0x/mesh-browser` or `@0x/mesh-browser-lite` packages, you can subscribe to order | ||
| events directly via the [`onOrderEvents` callback](). <!-- FIXME(albrow) put the correct link here --> | ||
|
|
||
| **Note:** Updates refer to updating the order's `fillableTakerAssetAmount` in the DB. | ||
| Whenever you receive an order event from this subscription, make the appropriate updates to your DB. Each | ||
| order event has an associated [OrderEventEndState](https://godoc.org/github.com/0xProject/0x-mesh/zeroex#pkg-constants). | ||
|
|
||
| **Note 2:** If we receive any event other than `ADDED` and `FILLABILITY_INCREASED` for an order we do not find in our database, we ignore the event and noop. | ||
| | End state | DB operation | | ||
| | ------------------------------------------------------------ | ---------------- | | ||
| | ADDED, FILLED, FILLABILITY_INCREASED, UNEXPIRED | Insert or Update | | ||
| | FULLY_FILLED, EXPIRED, CANCELLED, UNFUNDED, STOPPED_WATCHING | Remove | | ||
|
|
||
| #### 2. Get all orders currently stored in Mesh | ||
| **Note:** If you receive any event other than `ADDED`, `FILLABILITY_INCREASED`, or `UNEXPIRED` | ||
| for an order we do not find in our database, we ignore the event and noop. | ||
|
|
||
| There might have been orders stored in Mesh that the DB doesn't know about at this time. Because of this, we must fetch all currently stored orders in the Mesh node and upsert them in the database. This can be done using the [mesh_getOrders](rpc_api.md#mesh_getorders) JSON-RPC method. This method creates a snapshot of the Mesh node's internal DB of orders when first called, and allows for subsequent paginated requests against this snapshot. Because we are already subscribed to order events, any new orders added/removed after the snapshot is made will be discovered via that subscription. | ||
| #### 2. Get all orders currently stored in Mesh | ||
|
|
||
| **Note:** The [Mesh Typescript client](json_rpc_clients/typescript/README.md) has a convenience method that does the multiple paginated requests for you under-the-hood. You can simply call the [getOrders](json_rpc_clients/typescript/reference.md#getordersasync) method. | ||
| There might have been orders stored in Mesh that your DB doesn't know about at this time. Because | ||
| of this, you should fetch all currently stored orders in the Mesh node and upsert them in the database. | ||
| This can be done using the [orders](graphql_api.md#querying-and-filtering-orders) GraphQL query. If you | ||
| are using our TypeScript GraphQL client, you can use the [`getOrdersAsync`]() <!-- FIXME(albrow) put the correct link here --> | ||
| method. If you are using the `@0x/mesh-browser` or `@0x/mesh-browser-lite` packages, you can use | ||
| the method by the same name, [`getordersAsync`](). <!-- FIXME(albrow) put the correct link here --> | ||
|
|
||
| #### 3. Add all database orders to the Mesh node | ||
| Orders may be added or removed while you are getting existing orders from the Mesh DB. For this reason, | ||
| it is important to account for any order events received from step (1) while or after you get the existing | ||
| orders in this step. | ||
|
|
||
| Since there might also be orders added to the database that Mesh doesn't know about, we must also add all DB orders to Mesh. We can do this using the [mesh_addOrders](rpc_api.md#mesh_addorders) JSON-RPC method. This method accepts an array of signed 0x orders and returns which have been accepted and rejected. The accepted orders are returned with their `fillableTakerAssetAmount` and so these amounts should be updated in the database. Rejected orders are rejected with a specific [RejectedOrderStatus](https://godoc.org/github.com/0xProject/0x-mesh/zeroex#pkg-variables), including an identifying `code`. | ||
| **Note:** The [Mesh Typescript client](graphql_clients/typescript/README.md) has a convenience method | ||
| that does the multiple paginated requests for you under-the-hood. You can simply call the | ||
| [getOrders](graphql_clients/typescript/reference.md#getordersasync) method. | ||
|
|
||
| | Code | Reason | Should be retried? | | ||
| | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------- | ------------------ | | ||
| | EthRPCRequestFailed, CoordinatorRequestFailed, CoordinatorEndpointNotFound, InternalError | Failure to validate the order | Yes | | ||
| | MaxOrderSizeExceeded, OrderMaxExpirationExceeded, OrderForIncorrectChain, SenderAddressNotAllowed | Failed Mesh-specific criteria | No | | ||
| | OrderHasInvalidMakerAssetData, OrderHasInvalidTakerAssetData, OrderHasInvalidSignature, OrderUnfunded, OrderCancelled, OrderFullyFilled, OrderHasInvalidMakerAssetAmount, OrderHasInvalidTakerAssetAmount, OrderExpired | Invalid or unfillable order | No | | ||
| #### 3. Add all database orders to the Mesh node | ||
|
|
||
| If an order was rejected with a code related to the "failure to validate the order" reason above, you can re-try adding the order to Mesh after a back-off period. For all other rejection reasons, the orders should be removed from the database. | ||
| Since there might also be orders in your database that Mesh doesn't know about, you should also | ||
| add those orders to Mesh. We can do this using the [addOrders](graphql_api.md#adding-orders) | ||
| GraphQL mutation. If you are using our TypeScript GraphQL client, you can use the [`addOrdersAsync`]() <!-- FIXME(albrow) put the correct link here --> | ||
| method. If you are using the `@0x/mesh-browser` or `@0x/mesh-browser-lite` packages, you can use | ||
| the method by the same name, [`addOrdersAsync`](). <!-- FIXME(albrow) put the correct link here --> | ||
|
|
||
| This method accepts an array of signed 0x orders and returns which have been accepted | ||
| and rejected. The accepted orders are returned with their `fillableTakerAssetAmount` and so these | ||
| amounts should be updated in the database. Rejected orders are rejected with a specific | ||
| [RejectedOrderStatus](https://godoc.org/github.com/0xProject/0x-mesh/zeroex/ordervalidator), including | ||
| an identifying `code`. The following codes indicate temporary errors and you may try submitting the | ||
| order again (typically with exponential backoff): `INTERNAL_ERROR`, `ETH_RPC_REQUEST_FAILED`, or | ||
| `DATABASE_FULL_OF_ORDERS`. For any other code, the order has been rejected by Mesh and should not | ||
| be retried. If the order exists in your database is should be removed. | ||
|
|
||
| #### 4. Handle dropped connections | ||
|
|
||
| After performing the first 3 steps above, the Mesh node and database will be in-sync, and continue to remain in-sync thanks to the active order event subscription. If any new orders are added to the database, they will also need to be added to Mesh of course. But what if the WebSocket connection to the Mesh node goes down? In that case, it must be re-established and steps 1, 2 & 3 must be performed once again. | ||
|
|
||
| **Note:** The [Mesh Typescript client](json_rpc_clients/typescript/reference.md#getordersasync) takes care of re-connecting and re-establishing _all_ active subscriptions once it detects a disconnection. You can subscribe to a `reconnected` event using the [onReconnected](json_rpc_clients/typescript/reference.md#onreconnected) method. Whenever this callback is fired is when you need to re-run steps 2 and 3. | ||
|
|
||
| **Note 2:** With some WebSocket clients, we've noticed that the client is not always aware of when the connection has been dropped. It can be hard for clients to discern between a network disruption and latency. Because of this, we added an explicit [heartbeat subscription](rpc_api.md#mesh_subscribe-to-heartbeat-topic) to Mesh that you can subscribe to. If a heartbeat isn't received after some interval (e.g., 20 seconds), the client can forcible drop the connection and re-establish a new one. This too is already taken care of under-the-hood for those using the [Mesh Typescript client](json_rpc_clients/typescript/reference.md#getordersasync). | ||
| After performing the first 3 steps above, your database will be in-sync with the Mesh database, and continue to remain | ||
| in-sync thanks to the active order event subscription. If any new orders are added to the database, they will also need | ||
| to be added to Mesh of course. But what if the WebSocket connection to the Mesh node goes down? In that case, it | ||
| must be re-established and steps 1, 2 & 3 must be performed once again. | ||
|
|
||
| **Note:** With some WebSocket clients, we've noticed that the client is not always aware of when the connection has been | ||
| dropped. It can be hard for clients to discern between a network disruption and latency. Because of this, our GraphQL server | ||
| uses `GQL_CONNECTION_ACK` and `GQL_CONNECTION_KEEP_ALIVE` messages as described in | ||
| https://github.com/apollographql/subscriptions-transport-ws/blob/master/PROTOCOL.md. Our | ||
| [Typescript GraphQL client](graphql_clients/typescript) automatically handles these messages, and | ||
| most other GraphQL clients will too. | ||
|
|
||
| Happy database syncing! |
Oops, something went wrong.