The EOSIO History Tools is a legacy, proof-of-concept (PoC) application that demonstrates scalable and efficient access to finalized blockchain data via the nodeos state-history plugin.
The following History Tools components have been migrated to the following EOSIO repositories:
History Tools consists of the following components:
- database fillers - connect to the nodeos state-history plugin and populate databases
- wasm-ql servers - answer incoming queries by running server WASMs, which have read-only access to the databases
- wasm-ql library - when combined with the CDT library, provides utilities that server WASMs and client WASMs need
- examples of server WASMs and client WASMs
|App||Fills RocksDB||wasm-ql with RocksDB||Fills PostgreSQL||wasm-ql with PostgreSQL|
Note: by default,
history-tools does nothing; use the
--plugin option to select plugins.
This is an alpha release of the EOSIO History Tools. It includes database fillers
fill-rocksdb) which pull data from nodeos's State History Plugin, and a new
query engine (
wasm-ql-rocksdb) which supports queries defined by wasm, along
with an emulation of the legacy
/v1/ RPC API.
This alpha release is designed to solicit community feedback. There are several potential directions this toolset may take; we'd like feedback on which direction(s) may be most useful. Please create issues about changes you'd like to see going forward.
Since this is an alpha release, it will likely have incompatible changes in the future. Some of these may be driven by community feedback.
This release supports nodeos 1.8.x. It does not support 1.7.x or the 1.8 RC versions. This release includes the following:
This release adds temporary workarounds to
fill-pg to support Nodeos 2.0. It also disables the remaining tools. If you would
like to test rocksdb support or wasm-ql support, stick with Nodeos 1.8 and the Alpha 0.2.0 release of History Tools.
- Removed the
- Removed the
- Temporarily disabled building everything except
There are now 2 self-contained demonstrations in public Docker images. See container-demos for details.
- Talk: this demonstrates using wasm-ql to provide messages from on-chain conversations to clients in threaded order.
- Partial history: this demonstrates some of wasm-ql's chain and token queries on data drawn from one of the public EOSIO networks.
Added RocksDB and removed LMDB. This has the following advantages:
- Filling outperforms both PostgreSQL and LMDB by considerable margins, both for partial history and for full history on large well-known chains.
- Database size for full history is much smaller than PostgreSQL.
- Database fillers have a new option
--fill-trxto filter transaction traces.
- Database fillers no longer need
--fill-skip-towhen starting from partial history.
- Database fillers now automatically reconnect to the State History Plugin.
- wasm-ql now uses a thread pool to handle queries.
--wql-threadscontrols the thread pool size.
- wasm-ql now uses eos-vm instead of SpiderMonkey. This simplifies the build process.
- wasm-ql can now serve static files. Enabled by the new
- SHiP connection handling moved to
state_history_connection.hpp. This file may aid users needing to write custom solutions which connect to the State History Plugin.
fill-pg fills postgresql with data from nodeos's State History Plugin. It provides nearly all
data that applications which monitor the chain need. It provides the following:
- Header information from each block
- Transaction and action traces, including inline actions and deferred transactions
- Contract table history, at the block level
Tables which track the history of chain state, including
- Accounts, including permissions and linkauths
- Account resource limits and usage
- Contract code
- Contract ABIs
- Consensus parameters
- Activated consensus upgrades
fill-pg keeps action data and contract table data in its original binary form. Future versions
may optionally support converting this to JSON.
To conserve space,
fill-pg doesn't store blocks in postgresql. The majority of apps
don't need the blocks since:
- Blocks don't include inline actions and only include some deferred transactions. Most applications need to handle these, so should examine the traces instead. e.g. many transfers live in the inline actions and deferred transactions that blocks exclude.
- Most apps don't verify block signatures. If they do, then they should connect directly to
nodeos's State History Plugin to get the necessary data. Note that contrary to
popular belief, the data returned by the
/v1/get_blockRPC API is insufficient for signature verification since it uses a lossy JSON conversion.
- Most apps which currently use the
/v1/get_blockRPC API (e.g.
eosjs) only need a tiny subset of the data within block;
fill-pgstores this data. There are apps which use
/v1/get_blockincorrectly since their authors didn't realize the blocks miss critical data that their applications need.
fill-pg supports both full history and partial history (
trim option). This allows users
to make their own tradeoffs. They can choose between supporting queries covering the entire
history of the chain, or save space by only covering recent history.
EOSIO contracts store their data in a format which is convenient for them, but hard
on general-purpose query engines. e.g. the
/v1/get_table_rows RPC API struggles to provide
all the necessary query options that applications need.
wasm-ql-pg allows contract authors
and application authors to design their own queries using the same
toolset that they use to design contracts. This
gives them full access to current contract state, a history of contract state, and the
history of actions for that contract.
fill-pg preserves this data in its original
format to support
wasm-ql supports two kinds of queries:
- Binary queries are the most flexible. A client-side wasm packs the query into a binary format, a server-side wasm running in wasm-ql executes the query then produces a result in binary format, then the client-side wasm converts the binary to JSON. The toolset helps authors create both wasms.
- An emulation of the
We're considering dropping client-side wasms and switching the format of the first type of query to JSON RPC, Graph QL, or another format. We're seeking feedback on this switch.
These function identically to
wasm-ql-pg, but store data using RocksDB
instead of postgresql. Since RocksDB is an embedded database instead of a database server,
this option may be simpler to administer. RocksDB also saves space and fills quicker.
combo-rocksdb: Fills the database and answers queries. Use this for queries against a live database.
fill-rocksdb: Use this when filling a database for the first time. It fills faster than
combo-rocksdbbut can't answer queries. Switch to
combo-rocksdbafter the database catches up with the chain.
wasm-ql-rocksdb: Rarely used. Queries a database that isn't being filled.