The tools presented in this module so far are fully public: everyone can use them. However, they are meant for public networks, and the user is dependent on the infrastructure and configuration of these services. Some organizations are more interested in deploying their private Tezos network, either for private testing before going public or for entirely private use. Refer to the Private blockchain module for more info. Therefore, it is possible to use an indexer on private networks.
Fortunately, many public developments on Tezos are open-source and can easily be adapted for a private network, i.e. an indexer can be deployed and configured to watch a private network.
In this chapter, we will see how the source code of TzIndex and TzStats can be used to make a private indexer, either for use on a private network or a public network.
This chapter is divided into three parts:
- how to quickly set up a private network.
- how to install and configure these tools
- how to use them
We will simulate a private network that will be indexed by TzIndex. Therefore, this private network should hold the basic expected elements on any network:
- a few funded accounts
- a smart contract
- some operations
Let's start by setting up a private network. The process is quite difficult as it requires starting and configuring Tezos nodes (please refer to the Private Blockchain module for more info). Instead, let's choose an easier way by only simulating a private network using Ganache, a node module used for tests and simulations of Tezos and Ethereum blockchains. Ganache launches the Flextesa docker image, which runs Tezos nodes, bakers and endorsers in a sandbox environment. It also provides ten funded accounts from the start.
The complete source code of the raffle contract can be found here.
It contains a ganache configuration (with predefined accounts), three smart contracts and their associated migrations:
- the first contract holds a big map and does not do anything. This dummy contract is used to bypass a TzStats bug regarding big maps: the first big map (whose index is 0) is not fetched by the frontend.
- a raffle contract using a map
- a raffle using a big map
The second and third smart contracts will highlight the difference between the way maps and big maps are handled by indexers.
TzIndex is written in Go, a statically typed compiled programming language developed by Google. Install it using the instructions at golang.org/doc/install.
The next action is to install the docker images of TzIndex. Install Docker using the instructions at docs.docker.com/get-docker/.
package.json, one script is defined:
The npm repository of ganache can be found here: There are several versions of `ganache-cli`. Only the versions suffixed by `-tezos` support the Tezos blockchain (you can find them under the versions tab). The specific version used in the project can be found in the `devDependencies` section in `package.json`. When this article was written, the version was 6.12.1-tezos.0
This script starts a simulated private Tezos blockchain with ganache:
Notice all the funded accounts created by Ganache.
The contracts can be migrated onto the private network. More details can be found about the Truffle tool in the Build a Dapp module.
First, the private network has to be defined in the truffle-config.js file. A
development subsection should be found
It defines a localhost network, matching the ganache private network.
We're going to use the accounts from the
scripts/sandbox/account.js file. Our three contracts can now be migrated with this command:
Three contracts are now deployed onto our private network:
The migration files also include some operations to open a raffle and buy a ticket automatically once the contracts are deployed.
The same smart contract is deployed twice: one with _maps_ and the other with _big maps_. Later on, these two smart contracts will be used to highlight the difference between the processing of _maps_ and _big maps_ by _TzStats_.
TzIndex is an open-source indexer: it can freely be used and modified. The source code is available at github.com/blockwatch-cc/tzindex
To start the application, proceed as follows:
The last command should output "tzindex" (to check if a tzindex has indeed been built) A TzIndex binary is created in the same directory. The indexer can now watch the private network with this command:
Note the following options:
- --rpcurl: url of the ganache private network rpc node
- --notls: option to use http instead of https
- --enable-cors: TzIndex API will be queried by Tzstat (exposed on a different port): CORS need to be enabled
TzIndex now exposes its API on http://localhost:8000.
db/ folder is created when launching TzIndex for the first time: it contains all the data indexed about the
When restarting ganache, a blank new network is created. The `db/` folder, containing the old data has to be removed in order to make _TzIndex_ index the new network.
TzStats is the open-source frontend made to display the data from TzIndex. The source code is available at github.com/blockwatch-cc/tzstats
TzStats can interact with TzIndex by setting the TZSTATS_API_URL variable to the url of the TzIndex API.
The application can be launched with:
An error might occur during the npm install: ```shell /bin/sh: 1: sass: not found ``` If so, modify the `sass` field under `scripts` in the `package.json` file as follows: ```json "sass": "npx sass src/styles/scss/index.scss:src/styles/css/index.css" ``` Then run `npm install` again.
TzStats will now launch in your web browser.
At this point, the working context is:
- a simulated private network running through Ganache
- three migrated contracts
- two contract calls
- TzIndex indexing the private network
- TzStats connected to TzIndex, and running
This means that there has already been some activity that we can watch on our network.
TzStats is running at http://localhost:3000:
Notice that the frontend is different from its public version on TzStats.com but the inner workings are the same.
On top, the search bar allows you to search for transactions, blocks, addresses, etc. The left panel contains various information displayed, such as the number of baked blocks.
Click on it to get more info on that block:
Many block details are displayed; most of them can be clicked on for even more info. The block above only contains one operation, which is a call to our smart contracts. Click the hash to open the operation page:
The sender does match Alice's
pkh from scripts/sandbox/account.js.
pkh means public key hash: it is an address on a Tezos network.
Notice the contract address matches the returned address from the migration output logs.
Click on the address to inspect the smart contract:
You can see the history of calls, the entrypoints, the storage, etc. Notice the two calls made by Truffle: the origination of the contract and the purchase of a ticket.
The storage page displays the storage definition, the type of each field and their associated current value. Note that one participant is registered in the raffle and the contract holds one Tez (from selling one ticket).
Note that the sold_tickets big map should hold an
address as value, but it displays
1 in the storage. Indeed,
since big maps are meant to hold unbounded lists of data, they cannot be loaded directly into the storage (out of
performance concerns). Each big map is indexed and the number
1 being displayed is the index number of the big map
. Its data can actually be accessed by clicking on the
Bigmap 1 section.
Regular maps are meant to be used with limited data size as the data is directly retrieved from the storage section. You can see an example of such a map in the second migrated smart contract (baked in the fourth block):
The TzStats interface is user-friendly and every piece of information can easily be read by clicking on each element. Remember that all this information is coming from the data retrieved by the indexer: TzStats just displays them.
The same pieces of information can be retrieved without the frontend by just using the TzIndex API. For each page opened on TzStats, an API call is made to TzIndex. Each API call can be seen in the network explorer by pressing F12 in the network section.
The request URL shows the called API endpoints:
A lot of information is displayed, such as the contract's address, but some data is also missing, like the storage. Another API call has to be made for that:
All the available endpoints can be found here: tzstats.com/docs/api#explorer-endpoints
If you don't want your infrastructure to rely on third-party public indexers to monitor public networks, you can also use a local indexer. It just has to monitor a node: either a local node, or a public node (listed at tezostaquito.io/docs/rpc_nodes).
There are three operation modes, which retrieve more or less data:
- Full regular: all indexes are built (default mode)
- Light light-weight: consensus and governance indexes are not built (use --light in the CLI)
- Validate: state validation mode for checking accounts and balances at each block/cycle (use --validate in the CLI)
Note that whichever mode your choose, indexing a public network will take a significant amount of time due to the size of data to be fetched.
According to the Tzstats team, it is possible to index 13 000 blocks per minute from a local node connected to the mainnet (Cost and benefice section from this tzstats post). When indexing the mainnet from a public node (Smartpy node on our case), we could index about 10 blocks per seconde.
The mainnet was about 1 400 000 blocks, so to index the mainnet, it should take:
- about an hour and fifty minutes from a local node
- about a day and a half from a public node.
Of course, these figures depend on your network connectivity and hardware.
Running a private network does not condemn its user to be blind to whatever is happening on their network: private indexers and explorers can be set up to index and monitor a private network.
Public networks (mainnet, testnets) are indexed by public tools ([https://tzstats.com/] for instance). However, some use cases may required not to rely on third party services: private indexers and explorers can be used to that purpose.
Finally, indexers, since they expose a complete API, can be used to replace libraries (such as Taquito) or wallets. Decentralised apps can take advantage of these API calls to fetch data from a Tezos network, especially when it comes down to big maps.