Offchain operations

There is often a need to query and/or process offchain data before it can be included in the on-chain state. The conventional way of doing this is through oracles. While this approach works, it still has several flaws with respect to security, scalability, and infrastructure efficiency.

To make the offchain data integration secure and more efficient, Substrate provides offchain features:

  • Offchain worker (OCW) subsystems allows execution of long-running and possibly non-deterministic tasks (e.g. web requests, encryption/decryption and signing of data, random number generation, CPU-intensive computations, enumeration/aggregation of on-chain data, etc.) that could otherwise require longer than the block execution time.
  • Offchain storage offers storage that is local to a Substrate node that can be accessed both by offchain workers (both read and write access) and on-chain logic (write access via offchain indexing but not read access). This allows different worker threads to communicate to each other and to store user or node-specific data that does not require consensus over the whole network.
  • Offchain indexing allows the runtime, if opted-in, to write directly to the offchain storage independently from OCWs. This serves as a local/temporary storage for on-chain logic and complement to its on-chain state.

Off-chain features run in their own Wasm execution environment outside of the Substrate runtime. This separation of concerns makes sure that block production is not impacted by long-running offchain tasks. However, as the offchain features are declared in the same code as the runtime, they can easily access on-chain state for their computations.

<Image is missing>

Off-chain workers

Off-chain workers have access to extended APIs for communicating with the external world:

  • Ability to submit transactions (either signed or unsigned) to the chain to publish computation results.
  • A fully-featured HTTP client allowing the worker to access and fetch data from external services.
  • Access to the local keystore to sign and verify statements or transactions.
  • An additional, local key-value database shared between all offchain workers.
  • A secure, local entropy source for random number generation.
  • Access to the node's precise local time.
  • The ability to sleep and resume work.

Note that the results from offchain workers are not subject to regular transaction verification. A verification mechanism (e.g. voting, averaging, checking sender signatures, or simply "trusting") should be implemented to determine what information gets into the chain.

Off-chain storage

As its name indicates, the storage is not stored on-chain. It can be accessed by offchain worker threads (both read and write access) and on-chain logic (write only, refer to offchain indexing below). This storage is not populated among the blockchain network and does not need to have consensus computation over it.

As an offchain worker thread is being spawned off during each block import, there could be more than one offchain worker thread running at any given time. So, similar to any multi-threaded programming environment, there are also utilities to mutex lock the storage when accessing them for data consistency.

Offchain storage serves as a bridge for various offchain worker threads to communicate to each others and between offchain and on-chain logics. It can also be read using remote procedure calls (RPC) so it fits the use case of storing indefinitely growing data without over-consuming the on-chain storage.

Offchain indexing

Storage in the context of blockchain is mostly about on-chain state. But it is expensive (as it is populated to each node in the network) and not recommended for historical or user-generated data which grow indefinitely over time.

We have offchain storage for this purpose. In addition of being accessible by OCWs, Substrate also includes a feature called "offchain indexing" allowing the runtime to write directly to the offchain storage independently from OCWs. Nodes have to opt-in for persistency of this data via --enable-offchain-indexing flag when starting up the Substrate node.

Unlike OCWs, which are not executed during initial blockchain synchronization, offchain indexing is populating the storage every time a block is processed, so the data is always consistent and will be exactly the same for every node with indexing enabled.

Learn more

To look at concrete examples of offchain workers and how to use them in runtime development, refer to the following sections in Substrate Recipes: