try-runtime

The try-runtime tool is built to query a snapshot of runtime storage, using an in-memory-externalities to store state. In this way, it enables runtime engineers to write tests for a specified runtime state, for testing against real chain state before going to production. It is designed to be used as a command line interface to specify at which block to query state.

In its simplest form, try-runtime is a tool that enables:

  1. Connecting to a remote node and calling into some runtime API.
  2. Scraping the specified state from a node at a given block.
  3. Writing tests for that data.

Motivation

The initial motivation for try-runtime came from the need to test runtime changes against state from a real chain. Prior TestExternalities and BasicExternalities existed for writing unit and integrated tests with mock data, but lacked an avenue to test against a chain's actual state. try-runtime extends TestExternalities and BasicExternalities by scraping state (which is stored with key value pairs) via a node's RPC endpoints getStorage and getKeysPaged and inserting them into TestExternalities.

How it works

The try-runtime tool has its own implementation of externalities called remote_externalities which is just a builder wrapper around TestExternalities that uses a generic key-value store where data is type encoded.

The diagram below illustrates the way externalities sits outside a compiled runtime as a means to capture the storage of that runtime.

Storage externalities

Storage externalities

Testing with externalities

Testing with externalities

With remote_externalities, developers can capture some chain state and run tests on it. Essentially, RemoteExternalities will populate a TestExternalities with a real chain's data.

In order to query state, try-runtime makes use of Substrate's RPCs, namely StateApi. In particular:

  • storage: A method which returns a storage value under the given key.
  • storage_key_paged: A method which returns the keys with prefix with pagination support.

Usage

The most common use case for try-runtime is with storage migrations and runtime upgrades.

There are a number of flags that need to be preferably set on a running node in order to work well with try-runtime’s expensive RPC queries, namely:

  • set --rpc-max-payload 1000 to ensure large RPC queries can work.
  • set --rpc-cors all to ensure ws connections can come through.

You can combine try-runtime with fork-off-substrate to test your chain before production. Use try-runtime to test your chain's migration and its pre and post states. Then, use fork-off-substrate if you want to check that block production continues after the migration.

Calling into hooks from OnRuntimeUpgrade

By default, there are two ways of defining a runtime upgrade in the runtime. The OnRuntimeUpgrade trait provides the different methods to achieve this.

  • From inside a runtime. For example:

    struct Custom;
    	impl OnRuntimeUpgrade for Custom {
    		fn on_runtime_upgrade() -> Weight {
    		// -- snip --
    	}
    }
  • From inside a pallet. For example:

    #[pallet::hooks]
    impl<T: Config> Hooks<BlockNumberFor<T>> for Pallet<T> {
    	fn on_runtime_upgrade() -> Weight {
    	// -- snip --
    	}
    }

These hooks will specify what should happen upon a runtime upgrade. For testing purposes, we prefer having hooks that allow us to inspect the state before and after a runtime upgrade as well.

These hooks are not available by default, and are only available under a specific feature flag, named try-runtime.

The new hooks are as follows:

#[cfg(feature = "try-runtime")]
fn pre_upgrade() -> Result<(), &'static str> { Ok(()) }

#[cfg(feature = "try-runtime")]
fn post_upgrade() -> Result<(), &'static str> { Ok(()) }

Helper functions

OnRuntimeUpgradeHelpersExt are a set of helper functions made available from frame_support::hooks in order to use try-runtime for testing storage migrations. These include:

  • storage_key: Generates a storage key unique to this runtime upgrade. This can be used to communicate data from pre-upgrade to post-upgrade state and check them.
  • set_temp_storage: Writes some temporary data to a specific storage that can be read (potentially in the post-upgrade hook).
  • get_temp_storage : Gets temporary storage data written by set_temp_storage.

Using the frame_executive::Executive struct, these helper functions in action would look like:

pub struct CheckTotalIssuance;
impl OnRuntimeUpgrade for CheckTotalIssuance {
	#[cfg(feature = "try-runtime")]
	fn post_upgrade() {
		// iterate over all accounts, sum their balance and ensure that sum is correct.
	}
}

pub struct EnsureAccountsWontDie;
impl OnRuntimeUpgrade for EnsureAccountsWontDie {
	#[cfg(feature = "try-runtime")]
	fn pre_upgrade() {
		let account_count = frame_system::Accounts::<Runtime>::iter().count();
		Self::set_temp_storage(account_count, "account_count");
	}

	#[cfg(feature = "try-runtime")]
	fn post_upgrade() {
		// ensure that this migration doesn't kill any account.
		let post_migration = frame_system::Accounts::<Runtime>::iter().count();
		let pre_migration = Self::get_temp_storage::<u32>("account_count");
		ensure!(post_migration == pre_migration, "error ...");
	}
}

pub type CheckerMigrations = (EnsureAccountsWontDie, CheckTotalIssuance);
pub type Executive = Executive<_, _, _, _, (CheckerMigrations)>;

CLI interface

To use try-runtime from the command line, run your node with the --features=try-runtime flag.

The possible sub-commands include:

  • on-runtime-upgrade: Executes "tryRuntimeonruntime_upgrade" against the given runtime state.
  • offchain-worker: Executes "offchainWorkerApioffchainworker" against the given runtime state.
  • execute-block: Executes "coreexecuteblock" using the given block and the runtime state of the parent block.
  • follow-chain: Follows a given chain's finalized blocks and applies to all its extrinsics. This allows the behavior of a new runtime to be inspected over a long period of time, with real transactions coming as input.

For example, running try-runtime with the "on-runtime-upgrade" subcommand on a chain running locally:

cargo run --release --features=try-runtime try-runtime on-runtime-upgrade live ws://localhost:9944

Other scenarios

Using it to re-execute code from a ElectionProviderMultiPhase offchain worker on localhost:9944:

cargo run -- --release \
   --features=try-runtime \
   try-runtime \
   --execution Wasm \
   --wasm-execution Compiled \
   offchain-worker \
   --header-at 0x491d09f313c707b5096650d76600f063b09835fd820e2916d3f8b0f5b45bec30 \
   live \
   -b 0x491d09f313c707b5096650d76600f063b09835fd820e2916d3f8b0f5b45bec30 \
   -m ElectionProviderMultiPhase
   --uri wss://localhost:9944

You can pass in the --help flag after each subcommand to see the command's different options.

Run the migrations of the local runtime on the state of SomeChain, for example:

RUST_LOG=runtime=trace,try-runtime::cli=trace,executor=trace \
   cargo run try-runtime \
   --execution Native \
   --chain somechain-dev \
   on-runtime-upgrade \
   live \
   --uri wss://rpc.polkadot.io

Running it at a specific block number's state:

RUST_LOG=runtime=trace,try-runtime::cli=trace,executor=trace \
   cargo run try-runtime \
   --execution Native \
   --chain dev \
   --no-spec-name-check \
   on-runtime-upgrade \
   live \
   --uri wss://rpc.polkadot.io \
   --at <block-hash>

Notice --no-spec-name-check in particular is needed.

Where to go next