settings
Test Driven Development

Introduction

The Holochain system includes a testing harness that allows you to test your application functions. In addition we have built integration that simultaneously runs a set of tests in separate virtual nodes. We strongly recommend using both these tools for application development.

 

HoloWorld Tutorial: Writing Tests

 

 Running basic single-node tests

Assuming your current working directory is my_app you can run all the single-node tests (test/*.json) with this command:

$ hcdev test

This command loads a new copy of my_app into the ~/.holochaindev directory, and then finds all the .json extension files in the test directory and runs them in a random order. Note that this doesn't run the test files in the scenario directories, only the top-level test files.

You can run a single test file with (note you don't need the .json extension).: 

$ hcdev test <test-name>

 

Test Files

A test file is essentially a collection of function calls into your application with specified inputs and expected outputs, or error values. A test file consists of an object of the form:

{
  Identity: string   // identity to be used for Agent.String
  Tests: [
    {
      Convey:   string        // a human readable description of the tests intent
      Zome:     string        // the zome in which to find the function
      FnName:   string        // the application function to call
      Input:    string        // the function's input
      Output:   string        // the expected output to match against (full match)
      Err:      string        // the expected error to match against
      ErrMsg:   string        // the expected error message to match against
      Regexp :  string        // the expected out to match again (regular expression)
      Time:     int           // offset in milliseconds from the start of the test at which to run this test.
      Wait:     int           // time in milliseconds to wait before running this test from when the previous ran
      Exposure: string        // the exposure context for the test call (defaults to ZOME_EXPOSURE)
      Raw:      bool          // set to true if we should ignore fnName and just call input as raw code in the Ribosome, useful for testing helper functions and validation functions, or doing more complex testing
      Repeat:   int           // number of times to repeat this test, useful for scenario testing
    },
    {...}
  ]
}

Replacement Strings

Input and Output values can include special replacement values that allow you to substitute for various in-system values for your tests.

  • %h%,%h1%,%h2%,...: the hash of the nth entry on the source chain. Allows you to test that value was committed. %h% would mean most recently written, and %h1% second most recent, and so on.

  • %r1%,%r2%,%r3%: the three most recent test results treated as a stack.  Useful if result is a string (not an object).

  • %result0%,%result1%, ... : the results from previous tests in order of the test. Useful if result was a string (not an object)
  • {"%result%":X},: result from the Xth test.  Use this form when that result was an object, not a string.

  • %dna%: the DNA hash.

  • %agent%: the hash of the second entry on the chain which is always, the permanent agent entry.

  • %agenttop%: the hash of the most recent agent entry on the chain (because of key revocation or identity change).

  • %agentstr%: the contents of the current agent entry.

  • %key%: the node address or key, which is also a hash.

  • %mX.Y%: match from previously matched regular expression test output.  X & Y are optional and represent the Xth match, Yth grouping match.

  • %reps% : The current repetition number if you have specified that the test is to be repeated multiple times with the Repeat value.

When running scenario tests you also have available the following substitutions:

  • %<role>_key%: the hash of the address of the given role.
  • %<role>_str%: the identity string the given role.
  • %clone%: the clone number if this role is cloned.

 

Test Folder Structure

A Holochain application directory (with the test directory expanded) should look something like this:

  • my_app

    • dna

    • ui

    • test

      • singleNodeTest.1.json

      • singleNodeTest.2.json

      • scenario.myScenario.1

        • role1.json

        • role2.json

        • ...

        • roleN.json

        • _config.json

      • scenario.myScenario.2

        • etc...

 

Muilt-node tests (scenarios)

Multi-node tests are can be though of as a scenario with a number of roles, where each role represents one node in the multi-node test, and is defined by a test file in the scenario sub-directory.

To run a mulit-node scenario test, use this command:

hcdev scenario <your-scenario-directory-name-here>

Video showing tests being run (doesn't open new tab, maybe ctrl-click or w/e)

Each test/scenario sub-directory should contain one test file for each role required to model the test. The scenario command requires the name of the scenario directory [my_app/test/<scenarioName>] as a parameter. Test filenames are automatically discovered.

Tests pass or fail on the basis of the content of messages passed between roles/nodes on the network. In order to test the content of messages passed between roles, it is necessary for tests to account for the amount of time it takes for messages to travel between nodes on the network. This is achieved with the Time parameter of the test object. If roles 1 and 2 are called back and forthrespectively, then when forth sends a message, back should wait at least 50ms for checking to see if there are messages that contain the expected content. If the message has not yet arrived, then the test will fail.

You can add a `_config.json` file to the scenario directory of the following format:

{
    "GossipInterval":500,
    "Duration":3,
    "Clone":[{"Role":"sender","Number":20}]
}

This config file allows you:

  • set the gossip interval for your test

  • a minimum duration that all the nodes should be kept running for the test to succeed, i.e. so other nodes that still have tests pending can gossip with them.

  • a list of roles for which you want multiple clones to run at the same time, plus the count of clones of that role.

In the chat sample application you will find a backnforth scenario with two roles defined: person1 & person2. Check out these examples to see how things work.

 

Testing Error Conditions

To test error conditions you can specify your expected error in either the `Err` or `ErrMsg` attributes.  `Err` matches against an entire error object, so usually it's easier to use `ErrMsg` which just matches against the `errorMessage` attributed within the error object.  Note that if you have set `ErrorHandling` in the javascript zome `Config` to `"returnErrorValue"` the error object will be returned as a value from a any calls you make and thus won't show up in `Err` unless you explicitly `throw` it.

Notes on the Time parameter:

  • if Time is null or 0, tests are executed in the order discovered in the test file

  • all other values of the Time parameter are used in multi-node testing to allow sufficient space for gossip operations between nodes on the DHT.

  • to test a sequence of messages sent and received between Holochain nodes, between 50 to 200 millis is required for message arrival.

    • role1.json >> "send message, Time = 0"

    • role2.json >> "check for message, Time = 100"

  • the 50 to 200 millis number works well within the test harness network. Tests crossing external networks may require much more time for messages to be delivered.

TODO Note, it might make sense to implement both an automatic backnforth test generator, and also an asynchronous method of testing for the test message arrival (e.g. a message id)