• Harness

Harness/Scalebench is a script that tests Barrelfish.

Scalebench

Scalebench is a Python script in tools/harness/scalebench.py in the Barrelfish source repo. It contains a number of tests and benchmarks and can be used by anyone, but needs extra configuration when used outside of Microsoft or ETH. In particular, it needs to know about the configuration of the machines it is going to execute tests on, how to boot these machines and how to read their output. There are a number of QEMU-emulated machines preconfigured, which should work for everyone without further configuration.

Runnning a test

Harness can execute tests on a variety of hardware.

Write a new Scalebench ("harness") test

Create a new python class that inherits from TestCommon in a file in tools/harness/tests and implement the following methods (minimum):

import tests
from common import TestCommon
from results import PassFailResult

# need this decorator for our test to show up in list of available tests
@tests.add_test
class MyTest(TestCommon):
    '''Give a short description of your test here'''

    # this is the name that you will later use to run your test:
    # scalebench.py -t mytest ...
    name = "mytest"

    def get_modules(self, build, machine):
        '''Here you can provide one or more modules (binaries) that should be run for your test'''

        # get default set of modules from our superclass
        modules = super(MyTest, self).get_modules(build, machine)
        # add our test binary, this needs to be buildable as `make mytest`
        modules.add_module("mytest")
        return modules

    def process_data(self, testdir, rawiter):
        '''Here you can process the output of your test to determine
        pass/fail. `rawiter` is a raw iterator over the output of the test
        (e.g. qemu or console). `testdir` is a directory where you can
        store additional processed output, if desired.
        '''

        # iterate over all lines of output
        for line in rawiter:
            if line.startswith("mytest passed"):
                # found the line that `mytest` prints when the test
                # passes, return PASS
                return PassFailResult(True)
            
            # didn't find line of test passing at all, return FAIL
            return PassFailResult(False)