2.3. Experiment creation — `experiments`

The goal of the experiments module is to define a set of experiments and create a self-contained directory with all the files necessary for running these experiments. By making the directory self-contained, it can be moved into the system(s) where the experiments must be run (e.g., a cluster, or some machine different from the development one).

This guide will show how to generate experiments to evaluate all benchmarks on a simple benchmark suite (mybenchsuite), where each benchmark is run with different input sets, and benchmarks run inside a computer simulator program (mysim) that uses different a per-experiment configuration file.

Each experiment will thus be defined by the benchmark name, the benchmark’s input set, and the different configuration parameter permutations defined by the user. The initial file organization is the following:

.
|- generate.py       # the experiment-generation script described here
|- mysimulator       # source code for the simulator
|  |- Makefile
|  `- mysim.c
|- mysim.cfg.in      # template configuration file for the simulator
`- mybenchsuite      # benchmark suite
   |- 1.foo          # source code for a specific benchmark
   |  |- source.c
   |  `- Makefile
   |- 2.broken
   |  |- source.c
   |  `- Makefile
   |- 3.baz
   |  |- source.c
   |  `- Makefile
   |- README         # files that can be ignored
   `- NEWS

This is the roadmap to create an experiments directory that will contain all the necessary pieces to run our experiments:

Execute external programs to compile the simulator and the benchmarks.
Copy files for the simulator and each of the selected benchmarks into the experiments directory.
Define different sets of arguments to run the benchmarks with different inputs.
Define different configuration parameter combinations for the simulator.
Generate a simulator configuration file for each set of simulation parameters, and generate a script for each combination of simulator parameters, benchmark and benchmark input sets. These files are generated from templates that must be “translated” by the parameters of each experiment.

This example also showcases some of the more advanced features of experiments, but you can first take a look at Quick example for a much shorter and simpler example.

2.3.1. Script preparation

All the functions typically used in a experiments script are available in the sciexp2.expdef.env module, so we can import its contents to make them available at the top level:

#!/usr/bin/env python
# -*- python -*-

from sciexp2.expdef.env import *

# file contents ...

2.3.2. Directory preparation

First, we create a Experiments object with its output directory set, where all generated files will be placed:

e = Experiments(out="./experiments")

This object is initially empty, and only has the output directory set:

>>> e
Experiments(out='./experiments')

Note

It is usually recommended to not remove the output directory when re-executing a experiments script, since files will be copied and generated only if their contents have changed. This is later detected by the launcher program to re-run only those experiments whose scripts or configuration files have been updated (e.g., generated with new contents since last run).

2.3.3. Compile and copy the simulator

We will use execute to execute make from the current directory, and then use pack to copy the resulting binary into the output experiments directory:

e.execute("make", "-C", "./mysimulator")
# copied into 'experiments/bin/mysim'
e.pack("./mysimulator/mysim", "bin/mysim")

2.3.4. Find, compile and copy benchmarks

Instead of hard-coding our list of benchmarks, we will dynamically detect them, and drop the one that is named broken:

finder = from_find_files("./mybenchsuite/[0-9]*\.{{benchmark}}/", path="FILE"))
e.params(BENCH_DIR=finder.param("FILE"),
         benchmark=finder.param("benchmark"))
e = Experiments(e.view("benchmark != 'broken'"))

First, we create our file finder with from_find_files to find all benchmark directories (unsurprisingly, it finds both files and directories). Then we use params to add the path to the benchmark directories (BENCH_DIR) and the benchmark name (benchmark) into the parameters of our experiment set. After doing this, we create a new experiment set that does not contain the broken benchmark using view.

Note

The trailing slash in the file name template prevents matching the README and NEWS files under the mybenchsuite directory.

The result is that our experiment set now contains one element for each of the benchmarks we found:

>>> e
Experiments([{'benchmark': 'foo', 'BENCH_DIR': './mybenchsuite/1.foo/'},
             {'benchmark': 'baz', 'BENCH_DIR': './mybenchsuite/3.baz/'}],
            out='./experiments')

Then, we call make into each of the selected benchmark directories, and copy the resulting binaries into the output directory:

# results in executing the following commands:
#   make -C ./mybenchsuite/1.foo/
#   make -C ./mybenchsuite/3.baz/
e.execute("make", "-C", "{{BENCH_DIR}}")

# results in the following copies:
#   ./mybenchsuite/1.foo/foo -> ./experiments/benchmarks/foo
#   ./mybenchsuite/3.baz/baz -> ./experiments/benchmarks/baz
e.pack("{{FILE}}/{{benchmark}}", "benchmarks/{{benchmark}}")

Both command execution and file copying use templated arguments, so that the parameters on each experiment can be used to get a per-experiment string. This results in executing make on each of the benchmark directories (since the command only references the {{BENCH_DIR}} parameter), and copying each of the per-benchmark binaries we just compiled.

2.3.5. Define experiment parameters

Defining the experiment parameters is one of the heavy-weight operations, which is encapsulated in params. First of all, we want each benchmark to execute with different arguments, which are benchmark specific.

Let’s start with the simpler foo benchmark, which has two possible input values (small or big). For that, we use view to get the sub-set of experiments for that benchmark, and define their inputset and args parameter by applying params on that sub-set:

with e.view("benchmark == 'foo'") as v:
    v.params(inputset="{{args}}",
             args=["small", "big"])

If we look at our experiments, they now have the parameters we just defined with params only on the sub-set of experiments we got from view:

>>> e
Experiments([{'benchmark': 'foo', 'inputset': '{{args}}', 'args': 'small', 'BENCH_DIR': './mybenchsuite/1.foo/'},
             {'benchmark': 'foo', 'inputset': '{{args}}', 'args': 'big', 'BENCH_DIR': './mybenchsuite/1.foo/'},
             {'benchmark': 'baz', 'BENCH_DIR': './mybenchsuite/3.baz/'}],
            out='./experiments')

The baz benchmark example is a bit more involved, since it has three input arguments (arg1, arg2 and arg3). The first two take any value in the 2-element range starting at zero, and the third takes the base-two logarithm of the sum of the first two arguments:

import math
def fun(exp):
    return math.log(exp["arg1"] + exp["arg2"], 2)
with e.view("benchmark == 'baz'") as v:
    v.params("arg1 != 0 or arg2 != 0",
             inputset="{{arg1}}{{arg2}}",
             args="{{arg1}} {{arg2}} {{arg3}}",
             arg1=range(2),
             arg2=range(2),
             arg3=with_exp(fun))

In this case, we define the argument list that we will later use to run the benchmark as a string with the benchmark arguments (args). Since we must define the value of the third argument as a function of the first two, we will lazily calculate it with with_exp when necessary. Note that in this case, params also has a filter to avoid having the first two arguments both at zero, since the logarithm is infinite.

Note that in all benchmarks we generate the inputset variable, which will help us to uniquely identify each of the benchmark’s input sets (many benchmark suites already have unique input set names).

Finally, we also need to define the parameters we will use with our computer simulator (cores, l1, l2, l1_assoc, and l2_assoc), together with filtering-out some configurations that the simulator does not support. Again, this will take each of the benchmark configurations and “extend” each of them with each of the simulator parameter combinations.:

e.params("l1 <= v_.l2",
         "l1_assoc <= v_.l2_assoc",
         cores=range(1, 5),
         l1=[2**x for x in range(1,  6)], # size in KB
         l2=[2**x for x in range(1, 10)],
         l1_assoc=[1, 2, 4],
         l2_assoc=[1, 2, 4, 8])

Note

Using Python’s with statement with view is not mandatory, but can improve code readability in these cases. The canonical way to use it instead would be to treat its result as a regular object:

v = e.view(...)
v.params(...)

More generally, params accepts values that can be either, immediate values (e.g., an integer or string), value sequences (e.g., a list), or the result of the helper functions with_* and from_* in experiments. As a result, the experiment set will contain the cartesian product of the original contents and the permutations of the newly defined parameters. If this is not desired, you can use different Experiments objects, or can use the params_append method to append new entries instead of recombining them with the existing contents.

2.3.6. Generate simulator configuration files

The contents of an experiment set can be used to generate files from an input template, by substituting variable references with the specific values on each instance. In this example, we have a template simulator configuration file in mysim.cfg.in with the following contents:

cores = {{cores}}
l1_size  = {{l1}}         # KBytes
l1_assoc = {{l1_assoc}}
l2_size  = {{l2}}         # KBytes
l2_assoc = {{l2_assoc}}

With generate, we can create a new configuration file from our template ("conf/{{cores}}-{{l1}}-{{l1_assoc}}-{{l2}}-{{l2_assoc}}.cfg") for each parameter combination we defined above:

l.generate("mysim.cfg.in", "conf/{{cores}}-{{l1}}-{{l1_assoc}}-{{l2}}-{{l2_assoc}}.cfg")

What generate does is, for each per-experiment expansion of the second argument, take the file in the first argument (which could also be a template), and use the experiment corresponding to that expansion “translate” the file contents (the input file is, in fact, treated as a string whose contents are then translated).

Warning

For each possible simulation parameter combination, there exist multiple benchmark/argument combinations. That is, there are multiple experiments with the same configuration file. When such things happen, the output file will only be generated once with the first matching experiment, and subsequent experiments will simply log a message that a repeated file has been produced.

2.3.7. Generate an execution script for each experiment

The final step is to generate some scripts to actually run our experiments with all the selected benchmark, inputs, and simulation parameter combinations. We could simply use generate, but generate_jobs is an extension of it that already has some pre-defined job templates, and produces some extra metadata to manage experiment execution with the launcher program. We first have to decide which pre-defined template to use; all of them can be seen with launcher --list-templates. With that, we can now use launcher --show-template to inspect the template and see what parameters we need to define for it to work.

In this example we will use the shell template. Looking at the output of launcher --show-template shell we can see that we only need to defined the CMD parameter, which contains the actual command-line that will execute our experiment. Therefore, this will produce our experiment scripts:

e.params(# save some typing by defining these once and for all
         ID="{{benchmark}}-{{inputset}}-{{SIMID}}",
         SIMID="{{cores}}-{{l1}}-{{l1_assoc}}-{{l2}}-{{l2_assoc}}",

         CMD="""
# Python multi-line strings are handy to write commands in multiple lines
./bin/mysim -config conf/{{SIMID}}.cfg -output {{DONE}} -bench ./benchmarks/{{benchmark}} {{args}}
""")
e.generate_jobs("shell", "jobs/{{ID}}.sh")

We first define the parameters we need for the job script (CMD), and additional variables used by the former ones (ID and SIMID, which are used to save some typing). The CMD variable contains the command-line to run the simulator with the specified configuration file, as well as a specific benchmark along with its arguments. It also instructs the simulator to save its output in the value of the DONE variable. Note that DONE (and FAIL) is defined by the shell template, but has a default value (we can override it with params).

Finally, this also generates the file jobs.jd in the output directory. The launcher program will use this file to detect the available experiments, and will use the values of the DONE and FAIL variables to known which experiments have already been run and, if so, which of these failed.

Note

You should take a look at generate_jobs for additional features, like exporting experiment parameters so that they can be used to filter which jobs we want to manipulate with the launcher program (e.g., run a sub-set of the experiments depending on their configuration parameters).

There also are options to specify the dependencies of each job, so that the launcher program can automatically detect when jobs are outdated (e.g., when generating new configuration files for some experiments).

2.3.8. Writing new templates

The pre-defined templates might not cover your needs, but you can override the contents of an existing template by creating a file with the same name as the template (e.g., for the previous example, create shell.tpl in the same directory where you have generate.py).

For even greater flexibility, you can also extend the set of available templates by creating the appropriate files, which can reside in any of the directories listed in SEARCH_PATH, which by default includes the current directory.

2.3.9. Wrap-up

To wrap things up, here’s the contents of the generate.py file covering the whole example:

#!/usr/bin/env python
# -*- python -*-

import math

from sciexp2.expdef.env import *

e = Experiments(out="./experiments")

# compile & copy simulator
e.execute("make", "-C", "./mysimulator")
e.pack("./mysimulator/mysim", "bin/mysim")

# find & compile & copy benchmarks
finder = from_find_files("./mybenchsuite/[0-9]*\.{{benchmark}}/", path="FILE"))
e.params(BENCH_DIR=finder.param("FILE"),
         benchmark=finder.param("benchmark"))
e = Experiments(e.view("benchmark != 'broken'"))
e.execute("make", "-C", "{{BENCH_DIR}}")
e.pack("{{FILE}}/{{benchmark}}", "benchmarks/{{benchmark}}")

# benchmark parameters
with e.view("benchmark == 'foo'") as v:
    v.params(inputset="{{args}}",
             args=["small", "big"])
import math
def fun(exp):
    return math.log(exp["arg1"] + exp["arg2"], 2)
with e.view("benchmark == 'baz'") as v:
    v.params("arg1 != 0 or arg2 != 0",
             inputset="{{arg1}}{{arg2}}",
             args="{{arg1}} {{arg2}} {{arg3}}",
             arg1=range(2),
             arg2=range(2),
             arg3=with_exp(fun))

# simulation parameters
e.params("l1 <= v_.l2",
         "l1_assoc <= v_.l2_assoc",
         cores=range(1, 5),
         l1=[2**x for x in range(1,  6)], # size in KB
         l2=[2**x for x in range(1, 10)],
         l1_assoc=[1, 2, 4],
         l2_assoc=[1, 2, 4, 8])

# simulator config file
l.generate("mysim.cfg.in", "conf/{{cores}}-{{l1}}-{{l1_assoc}}-{{l2}}-{{l2_assoc}}.cfg")

# generate execution scripts
e.params(# save some typing by defining these once and for all
         ID="{{benchmark}}-{{inputset}}-{{SIMID}}",
         SIMID="{{cores}}-{{l1}}-{{l1_assoc}}-{{l2}}-{{l2_assoc}}",

         CMD="""
# Python multi-line strings are handy to write commands in multiple lines
./bin/mysim -config conf/{{SIMID}}.cfg -output {{DONE}} -bench ./benchmarks/{{benchmark}} {{args}}
""")
e.generate_jobs("shell", "jobs/{{ID}}.sh")

Although this might look unnecessarily long, the ability to concisely specify parameter permutations and apply filters on them can keep large parameter explorations under control. If you couple that with the ability to track the execution state of experiments with the launcher program, that becomes even more convenient.

2.3. Experiment creation — experiments

2.3.1. Script preparation

2.3.2. Directory preparation

2.3.3. Compile and copy the simulator

2.3.4. Find, compile and copy benchmarks

2.3.5. Define experiment parameters

2.3.6. Generate simulator configuration files

2.3.7. Generate an execution script for each experiment

2.3.8. Writing new templates

2.3.9. Wrap-up

2.3. Experiment creation — `experiments`