The Build Generator

In this tutorial, the build-generator usage will be explained. In order to fully understand the purpose of the build-generator, please refer to Writing Recipes and Writing Distributions or even CITK Vision and Concept. In short, the build generator a) parses a distribution file and the included recipes, b) clones the corresponding source code repositories, c) analyses the source code repositories in order to obtain, e.g., meta information and a dependency graph and d) creates build jobs on a CI server based-on a) - c). In the following we will explain the most basic options for the build-generator. For now, the build-generator is distributed within the jenkins.tar.gz.

CITK Developer Toolchain

$HOME/citk/jenkins/job-configurator --on-error=continue -d $HOME/citk/dist/distributions/taverna-workbench-2.5.distribution -m toolkit -D toolkit.volume=$HOME/citk/systems -u $USER -p $PASS

"--on-error=continue": Abort when encountering errors? Either "abort" or "continue".

"-d": Full path to the distribution file (system) you want to install/replicate.
You might use variables here, e.g., -d $HOME/dev/citk/distributions/taverna-workbench-2.5.distribution

"-m": Load templates. Usually, you want to use the toolkit templates. The toolkit templates are designed to build and install the desired system on your machine. There are, however, additional templates providing different functionalities. Another possibility would be ci and ci-deploy templates.

  • The ci templates are designed to also invoke unit-test or code assessment metrics like loc and sloccount — if available. Furthermore, ci template-based jobs will be triggered automatically when a change in the source code repository is detected. Here, up- and downstream jobs will be triggered. This kind of template reflects the original CI paradigm.

  • The ci-deploy templates inherit this behavior, but additionally the install step in each job is invoked.

"-D": Overwrite a variable after loading the distribution. Arguments to this option have to be of the form VARIABLE-NAME=VALUE. This option can be supplied multiple times. As an example the variable $toolkit.volume is pre-defined in the distribution file and points to the default installation prefix that is "/tmp" in this case. If you want to install the target system in a different prefix provide, e.g., -D toolkit.volume=$HOME when invoking the build-generator.

"-u" and "-p": Provide your jenkins username and password in order to create jobs. You can also use -a and provide your Jenkins API token.

Useful Variables

"-b": Jenkins base URI. If you need to create build-jobs on a remote Jenkins, e.g., one that is running in another network, just provide the correct URI.

"--cache-directory": The build generator is capable of creating a cache for repository mirrors. By using this argument, subsequent generator runs will be accelerated since the target repositories are not cloned again. Example: --cache-directory=/tmp/generator-cache

"--trace-variable": Trace usage of the given variable. Example --trace-variable=ROS

"--report-directory": Write analysis results into a log directory. This directory contains two interesting files: DISTRIBUTION.json: a fully merged json file comprising all evaluated variables and all projects definitions and dependency graph of projects within the distribution. Inspect with: xdot Example: --report-directory=/tmp

"--build-flow-fail": If you provide this argument, the build-flow job (that orchestrates all other build jobs) fails if one of the coordinated jobs fails. This is recommended if you want to strictly check if your systems is building correctly.

"-D global.O": Sets C/CXX flags globally. In this case the level of compiler optimization. The default is -D global.O=2

"-D global.march": Sets the instruction set the compiler can use. The default is -D global.march=core2

"-D make.threads": Sets the number of threads for jobs using a C/C++ compiler and make. The default is -j 1. Be careful with this option since it may easily eat up all your CPU cycles and system memory when building massively in parallel. As a rule of thumb calculate this option as follows: (AVAILABLE_CPU_CORES / (NUMBER_OF_JENKINS_EXECUTORS * MAKE_THREADS)) >= 2. As an example, on an 8-core machine you should not choose a make.threads value bigger than 2: (8 / (2*2)) = 2. The default value for number of Executors of our Jenkins is 2. Besides that you should monitor the amount of RAM which is consumed by your Jenkins and the running build jobs. -D make.threads=2

Updating The Build Generator

In case you haven't updated your Jenkins instance or the included build generator for a long time you might get errors like:

The template requires generator version x.x.x, but this generator is version x.x.x

In order to update your generator, go to: and choose your platform. Download the 'build-generator' binary file.

Next, go to your jenkins folder, the default is $HOME/citk/jenkins. In this folder, replace the existing build-generator binary with the newly downloaded file. Also, do not forget to chmod +x the new file.