Table of Content

Step #1: General Concept

This tutorial will provide a high-level overview of how to write CITK recipes. A more technical and detailed README resides in the CITK git repository. The examples presented on this page will "only" cover the very basics. If you need more detailed information please refer to the README. In case your questions are not answered in either one of the two sources, please subscribe to: citk-user [AT] lists.cit-ec.uni-bielefeld DOT de

If you did not read the Vision and Concept introduction, here's a short primer on recipes and the corresponding tool chain.

CITK Developer Toolchain

In general, we assume that researchers make use of multiple artifacts in their daily working routine. These artifacts are for instance source code, binary executables or libraries, data sets, configuration or launch files, and many more. Since these kind of artifacts are extremely heterogeneous, due to their nature (binary vs. source code for instance) and thus are usually handled and deployed differently, we provide a template based recipe solution in order to describe them.

Recipes are a textual (JSON) description that contains, e.g., meta information about the artifact, info about required dependencies, and info about how the artifact is deployed (e.g. the build system). Moreover, recipes contain information about available versions of an artifact. As an example, a software repository can have multiple branches and tags. Usually, you will write one recipe per artifact.

Moreover, multiple recipes can be combined into a so-called distribution. A distribution, like a recipe, is also a text-based JSON file that contains meta information as well as references to recipes and the desired version of the recipe, the artifact respectively. Distribution files are read by a build-generator that creates build jobs on a CI server for each and every recipe that is included in a distribution file. Besides jobs for the artifacts, an orchestration job (build-flow job) is created which, once it has been triggered, coordinates the build order of the jobs respecting the global dependency graph. The build-generator is the subject of the Using the Build-Generator tutorial. Distributions are explained in the Writing Distributions Tutorial

Step #2: Writing a Recipe

As mentioned earlier, recipes are template based. This means we provide templates for various build systems, such as CMake, Maven, Setuptools, Autotools, etc. Additionally, we also provide a 'freestyle' template where you can basically execute any arbitrary command that is required in order to deploy your artifact.

Variables that are provided by the templates can be accessed using ${variable.name}. The most interesting one is probably ${toolkit.dir} which contains the actual installation prefix of the target system (see Step #7: The Freestyle Template).

In the following we will explain a recipe step-by-step. We will start with an exemplary C++ software component. The name of the file is example-cpp-recipe.project.


{
  "catalog": {
    "title": "Catalog Title of the Component",
    "description": "opensource://default.md", 
    "image": [ "catalog://system_image_default.jpg" ]
  },
  "name": "example-cpp-recipe", 
  "templates": [ "cmake-cpp", "base" ], 
  "variables": {
     "access": "private", 
     "branches": [ "master",  "1.1", "2.0"  ], 
     "description": "This is a short textual description of the software component.", 
     "extra-provides": [  [ "cmake-cpp", "example-cpp-recipe", "1.1" ],  [ "cmake-cpp",  "example-cpp-recipe", "2.0 ]  ], 
     "extra-requires": [  [ "pkg-config", "xmltio", "1.3" ]  ], 
     "repository": "https://my.scm/git/btl.git", 
     "scm.credentials": "my.scm",
     "platform-requires": {
       "ubuntu": { "trusty": { "packages": [ "libgstreamer-plugins-base0.10-0", "libc6-dev" ] } }
     },
    "recipe.maintainer": [ "avalid@email", "availd@email" ]
  },
  "versions": [
    {
      "name": "experiment-0.1",
      "variables": {
        "access": "private",
        "description": "This is a special version which has also been moved to another repository",
        "commit": "df7a7cde23c922a299224ceac23fcfc4dbb6d919",
        "repository": "https://my-other-repro.scm/git/btl.git",
        "scm.credentials": "my-other-repro.scm"
      }
    }
  ]
}

Step #3: Catalog Block

First of all let's have a look at the catalog block. The catalog block is mandatory and contains information about the representation in the CITK web catalog (this one!).

{
  "catalog": {
    "title": "Catalog Title of the Component",
    "description": "opensource://default.md", 
    "image": [ "catalog://system_image_default.jpg" ]
},

"title": Since recipes are transformed and synced to this catalog in order to provide a human readable and browsable representation, you can choose a "fancy" title for your software component which is more self-explanatory than the recipe "name" field.

"description": This field contains an uri to a component description which should provide detailed information about the software (in contrast to the top level description field which is intended to provide short info). You can make use of the uri pattern as depicted in the example above, or use a FQD (e.g. a readme hosted on github). Here, the readme is located in the CITK git repository itself. Descriptions reside in assets/project-descriptions. The pattern opensource:// indicates that a description located in the CITK reprository is used. Descriptions must be written in the [m]ark[d]own syntax.

"image": This is an array of images used to depict the software component such as, screenshots, logos, etc. Similar to the description field, images that are already available (re-use!) in the catalog via Media Sets can be referenced using catalog://. Here, you just need to provide the filename, e.g. "image": [ "catalog://nao-minimal.png" ] . Multiple images are referenced like the following: "image": [ "catalog://nao-minimal.png", "catalog://nao-minimal_1.png" ] . You can also use regular URLs. If so, please refrain from violating copyrights. Please also note that 3rd party images may vanish over time.

Step #4: Name and Templates

Now let's have a look at name and templates

  "name": "example-cpp-recipe", 
  "templates": [  "cmake-cpp", "base"  ],

"name": This is simple, must be the same as the filename without the *.suffix. my-example-recipe.project will become: "name": "my-example-recipe"

"templates": As mentioned earlier recipes are template based. In order to make the integration of different build-systems simple and to hide the inherent complexity (of some), you just need to reference the template suitable for your build system. As a rule of thumb you can use the name of your build system, e.g., setuptools, autotools, cmake-cpp, cmake-catkin, make, maven, etc... The templates are used to apply a set of CITK default values, e.g., global environment and build variables, build flags, directory structures and so on. The minimal tuple (as depicted above) is the used build tool and the base template. An exhaustive list of templates can be found in the CITK repository. Below is a list of the most popular ones.


Java Maven

"templates": [
    "maven", 
    "base"
  ],

Python Setuptools/Distutils

"templates": [
    "setuptools", 
    "base"
  ],

CMake

"templates": [
    "cmake-cpp", 
    "base"
  ],

ROS wstool

"templates": [
    "wstool", 
    "base"
  ],

ROS Catkin

"templates": [
    "cmake-catkin", 
    "base"
  ],

Freestyle

"templates": [
    "freestyle", 
    "base"
  ],

The GNU Build System, also known as the Autotools

"templates": [
    "autotools", 
    "base"
  ],


Step #5: Variables

The upcoming topic is variables. The variables section describes a set of values that are essential for this (software) artifact.

  "variables": {
     "access": "private", 
     "branches": [ "master",  "1.1", "2.0"  ], 
     "description": "This is a short textual description of the software component.", 
     "extra-provides": [  [ "cmake-cpp", "example-cpp-recipe", "1.1" ],  [ "cmake-cpp",  "example-cpp-recipe", "2.0 ]  ], 
     "extra-requires": [  [ "pkg-config", "xmltio", "1.3" ]  ], 
     "repository": "https://my.scm/git/btl.git", 
     "scm.credentials": "my.scm",
     "platform-requires": {
       "ubuntu": { "trusty": { "packages": [ "libgstreamer-plugins-base0.10-0", "libc6-dev" ] } }
     },
    "recipe.maintainer": [ "avalid@email", "availd@email" ]
  }

"access": If access (read) is allowed without providing credentials the value for this field should be "public", otherwise the field is "private"

"branches": A list of branches that are availble in the source code repository.

"description": A short and meaningful description of the artifact. A more detailed version can be referenced in the catalog block (see above).

"extra-provides" and "extra-requires": In general, the build-job generator calculates the dependency graph of all recipes included in a distribution. If a dependency of an artifact cannot be derived from the build system, e.g., dependend components are not using the same build system or you want to model a runtime depencency you need to define "extra-requires" and "extra-provides" accordingly. The suggested syntax is as follows: "extra-provides": [ [ "$template", "$name", "$version" ]. This pattern must be referenced in the recipe "extra requiring" the "extra provider". Example: recipe_a defines "extra-provides": [ [ "maven", "slow_java", "1.0" ] ]. recipe_b has a runtime dependency on recipe_a, so recipe_b would define "extra-requires": [ [ "maven", "slow_java", "1.0" ] ]

"repository": Here you need to provide the URL of your source code repository (git, svn, etc.).

"scm.credentials": You only need to provide this field if access to your code is protected by a password. The suggested usage pattern for this field is to use the Top Level Domain of your repository without the protocol, e.g. http(s). If your code is open source for example, you can mark it as public and remove this field.

"platform-requires": This field is intended to list your platform dependencies. In this case for Ubuntu 14.04 Trusty Tahr. Please collect and reference all ubuntu packages that are required in order to build or run your software component.

"recipe.maintainer": This is an array of developers who are mainly responsible for the recipe, in case there are questions or requests/suggestions for a change in the recipe.

Step #6: Versions

Versions are a special part of the recipe. In a version, you can basically redefine every aspect of the recipe if that's required for a specific release version of your software. As an example, your software has been cloned in another repository and has been modified for a special purpose. This version is named "experiment-0.1" and contains different values for, e.g., the repository url and a special commit hash which is used to build the software. In a distribution, you can reference recipe versions as well as branches using the respective identifier. Versions are not mandatory, but are strongly recommended if you need to freeze or document a (stable) release.

 "versions": [
    {
      "name": "experiment-0.1",
      "variables": {
        "access": "private",
        "description": "This is a special version which has also been moved to another repository" 
        "commit": "df7a7cde23c922a299224ceac23fcfc4dbb6d919",
        "repository": "https://my-other-repro.scm/git/btl.git",
        "scm.credentials": "my-other-repro.scm"
      }
    }
]

Step #7: The Freestyle Template

The freestyle template is a special recipe template due to the fact that you can define any arbitrary command "shell.command": "ls -la" in order to download a data set for instance. An example can be found below.


{
  "catalog": {
    "description": "opensource://default.md", 
    "image": [
      "catalog://system_image_default.jpg"
    ], 
    "title": "Data set for DLIB face pose estimation"
  }, 
  "name": "dataset-dlib",
  "templates": [
    "freestyle", 
    "base"
  ], 
  "variables": {
    "platform-requires": {
      "ubuntu": {
        "packages": [
          "bzip2"
        ]
      }
    },
    "description": "Downloads and deploys the DLIB face pose estimation dat file",
    "keywords": [
      "data set", 
      "face recognition", 
      "learning", 
      "dlib"
    ],
    "recipe.maintainer": [ "availd@email" ],
    "shell.command": "
            mkdir -p ${toolkit.dir}/share/dlib && \\
            wget -N --progress=dot:mega http://dlib.net/files/shape_predictor_68_face_landmarks.dat.bz2 && \\
            bzcat shape_predictor_68_face_landmarks.dat.bz2 > ${toolkit.dir}/share/dlib/shape_predictor_68_face_landmarks.dat
            "
  },
  "versions": [
     { "name": "dlib.net" }
  ]
}

Step #8: The Runnable Template

Another special template is the runnable template. As the name indicates, runnable templates "run" something. These templates are designed to execute a program or even an entire system. The specialty here is, that these recipes are not included in the build-flow. In other words, as soon as the build-generator has created all build jobs, and the orchestration job is triggered, all jobs except for the runnables are built. Runnables are triggered by the user only (manually). An example can be found below.


{
  "catalog": {
    "title": "Flobi Simulation Integration Tests Runner",
    "description": "opensource://default.md", 
    "image": [
      "catalog://system_image_default.jpg"
    ],
  }, 
  "name": "runnable-csra-flobi-sim",
  "templates": [
   "runnable",
    "base"
  ], 
  "variables": {
    "access": "private",
    "branches": [
      "master"
    ], 
    "description": "FSMT LSP-CSRA Integration Tests for Flobi Simulation Startup", 
    "keywords": [
      "python", 
      "fsmt", 
      "programming language", 
      "testing", 
      "integrations tests"
    ], 
    "recipe.maintainer": [ "availd@email", "availd@email" ],
    "shell.command": "#!/bin/bash
        export DISPLAY=:0.0;
        . ${toolkit.dir}/etc/vdemo_scripts/lsp-csra-base-localhost.sh;
        $prefix/bin/fsmt -p -c -o $WORKSPACE --loglevel=WARNING ${toolkit.dir}/etc/fsmt-experiments/simulation/flobi/flobi-hlc-sim-local.scxml;"
  }
}