Table of Content

Distribution Files

Before you start creating a distribution file, you should read the writing recipes tutorial in order to catch up on the general tool chain concept and tool chain pipeline. If you did that already, you can start creating distribution files based on this tutorial that will cover the basics.

CITK Developer Toolchain

As mentioned in the recipe tutorial, a distribution file is a composite of multiple recipes. A distribution file always describes a specific system version. In the following, we will discuss an exemplary distribution file and its fields. Remember, the build-generator creates build jobs based on a distribution file and the incorporated recipes. Besides the build jobs for the recipes, the generator creates 3 additional jobs: a) the build-flow-job, which orchestrates the build order of all jobs that belong to a distribution, and b) a prepare and finish job which can be used to, e.g., initially create folders or set permission of the freshly installed distribution in the file system after the actual install step. The build-generator is the subject of the Using the Build-Generator tutorial.

The exemplary distribution file is named nao-minimal-nightly.distribution. The naming convention for distribution files is $name-$version.distribution. Variables in distribution files can be accessed using ${variable.name}.


{
  "catalog": {
    "title": "NAO minimal",
    "description": "opensource://nao-minimal-nightly.md",
    "image": [
        "catalog://nao-minimal.png"
    ],
    "version": "nightly"
  },
  "name": "nao-minimal-nightly",
  "variables": {
    "toolkit.volume": "$HOME/citk/systems/",
    "toolkit.dir": "${toolkit.volume}/${distribution-name}",
    "recipe.maintainer": [ "mwiechmann@techfak.uni-bielefeld.de", "flier@techfak.uni-bielefeld.de" ],
    "buildflow-name": "distribution-buildflow-${distribution-name}",
    "prepare-hook-name": "distribution-prepare-${distribution-name}",
    "prepare-hook/unix": "mkdir -p \"${toolkit.dir}/opt\"; chmod 2755 \"${toolkit.dir}\"",
    "finish-hook-name": "distribution-finish-${distribution-name}",
    "finish-hook/unix": "find \"${toolkit.dir}\" -type d -exec chmod 2755 {} \\;"
  },
  "platform-requires": {
    "ubuntu": {
      "xenial": {
        "packages": [
          "cmake",
          "build-essential",
          "libgstreamer-plugins-base0.10-0",
          "python2.7",
          "xdotool"
        ]
      }
    }
  },
  "versions": [
    [
      "aldebaran-choregraphe",
      "2.1.4.13"
    ],
    [
      "libicu",
      "release-52-1"
    ],
    [
      "libhdf5",
      "1.8.13"
    ],
    [
      "aldebaran-naoqi-sdk-cpp",
      "2.1.4.13"
    ],
    [
      "aldebaran-naoqi-sdk-python27",
      "2.1.4.13"
    ],
    [
      "fsmt",
      "0.19"
    ],
    [
      "pyscxml",
      "v.0.8.5-fsmt"
    ],
    [
      "fsmt-exp-nao-minimal",
      "master"
    ],
    [
      "runnable-nao-minimal",
      "master"
    ]
  ]
}


Step #1: Catalog Block

First of all let's have a look at the catalog block. The catalog block is mandatory and contains information about the representation in the CITK web catalog (this one!).

{
  "catalog": {
    "title": "NAO minimal",
    "description": "opensource://nao-minimal-nightly.md",
    "image": [
        "catalog://nao-minimal.png"
    ],
    "version": "nightly"
  },

"title": Since distribution files are also (like the included recipes) transformed and synced to this catalog in order to provide a human readable and browsable representation, you can choose a "fancy" title for your system which is more self-explanatory than the "name" field.

"description": This field contains an uri to a system description which should provide detailed information about the system. You can make use of the uri pattern as depicted in the example above, or use a FQD (e.g. a readme hosted on github). Here, the readme is located in the CITK git repository itself. Descriptions reside in assets/distribution-descriptions. The pattern opensource:// indicates that a description located in the CITK reprository is used. Descriptions must be written in the [m]ark[d]own syntax.

"image": This is an array of images used to depict the system such as, screenshots, logos, etc. Similar to the description field, images that are already available (re-use!) in the catalog via Media Sets can be referenced using catalog://. Here, you just need to provide the filename, e.g. "image": [ "catalog://nao-minimal.png" ] . Multiple images are referenced like the following: "image": [ "catalog://nao-minimal.png", "catalog://nao-minimal_1.png" ] . You can also use regular URLs. If so, please refrain from violating copyrights. Please also note that 3rd party images may vanish over time.

"version": Since the actual version of a distribution file cannot be derived from the filename or any other metric, you need to explicitly assign a value to this field. The title in the catalog is generated based on the title and the version fields. The catalog title of this distribution will subsequently become: NAO minimal-nightly

Step #2: Variables

Values inside the variables block define how directory structures of an installation are organized and what is done before and after the installation process.

"variables": {
    "toolkit.volume": "$HOME/citk/systems/",
    "toolkit.dir": "${toolkit.volume}/${distribution-name}",
    "recipe.maintainer": [ "mwiechmann@techfak.uni-bielefeld.de", "flier@techfak.uni-bielefeld.de" ],
    "buildflow-name": "distribution-buildflow-${distribution-name}",
    "prepare-hook-name": "distribution-prepare-${distribution-name}",
    "prepare-hook/unix": "mkdir -p \"${toolkit.dir}/opt\"; chmod 2755 \"${toolkit.dir}\"",
    "finish-hook-name": "distribution-finish-${distribution-name}",
    "finish-hook/unix": "find \"${toolkit.dir}\" -type d -exec chmod 2755 {} \\;"
  },

"toolkit.volume": This variable defines the top level installation folder. In this example it is $HOME/citk/systems/ which is the CITK default. This value can be overwritten in the build-generator call using -D toolkit.volume=/some/path.

"toolkit.dir": This variable defines the absolute installation prefix or path of your system. The path is generated based on: ${toolkit.volume}/${distribution-name}. So in this example the entire system will be installed in /tmp/nao-minimal-nightly

"recipe.maintainer": This is an array of developers who are mainly responsible for the system, in case there are questions or requests/suggestions for a change in the distribution file.

"buildflow-name": The name of the build job that is responsible for the correct build order of all jobs. In this case the name of the job will be distribution-buildflow-nao-minimal-nightly. The same logic applies to prepare-hook-name and finish-hook-name.

"prepare-hook/unix" and "finish-hook/unix": As mentioned earlier, before the actual installation step unix command can be issued in order to prepare the environment, e.g., creating folder structures. Analogously, the finish-hook can be used to set permissions on the installation. The prepare-hook is called before the first artifact is installed, the finish-hook is called after the last artifact has been installed.

Step #3: Platform Requires

"platform-requires": {
    "ubuntu": {
      "xenial": {
        "packages": [
          "cmake",
          "build-essential",
          "libgstreamer-plugins-base0.10-0",
          "python2.7",
          "xdotool"
        ]
      }
    }
  },

"platform-requires": This field is intended to list your platform dependencies. In this case for Ubuntu 16.04 Xenial Xerus. In general, the dependencies are collected and aggregated (by the build-generator) in recipes, since they define the functional building blocks of your system. However, sometimes there are additional dependencies that are required by the system, but are not required by any of the included recipe. It seems more convenient to collect all dependencies in the distribution file because this means you only need to edit one file. This is absolutely not recommended since it renders re-use of dependency information useless. If a dependency originates in an included recipe, it must be defined inside the recipe and not in the distribution file.

Step #4: Versions

Versions are the essential part of a distribution. Version entries reference artifacts (recipes) that belong to a system version. The syntax is [ $recipe.name, $version ]. In this example the software component (recipe) aldebaran-choregraphe is required in version 2.1.4.13. This implies that the corresponding recipe aldebaran-choregraphe.project defines the version 2.1.4.13 as either a branch or as a specific version section in the recipe (cf. writing recipes tutorial)

"versions": [
    [
      "aldebaran-choregraphe",
      "2.1.4.13"
    ],
    [
      "libicu",
      "release-52-1"
    ]
}