ROS Build Farms

The ROS build farms are an important infrastructure to support the ROS ecosystem, provided and maintained by Open Robotics. They provide building of source and binary packages, continuous integration, testing, and analysis for ROS 1 and ROS 2 packages. There are two hosted instance for open source packages:

  1. https://build.ros.org/ for ROS 1 packages

  2. https://build.ros2.org/ for ROS 2 packages

If you are going to use any of the provided infrastructure please consider signing up for the build farm discussion forum in order to receive notifications, e.g., about any upcoming changes.

Jobs and Deployment

The ROS build farms perform several different jobs. For each job type you will find a detailed description of what they do and how they work:

  • release jobs generate binary packages, e.g., deb packages

  • devel jobs build and test ROS packages within a single repository on a polling basis

  • pull_request jobs build and test ROS packages within a single repository triggered by webhooks

  • CI jobs build and test ROS packages across repositories with the option of using artifacts from other CI jobs to speed up the build

  • doc jobs generate the API documentation of packages and extract information from the manifests

  • miscellaneous jobs perform maintenance tasks and generate informational data to visualize the status of the build farm and its generated artifacts

Creation and Deployment

The above jobs are created and deployed when packages are bloomed, i.e. released for ROS 1 or ROS 2. Once blooming is successful and a package is incorporated in one of the ROS distributions (via pull request to rosdistro), the according jobs will be spawned. The names of the jobs encode their type and purpose: [1]

  • release jobs:

    • {distro}src_{platf}__{package}__{platform}__source build source packages of releases

    • {distro}bin_{platf}__{package}__{platform}__binary build binary packages of releases

    For instance, the binary packaging job of rclcpp on ROS 2 Iron (running on Ubuntu Jammy amd64) is named Ibin_uJ64__rclcpp__ubuntu_jammy_amd64__binary.

  • devel jobs:

    • {distro}dev__{package}__{platform} perform a CI build for the releasing branch

  • pull_request jobs

    • {distro}pr__{package}__{platform} perform a CI build for a pull request

    For instance, the PR job for rclcpp on ROS 2 Iron (running on Ubuntu Jammy amd64) is named Ipr__rclcpp__ubuntu_jammy_amd64.

Execution

Execution of the jobs depends on the type of the job:

  • devel jobs will be triggered every time a commit is done to the respective branch polling based on a configured frequency.

  • pull_request jobs will be triggered by webhooks from respective pull request of the upstream [2] repository

  • release jobs will be triggered once every time a new package version is released, i.e. a new rosdistro pull request was accepted for this package. The source jobs are triggered by a version change in the rosdistro distribution file, the binary jobs are triggered by their source counterpart.

Frequency Asked Questions (FAQ) and Troubleshooting

  1. I get Jenkins mails from failing build farm jobs. What do I do?

    Go to the job that raised the issue. You find the link on top of the Jenkins email. Once you followed the link to the build job, click Console Output on the left, then click Full Log. This will give you the full console output of the failing build. Try to find the top-most error as it is usually the most important and other errors might be follow-ups.

    The bottom of the email might read 'apt-src build [...]' failed. This is usually because of an error building the package. This usually hints at missing dependencies, see 2.

  2. I seem to be missing a dependency, how do I find out which one?

    You basically have two options, a. is easier but may take several iterations, b. is more elaborate and gives you the full insight as well as local debugging.

    1. Inspect the release job that raised the issue (see 1.) and localize the cmake dependency issue. To do so, browse to the cmake section, e.g., navigate to the build binarydeb section through the menu on the left in case of a Ubuntu/Debian build job. The CMake Error will typically hint at a dependency required by the cmake configuration but missing in the package manifest. Once you have fixed the dependency in the manifest, do a new release of your package and wait for feedback from the build farms or…

    2. To get the full insight and faster, local debugging, you can run the release jobs locally. This allows to iterate the manifest locally until all dependencies are fixed.

  3. Why do release jobs fail when devel jobs / my github actions / my local builds succeed?

    There are several potential reasons for this. First, release jobs build against a minimal ROS installation to check if all dependencies are properly declared in the package manifest. Devel jobs / github actions / local builds may be performed in an environment that has the dependencies already installed, therefore does not notice dependency issues. Second, they might build different versions of the source code. While devel jobs / github actions / local builds usually build the latest version from the upstream [2] repository, release jobs build the source code of the latest release, i.e. the source code in the respective upstream branches of the release repository [3].

Further Reading

The following links provide more details and insights into the build farms: