YCM-SUPERBUILD(7)

📦
version 0.13.0 · 2 versions
, in ubuntu noble, ubuntu questing

💖 YCM-SUPERBUILD(7) — YCM 💖

okay bestie, welcome to the YCM Superbuild manual, straight out of the YCM corner of your man pages. this one's a whole saga so we're doing it in three parts — buckle up, hydrate, and let's get into it 🫶

✨ NAME ✨

ycm-superbuild — the YCM Superbuild User Manual. that's it, that's the name, short and slay 💅

🌈 YCM SUPERBUILD 🌈

so. a YCM Superbuild is a CMake project whose entire personality, its whole one job, its main character energy, is: download and build several other projects. that's it. that's the gig. 🎀

here's the vibe: the superbuild checks if a package is already available on your system, and if it's not — if your system is giving nothing — it goes and downloads the source code, builds it, and installs it for you. no cap.

a YCM superbuild supports, out of the box, zero setup required:

  • User mode 👑
  • Developer mode 🧚
  • Demos ✨
  • Automatic integration with CDash. 🔥
  • Automatic documentation generation using doxygen. 📚
  • Generation of dependency graphs using dot. 🕸️

depending on why you're here, skip to the section that matches your situation, bestie:

  • if you literally just want to build the YCM superbuild project with all the sub-projects and start using it, go read the YCM Superbuild Manual for Basic Users.
  • if you want to download, build and modify the source code of one or some of the YCM superbuild sub-projects, go read the YCM Superbuild Manual for Developers.
  • if you want to create a new superbuild or modify an existing one, or you're nosy about the internals of a YCM superbuild (valid), go read the YCM Superbuild Manual for Maintainers.

🌸 YCM SUPERBUILD MANUAL FOR BASIC USERS 🌸

good news, this is the easy part. a YCM superbuild can be built like any other CMake project. like, genuinely. nothing cursed here.

🐧 Linux 🐧

say <YOUR_PROJECT> is the directory where you downloaded the superbuild project you want to build. (it is. trust.) then:

cd <YOUR_PROJECT>

mkdir build
cd build
cmake ..

now, if you run

make

the superbuild will download and install all the required projects that cannot be found on the system. it's giving helpful, honestly.

after the build, all the subprojects will be installed inside the ${YCM_EP_INSTALL_DIR} folder — by default that's ${CMAKE_BINARY_DIR}/install, i.e. build/install — so in order to actually use it you're gonna have to adjust some environment variables:

export PATH=$PATH:${PROJECT_SOURCE_DIR}/build/install/bin/
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:${PROJECT_SOURCE_DIR}/build/install/lib/
export CMAKE_PREFIX_PATH=$CMAKE_PREFIX_PATH:${PROJECT_SOURCE_DIR}/build/install/

you can add these lines to your ~/.bashrc file (just replace ${PROJECT_SOURCE_DIR} with the folder where you downloaded your project) if you don't want to have to execute them manually every single time. save yourself the pain, bestie 💜

in order to compile just one project (and all the projects on which this project depends — they come as a package deal, sorry), you can just run instead:

make <project>

in order to update the external projects, you will have to run

make update-all

or, if you want to update just one project, you can run

make <project>-update

after updating some project, you should then rebuild:

make
okay so important: update alone is not enough. you update, then you rebuild. that's the order. periodt. 💅

🍎 OS X 🍎

on OS X you have the option to generate a GNU Makefile or an Xcode project. choose your fighter.

if you choose to use the Makefile then you can follow the same steps of the Linux installation guide. literally the same. only the environmental variables change, as explained later.

if you choose to generate the Xcode project, you have to follow the following steps:

mkdir build
cd build
cmake .. -G Xcode

now, if you run

xcodebuild

the superbuild will download and install all the required projects that cannot be found on the system. the above command builds the project with the default configuration of Xcode. you can also open the project into the Xcode IDE and build it from there (very GUI of you ✨), or explicitly specify the configuration at command line:

#Debug
xcodebuild -configuration Debug
#Release
xcodebuild -configuration Release

after the build, all the subprojects will be installed inside the ${YCM_EP_INSTALL_DIR} folder — by default ${CMAKE_BINARY_DIR}/install, i.e. build/install — so in order to use it you will have to adjust some environment variables:

export PATH=$PATH:${PROJECT_SOURCE_DIR}/build/install/bin/
export DYLD_LIBRARY_PATH=$DYLD_LIBRARY_PATH:${PROJECT_SOURCE_DIR}/build/install/lib/
export CMAKE_PREFIX_PATH=$CMAKE_PREFIX_PATH:${PROJECT_SOURCE_DIR}/build/install/

you can add these lines to your ~/.bashrc file (or the correct file for your shell — read the room, know your shell) if you don't want to have to execute them manually every time. just replace ${PROJECT_SOURCE_DIR} with the folder where you downloaded your project.

in order to compile just one project (and all the projects on which this project depends) you can just run instead:

xcodebuild -target <project>

in order to update the external projects, you will have to run

xcodebuild -target ALL_UPDATE

or if you want to update just one project, you can run

xcodebuild -target <project>-update

after updating some project, you should then rebuild:

xcodebuild

and if you don't remember the name of the targets (no shame, there's a lot going on) you can type

xcodebuild -list

for a list of the targets in the project. iconic little helper command, honestly 🎀

📝 Todo 📝

Add Xcode screenshot and instructions for using the GUI

🪟 Windows 🪟

📝 Todo 📝

Add Windows documentation

YCM SUPERBUILD MANUAL FOR DEVELOPERS

Okay bestie, if you're a developer, this is your era. A developer is someone who doesn't just wanna build the superbuild — they also wanna get in there and modify some of the subprojects. 💅✨ Periodt.

NOTE: If a package should be built with a YCM Superbuild, this should not be available on the system. If it is, then the source code will not even be downloaded. In order to keep 2 different versions, the superbuild should ignore the system version, and download and build it instead. This can be done by setting the USE_SYSTEM_<PACKAGE> variable to FALSE or by setting the YCM_DISABLE_SYSTEM_PACKAGES variable to TRUE to do it for all the packages of the superbuild.

You can do this by running ccmake or cmake-gui and changing the value, or by adding -DUSE_SYSTEM_<PACKAGE>:BOOL=FALSE to the cmake command line. 🫶

📁 Directories 📁

The superbuild will download each subproject inside the build tree of the project in the folder:

${PROJECT_SOURCE_DIR}/<component>/<project name>

The <component> is assigned by the maintainer, and has the main purpose to split the source code into conceptual units.

The <project name> should be the name that is used in find_package() calls to find that package.

Each project will be configured and built in the folder:

${PROJECT_BINARY_DIR}/<component>/<project name>

${PROJECT_BINARY_DIR} is the folder where you are building the YCM superbuild project, usually ${PROJECT_SOURCE_DIR}/build/.

The superbuild will run the configure, build and install step for each project.

Each project will be installed in ${YCM_EP_INSTALL_DIR} (by default ${PROJECT_BINARY_DIR}/install). You should not change the YCM_EP_INSTALL_DIR variable, unless you wish to build the superbuild only once, and discard the build directory. In this case you should change this variable to the final destination of the build since, for many projects, the build is not relocatable. Please also note that, if you change it to a folder that is not writable by current user, you will have to run the whole build as superuser. 👑

💻 Developer Mode 💻

A developer usually works on a limited set of projects.

For each superproject that the developer will modify, they should enable the YCM_EP_DEVEL_MODE_<PROJECT> CMake cached variable.

You can do this by running ccmake or cmake-gui and changing the value, or by adding -DYCM_EP_DEVEL_MODE_<PACKAGE>:BOOL=FALSE to the cmake command line.

Note that the update target will be disabled for projects in YCM_EP_DEVEL_MODE_<PROJECT>, and they will not be updated unless the user updates them manually. An automatic update in a modified project would require a manual intervention anyway. Also when working with branches, the update performed by ExternalProject could switch branch or checkout a specific tag or commit, and, even though no work should be lost, the user might not know how to recover it. For this reason the update target is disabled for the projects that the user will modify, and it is enabled only if the YCM_EP_EXPERT_MODE variable is enabled.

🧠 Expert Mode 🧠

The YCM_EP_EXPERT_MODE variable will set the YCM superbuild in “expert mode”. This is disabled by default. This means that all the projects that are in “developer mode” will have all the targets enabled (including the update step) and that the update and similar targets will keep these targets as well.

🎯 Targets 🎯

Global Targets

These targets influence all the whole YCM superbuild. In brackets is the name of the target on IDEs like Visual Studio and Xcode.

  • all (ALL) — Build all sub-projects. Slay the whole squad. 🏗️
  • test (TEST) — Run tests for all sub-projects (only if tests are enabled, see enable_testing(). Vibe check the code. 🧪
  • update-all (ALL_UPDATE) — Update all sub-projects, except for those in YCM_EP_DEVEL_MODE_<PROJECT>. No cap, those stay put. 🔄
  • fetch-all (ALL_FETCH) — Runs git fetch for all the sub-projects in YCM_EP_DEVEL_MODE_<PROJECT> (git sub-projects only). 🐙
  • status-all (ALL_STATUS) — Prints the status (using the appropriate SCM command) for all the sub-projects in YCM_EP_DEVEL_MODE_<PROJECT>. 📊
  • clean-all (ALL_CLEAN) — Todo: Missing docs
  • print-directories-all (ALL_PRINT_DIRECTORIES) — Prints the source and binary directories for all the sub-projects in YCM_EP_DEVEL_MODE_<PROJECT>. 📂
  • install — Todo: Add a proper installation.
WARNING: YCM does not create an install target (yet). In some cases, i.e. if you include some CMake module that installs some file, you might find an install target, but it will not install the whole superbuild, but just these files. One known module that adds the install target is catkinConfig.cmake distributed with ROS Hydro, and usually included by running find_package(catkin).

Component Targets

These targets influence a specific COMPONENT, for example external.

Todo: Component targets.
  • <COMPONENT> — Todo: Missing docs
  • <COMPONENT>-update — Todo: Missing docs

Project Targets - Common

These targets are always available for all the sub-projects:

  • <PROJECT> — Builds a sub-project and all its dependees.
  • <PROJECT>-test — Builds a sub-project and all its dependees and executes its tests (only if tests are enabled, see enable_testing().

Project Targets - Basic Mode

These targets are available only for the sub-projects that are not in YCM_EP_DEVEL_MODE_<PROJECT>.

  • <PROJECT>-update — Update a sub-project.

Project Targets - Development Mode

These targets are available only for the sub-projects that are in YCM_EP_DEVEL_MODE_<PROJECT>.

  • <PROJECT>-configure — Configure a sub-project.
  • <PROJECT>-fetch — Runs git fetch a sub-project (git sub-projects only).
  • <PROJECT>-status — Prints the status (using the appropriate SCM command) for a sub-project.
  • <PROJECT>-clean — Todo: Missing docs
  • <PROJECT>-edit-cache — Edit CMake cache for a sub-project (CMake sub-projects only).
  • <PROJECT>-open — Open the project for editing.
NOTE: MSVC and Xcode only
  • <PROJECT>-print-directories — Prints the source and binary directories for a sub-project.
  • <PROJECT>-dependees — Builds all the sub-projects required by this sub-project.
  • <PROJECT>-dependees-update — Update all the sub-projects that are not in YCM_EP_DEVEL_MODE_<PROJECT> and that are required by this sub-project.
  • <PROJECT>-dependers — Builds all the sub-projects that require this sub-project.
  • <PROJECT>-dependers-update — Update all the sub-projects that are not in YCM_EP_DEVEL_MODE_<PROJECT> and that require this sub-project.

Project Targets - Special Components

Projects in some special components behave in a different way.

documentation Projects

These projects usually don’t have build, configure, or install step. The only target enabled is the <PROJECT> target.

The update step for these projects is always performed with the <PROJECT> target.

If one of the steps is added manually, this step is performed with the <PROJECT> target.

These projects are not added to the global targets.

examples Projects

These projects are not added to the global targets.

templates Projects

These projects are not added to the global targets.

🖥️ IDEs 🖥️

Todo: Add documentation about how to use the most common IDEs with YCM.
  • Xcode
  • QtCreator
  • MSVC
  • Maybe more.

💖 YCM SUPERBUILD MANUAL FOR MAINTAINERS 💖

ok so, bestie, a YCM superbuild rides on the ExternalProject CMake module. The version of ExternalProject that ships inside YCM is basically the same one, except it's got a few extra patches stacked on top — patches that make it do more, and that fix a couple of issues so you can actually work with the code that ExternalProject downloads without risking to lose work 😭. The reason those patches live here and not upstream? So they can be tested and used first, before getting submitted upstream. The long-term goal is to merge all the features we need into the module upstream. slay, eventually.

Real talk though: the modified CMake modules do not get included automatically when you include YCM. To use the version that comes with YCM you have to set the YCM_USE_CMAKE_PROPOSED variable to ON before you search for YCM with find_package(YCM) or bootstrap it. like, before. not after. 💅

# Enable cmake-proposed modules
set(YCM_USE_CMAKE_PROPOSED ON)

# Now bootstrap YCM
include(YCMBootstrap)

And because a superbuild downloads all the external projects off the net, bootstrapping treats YCM like any other external project — with one glow-up difference: it's downloaded and built at configure time instead of at compile time. Which means you get to use every YCM module right after the bootstrap. iconic.

The other important modules for making a superbuild, bestie:

  • YCMEPHelper — a helper for ExternalProject that does some extra setup and adds some extra targets ✨
  • FindOrBuildPackage — makes sure a package is available and, if it isn't, downloads and builds it 🫶

SEE ALSO: Superbuild Helper Modules

NOTE: A superbuild usually does not contain source code, but just the CMake files to build all the subprojects. It can build code, but the target dependencies should be handled properly.

WARNING: CMake modules installed by packages built by the superbuild will not be available during the configure phase of the superbuild, so you cannot include them in the CMakeLists.txt file, and you cannot use the functions and macros that these files declare. This is often a really good reason for not including source code in the superbuild. 🚫

🧱 Build Modules 🧱

Every subproject “Project” you want to build should have a BuildProject.cmake file sitting in one of the folders in CMAKE_MODULE_PATH. YCM ships a few of them (see Build Package Modules), and they get found automatically once YCM was found by find_package(YCM) or bootstrapped. You can add more Build modules to your superbuild by dropping them in some folder and appending that folder to the CMAKE_MODULE_PATH variable. You can also use this variable to replace one of the Build files included in YCM — but in that case your folder has to be in CMAKE_MODULE_PATH before you search for YCM:

# Build modules are in cmake folder
list(APPEND CMAKE_MODULE_PATH "${CMAKE_CURRENT_SOURCE_DIR}/cmake")

# Now search (or bootstrap) YCM
find_package(YCM REQUIRED)
# or
# include(YCMBootstrap)

Each build file should handle properly the dependencies required by the project that will be built. By managing the dependencies for each project, the superbuild can ensure the dependencies are available when the build starts. This is especially important when you run parallel builds (i.e. make -j8) because builds

Here's a basic example, for a project “Bar” that depends on project “Foo”, using YCMEPHelper and FindOrBuildPackage to build the code:

include(YCMEPHelper)
include(FindOrBuildPackage)

# Ensures that the superbuild knows about the Foo package
find_or_build_package(Foo QUIET)

ycm_ep_helper(Bar TYPE GIT
                  SYLE FOO_STYLE
                  REPOSITORY foo/bar.git
                  TAG master
                  DEPENDS Foo) # Explicitly declare that Bar depends on Foo

The main CMakeLists.txt will then include:

include(FindOrBuildPackage)

# Build the Bar package. Foo will be handled automatically
# You don't need a "find_or_build_package(Foo)" call here if the Foo
# package is not conceptually part of your superbuild
find_or_build_package(Bar)

🛠️ Non CMake Projects 🛠️

Projects written with CMake can be pulled into a superbuild in, like, a few minutes. Other projects using for example autotools or other build systems can still be included, but they take a bit more effort to add to the build system. no cap.

In practice, the configure step is the one that usually needs modifying. You do that by passing the CONFIGURE_COMMAND argument to the ycm_ep_helper() command. An important thing to configure is the prefix where the package will be installed — otherwise it gets installed to the system default (usually in /usr/local/) and that folder might not be writable by the user. 🙈

The following strings (including the < and > characters) can be used to configure the steps:

<SOURCE_DIR>   # Source directory
<BINARY_DIR>   # Binary directory
<INSTALL_DIR>  # Install directory
<TMP_DIR>      # Directory for temporary files.

For example, for an automake project, the ycm_ep_helper() call could look something like this:

ycm_ep_helper(Foo TYPE GIT
                  STYLE FOO_STYLE
                  REPOSITORY foo/foo.git
                  TAG v1.0
                  CONFIGURE_COMMAND <SOURCE_DIR>/configure --prefix=<INSTALL_DIR>)

🎀 Overriding Parameters 🎀

Every parameter of the ycm_ep_helper() function can be overridden using cmake variables. Like, to overwrite the TAG for project FOO, and the COMPONENT for project BAR, you just add somewhere before the relevant ycm_ep_helper() call:

set(FOO_TAG v1.0)
set(BAR_COMPONENT important)

See the YCMEPHelper module documentation for other details about the parameters that can be modified with a variable. 📖

🌈 Specifying additional CMake arguments for all subprojects 🌈

Sometimes it's convenient to specify some additional command line arguments that get passed during the cmake invocation of all CMake-based subprojects. That's possible thanks to the YCM_EP_ADDITIONAL_CMAKE_ARGS CMake cache variable, which you can set as a command line argument passed to the superbuild cmake invocation:

or as a CMake variable set in the CMake code:

set(YCM_EP_ADDITIONAL_CMAKE_ARGS "-DBOOL_OPTION:BOOL=ON -DLIST_VARIABLE:STRING=foo;bar")

Note that in the latter case, you need to make sure you set YCM_EP_ADDITIONAL_CMAKE_ARGS before the first inclusion of the YCMEPHelper module in your project. timing is everything, bestie. ⏰

💜 Components 💜

YCMEPHelper assigns to each sub-project a COMPONENT (the default component is external). This is useful to separate your project into conceptual units.

You can add any other component for your superbuild.

This will influence the superbuild in some ways:

  1. The superbuild will create targets for each component.

SEE ALSO: Component Targets for details.

  1. The component will influence the folders where the project is downloaded and built.

SEE ALSO: Directories for details.

  1. Some special components are handled in a slightly different way:
    • external component is for packages that the users of the superbuild will not modify.
    • documentation component contains only documentation and is not necessary for building the superbuild.
    • example and template components are not necessary for building the superbuild.

SEE ALSO: Project Targets - Special Components for details.

🌿 Changing TAG (git repository only) 🌿

Todo

Perhaps this should be in the YCMEPHelper module documentation?

ycm_ep_helper() allows you to set a TAG for git repositories. This TAG can be any ref known by the git repository, i.e. a commit hash, a tag or a branch.

ExternalProject handles this tag in different ways depending on whether it's a head or a fixed commit.

In the first case, git will perform a checkout of the tag the first time, and then it will perform a rebase when the update target is executed.

This is the recommended mode to use when you need to work on one project inside your superbuild. ✨

Todo

Changing branch issues

In the latter case, git always performs a checkout of the specific commit, leaving the user in ‘detached HEAD’ state. 💀

This is the recommended mode to use for projects that are just dependencies for projects inside your superbuild, and that developers will not modify.

Todo

Changing tag issues

🎨 Styles 🎨

Todo

Perhaps this should be in the YCMEPHelper module documentation?

Todo

Missing docs

📊 CDash Integration 📊

Todo

Unit tests are not well integrated yet, see YCM issue #17

🤖 Non Interactive Builds 🤖

For build machines, the NON_INTERACTIVE_BUILD variable should be set to true.

NOTE: This should either be set by running cmake -DNON_INTERACTIVE_BUILD:BOOL=TRUE, or using an initial cache file and running cmake -C <file>

📦 Install Step 📦

An important thing to notice: if the subprojects are written in a proper way, the user will have all the files that he needs to use the projects in ${PROJECT_BINARY_DIR}/install

Which means it's pretty important for your subproject to install the files in the install step, and that all the files are installed inside the install prefix. For CMake projects this usually means the DESTINATION argument for the install() command should be a relative path instead of absoulute. 🫶

👑 Maintainer Mode 👑

If the YCM_EP_MAINTAINER_MODE CMake variable is enabled, all the targets for all the projects will be enabled, including the update step.

This is a useful variable for maintainers, but it is not recommended for developers. 🙅‍♀️

⭐ Examples ⭐

This is a list of known projects using YCM.

  • WALK-MAN FP7 EU project ([image: lock] [image] WALK-MAN Superbuild Repository)
  • CoDyCo FP7 EU project (CoDyCo Superbuild Repository)

Copyright 2012-2021 Istituto Italiano di Tecnologia (IIT)

and that's a wrap on the whole page, bestie 🎀 this was version 0.13., dated Nov 05, 2023 — go build something iconic with it. mwah 💅✨

📦 where this page comes from