1. Goals

2. Design

The pbuild tool is a level above "the build script" called "build". "build" is handling single build jobs, while pbuild is managing the dependencies between this build jobs. Both can get executed on any Linux system directly.

pbuild is processing local sources, but may download remote sources and binaries when referenced. The local sources can get managed by any source revision management tool (eg. git).

The tasks of the build script are: * Building a defined and reproducable environement for one build job * Calling the build tool for any supported format (rpm, deb, kiwi image, docker, …) * Calling validations as provided by the build environement * Extract build results * Parse dependencies in binaries and build description files

The tasks of pbuild are: * calculating the dependencies between the build jobs for a build order * calculating the need of rebuilds due to changed sources or dependencies * manage multiple running build jobs in the background * organize build results on the disk * download remote assets (sources or binaries)

2.1. Filesystem Structure

pbuild expects to get call in a directory where it’s project configuration files are living. Package sources live in directories inside of this project directory:

  • Top directory

    • Package A

      • source files

    • Package B

      • source files

    • _pbuild

    • _config

    • _build.<preset>

      • _jobhistory

      • Package A results

        • Build results

      • Package B results

        • Build results

      • .pbuild

2.1.1. _config

Can define build configurations. Usually defaults from a distribution are used, but this file can provide additional rules. These define which packages need to become available in the build environment, macro definitions and many more. See the build config topic for a detailed list.

2.1.2. _pbuild

This file defines one or more presets. A preset defines which

  • distribution configuration

  • repositories

  • hardware architectures

2.1.3. _build.<preset>

contains all binaries for the given preset. These are either downloaded or build local. Additional underscore prefixed files contain informations about the build history and current states of the build.

2.2. Important Options

pbuild does not need any parameters by default. It will build inside of KVM by default which can work as user (non-root) in a proper environment. KVM will also ensure that the correct kernel is running during build. However, builds in chroot or containern is also possible via

--vm-type=chroot
--vm-type=docker

Define the amount of provided memory in KVM via

--vm-memory=MEGABYTES

The build environment disk space can get defined via

--vm-disk-size=MEGABYTES

The KVM is fully isolated by default. This ensures that the build can become fully reproducibile and also that no attack by using untrusted sources or binaries can happen. However, one may want to use remote resources during build. In that case network access can get enabled via

--vm-network

Keep in mind that this can get usually avoided by using remote assets instead.

2.3. Important Debugging Options

pbuild always creates a consistent state. That means it may not build actually anything, when the last build result look current. One can enforce a rebuild by using:

--rebuild <PACKAGE>

The build log is by default only stored on disk. The live build log can be made available by using the

--showlog

option. If you want to debug a problem with a single package, you can use the

--single <PACKAGE>

option to make pbuild only look at the specified package. This also triggers an rebuild of the package. It often makes sense to use

--single <PACKAGE> --shell-after-fail

to get a shell prompt in case a build failure occurs.

Alternatively, one can reuse last build enviroment or create a new one by using the

--shell

option. This will give a prompt after the environment has been initialized.

Extra packages needed for debugging can be specified with

-x binary_package

(eg. gdb, strace, valgrind or assumed missing dependencies for testing).

2.4. Remote Assets

Remote Assets can be defined inside of the build descriptions. They are resources from a remote servers and will be downloaded and validated by pbuild before the corresponding build is started.

Supported protocols for resources using an url are:

  • http

  • https

  • git+http

  • git+https

This depends on the build description format as described below.

2.4.1. rpm spec files

Source lines in spec files can declare remote assets. Rpm is already supporting to use an URL as source definition. A comment before such a line marks it as remote asset for pbuild:

#!RemoteAsset
Source: <URL>

It is also possible to specify a checksum to make sure that the asset is correct.

#!RemoteAsset: <CHECKSUM>
Source: <URL>

An alternative is just to use the

#!RemoteAssetUrl: <URL>

marker without any Source: tag when the downloaded file should not become part of the source rpm.

2.4.2. kiwi build description

The kiwi format is not supporting remote source specifications, but they can be specified via XML comments.

<!-- OBS-RemoteAsset: <URL> -->

The resulting downloaded file can be copied into the image using

<!-- OBS-CopyToImage: <source> <destination> -->

The kiwi profile name is added to container names by default. This can be disabled via

<!-- OBS-DoNotAppendProfileToContainername -->

2.4.3. Dockerfile description

Has no support for assets yet.

2.4.4. Arch Linux PKGBUILD

PKGBuild files already contain the needed URLs and Checksums, so no extra treatment is needed.

2.4.5. FedPkg "sources" file

The sources file contains a list of file names with checksums. PBuild will automatically download the missing files from a FedPkg Server that needs to be configured in the project config via the "AssetsURL:" directive or set with the "--assets" option.

3. About the Build Configuration

The build configuration syntax is

Switch on or off certain features during the build.

Decide which package is installed during build if there are circular dependencies.

Handle user decisions like package providing the same or special macros, packages, or flags in the build environment.

3.1. Configuration File Syntax

The syntax is basically the same than in RPM spec files. However, it is independent of the used packaging format. The project configuration is parsed by &obsa;. This means, you can use RPM features like macros or conditions in the configuration. All lines have the form:

keyword: arguments

In addition to that rpm macro statement can get used, for example to activate some rules only for specific architectures via conditions.

The following list contains a list of allowed keywords in the project configuration:

Available Keywords in Project Configuration

3.1.1. BinaryType: TYPE

The binary type is the format of the files which will be the result of the build jobs. This gets usually set depending on the build recipe type. In some situations, for example a &kiwi; build job result gets converted into an rpm, it can be used to overwrite it. Possible values are: rpm, deb or none.

Sets the binary format used to set up the build environment. For example a package with spec build description may use and generate deb packages instead of RPMs. If no binary type is specified, OBS deduces it from the build recipe type. If the recipe type is also not set, OBS looks at the Preinstall package list for a hint.

3.1.2. BuildEngine: ENGINE

Use an alternative build engine. This is still chained inside of the build script for security reasons. Alternatives are mock (for Fedora and Red Hat) and debootstrap (for Debian). This will avoid differences in the build environment setup, but it will also have an effect on speed and reduced features. It should only be used when you want to emulate the distribution build. debbuild engine will build deb files out of a spec file description. It can be used by the following definition inside of the project build config:

Repotype: debian
Type: spec
Binarytype: deb
BuildEngine: debbuild
Support: pax
Support: debbuild
Keep: debbuild

3.1.3. BuildFlags: FLAG:VALUE

The BuildFlags keyword defines flags for the build process. The following values for FLAG are usable. All flags with a leading colon (:) require an additional parameter directly following them.

  • vmfstype:TYPE

Defines a specific file system when building inside of a VM. Possible values are ext2, ext3, ext4, btrfs, xfs, reiserfs (v3).

  • vmfsoptions:OPTIONS

There is also an option to disable directory indexing for ext2, ext3, and ext4 file systems. This makes file ordering inside of directories reproducible but may have a negative performance impact. To apply this, specify vmfsoptions:nodirindex

  • kiwiprofile:PROFILE

builds the selected profile in &kiwi; appliance builds.

  • logidlelimit:SECONDS

Build jobs which don’t create any output get aborted after some time. This flag can be used to modify the limit. Specify the seconds behind flag.

  • excludebuild:PACKAGE

Exclude a package from building. _multibuild flavors can be specified via package:flavor syntax.

  • onlybuild:PACKAGE

DANGER: this may remove many build results when introduced the first time! It can be used to maintain a whitelist of packages to be built. All other packages will turn to excluded state.

  • ccachetype:TYPE

Defines the ccache implementation, possible values are: ccache, sccache

  • useccache:PACKAGE

Configure usage of ccache when building packages that match the value of this parameter. The ccache package will automatically be installed and configured. The directory /.ccache/ will be configured as cache directory. To configure ccache, the file /.ccache/ccache.conf can be modified as part of the build process by the $BUILD_USER environment variable.

In some cases, there is no archive for the current package, such as when the package was newly branched or when binaries were deleted. In these cases, &obs; will check whether there is a package of the same name built for the same architecture within one of the repositories configured in the project’s meta configuration. If so, the archive of that package will be used. The repositories will be searched in the order they are configured in the meta configuration, starting from the top.

An alternative way to enable caching based on build dependencies is to add "--enable-cache" as dependency, for example via a Substitute rule:

Substitute: gcc-c++ gcc-c++ --enable-ccache

This will always enable ccache when a direct build depdency to gcc-c++ is required.

It is also possible to set the type, eg:

Substitute: cargo cargo --enable-ccache=sccache
  • obsgendiff

OBS can run an external program that has access to the current build and the previously successful result, e.g. to generate a difference or a changelog from the diff.

build will run all scripts in /usr/lib/build/obsgendiff.d/ on the build host (not in the %buildroot) when this flag is set. If one of the scripts fails to run or no scripts are found, then the overall build fails. I.e. if BuildFlags: obsgendiff is set, then you must provide at least one script in /usr/lib/build/obsgendiff.d/, otherwise your build will fail.

A common use case for obsgendiff is to run release-compare after the build.

3.1.4. Constraint: SELECTOR STRING

Caution OBS only

Define build constraints for build jobs. The selector is a colon-separated list which gets a string assigned. See the build job constraints page for details.

3.1.5. ExpandFlags: FLAG

Flags which modify the behaviour during dependency resolution.

  • unorderedimagerepos (OBS 2.10 or later)

The priority of repositories defined in an image build is usually important. This is to avoid switching repositories when the same package is available in multiple repositories. However, it might be wanted to ignore that and just pick the highest version. This can be achieved by defining this flag

  • preinstallexpand

Preinstall also all dependencies of a preinstalled package. Instead of manually listing all packages for a working package tool one can just install dependencies of it. However, these might be more then actually needed depending on the distribution.

  • module:NAME-STREAM

Enable Red Hat-specific module support in repo md repositories. By default, modules are not used, but content can be made available by specifying the module name. To remove a module, add an exclamation mark (!) as prefix.

3.1.6. ExportFilter: REGEX ARCH

The export filter can be used to export build results from one architecture to others. This is required when one architecture needs packages from another architecture for building. The REGEX placeholder must match the resulting binary name of the package. It will export it to all listed scheduler architectures. Using a single dot will export it to the architecture which was used to build it. So not using a dot there will filter the package.

3.1.7. FileProvides: FILE PACKAGES

&obsa; ignores dependencies to files (instead of package names) by default. This is mostly done to reduce the amount of memory needed, as the package file lists take up a considerable amount of repository meta data. As a workaround, FileProvides can be used to tell the systems which packages contain a file. The File needs to have the full path.

3.1.8. HostArch: HOST_ARCH

This is used for cross builds. It defines the host architecture used for building, while the scheduler architecture remains the target architecture.

3.1.9. Ignore: PACKAGES

Ignore can be used to break dependencies. This can be useful to reduce the number of needed packages or to break cyclic dependencies. Be careful with this feature, as breaking dependencies can have surprising results.

3.1.10. Ignore: PACKAGE_A:PACKAGES

It is possible to define the ignore only for one package. This package must be listed first with a colon.

3.1.11. Keep: PACKAGES

To eliminate build cycles the to-be-built package is not installed by default, even when it is required. Keep can be used to overwrite this behavior. It is usually needed for packages like make that are used to build itself. Preinstalled packages are automatically kept, as the package installation program needs to work all the time.

3.1.12. OptFlags: TARGET_ARCH FLAGS (RPM only)

Optflags exports compiler flags to the build. They will only have an effect when the spec file is using $RPM_OPT_FLAGS. The target architecture may be * to affect all architectures.

3.1.13. Order: PACKAGE_A:PACKAGE_B

The build script takes care about the installation order if they are defined via dependencies inside of the packages. However, there might be dependency loops (reported during setup of the build system) or missing dependencies. The Order statement can be used then to give a hint where to break the loop.

The package in PACKAGE_A will get installed before the package in PACKAGE_B.

3.1.14. Patterntype: TYPES

Defines the pattern format. Valid values are: none (default), ymp, comps.

3.1.15. Prefer: PACKAGES

In case multiple packages satisfy a dependency, the OBS system will complain about that situation. This is unlike like most package managing tools, which just pick one of the package. Because one of OBS’ goal is to provide reproducible builds, it reports an error in this case instead of choosing a random package. The Prefer: tag lists packages to be preferred in case a choice exists. When the package name is prefixed with a dash, this is treated as a de-prefer.

3.1.16. Prefer: PACKAGE_A:PACKAGES

It is possible to define the prefer only when one package is creating the choice error. This package must be listed first with a colon.

3.1.17. Preinstall: PACKAGES

Are needed to run the package installation program. These packages get unpacked before the VM gets started. Included scripts are not executed during this phase. However, these packages will get installed again inside of the VM including script execution.

3.1.18. PublishFlag: FLAG

Caution OBS only

Flags which modify the behaviour during repository generation.

  • create_empty

Create a repository even with no content, but with meta data.

  • noearlykiwipublish

Only publish kiwi build results after entire repository has finished building. Without this kiwi build results get published immediately after the build is finished.

  • nofailedpackages

Block publishing if any build result was failed, broken, or unresolvable. This is evaluated individually for each architecture. That means, packages can be published for an architecture on which it builds, even if a package fails to build on another architecture.

  • withreports

Also publish internal content tracking files (.report files).

  • ympdist:NAME (OBS 2.11 or later)

Defines the distversion to be used in group element of ymp files. This is used by the installer to check if the repository is suitable for the installed distribution.

3.1.19. PublishFilter: REGEXP [REGEXP]

Caution OBS only

Limits the published binary packages in public repositories. Packages that match any REGEXP will not be put into the exported repository. There can be only one line of PublishFilter for historic reasons. However, multiple REGEXP can be defined.

3.1.20. Repotype: TYPE[:OPTIONS]

Defines the repository format for published repositories. Valid values are: none, rpm-md, suse, debian, hdlist2, arch, staticlinks and vagrant. The OPTIONS parameter depends on the repository type, for rpm-md the known options are ‘legacy’ to create the old rpm-md format, ‘deltainfo’ or ‘prestodelta’ to create delta rpm packages, ‘rsyncable’ to use rsyncable gzip compression. To split the debug packages in an own published repository the option splitdebug:REPOSITORY_SUFFIX can be appended, e.g.

3.1.21. Repotype: rpm-md splitdebug:-debuginfo

(the repository format may even be omitted to use the default type). This results in a debuginfo package repository being created in parallel to the package repository.

3.1.22. Required: PACKAGES

Contain one or more packages that always get installed for package builds. A change in one of these packages triggers a new build.

3.1.23. Runscripts: PACKAGES

Defines the scripts of preinstalled packages which needs to be executed directly after the preinstall phase, but before installing the remaining packages.

3.1.24. Substitute: PACKAGE_A PACKAGES

It is possible to replace to BuildRequires with other packages. This will have only an effect on directly BuildRequired packages, not on indirectly required packages.

3.1.25. Support: PACKAGES

Contain one or more packages which also get installed for package builds, but a change in one of the packages does not trigger an automatic rebuild.

This is useful for packages that most likely do not influence the build result, for example make or coreutils.

3.1.26. Target: TARGET_ARCH (RPM only)

Defines the target architecture. This can be used to build for i686 on i586 schedulers for example.

3.1.27. Target: GNU_TRIPLET (DEB only)

Defines the target architecture via a gnu triplet (not the debian architecture!). For example arm-linux-gnueabihf for armv7hl builds.

3.1.28. Type: TYPE

Build recipe type. This is the format of the file which provides the build description. This gets usually autodetected, but in some rare cases it can be set here to either one of these: spec, dsc, kiwi, livebuild, arch, preinstallimage.

Defines the build recipe format. Valid values are currently: none, spec, dsc, arch, kiwi, preinstallimage. If no type is specified, OBS deduces a type from the binary type.

3.1.29. VMInstall: PACKAGES

Like Preinstall, but these packages get only installed when a virtual machine like Xen or KVM is used for building. Usually packages like mount are listed here.

3.2. Macros

Macros are defined at the end of the project configuration. The macro section is only used on RPM builds.

The project configuration knows two possible definitions:

%define Macro Definition

Starting with a %define line and are used in the project configuration only. These definitions are not available inside the build root. These are only needed when these macros are used elsewhere in the build config.

For example:

%define _use_profiler 1
%if 0%{?_use_profiler}
Require: gprof
%endif

Another example doing it by architecture:

%ifarch x86_64
Support: x86_64_only_package
%endif

3.3. Macro Definitions

Starting after the Macros: line and are exported into the .rpmmacros file of the build root. As such, these macro definitions can be used in a spec file.

For example, you can define

%define _with_pulseaudio 1

Macros Used in Spec Files Only

The macro definition in the project configuration is located at the end and has the following structure:

Structure of a Macro Definition

Macros: # add your macro definitions here %_hardened_build 0 :Macros

Everything that starts with a hash mark (#) is considered a comment.

4. Functionalities

4.1. package builds

Package builds are supported for rpm, deb, arch and their variations.

4.2. image builds

4.3. container builds

4.4. Cross Architecture Build

Cross architecture builds specify two sets of repositories, one for the build environement and one for the environment for the build result.

4.4.1. Terminology

The terminiology for this is unfortunatly not standarised and tools have conflicting defintions.

We use the following

Host: The architecture build environment which executes the for example the compiler.

cmake is calling this HOST as well (eg. CMAKE_HOST_SYSTEM_PROCESSOR)

GNU tools (esp. the configure scripts generated by autoconf) is using BUILD as abbrevasion here

Target: The architecture of the resulting binaries

cmake has this as default without any abbrevasion (eg. CMAKE_SYSTEM_PROCESSOR)

GNU tools (esp. the configure scripts generated by autoconf) is using HOST as abbrevasion here: The "TARGET" defintion of them is something else, usually only used for compilers where it defines the target architecture a from a cross build compiler.

The compatible terms can be found in the following table:

Unfortunatly the naming of architecture areas is not consistent via the various build tooling. Please find the terminilogy mapping from our "build" script and other toolings in the list below.

build

host

target

-/-

cmake

HOST

default

-/-

GNU

build

host

target

rpm

build

target

-/-

Our build script just knows the host and target definition as described above.

cmake is similar, just uses different namings.

GNU (esp. in automake and autoconf) has actually three definitions, mainly for compiler toolchain. "build" is the environment which creates binaries. Host are the resulting binaries. And target is important when these binaries can build binaries as well for another target.

4.4.2. Dependency handling

Dependencies need to be resolved always twice during a cross build. One time for the host architecture and one time for the target architecture. Build dependencies of a package get only installed for the target architecture by default.

However, global rules in the build configuration can get created to modify this behaviour.

AlsoNative: PACKAGES

Can be used to mark a package to get installed into the target and in the host system.

OnlyNative: PACKAGES

Can be used to to install a package in the hostsystem, but not in the target system. Eg. for build tooling or code generators.

The target base system

All packages for a host system which are configured as "Requrired" get installed only in the host system, but not in the target system. The reason behind is that this packages are only supposed to be able to execute commands and therefore they are not needed in the target environment. Only the build dependencies get installed there.

However, some distributions may lack a complete set of dependencies (esp. debian based). In this case the base system for the target system can manually get configured via

Substitute: sysroot-packages PACKAGES

4.4.3. Example Calls in spec files for cross build

Note: the basic design of rpm was the GNU style, but it changed in the way that you have to specify the host architecture via %target macro on most distributions:

./configure --build=%build --host=%target

4.5. Repository setup

4.6. Build Config