1. Goals
2. Design
The pbuild tool is a level above "the build script" called "build". "build" is handling single build jobs, while pbuild is managing the dependencies between this build jobs. Both can get executed on any Linux system directly.
pbuild is processing local sources, but may download remote sources and binaries when referenced. The local sources can get managed by any source revision management tool (eg. git).
The tasks of the build script are: * Building a defined and reproducable environement for one build job * Calling the build tool for any supported format (rpm, deb, kiwi image, docker, …) * Calling validations as provided by the build environement * Extract build results * Parse dependencies in binaries and build description files
The tasks of pbuild are: * calculating the dependencies between the build jobs for a build order * calculating the need of rebuilds due to changed sources or dependencies * manage multiple running build jobs in the background * organize build results on the disk * download remote assets (sources or binaries)
2.1. Filesystem Structure
pbuild expects to get call in a directory where it’s project configuration files are living. Package sources live in directories inside of this project directory:
-
Top directory
-
Package A
-
source files
-
-
Package B
-
source files
-
-
…
-
_pbuild
-
_config
-
_build.<preset>
-
_jobhistory
-
Package A results
-
Build results
-
-
Package B results
-
Build results
-
-
.pbuild
-
-
2.1.1. _config
Can define build configurations. Usually defaults from a distribution are used, but this file can provide additional rules. These define which packages need to become available in the build environment, macro definitions and many more. See the build config topic for a detailed list.
2.1.2. _pbuild
This file defines one or more presets. A preset defines which
-
distribution configuration
-
repositories
-
hardware architectures
2.1.3. _build.<preset>
contains all binaries for the given preset. These are either downloaded or build local. Additional underscore prefixed files contain informations about the build history and current states of the build.
2.2. Important Options
pbuild does not need any parameters by default. It will build inside of KVM by default which can work as user (non-root) in a proper environment. KVM will also ensure that the correct kernel is running during build. However, builds in chroot or containern is also possible via
--vm-type=chroot
--vm-type=docker
Define the amount of provided memory in KVM via
--vm-memory=MEGABYTES
The build environment disk space can get defined via
--vm-disk-size=MEGABYTES
The KVM is fully isolated by default. This ensures that the build can become fully reproducibile and also that no attack by using untrusted sources or binaries can happen. However, one may want to use remote resources during build. In that case network access can get enabled via
--vm-network
Keep in mind that this can get usually avoided by using remote assets instead.
2.3. Important Debugging Options
pbuild always creates a consistent state. That means it may not build actually anything, when the last build result look current. One can enforce a rebuild by using:
--rebuild <PACKAGE>
The build log is by default only stored on disk. The live build log can be made available by using the
--showlog
option. If you want to debug a problem with a single package, you can use the
--single <PACKAGE>
option to make pbuild only look at the specified package. This also triggers an rebuild of the package. It often makes sense to use
--single <PACKAGE> --shell-after-fail
to get a shell prompt in case a build failure occurs.
Alternatively, one can reuse last build enviroment or create a new one by using the
--shell
option. This will give a prompt after the environment has been initialized.
Extra packages needed for debugging can be specified with
-x binary_package
(eg. gdb, strace, valgrind or assumed missing dependencies for testing).
2.4. Remote Assets
Remote Assets can be defined inside of the build descriptions. They are resources from a remote servers and will be downloaded and validated by pbuild before the corresponding build is started.
Supported protocols for resources using an url are:
-
http
-
https
-
git+http
-
git+https
This depends on the build description format as described below.
2.4.1. rpm spec files
Source lines in spec files can declare remote assets. Rpm is already supporting to use an URL as source definition. A comment before such a line marks it as remote asset for pbuild:
#!RemoteAsset
Source: <URL>
It is also possible to specify a checksum to make sure that the asset is correct.
#!RemoteAsset: <CHECKSUM>
Source: <URL>
An alternative is just to use the
#!RemoteAssetUrl: <URL>
marker without any Source: tag when the downloaded file should not become part of the source rpm.
2.4.2. kiwi build description
The kiwi format is not supporting remote source specifications, but they can be specified via XML comments.
<!-- OBS-RemoteAsset: <URL> -->
The resulting downloaded file can be copied into the image using
<!-- OBS-CopyToImage: <source> <destination> -->
The kiwi profile name is added to container names by default. This can be disabled via
<!-- OBS-DoNotAppendProfileToContainername -->
2.4.3. Dockerfile description
Remote assets that will be downloaded before the build can be added via the
comment #!RemoteAssetUrl: https://my.url/asset.tar
. pbuild
will fetch the
asset before the build and will make it available in the buildroot so that the
asset can copied into the container as follows:
2.4.4. Arch Linux PKGBUILD
PKGBuild files already contain the needed URLs and Checksums, so no extra treatment is needed.
2.4.5. FedPkg "sources" file
The sources file contains a list of file names with checksums. PBuild will automatically download the missing files from a FedPkg Server that needs to be configured in the project config via the "AssetsURL:" directive or set with the "--assets" option.
3. About the Build Configuration
The build configuration syntax describes the build environment used for building packages. It is used to:
-
Switch on or off certain features during the build
-
Decide which packages are installed during build
-
Resolve dependency problems like when there are multiple providers for a dependency
-
Handle user decisions like macro settings
3.1. Configuration File Syntax
The syntax is basically the same as in RPM spec files. However, it is independent of the used packaging format. You can use spec file features like macros or conditions in the configuration.
All lines have the form:
keyword: arguments
Use %if
or %ifarch
if the line should only be used in some condition
Many keywords like Require
or BuildFlags
do not replace existing
data, but add to it. This means you can have multiple BuildFlags
lines instead of having one line with all the flags you need.
You can prepend an exclamation mark !
to the argument to remove
an existing entry from the data.
3.2. Available Keywords in Project Configuration
The following list contains all the allowed keywords in the project configuration:
3.2.1. AssetsURL: TYPE@URL
Define a url for automatic asset downloading. Supported types
are currently fedpkg
and goproxy
.
3.2.2. BinaryType: TYPE
The binary type is the format of the packages that make up the build
environment. This is usually set automatically depending on the recipe
type and preinstall package list.
Currently understood values are: rpm
, deb
, and arch
.
3.2.3. BuildEngine: ENGINE
Use an alternative build engine. Examples are mock
(for Fedora and
Red Hat) and debootstrap
(for Debian), debbuild
(to build debian
packages with spec files), podman
(container builds).
Here is an example config for debbuild
:
Type: spec
Repotype: debian
Binarytype: deb
BuildEngine: debbuild
Support: pax debbuild
3.2.4. BuildFlags: FLAG[:VALUE]
The BuildFlags keyword defines flags for the build process. The following values for FLAG are usable.
-
vmfstype:TYPE
Defines a specific file system when building inside of a VM. Possible
values are ext2
, ext3
, ext4
, btrfs
, xfs
, reiserfs
(v3).
-
vmfsoptions:OPTIONS
Sets options for file system creation. Currently only the nodirindex
option is supported, which disables directory indexing for ext file
systems. This makes file ordering inside of directories reproducible
but may have a negative performance impact.
-
kiwiprofile:PROFILE
Selects the profiles to build in kiwi appliance builds.
-
logidlelimit:SECONDS
Build jobs which do not create any output are aborted after some time. This flag can be used to modify the limit.
-
excludebuild:PACKAGE
Exclude a package from building. If a package builds multiple flavors
("multibuild"), the corresponding flavor can be specified via the
package:flavor
syntax.
-
onlybuild:PACKAGE
This can be used to maintain a whitelist of packages to be built. All other packages will turn to excluded state (and the old build results will be erased!).
-
ccachetype:TYPE
Defines the ccache implementation, possible values are: ccache
, sccache
.
-
useccache:PACKAGE
Configure usage of ccache when building the specified package.
-
obsgendiff
OBS can run an external program that has access to the current build and the previously successful result, e.g. to generate a difference or a changelog from the diff.
build will run all scripts in /usr/lib/build/obsgendiff.d/ on the build host (not in the %buildroot) when this flag is set. If one of the scripts fails to run or no scripts are found, then the overall build fails. I.e. if BuildFlags: obsgendiff is set, then you must provide at least one script in /usr/lib/build/obsgendiff.d/, otherwise your build will fail.
A common use case for obsgendiff is to run release-compare after the build.
-
setvcs
Adds the SCM URL to binary results when the package sources are managed via the scmsync mechanic. The url is written into the VCS tag of rpms when enabling this functionality.
-
sbom:FORMAT
Enables generation of SBOM (Software Bill Of Material) data. Supported formats are spdx and cyclonedx.
-
container-compression-format:FORMAT
Sets a compression format for container layers. Possible values are gzip
, zstd
,
zstd:chunked
. The usage of values other than gzip is only supported by the
podman engine.
-
container-build-format:FORMAT
For podman container builds, it specifies the container config format. Possible values are docker and oci. The default is docker. The docker format allows a few extensions like ONBUILD, SHELL, DOMAINNAME, COMMENT, HEALTHCHECK amongst others.
3.2.5. Conflict: PACKAGE
Specify that a package must not be installed in the build environment.
3.2.6. Conflict: PACKAGE_A:PACKAGE_B
Specify a synthetic conflict between to packages.
3.2.7. Constraint: SELECTOR STRING
OBS only |
Define build constraints for build jobs. The selector is a colon-separated list which gets a string assigned. See the build job constraints page for details.
3.2.8. DistMacro: NAME VALUE
Define a macro to be used when parsing the spec files of packages.
This is similar to using a Macros:
section with the difference
that the macro will not be written to the .rpmmacros file. It should
therefore be used for macros that come from packages of the
distributions.
Note that the lines of the project config are macro expanded while
parsing, so you have to use %%
for a literal percent sign in
the value.
3.2.9. ExpandFlags: FLAG[:VALUE]
Flags which modify the behaviour during dependency resolution.
-
unorderedimagerepos
The priority of repositories defined in an image build is usually important. This is to avoid switching repositories when the same package is available in multiple repositories. However, it might be wanted to ignore that and just pick the highest version. This can be achieved by defining this flag
-
preinstallexpand
Preinstall also all dependencies of a preinstalled package. Instead of manually listing all packages for a working package tool one can just install dependencies of it. However, these might be more then actually needed depending on the distribution.
-
module:NAME-STREAM
Enable Red Hat-specific module support in repo md repositories. By default, no module is used, so every module needed needs to be specified in the configuration. To remove a module, add an exclamation mark (!) as prefix.
-
dorecommends
Try to install all recommended packages. Packages with dependency conflicts are ignored.
-
dosupplements
Try to install all supplemented packages. Packages with dependency conflicts are ignored. This has the downside that new packages can cause different dependency expansion, so this should only be enabled for special use cases.
-
ignoreconflicts
Ignore defined conflicts of packages. By default these are reported as unresolvable. This switch may be useful when packages get not installed in the build environment, but getting processed afterwards. That tool, eg. some image building tool, must be able to handle the situation (eg. by just using a subset of the packages).
-
kiwi-nobasepackages
Do not put the require/support/preinstall packages in the repositories offered to the kiwi build tool. This should have been the default.
-
keepfilerequires
Dependencies on files are only fulfilled if matching FileProvides are specified in the project configuration. If those are missing, the dependency results in an "unresolvable" state for directly required files or in silent breaking of the dependency for indirectly required files. With this option, all file requires are honoured by default and lead to "unresolvable" if there are no matching FileProvides defined.
3.2.10. ExportFilter: REGEX ARCHITECTURES
The export filter can be used to export build results from one
architecture to others. This is required when one architecture needs
packages from another architecture for building. The REGEX placeholder
must match the resulting binary name of the package. It will export it
to all listed scheduler architectures. Exported packages are not
used in the built architecture by default, add a .
pseudo architecture
to also use them locally.
3.2.11. FileProvides: FILE PACKAGES
Due to memory consumption reasons dependencies to files as supported by rpm are ignored by default. As a workaround, FileProvides can be used to tell the systems which packages contain a file. The file needs to have the full path.
3.2.12. HostArch: HOST_ARCH
This is used for cross builds. It defines the host architecture used for building, while the scheduler architecture remains the target architecture.
3.2.13. Ignore: PACKAGE_OR_DEPENDENCY
Ignore can be used to break dependencies. This can be useful to reduce the number of needed packages or to break cyclic dependencies. If a package is specified, all capabilities provided by the package are ignored.
Be careful with this feature as breaking dependencies can have unwanted results. It is usually better to limit its usage by also specify the originating package as described in the following section.
3.2.14. Ignore: ORIGIN_PACKAGE:PACKAGE_OR_DEPENDENCY
Ignore a dependency coming from ORIGIN_PACKAGE. See the previous section for more details.
3.2.15. Keep: PACKAGES
To eliminate build cycles the to-be-built packages are not installed by
default. Keep can be used to overwrite this behavior. It is usually needed
for packages like make
that are used to build itself. Preinstalled
packages are automatically kept, as the package installation program needs
to work all the time.
3.2.16. Macros:
Defines the start of a literal macros block. The block is ended
by either reaching the end of the config or by a literal
Macros:
line. See the section about macro definitions below for
more information.
3.2.17. OptFlags: TARGET_ARCH FLAGS (RPM only)
Optflags exports compiler flags to the build by adding lines to rpm’s
rpmrc
file. They will only have an effect when the spec file is using
$RPM_OPT_FLAGS
or %{optflags}
. The target architecture may be set
to *
to affect all architectures.
3.2.18. Order: PACKAGE_A:PACKAGE_B
The build script takes care about the installation order if they are defined via dependencies inside of the packages. However, there might be dependency loops (reported during setup of the build system) or missing dependencies. The Order statement can be used then to give a hint where to break the loop.
The package in PACKAGE_A will get installed before the package in PACKAGE_B.
3.2.19. Patterntype: TYPE
Defines the pattern format. Valid values are: none
(default), ymp
,
comps
. Multiple types can be specified.
3.2.20. Prefer: PACKAGE
In case multiple packages satisfy a dependency, the dependency expansion will fail. This is unlike like most package managing tools, which just pick one of the package. It is done that way to provide reproducible builds and reduce the chance of surprising changes when new packages are added to the repository. The Prefer directive lists packages to be preferred in case a choice exists. When the package name is prefixed with a minus sign, it is treated as a de-prefer.
3.2.21. Prefer: ORIGIN_PACKAGE:PACKAGE
It is possible to define the prefer only when the dependency comes from the specified originating package.
3.2.22. Preinstall: PACKAGE
This is used to specify packages needed to run the package installation program. These packages are unpacked so that the native installation program can be used to install the build environment. Included scripts are not executed during this phase. However, these packages will be re-installed later on including script execution.
3.2.23. PublishFilter: REGEXP [REGEXP]
OBS only |
Limits the published binary packages in public repositories. Packages that match any REGEXP will not be put into the generated repository.
There can be only one line of PublishFilter for historic reasons. However, multiple REGEXP can be defined.
3.2.24. PublishFlags: FLAG[:VALUE]
OBS only |
Flags which modify the behaviour during repository generation.
-
create_empty
Create a repository even with no content, but with meta data.
-
noearlykiwipublish
Only publish kiwi build results after entire repository has finished building. Without this kiwi build results get published immediately after the build is finished.
-
nofailedpackages
Block publishing if any build result was failed, broken, or unresolvable. This is evaluated individually for each architecture. That means, packages can be published for an architecture on which it builds, even if a package fails to build on another architecture.
-
withreports
Also publish internal content tracking files (.report files).
-
withsbom
Also publish SBOM data in the repostory. Container SBOM data is always pushed to the registries.
-
ympdist:NAME
Defines the distversion to be used in group element of ymp files. This is used by the installer to check if the repository is suitable for the installed distribution. (OBS 2.11 or later)
-
singleexport
If multiple packages contain different versions of a rpm package, only
publish the one from the first package. If the project is of the type
maintenance_release
, this will be the package with the highest
incident number.
-
artifacthub:REG_REPO:ID:[NAME[:EMAIL]]
Specify data for artifacthub repository verification. This will be added to the registry repository when pushing a container to it.
3.2.25. RegistryURL: URL
Define a url for the downloading of containers.
3.2.26. Repotype: TYPE[:OPTIONS]
Defines the repository format for published repositories. Valid values
are: none
, rpm-md
, suse
, debian
, hdlist2
, arch
, staticlinks
and vagrant. Multiple types can be specified to generate more than
one metadata type.
The OPTIONS parameter depends on the repository type, for rpm-md the
known options are legacy
to create the old rpm-md format, deltainfo
or prestodelta
to create delta rpm packages, rsyncable
to use
rsyncable gzip compression.
To split the debug packages in an own published repository the type
splitdebug:REPOSITORY_SUFFIX
can be appended, e.g.:
Repotype: rpm-md splitdebug:-debuginfo
This results in a debuginfo package repository being created in parallel to the package repository.
3.2.27. RepoURL: [TYPE@]URL
Define a url for the downloading of repository packages. Supported types
are currently arch
, debian
, hdlist2
, rpmmd
, suse
. If the
type is not specified, it is guessed from the build type.
3.2.28. Required: PACKAGE
Specify a package that always is installed for package builds. A change in one of these packages triggers a new build.
3.2.29. Runscripts: PACKAGE
Execute the scriptlets of the specified preinstalled package. Scriptlet execution takes place after the preinstall phase, but before installing the remaining packages.
3.2.30. Substitute: OLD_DEPENDENCY [NEW_DEPENDENCY… ]
It is possible to replace BuildRequires dependencies with other dependencies. This will have only an effect on directly BuildRequired packages, not on indirectly required packages.
3.2.31. Support: PACKAGE
Specify a package that always is installed for package builds.
Unlike Required:
, a change in one of these packages does not trigger an
automatic rebuild.
This is useful for packages that most likely do not influence the build
result, for example make
or coreutils
.
3.2.32. Target: GNU_TRIPLET
Defines the target architecture via a gnu triplet (not the debian
architecture!). For example arm-linux-gnueabihf
for armv7hl builds,
or i686
for building i686 packages.
3.2.33. Type: TYPE
Build recipe type. This is the format of the file which provides the
build description (the "build recipe"). This is usually autodetected from
the binary type, but in some rare cases it may be needed to manually
configure the type. Currently the following types are understood:
spec
, dsc
, arch
, kiwi
, livebuild
, productcompose
, preinstallimage
.
3.2.34. VMInstall: PACKAGE
Like Preinstall, but these packages get only installed when a virtual
machine like Xen or KVM is used for building. Usually packages like
mount
are listed here.
3.3. Macros Definitions for the build configuration
You can use rpm macro definitions in the project config to improve
configurability.
Macros are defined with %define
or %global
. They are only known
in the project configuration but not available when the build is
done.
For example:
%define _use_profiler 1
%if 0%{?_use_profiler}
Require: gprof
%endif
Another example limiting a line to an architecture:
%ifarch x86_64
Support: x86_64_only_package
%endif
3.4. Macro Definitions for the build process
To specify macros for the building process, use the Macros:
keyword. All lines after Macros:
up to the end of the config
or to a :Macros
line are used when parsing the spec file and
also made available to the build by copying them to the .rpmmacros
file in the build root.
Example:
Macros:
# add your macro definitions here
%_hardened_build 0
:Macros
Note that the macro lines are copied verbatim, i.e. macro expansion does not take place.
3.5. Building with ccache or sccache
The usage of ccache or sccache can be enabled for each package by
seting the useccache:PACKAGE
build flag.
The ccache package will automatically be installed and configured. The directory /.ccache/ will be configured as cache directory. To configure ccache, the file /.ccache/ccache.conf can be modified as part of the build process by the $BUILD_USER environment variable.
In some cases, there is no archive for the current package, such as when the package was newly branched or when binaries were deleted. In these cases, the system will check whether there is a package of the same name built for the same architecture within one of the repositories configured in the project’s meta configuration. If so, the archive of that package will be used. The repositories will be searched in the order they are configured in the meta configuration, starting from the top.
An alternative way to enable caching based on build dependencies is to add "--enable-cache" as dependency, for example via a Substitute rule:
Substitute: gcc-c++ gcc-c++ --enable-ccache
This will always enable ccache when a direct build depdency to gcc-c++ is required.
It is also possible to set the type, eg:
Substitute: cargo cargo --enable-ccache=sccache
4. Functionalities
4.1. package builds
Package builds are supported for rpm, deb, arch and their variations.
4.2. image builds
4.3. container builds
4.4. Cross Architecture Build
Cross architecture builds specify two sets of repositories, one for the build environement and one for the environment for the build result.
4.4.1. Terminology
The terminiology for this is unfortunatly not standarised and various tools have conflicting defintions.
We use the following
Host Native: The architecture build environment which executes the for example the compiler.
cmake is calling this HOST as well (eg. CMAKE_HOST_SYSTEM_PROCESSOR)
GNU tools (esp. the configure scripts generated by autoconf) is using BUILD as abbrevasion here
Target: The architecture of the resulting binaries
cmake has this as default without any abbrevasion (eg. CMAKE_SYSTEM_PROCESSOR)
GNU tools (esp. the configure scripts generated by autoconf) is using HOST as abbrevasion here: The "TARGET" defintion of them is something else, usually only used for compilers where it defines the target architecture a from a cross build compiler.
The compatible terms can be found in the following table:
Unfortunatly the naming of architecture areas is not consistent via the various build tooling. Please find the terminilogy mapping from our "build" script and other toolings in the list below.
pbuild |
host |
target |
-/- |
|
cmake |
HOST |
default |
-/- |
|
GNU |
build |
host |
target |
|
rpm |
build |
target |
-/- |
Our build script just knows the host and target definition as described above.
cmake is similar, just uses different namings.
GNU (esp. in automake and autoconf) has actually three definitions, mainly for compiler toolchain. "build" is the environment which creates binaries. Host are the resulting binaries. And target is important when these binaries can build binaries as well for another target.
4.4.2. Dependency handling
Dependencies need to be resolved always twice during a cross build. One time for the host architecture and one time for the target architecture. Build dependencies of a package get only installed for the target architecture by default.
However, global rules in the build configuration can get created to modify this behaviour.
AlsoNative: PACKAGES
Can be used to mark a package to get installed into the target and in the host system.
OnlyNative: PACKAGES
Can be used to to install a package in the hostsystem, but not in the target system. Eg. for build tooling or code generators.
The target base system
All packages for a host system which are configured as "Requrired" get installed only in the host system, but not in the target system. The reason behind is that this packages are only supposed to be able to execute commands and therefore they are not needed in the target environment. Only the build dependencies get installed there.
However, some distributions may lack a complete set of dependencies (esp. debian based). In this case the base system for the target system can manually get configured via
Substitute: sysroot-packages PACKAGES
4.4.3. Example Calls in spec files for cross build
Note: the basic design of rpm was the GNU style, but it changed in the way that you have to specify the host architecture via %target macro on most distributions:
./configure --build=%build --host=%target