PathInterposer¶

Table of contents
PathInterposer

Warnings¶

This page is a work in progress, so it might contains mistakes, inaccuracies, etc.

Introduction¶

Sometimes we need to use host tools for building Replicant.

This has several consequences:

When the tools that we inherit from AOSP or LineageOS lack certain features (like when the host xz provided for Android was compiled without the ARM BJC filter), we can easily use the host tools that have such features instead. The downside is that it makes the build process potentially less reproducible as the version of the tools provided by the host distributions can change with the distribution updates or when different people use different distributions.
We typically trust the host distributions like Trisquel way better than binaries provided by Google. So even if both are free software, we would prefer to trust the distribution tools instead.

Ideally we could make Replicant build most of the tools we need in a reproducible way to not use the host tools without having to trust prebuilt binaries, but this is not always practical as doing that could take a lot of time. In addition we do need tools to build replacement tools in the first place.

In any case when using tools from the host distribution, it's a really good idea to not hardcode the path of the tool (like /usr/bin/python) and instead make sure that the host distribution PATH is used instead.

This has several advantages:

It helps a lot making Replicant compatible with more distributions as:
- not all the distributions have the tools in the same path (/usr/bin vs /usr/sbin)
- some distributions (like Guix) do respect the PATH but don't use standard paths (repo could be in /gnu/store/4cw9j7c36cgv37g6bzs2dnygmfc5jxh1-git-repo-2.4.1/share/git-repo/repo or in ~/.guix-profile/bin/repo). And since Guix is reproducible and even bootstrapable we also want not to rule out this option, while also being compatible with other distributions (for simplicity and to not be stuck with Guix).
users might need to override tools in some cases (that happened with proguard in Replicant 6)

This article shows how to use host tools in a way that makes use of the host PATH and also how Android implements (with PathInterposer) the isolation between the tools provided by the host distribution and Android.

How it works¶

During the build:

The host path is saved to out/.path_interposer_origpath
The host path is changed to a new one
The path interposer is used to access host binaries. It uses the saved path for that.

With Trisquel 8, and Replicant 11, here's an example of how the path looks like for applications being launched through the path interposer¹:

/home/replicant/replicant-11/prebuilts/jdk/jdk11/linux-x86/bin
/home/replicant/replicant-11/prebuilts/jdk/jdk11/linux-x86/bin
/home/replicant/replicant-11/out/soong/host/linux-x86/bin
/home/replicant/replicant-11/out/host/linux-x86/bin
/home/replicant/replicant-11/prebuilts/gcc/linux-x86/arm/arm-linux-androideabi-4.9/bin
/home/replicant/replicant-11/development/scripts
/home/replicant/replicant-11/prebuilts/devtools/tools
/home/replicant/replicant-11/external/selinux/prebuilts/bin
/home/replicant/replicant-11/prebuilts/misc/linux-x86/dtc
/home/replicant/replicant-11/prebuilts/misc/linux-x86/libufdt
/home/replicant/replicant-11/prebuilts/clang/host/linux-x86/llvm-binutils-stable
/home/replicant/replicant-11/prebuilts/android-emulator/linux-x86_64
/home/replicant/replicant-11/prebuilts/asuite/acloud/linux-x86
/home/replicant/replicant-11/prebuilts/asuite/aidegen/linux-x86
/home/replicant/replicant-11/prebuilts/asuite/atest/linux-x86
/home/replicant/bin
/home/replicant/.local/bin
/usr/local/sbin
/usr/local/bin
/usr/sbin
/usr/bin
/sbin
/bin
/usr/games

Here we can deduce that the paths set by the Android build system take precedence over the host paths.

¹ This has been deduced by prints with the following patch:

--- a/cmd/path_interposer/main.go
+++ b/cmd/path_interposer/main.go
@@ -116,6 +116,10 @@ func Main(stdout, stderr io.Writer, interposer string, args []string, opts mainO
                return 1, fmt.Errorf("Failed to set PATH env: %v", err)
        }

+       fmt.Fprintln(os.Stderr, "####################################")
+       fmt.Fprintln(os.Stderr, os.Getenv("PATH"))
+       fmt.Fprintln(os.Stderr, "####################################")
+
        if config := opts.config(base); config.Log || config.Error {
                var procs []paths.LogProcess
                if opts.lookupParents != nil {

Launching host binaries without touching the prebuilts¶

In Replicant 10, we could use out/.path/python3 as python3 interpreter assuming that we enabled it in build/soong/ui/build/paths/config.go

for instance in the Mesa (external/mesa3d) Android.mk we had:

MESA_PYTHON2 := out/.path/python2
MESA_PYTHON3 := out/.path/python3

This worked because 'python2' and 'python3' were only available in the host path:

$ for path in $(echo /home/replicant/replicant-10/prebuilts/jdk/jdk9/linux-x86/bin:/home/replicant/replicant-10/prebuilts/jdk/jdk9/linux-x86/bin:/home/replicant/replicant-10/out/soong/host/linux-x86/bin:/home/replicant/replicant-10/out/host/linux-x86/bin:/home/replicant/replicant-10/prebuilts/gcc/linux-x86/arm/arm-linux-androideabi-4.9/bin:/home/replicant/replicant-10/development/scripts:/home/replicant/replicant-10/prebuilts/devtools/tools:/home/replicant/replicant-10/external/selinux/prebuilts/bin:/home/replicant/replicant-10/prebuilts/misc/linux-x86/dtc:/home/replicant/replicant-10/prebuilts/misc/linux-x86/libufdt:/home/replicant/replicant-10/prebuilts/android-emulator/linux-x86_64:/home/replicant/replicant-10/prebuilts/asuite/acloud/linux-x86:/home/replicant/replicant-10/prebuilts/asuite/aidegen/linux-x86:/home/replicant/replicant-10/prebuilts/asuite/atest/linux-x86:/home/replicant/bin:/home/replicant/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games | sed 's#:#\n#g') ; do ls "${path}/python3" ; done
ls: cannot access '/home/replicant/replicant-10/prebuilts/jdk/jdk9/linux-x86/bin/python3': No such file or directory
ls: cannot access '/home/replicant/replicant-10/prebuilts/jdk/jdk9/linux-x86/bin/python3': No such file or directory
ls: cannot access '/home/replicant/replicant-10/out/soong/host/linux-x86/bin/python3': No such file or directory
ls: cannot access '/home/replicant/replicant-10/out/host/linux-x86/bin/python3': No such file or directory
ls: cannot access '/home/replicant/replicant-10/prebuilts/gcc/linux-x86/arm/arm-linux-androideabi-4.9/bin/python3': No such file or directory
ls: cannot access '/home/replicant/replicant-10/development/scripts/python3': No such file or directory
ls: cannot access '/home/replicant/replicant-10/prebuilts/devtools/tools/python3': No such file or directory
ls: cannot access '/home/replicant/replicant-10/external/selinux/prebuilts/bin/python3': No such file or directory
ls: cannot access '/home/replicant/replicant-10/prebuilts/misc/linux-x86/dtc/python3': No such file or directory
ls: cannot access '/home/replicant/replicant-10/prebuilts/misc/linux-x86/libufdt/python3': No such file or directory
ls: cannot access '/home/replicant/replicant-10/prebuilts/android-emulator/linux-x86_64/python3': No such file or directory
ls: cannot access '/home/replicant/replicant-10/prebuilts/asuite/acloud/linux-x86/python3': No such file or directory
ls: cannot access '/home/replicant/replicant-10/prebuilts/asuite/aidegen/linux-x86/python3': No such file or directory
ls: cannot access '/home/replicant/replicant-10/prebuilts/asuite/atest/linux-x86/python3': No such file or directory
ls: cannot access '/home/replicant/bin/python3': No such file or directory
ls: cannot access '/home/replicant/.local/bin/python3': No such file or directory
ls: cannot access '/usr/local/sbin/python3': No such file or directory
ls: cannot access '/usr/local/bin/python3': No such file or directory
ls: cannot access '/usr/sbin/python3': No such file or directory
/usr/bin/python3
ls: cannot access '/sbin/python3': No such file or directory
ls: cannot access '/bin/python3': No such file or directory
ls: cannot access '/usr/games/python3': No such file or directory

For instance prebuilts/python/linux-x86/2.7.5/bin/python wasn't in the path.

Enabling new host tools in build/soong/ui/build/paths/config.go

If the tool we want to use isn't allowed by the PathInterposer, we will also need to allow it by modifying its configuration in build/soong. Here's an example for xz:

commit b896549c85f0a87fb94c9e879a3fcf2062efad10
Author: Denis 'GNUtoo' Carikli <GNUtoo@cyberdimension.org>
Date:   Mon Nov 23 15:54:08 2020 +0100

    Enable host xz to build the kernel

    Without that fix we have:
      XZKERN  arch/arm/boot/compressed/piggy_data
      xz: Unsupported filter chain or filter options
      make[3]: ***
      [.../kernel/replicant/linux/arch/arm/boot/compressed/Makefile:184:
      arch/arm/boot/compressed/piggy_data] Error 1
      make[3]: *** Deleting file 'arch/arm/boot/compressed/piggy_data'
      make[2]: ***
      [.../kernel/replicant/linux/arch/arm/boot/Makefile:64:
      arch/arm/boot/compressed/vmlinux] Error 2
      make[1]: *** [arch/arm/Makefile:328: zImage] Error 2
      make[1]: Leaving directory
      '.../out/target/product/i9300/obj/KERNEL_OBJ'
      make: *** [Makefile:185: __sub-make] Error 2

    Signed-off-by: Denis 'GNUtoo' Carikli <GNUtoo@cyberdimension.org>

diff --git a/ui/build/paths/config.go b/ui/build/paths/config.go
index b84e6245..3a3e759b 100644
--- a/ui/build/paths/config.go
+++ b/ui/build/paths/config.go
@@ -97,6 +97,7 @@ var Configuration = map[string]PathConfig{
        "sh":      Allowed,
        "tr":      Allowed,
        "unzip":   Allowed,
+       "xz":      Allowed,
        "zip":     Allowed,

        // Host toolchain is removed. In-tree toolchain should be used instead.