Building Docker images for Kubernetes running on ARM

December 4, 2019

Nearly every Docker image you’ve ever run on Kubernetes will not work on your Homelab Raspberry Pi cluster. Why? What do we need to do? This article introduces the “docker buildx” plugin to make it easy to produce mult-arch Docker images.

I’ve started a new home lab of Raspberry Pis (and soon to include some old Intel laptops) running k3s distribution of Kubernetes. In my first iteration, I have one Pi running Kubernetes master, and two Pis as workers only:

$ kubectl get nodes -o wideNAME                     STATUS     ROLES    OS-IMAGE
raspberrypi3-red         NotReady   <none>   Raspbian GNU/Linux 10 (buster)
raspberrypi3-white       Ready      <none>   Raspbian GNU/Linux 10 (buster)
raspberrypi3-lightblue   Ready      master   Raspbian GNU/Linux 10 (buster)

The joy of seeing my new cluster running was soon doused with a fun-retardant called CreateContainerError:

$ kubctl get pod -n kube-system
NAME                                      READY   STATUS                 RESTARTS   AGE
tiller-deploy-585d9c49b6-v646t            0/1     CreateContainerError   6          6h25m

What is wrong here?

Docker images will “run anywhere” only if the image was built for the same computer architecture. This was never a problem whilst our local computer, our CI system, and our production system all used the same architecture: the famous “Intel Inside” x86 and clone-makers like AMD.

$ uname -m
x86_64

What is my Raspberry Pi’s architecture?

$ kubectl run -i --rm --image busybox whats-my-arch --restart Never -- uname -m
armv7l

Are there Docker images that support multiple architectures?

This is a solved problem. For example, the Kubernetes project publishes Docker images that are built for multiple architectures:

$ docker container run mplatform/mquery gcr.io/google-containers/kube-apiserver:v1.16.3
...
Image: gcr.io/google-containers/kube-apiserver:v1.16.3
 * Manifest List: Yes
 * Supported platforms:
   - linux/amd64
   - linux/arm
   - linux/arm64
   - linux/ppc64le
   - linux/s390x

The support for multiple architectures in the gcr.io/google-containers/kube-apiserver:v1.16.3 image is actually a collection of architecture-specific images. When your container runtime system (containerd, docker daemon, etc) needs to pull down an image, it specifically asks for the one for its architecture.

Alas, our nemesis from above, the Helm Tiller image, does not:

$ docker container run mplatform/mquery gcr.io/kubernetes-helm/tiller:v2.16.1
Image: gcr.io/kubernetes-helm/tiller:v2.16.1
 * Manifest List: No
 * Supports: amd64/linux

The Helm v2 Tiller project never released Docker images for any flavour of ARM processor. Whilst this is no longer an issue since Helm v3.0.0 no longer includes the server-side Tiller project, it was my first contact with the challenges of bringing software into my Raspberry Pi/Kubernetes cluster, and as such I shall use it as my example.

It also means this anti-pattern, single-architecture Tiller Docker image will never change and this blog post will never need updating with another example.

Creating multi-arch Docker images

The specification for pushing multi-arch images to a registry is well defined, and a Docker blog post introduces it nicely.

For us to create our own multi-arch Docker images we will first need multi-arch base images. One place to consider is the Docker Official images, which have all been published as multi-arch since Sept 2017.

For example, the golang:alpine image supports our required ARM v7 architecture:

Let’s use this example official image to create a set of new images – one per architecture platform.

Consider a Dockerfile:

FROM golang:alpine AS build
ARG TARGETPLATFORM
ARG BUILDPLATFORM
RUN echo "I was built on a platform running on $BUILDPLATFORM, building this image for $TARGETPLATFORM" > /log
FROM alpine
COPY --from=build /log /log
CMD [ "cat", "/log" ]

Next, I need to enable the experimental features of Docker. On Docker for Mac, it can be found under Preferences > Daemon > Basic.

If I try to use docker buildx on our Mac now I still get an error:

$ docker buildx build --platform linux/amd64,linux/arm/v7 .
Multiple platforms feature is currently not supported for docker driver.
Please switch to a different driver (eg. "docker buildx create --use")

I needed to create an isolated Builder Instance that supported multiple platforms:

$ docker buildx create --use
inspiring_benz
$ docker buildx ls
NAME/NODE DRIVER/ENDPOINT STATUS  PLATFORMS
NAME/NODE         DRIVER/ENDPOINT             STATUS   PLATFORMS
inspiring_benz *  docker-container
  inspiring_benz0 unix:///var/run/docker.sock inactive
default           docker
  default         default                     running  linux/amd64, linux/arm64, linux/ppc64le, linux/s390x, linux/386, linux/arm/v7, linux/arm/v6

I can now build this Dockerfile for multiple platform architectures:

docker buildx build \
  --platform linux/amd64,linux/arm64,linux/arm/v7,linux/arm/v6 \
  --tag starkandwayne/hello-multiarch:cat-log \
  --push \
  .

I use --push to push the collection of Docker images to a registry. Why? The Docker Daemon used for docker buildx build is not the same one we have access to from Docker for Mac. In order for us to subsequently use the linux/amd64 version locally on Docker for Mac, we need to push it to a registry, and pull it back to the normal Docker Daemon. We also need to push it to a registry in order to run it upon our Raspberry Pi k3s kubernetes cluster in a minute.

I can confirm that we’ve built and published our image for multiple platform architectures:

$ docker container run mplatform/mquery starkandwayne/hello-multiarch:cat-log
Image: starkandwayne/hello-multiarch
 * Manifest List: Yes
 * Supported platforms:
   - linux/amd64
   - linux/arm64
   - linux/arm/v7
   - linux/arm/v6

Let’s try out our amd64/x86 image locally and confirm we get the linux/amd64 architecture image:

$ docker pull starkandwayne/hello-multiarch:cat-log
$ docker run -ti starkandwayne/hello-multiarch:cat-log
I was built on a platform running on linux/amd64, building this image for linux/amd64

Finally, I can run my Docker image on our Raspberry Pi Kubernetes:

$ kubectl run -i --image starkandwayne/hello-multiarch:cat-log --rm --restart=Never hello
I was built on a platform running on linux/amd64, building this image for linux/arm/v7

Build a Golang application for multi-arch images

Let’s do this once more, but with a Golang application compiled within a golang:alpine build image, and copied across to the final alpine image.

Consider a tiny Golang app main.go and a multi-stage Dockerfile :

package main
import (
	"fmt"
	"runtime"
)
func main() {
	fmt.Printf("Hello from %s architecture\n", runtime.GOARCH)
}

FROM golang:alpine AS build
ADD . /buildspace
WORKDIR /buildspace
RUN go build -o buildx-demo ./...
FROM alpine AS app
COPY --from=build /buildspace/buildx-demo /usr/bin/buildx-demo
CMD [ "/usr/bin/buildx-demo" ]

As above, we run docker buildx build and select many --platform targets:

$ docker buildx build \
  --platform linux/amd64,linux/arm64,linux/arm/v7,linux/arm/v6 \
  --tag starkandwayne/hello-multiarch:golang \
  --push .

I can now run the image on my Docker for Mac x86/amd64 Docker daemon:

$ docker run -ti starkandwayne/hello-multiarch:golang
Hello from amd64 architecture

I can run it on an x64/amd64 Kubernetes platform:

$ kubectl run -i --image starkandwayne/hello-multiarch:golang --rm --restart=Never hello
Hello from amd64 architecture

Finally, and with all the joy that comes from a home computing lab, I can run the image on my Raspberry Pi/ARM Kubernetes platform:

$ kubectl run -i --image starkandwayne/hello-multiarch:golang --rm --restart=Never hello
Hello from linux/arm/v7 architecture

Thanks and links

There is a great range of documentation and blog posts already existing. I recommend you check out the following helpful posts that made it possible for me to figure out multiarch Docker images using docker buildx:

Written by:
Ashley Gerwitz

Marketing Manager at Stark & Wayne and Qarik