Kubernetes and Buildah and microk8s

Kubernetes is awesome. But it is very annoying to be halfway through a guide just to have that guide “drop into” docker. Kubernetes _can_ use docker, but it doesn’t _need_ docker. In particular, this is true of microk8s. If you have your kubernetes cluster because of microk8s, you won’t have docker installed.

As an alternative to “docker build”, you can use buildah – “a tool that builds Open Container Initiative (OCI) container images”. All that needs done is (1) translate those “docker xxx” commands to buildah, and (2) navigate the registries so that your buildah-built image ends up somewhere that your kubectl can use it.

Dockerfile – original

FROM openjdk:11-jre-slim
WORKDIR /app
COPY target/java-hello-world-0.0.1.jar java-hello-world.jar
ENTRYPOINT ["java", "-jar", "/app/java-hello-world.jar"]
EXPOSE 8080

(Gads – a little side note here: WordPress by default allows .doc file types, but denies .txt file types “for security reasons”. It brings into doubt WordPress’s concept of ‘security’. )

Note: there are two machines here: “bld$” is the prompt of the buildah machine. “k8s$” is the prompt of the microk8s kubernetes machine.

0. Diversion – create the target/java-hello-world-0.0.1.jar file by installing java, cloning source from github. You can skip this step if you already have a java .jar file.

Diversion: install java, create java-hello-world-0.0.1.jar

bld$ sudo apt install openjdk-11-jdk-headless
bld$ git clone https://github.com/bmuschko/ckad-study-guide.git
bld$ cd ckad-study-guide/ch02/containerized-java-app
bld$ ./mvnw  package spring-boot:repackage
...snip...
bld$ ls -hl target/java-hello-world-0.0.1.jar
-rw-rw-r-- 17M Jun  8 23:53 target/java-hello-world-0.0.1.jar

1. Install buildah, fix the Dockerfile, and create the container image.

Command: install buildah and java

bld$ sudo apt-get -y install buildah

Dockerfile – fixed for buildah

FROM docker.io/openjdk:11-jre-slim
WORKDIR /app
COPY target/java-hello-world-0.0.1.jar java-hello-world.jar
ENTRYPOINT ["java", "-jar", "/app/java-hello-world.jar"]
EXPOSE 8080

Command: build with buildah

bld$ buildah bud -t java-hello-world:1.0.0 . 

-- NOTE: "buildah images" is the correct command to list images
bld$ buildah images
REPOSITORY                    TAG           IMAGE ID       CREATED          SIZE
localhost/java-hello-world    1.0.0         a24b49859451   7 weeks ago      241 MB

-- Note: "buildah list" == "buildah containers" == not the correct command
--       at this step in the process.  kept here only for historical purposes.
bld$ buildah list
CONTAINER ID  BUILDER  IMAGE ID     IMAGE NAME                       CONTAINER NAME
39d1427e4dad     *     a24b49859451 localhost/java-hello-world:1.0.0 java-hello-world-working-container

At this time, you have a OCI container image.

• (Diversion – create a registry using “buildah run”. Push the container image.)

Diversion: running a registry with buildah run

# NOTE: these steps are not required.  The goal is to get it to your microk8s registry, not to create another registry as done here:
bld$ registry=$(buildah from registry)
bld$ buildah run $registry
bld$ buildah push --tls-verify=false java-hello-world:1.0.0 docker://localhost:5000/java-hello-world/java-hello-world:1.0.0
 

2. Start microk8s registry, collect IP information

Command: Start the microk8s registry

k8s$ microk8s enable registry
The registry will be created with the default size of 20Gi.
You can use the "size" argument while enabling the registry, eg microk8s.enable registry:size=30Gi
Addon storage is already enabled.
Applying registry manifest
namespace/container-registry created
persistentvolumeclaim/registry-claim created
deployment.apps/registry created
service/registry created
configmap/local-registry-hosting configured
The registry is enabled



$ kubectl get po -A | grep registry
container-registry   registry-9b57d9df8-hp79v                     1/1     Running   0          3d19h
k8s$ kubectl get po --namespace=container-registry registry-9b57d9df8-hp79v                  -o wide
NAME                       READY   STATUS    RESTARTS   AGE     IP             NODE                NOMINATED NODE   READINESS GATES
registry-9b57d9df8-hp79v   1/1     Running   0          3d19h   10.1.119.169   ub2004microk8stwo              
k8s$ echo 10.1.119.169 is not "externally available"
k8s$ kubectl get no -o wide | grep ub2004microk8stwo
ub2004microk8stwo     Ready       211d   v1.19.10-34+8f8eec7c3f1428   10.0.1.80             Ubuntu 20.04 LTS   5.4.0-72-generic   containerd://1.3.7
k8s$ REGISTRY=10.0.1.80:32000
 

Command: Push from buildah machine to microk8s registry

bld$ REGISTRY=10.0.1.80:32000
bld$ buildah push --tls-verify=false java-hello-world:1.0.0 ${REGISTRY}/java-hello-world:1.0.0
 

Command: Confirm the image is in the registry

k8s$ REGISTRY=10.0.1.80:32000
k8s$ curl http://$REGISTRY/v2/_catalog
{"repositories":["java-hello-world"]}
k8s$ curl http://$REGISTRY/v2/java-hello-world/tags/list
{"name":"java-hello-world","tags":["1.0.0"]}

deployment.yaml – using minik8s registry as image source

apiVersion: apps/v1
kind: Deployment
metadata:
  name: exported-java-deployment
  labels:
    app: exportedjava
spec:
  selector:
    matchLabels:
      app: exportedjava
  template:
    metadata:
      labels:
        app: exportedjava
    spec:
      containers:
      - name: java-hello-world
        image: localhost:32000/java-hello-world:1.0.0      
        imagePullPolicy: Never
        ports:
        - containerPort: 8080

Command: Apply the deployment

k8s$ kubectl apply -f deployment.yaml 

Command: Test deployment

k8s$ kubectl get po -o wide
NAME                                        READY   STATUS    RESTARTS   AGE    IP             NODE                NOMINATED NODE   READINESS GATES
exported-java-deployment-54cbc44997-ttgxc   1/1     Running   0          103s   10.1.119.149   ub2004microk8stwo              
k8s$ curl http://10.1.119.149:8080
Hello World!

(Miscellaneous commands – query running containers for image names and sizes)

Command: Show running images – names and sizes

k8s$ kubectl get nodes -o json | jq '.items[].status.images[] | .names[1], .sizeBytes'
"docker.io/bitnami/mongodb:4.4.2-debian-10-r27"
180259295
"docker.io/kubernetesui/dashboard:v2.0.0"
66209190
"docker.io/coredns/coredns:1.6.6"
12932169
"k8s.gcr.io/pause:3.1"
317164
"localhost:32000/java-hello-world:1.0.0"
97008161

Command: Alternative way to copy/import image

-- FAILS, wrong way to "push"/export the image
bld$ buildah push 1bb7f7cb0d2c dir:/tmp/1bb7f7cb0d2c
Getting image source signatures
Copying blob 40093787e10f done  
Copying blob e1929b65aa97 done  
Copying blob 1b739069a094 done  
Copying blob 2655e3fdba88 done  
Copying blob d32026d4252e done  
Copying blob c76dc6af9411 done  
Copying blob 89efdf5cc8ae done  
Copying blob d9b44548153b done  
Copying blob 8f2b297d408f done  
Copying blob 89f7fb50be94 done  
Copying config 1bb7f7cb0d done  
Writing manifest to image destination
Storing signatures
$bld tar cvf 1bb7f7cb0d2c.tar ./1bb7f7cb0d2c/
$bld scp 1bb7f7cb0d2c.tar k8s:~

k8s$ mkcrok8s ctr image import 1bb7f7cb0d2c.tar
ctr: unrecognized image format
--   Fail.

-- Correct way to 'buildah push", but incorrect way to create .tar file, FAILS:
bld$ buildah push 1bb7f7cb0d2c oci:/tmp/oci-path:localhost/mysimpleapi:1.0.0
bld$ cd /tmp
bld$ tar cvf oci-path.tar  ./oci-path
k8s$ microk8s ctr image import oci-path.tar
ctr: unrecognized image format

-- Correct way to create the .tar file:  WORKS:
bld$ cd /tmp
bld$ cd oci-path
bld$ tar cvf ../mysimpleapi.tar .
bld$ scp
k8s$ microk8s ctr image import mysimpleapi.tar
unpacking localhost/mysimpleapi:1.0.0 (sha256:xxxxxx)...done
k8s$ microk8s ctr image list -q | grep mysimpleapi
localhost/mysimpleapi:1.0.0