Kubernetes and Buildah and microk8s

Kubernetes is awesome. But it is very annoying to be halfway through a guide just to have that guide “drop into” docker. Kubernetes _can_ use docker, but it doesn’t _need_ docker. In particular, this is true of microk8s. If you have your kubernetes cluster because of microk8s, you won’t have docker installed.

As an alternative to “docker build”, you can use buildah – “a tool that builds Open Container Initiative (OCI) container images”. All that needs done is (1) translate those “docker xxx” commands to buildah, and (2) navigate the registries so that your buildah-built image ends up somewhere that your kubectl can use it.

Dockerfile – original
FROM openjdk:11-jre-slim
COPY target/java-hello-world-0.0.1.jar java-hello-world.jar
ENTRYPOINT ["java", "-jar", "/app/java-hello-world.jar"]

(Gads – a little side note here: WordPress by default allows .doc file types, but denies .txt file types “for security reasons”. It brings into doubt WordPress’s concept of ‘security’. )

Note: there are two machines here: “bld$” is the prompt of the buildah machine. “k8s$” is the prompt of the microk8s kubernetes machine.

0. Diversion – create the target/java-hello-world-0.0.1.jar file by installing java, cloning source from github. You can skip this step if you already have a java .jar file.

Diversion: install java, create java-hello-world-0.0.1.jar
bld$ sudo apt install openjdk-11-jdk-headless
bld$ git clone https://github.com/bmuschko/ckad-study-guide.git
bld$ cd ckad-study-guide/ch02/containerized-java-app
bld$ ./mvnw  package spring-boot:repackage
bld$ ls -hl target/java-hello-world-0.0.1.jar
-rw-rw-r-- 17M Jun  8 23:53 target/java-hello-world-0.0.1.jar

1. Install buildah, fix the Dockerfile, and create the container image.

Command: install buildah and java
bld$ sudo apt-get -y install buildah
Dockerfile – fixed for buildah
FROM docker.io/openjdk:11-jre-slim
COPY target/java-hello-world-0.0.1.jar java-hello-world.jar
ENTRYPOINT ["java", "-jar", "/app/java-hello-world.jar"]
Command: build with buildah
bld$ buildah bud -t java-hello-world:1.0.0 . 

-- NOTE: "buildah images" is the correct command to list images
bld$ buildah images
REPOSITORY                    TAG           IMAGE ID       CREATED          SIZE
localhost/java-hello-world    1.0.0         a24b49859451   7 weeks ago      241 MB

-- Note: "buildah list" == "buildah containers" == not the correct command
--       at this step in the process.  kept here only for historical purposes.
bld$ buildah list
39d1427e4dad     *     a24b49859451 localhost/java-hello-world:1.0.0 java-hello-world-working-container

At this time, you have a OCI container image.

  • (Diversion – create a registry using “buildah run”. Push the container image.)
Diversion: running a registry with buildah run
# NOTE: these steps are not required.  The goal is to get it to your microk8s registry, not to create another registry as done here:
bld$ registry=$(buildah from registry)
bld$ buildah run $registry
bld$ buildah push --tls-verify=false java-hello-world:1.0.0 docker://localhost:5000/java-hello-world/java-hello-world:1.0.0

2. Start microk8s registry, collect IP information

Command: Start the microk8s registry
k8s$ microk8s enable registry
The registry will be created with the default size of 20Gi.
You can use the "size" argument while enabling the registry, eg microk8s.enable registry:size=30Gi
Addon storage is already enabled.
Applying registry manifest
namespace/container-registry created
persistentvolumeclaim/registry-claim created
deployment.apps/registry created
service/registry created
configmap/local-registry-hosting configured
The registry is enabled

$ kubectl get po -A | grep registry
container-registry   registry-9b57d9df8-hp79v                     1/1     Running   0          3d19h
k8s$ kubectl get po --namespace=container-registry registry-9b57d9df8-hp79v                  -o wide
NAME                       READY   STATUS    RESTARTS   AGE     IP             NODE                NOMINATED NODE   READINESS GATES
registry-9b57d9df8-hp79v   1/1     Running   0          3d19h   ub2004microk8stwo              
k8s$ echo is not "externally available"
k8s$ kubectl get no -o wide | grep ub2004microk8stwo
ub2004microk8stwo     Ready       211d   v1.19.10-34+8f8eec7c3f1428             Ubuntu 20.04 LTS   5.4.0-72-generic   containerd://1.3.7
Command: Push from buildah machine to microk8s registry
bld$ buildah push --tls-verify=false java-hello-world:1.0.0 ${REGISTRY}/java-hello-world:1.0.0
Command: Confirm the image is in the registry
k8s$ curl http://$REGISTRY/v2/_catalog
k8s$ curl http://$REGISTRY/v2/java-hello-world/tags/list

deployment.yaml – using minik8s registry as image source
apiVersion: apps/v1
kind: Deployment
  name: exported-java-deployment
    app: exportedjava
      app: exportedjava
        app: exportedjava
      - name: java-hello-world
        image: localhost:32000/java-hello-world:1.0.0      
        imagePullPolicy: Never
        - containerPort: 8080
Command: Apply the deployment
k8s$ kubectl apply -f deployment.yaml 
Command: Test deployment
k8s$ kubectl get po -o wide
NAME                                        READY   STATUS    RESTARTS   AGE    IP             NODE                NOMINATED NODE   READINESS GATES
exported-java-deployment-54cbc44997-ttgxc   1/1     Running   0          103s   ub2004microk8stwo              
k8s$ curl
Hello World!

(Miscellaneous commands – query running containers for image names and sizes)

Command: Show running images – names and sizes
k8s$ kubectl get nodes -o json | jq '.items[].status.images[] | .names[1], .sizeBytes'
Command: Alternative way to copy/import image

-- FAILS, wrong way to "push"/export the image
bld$ buildah push 1bb7f7cb0d2c dir:/tmp/1bb7f7cb0d2c
Getting image source signatures
Copying blob 40093787e10f done  
Copying blob e1929b65aa97 done  
Copying blob 1b739069a094 done  
Copying blob 2655e3fdba88 done  
Copying blob d32026d4252e done  
Copying blob c76dc6af9411 done  
Copying blob 89efdf5cc8ae done  
Copying blob d9b44548153b done  
Copying blob 8f2b297d408f done  
Copying blob 89f7fb50be94 done  
Copying config 1bb7f7cb0d done  
Writing manifest to image destination
Storing signatures
$bld tar cvf 1bb7f7cb0d2c.tar ./1bb7f7cb0d2c/
$bld scp 1bb7f7cb0d2c.tar k8s:~

k8s$ mkcrok8s ctr image import 1bb7f7cb0d2c.tar
ctr: unrecognized image format
--   Fail.

-- Correct way to 'buildah push", but incorrect way to create .tar file, FAILS:
bld$ buildah push 1bb7f7cb0d2c oci:/tmp/oci-path:localhost/mysimpleapi:1.0.0
bld$ cd /tmp
bld$ tar cvf oci-path.tar  ./oci-path
k8s$ microk8s ctr image import oci-path.tar
ctr: unrecognized image format

-- Correct way to create the .tar file:  WORKS:
bld$ cd /tmp
bld$ cd oci-path
bld$ tar cvf ../mysimpleapi.tar .
bld$ scp
k8s$ microk8s ctr image import mysimpleapi.tar
unpacking localhost/mysimpleapi:1.0.0 (sha256:xxxxxx)...done
k8s$ microk8s ctr image list -q | grep mysimpleapi

Posted in Software Engineering | Comments Off on Kubernetes and Buildah and microk8s

Naming and Security

Came across an interesting configuration file pattern today:

      insecure: true
      insecure: true
      insecure: true
      insecure: true
      verify_ssl: false

The interesting thing is that 4 “subprojects” choose the key “insecure”, yet the barbican subproject choose the correct key “verify_ssl”. The flag in question here does exactly what the barbican configuration says: it still uses SSL (hence, it is still secure), but the verify (host) check has been disabled. The others incorrectly imply that all security is lost, which is not the case. They still use encrypted communications. The key “insecure” is would only be correct if that configuration item would switch between “http:” and “https:” connections. None of the configuration items here do that – when set to “false”, the only difference is that the server’s certificate will not be validated.

The fun thing to consider here is the various cultures that exist in projects, teams and countries. If OpenStack operated as a “Cancel Culture”, the “verify_ssl” would be cancelled for showing the courage to be (a) different and (b) correct. Because ‘Cancel Culture’ emphasizes agreement over accuracy, consensus over truth, perception over reality.

Posted in Software Engineering | Comments Off on Naming and Security

WordPress Comment Spam

A quick note on WordPress spam in comments – Starting 2020 November 26th and continuing until 2020 December 17th, a spammer created a bunch of comments on the one page that had accidentally allowed comments – “About”

In total, there were 33,389 spam comments created. This was caused by (1) migrating this blog from the previous (amateurish) hosting company, and (2) WordPress creating one post by default with “Comments” still active. It was easy enough to stop new comments (edit Post, Discussion, uncheck “Allow Comments”). To remove 33,389 comments, I used the plugin “Delete All Comments of wordpress” by Navneet Soni.

All of the comments had IP addresses from a very small block (5.188.211.x):, count=2997, count=3346, count=3004, count=2975, count=2985, count=3326, count=3007, count=3070, count=2990, count=3074, count=2615

Whois result:

% Information related to ' -'

% Abuse contact for ' -' is 'abuse@pindc.ru'

inetnum: -
netname:        inf-net
country:        RU
admin-c:        MK19775-RIPE
tech-c:         MK19775-RIPE
status:         ASSIGNED PA
mnt-by:         MNT-PINSUPPORT
created:        2017-04-13T10:32:21Z
last-modified:  2017-04-13T10:32:21Z
source:         RIPE

person:         Makary Kwiatkowski
address:        ul. Zachodnia 20
address:        Bialystok 15-345
phone:          +48695689091
nic-hdl:        MK19775-RIPE
mnt-by:         MNT-PINSUPPORT
created:        2016-10-26T21:03:01Z
last-modified:  2016-10-26T21:14:53Z
source:         RIPE # Filtered

% Information related to ''
descr:          PIN DC
origin:         AS34665
mnt-by:         MNT-PIN
mnt-by:         MNT-PINSUPPORT
created:        2019-11-11T07:41:06Z
last-modified:  2019-11-11T07:41:06Z
source:         RIPE

Posted in Uncategorized | Comments Off on WordPress Comment Spam

RetroPie Raspberry Pi 4

Item Product Cost
Pi 4 CanaKit Raspberry Pi 4 Basic Kit 1.5GHz 64-bit quad core ARMv8, USB-C 3.5A Power, USB-C Power switch $55
MicroSD SanDisk 64GB Ultra microSDXC USH-I $12
Case RETROFLAG NESPi 4 Case with SSD Case, USB-C Power Supply, HDMI Adapter cooling Fan and heatsinks $40
HDMI Adapter Micro HDMI to HDMI Adapter 6.5ft $8
Controller Rii Game Controller, SNES Retro USB (x2) $9
Controller 8Bitdo SN30 Pro USB Gamepad, Wired $25
Tweezers Refine Tweezers Slant Tip, 3 count $5
SSD 500GB Samsung 860 EVO NZ076E500B/AM $70
OS RetroPie 4.6 for Raspberry Pi 4 and Retroflag-picase Safe Shutdown
OS NOOBS v3.5.0, 2329 MB
Total $224


  • Needed tweezers because the case only exposes 1/16″ inch of the microSD card
  • On this case, use the microHDMI furthest from the power. If you use the other port, you may not get video, and you definitely will not get audio-over-HDMI.
  • Started with the SNES controllers. These are authentic-looking, but they lack a bunch of buttons, including the all-important “Hotkey”, so upgraded to the 8Bitdo
  • The case has a “Safe Shutdown” switch. Use the fork (crcerror), not the original, github project code. After turning it off, changing the switch to “On”, booting, then installing the software and rebooting, the power/reset buttons change. The “Reset” button on the case now acts to (1) quit the game, (2) quit the emulator, and (3) restart Emulationstation itself. And now, the “Power” button will call “shutdown” instead of just turning off the power. The “power led” behavior changes a bit too – there is a delay on both power on and power off.
  • The case came with a small microHDMI adapter, but it was awkward, so purchased the microHDMI cable.
  • Ended up with two USB-C power supplies, because the PI case came with one too. CanaKit provied a 3.5W, the case provided a 3.0W
  • For the better controller, the triggers won’t register out-of-the-box. You have to compile the xboxdrv package (from RetroPie-Config, Manage Packages >> Manage Driver Packages >> xboxdrv). On a Raspberry Pi 4, this takes “a long time” (7 minutes). Remember to enable the driver as well.
Posted in Computer Builds | Comments Off on RetroPie Raspberry Pi 4

FreeNas 11 ZFS Replace Failed Drive

This documents the steps to replace a failed drive in a mirror configuration in FreeNAS 11.1-U7. This process is really easy with FreeNAS.

This scenario is in contrast to replacing a failed drive before it fails (in a Centos “ad-hoc” ZFS system).

(Note: at this time, FreeNAS 11.3-U4.1 is available. It has a different GUI from the description below.)

First, confirm FreeNAS agrees that the drive is dead. Your alert in the upper right corner will be showing red “Critical”.

Second, physically replace the drive with an equal-or-larger sized drive.

Third, from the FreeNAS 11 GUI:

  1. Click “Storage”
  2. Click on the pool name row (first row, not the second row)
  3. At the bottom, click “Volume Status”
  4. Click on the row below the “mirror-0” that says “UNAVAIL”

    note: the failed disk will probably have a number, like 11421970349345372421, instead of a device, like ada4p2

    after you do this, a “Replace” button appears at the bottom of the screen

  5. Click on “Replace”

    after you do this, a dialog “Replacing disk 11421970349345372421”

    The drop-down dialog will be pre-populated with the newly-inserted disk, by device name, like ada5 (X.Y TB)

  6. Push “Replace Disk”

    after you do this, the dialog changes to “Please wait…”

    after about 60 seconds, the dialog closes, and returns to your pool status

At this point, all of the disks should show “ONLINE”, and the top status changes to “Resilver”.
Eventually, a “Progress: NN.nn%” will start showing the resilvering status of the drive.
You can also run “zpool status” to see the resilver progress.

Extra tidbit: the 4TB drive resilver (2.7TB used) took 8h35m, which is 91MB/second.

Posted in Storage, ZFS | Comments Off on FreeNas 11 ZFS Replace Failed Drive

Crossword Puzzle Data Format

The biggest crisis in computer science today is the lack of a good crossword puzzle data format.

There are some existing formats:

There are also some terrible formats:

  • Across Lite
  • – BINARY format (!!) with “older TEXT version”. Enough said. The TEXT version does have no redundancy, however.

  • XwordInfo JSON – doesn’t even have a real name, and doesn’t deserve one. The format is full of redundancy, and is difficult to use. Example .json.

Special purpose formats:

Posted in Software Engineering | Comments Off on Crossword Puzzle Data Format

jq pattern for terrible JSON

Many JSON formats are completely brain dead. Instead of the natural { key : value }, these formats go “all meta’, using { Field: “name”, Value: “amateur” } or { “name” : “name”, “value”: “amateur” }. This “meta key-value” approach is a stupid format that doesn’t provide any extra extensibility, yet completely destroys the concept of a useful schema.

This poor design also complicates json query (jq) (see cookbook too)

The recipe to extract the value you want:

$ jq -r '.[] | select( .Field == "properties").Value' output.json
this is a terrible design

With this input file:

$ cat output.json
    "Field": "description",
    "Value": "human description of a terrible design"
    "Field": "enabled",
    "Value": true
    "Field": "id",
    "Value": "fa722ed8b4f56d14bcf77537"
    "Field": "name",
    "Value": "your-name"
    "Field": "properties",
    "Value": "this is a terrible design"
Posted in Software Engineering | Comments Off on jq pattern for terrible JSON

GraphQL with curl examples

GraphQL curl command examples, showing you both the curl command and the graphql schema.

Project link: graphql-java-codegen-gradle-plugin

This is documentation for the combination of:

  1. A realistic graphql schema
  2. An actual Java server that implements that graphql schema
  3. Using curl to issue graphql commands to that server

It is amazing difficult to find all three of these together in one place.


  1. clone git repository https://github.com/kobylynskyi/graphql-java-codegen-gradle-plugin
  2. cd graphql-java-codegen-gradle-plugin/graphql-codegen-gradle-plugin-example
  3. echo > settings.gradle
  4. gradle build
  5. start mongodb on port 27017 on localhost
  6. gradle run
    — this starts a server on port localhost:8080

Then, in a different window, run these sample curl commands:

* Shows syntax for types: String, Int, BigDecimal, and enum
* Only outputs ‘size’ because that is what the query asked to get
* Shows that “mutation” is still a “query”
* Shows the correct quotation mark escaping, and where quotes are required and where they are not required

curl  \
  -X POST \
  -H "Content-Type: application/json" \
  -d '{ "query":  "mutation {  newBike(bike : { type: ROAD, brand: \"foo\", size: \"big\", year: 2000, price: 123 }) { size } }" }' \


  "data": {
    "newBike": {

* shows formatting of “DateTime” field

curl  \
  -X POST \
  -H "Content-Type: application/json" \
  -d '{ "query":  "query {  bikes { id type brand size year price addedDateTime} }"  }' \


  "data": {
    "bikes": [
        "id": "5e28ac4d64f0e8088f8bce47",
        "type": "ROAD",
        "brand": "foo",
        "size": "big",
        "year": 2000,
        "price": 123,
        "addedDateTime": "2020-01-22 14:10:53 -0600"
Posted in Software Engineering | Comments Off on GraphQL with curl examples

AMD Ryzen Build

Some facts on the CPU: it is currently #135 on PassMark [16,9499] cpubenchmark.net.

The video card was the least expensive PCI available at Micro Center.

This build is intended for two distinct purposes. The first is a general-purpose, install linux, install multiple linux, experiment machine. The second is as a Windows gaming machine, which will require a “real” GPU and Windows 10 install.

Note: The AMD CPU came with a coupon code for a free game. However, redeeming the code requires “AMD Verification Tool”, and that tool is windows-only. So, don’t fall for AMD if you want to use linux. Probably, just don’t fall for AMD, period.

Item Product Cost
CPU AMD Ryzen 7 2700x 3.7Ghz 8 Core, 16 threads AM4 (4.3Ghz turbo) AM4, 105W $159
Cooler Wraith Prism Cooler
RAM G.Skill Aegis 16GB (2 x 8GB) DDR4 2400 (PC4 19200) CL17 F4-2400C17D-16G – Black/Red $68
Motherboard Gigabyte B450M Aorus AM4 mATX Motherboard $40
Power Supply Apea 500W reuse
Video EVGA GeForce 8400GS 1GB Single-Fan DDR3 $42
Video Gigabyte Radeon RX 570 4GG 256-bit GDDR4 x16 $115
Case NZXT GAMMA Classic ATX Mid Tower reuse $31
M.2 Drive Intel 660p Series 1TB – NVMe 3.0 x4 3D2, QLC SSD (SSDPEKNW010T8X1) $83
SS Drive None
HD Drive None
4-to-8 pin adapter Coboc EPSADLP48-6 6in ATX 12V P4 4-Pin LP4 to EPS 12V 8-pin $2
OS Ubuntu to start, then Windows 10 Pro 64-bit OEM
Total $394
Tax 7.529% $24
Grand Total $418

tax newegg 7.526%
tax microcenter 7.529%

Posted in Computer Builds | Comments Off on AMD Ryzen Build

ZFS replace drive before it fails

The main Virtual Machine Server was seeing hardware failures and ZFS “scrub not zero bytes”. One of the (cheap) Hitachi Ultrastar 2TB disks was starting to fail, after only 1.5 years. Smartctl was showing 53 recent errors.

  pool: vmstorage
 state: ONLINE
  scan: scrub repaired 1.75M in 5h12m with 0 errors on Tue Oct  1 07:12:22 2019

        NAME                                            STATE     READ WRITE CKSUM
        vmstorage                                       ONLINE       0     0     0
          mirror-0                                      ONLINE       0     0     0
            ata-Hitachi_HUA723020ALA641_YFG31Y3A-part2  ONLINE       0     0     0
            ata-Hitachi_HUA723020ALA641_YFG4GJ8A-part2  ONLINE       0     0     0

/var/log/messages was showing stuff like this, over and over:

Sep 15 03:03:42 dellt3600 smartd[19459]: Device: /dev/sdc [SAT], 831 Currently unreadable (pending) sectors

Looking in /dev/disk/by-id/*, the failing drive has a serial number of YFG4GJ8A.

So, the setup for these commands became:

export DISK_GOOD=/dev/disk/by-id/ata-Hitachi_HUA723020ALA641_YFG31Y3A-part2
export DISK_BAD=/dev/disk/by-id/ata-Hitachi_HUA723020ALA641_YFG4GJ8A-part2
export DISK_REPLACE=/dev/disk/by-id/ata-Hitachi_HUA723020ALA641_YGJ0JSYA-part2

The command to remove the failing, but not yet failed, drive from the mirror:

zpool detach vmstorage $DISK_BAD

(At this point, I shutdown the machine, and had to swap disks since there was only room for two 3.5″ HDDs).

After reboot, the command to add the new disk into the mirror:

zpool attach vmstorage $DISK_GOOD $DISK_REPLACE

Resilvering 1.25TB took 4h56m.

Note: if you “pre-partition” your ZFS disks (like I do), then you also need the “root” disk to run parted:

export DISK_REPLACE_ROOT=/dev/disk/by-id/ata-Hitachi_HUA723020ALA641_YGJ0JSYA

Use ‘unit s’ to create the partitions with exactly the same sector counts as the drive being replaced.

Just recording the replacement drive: $42 – HGST/Hitachi Ultrastar 7K3000 2TB 7200RPM Enterprise Grade Sata III For the record – the drive arrived new, with 0 hours power-on time. Vendor was DBSKY.

Posted in ZFS | Comments Off on ZFS replace drive before it fails