FreeNas 11 ZFS Replace Failed Drive

This documents the steps to replace a failed drive in a mirror configuration in FreeNAS 11.1-U7. This process is really easy with FreeNAS.

This scenario is in contrast to replacing a failed drive before it fails (in a Centos “ad-hoc” ZFS system).

(Note: at this time, FreeNAS 11.3-U4.1 is available. It has a different GUI from the description below.)

First, confirm FreeNAS agrees that the drive is dead. Your alert in the upper right corner will be showing red “Critical”.

Second, physically replace the drive with an equal-or-larger sized drive.

Third, from the FreeNAS 11 GUI:

  1. Click “Storage”
  2. Click on the pool name row (first row, not the second row)
  3. At the bottom, click “Volume Status”
  4. Click on the row below the “mirror-0” that says “UNAVAIL”

    note: the failed disk will probably have a number, like 11421970349345372421, instead of a device, like ada4p2

    after you do this, a “Replace” button appears at the bottom of the screen

  5. Click on “Replace”

    after you do this, a dialog “Replacing disk 11421970349345372421”

    The drop-down dialog will be pre-populated with the newly-inserted disk, by device name, like ada5 (X.Y TB)

  6. Push “Replace Disk”

    after you do this, the dialog changes to “Please wait…”

    after about 60 seconds, the dialog closes, and returns to your pool status

At this point, all of the disks should show “ONLINE”, and the top status changes to “Resilver”.
Eventually, a “Progress: NN.nn%” will start showing the resilvering status of the drive.
You can also run “zpool status” to see the resilver progress.

Extra tidbit: the 4TB drive resilver (2.7TB used) took 8h35m, which is 91MB/second.

Posted in Storage, ZFS | Comments Off on FreeNas 11 ZFS Replace Failed Drive

Crossword Puzzle Data Format

The biggest crisis in computer science today is the lack of a good crossword puzzle data format.

There are some existing formats:

There are also some terrible formats:

  • Across Lite
  • – BINARY format (!!) with “older TEXT version”. Enough said. The TEXT version does have no redundancy, however.

  • XwordInfo JSON – doesn’t even have a real name, and doesn’t deserve one. The format is full of redundancy, and is difficult to use. Example .json.

Special purpose formats:

Posted in Software Engineering | Comments Off on Crossword Puzzle Data Format

jq pattern for terrible JSON

Many JSON formats are completely brain dead. Instead of the natural { key : value }, these formats go “all meta’, using { Field: “name”, Value: “amateur” } or { “name” : “name”, “value”: “amateur” }. This “meta key-value” approach is a stupid format that doesn’t provide any extra extensibility, yet completely destroys the concept of a useful schema.

This poor design also complicates json query (jq) (see cookbook too)

The recipe to extract the value you want:

$ jq -r '.[] | select( .Field == "properties").Value' output.json
this is a terrible design

With this input file:

$ cat output.json
    "Field": "description",
    "Value": "human description of a terrible design"
    "Field": "enabled",
    "Value": true
    "Field": "id",
    "Value": "fa722ed8b4f56d14bcf77537"
    "Field": "name",
    "Value": "your-name"
    "Field": "properties",
    "Value": "this is a terrible design"
Posted in Software Engineering | Comments Off on jq pattern for terrible JSON

GraphQL with curl examples

GraphQL curl command examples, showing you both the curl command and the graphql schema.

Project link: graphql-java-codegen-gradle-plugin

This is documentation for the combination of:

  1. A realistic graphql schema
  2. An actual Java server that implements that graphql schema
  3. Using curl to issue graphql commands to that server

It is amazing difficult to find all three of these together in one place.


  1. clone git repository
  2. cd graphql-java-codegen-gradle-plugin/graphql-codegen-gradle-plugin-example
  3. echo > settings.gradle
  4. gradle build
  5. start mongodb on port 27017 on localhost
  6. gradle run
    — this starts a server on port localhost:8080

Then, in a different window, run these sample curl commands:

* Shows syntax for types: String, Int, BigDecimal, and enum
* Only outputs ‘size’ because that is what the query asked to get
* Shows that “mutation” is still a “query”
* Shows the correct quotation mark escaping, and where quotes are required and where they are not required

curl  \
  -X POST \
  -H "Content-Type: application/json" \
  -d '{ "query":  "mutation {  newBike(bike : { type: ROAD, brand: \"foo\", size: \"big\", year: 2000, price: 123 }) { size } }" }' \


  "data": {
    "newBike": {

* shows formatting of “DateTime” field

curl  \
  -X POST \
  -H "Content-Type: application/json" \
  -d '{ "query":  "query {  bikes { id type brand size year price addedDateTime} }"  }' \


  "data": {
    "bikes": [
        "id": "5e28ac4d64f0e8088f8bce47",
        "type": "ROAD",
        "brand": "foo",
        "size": "big",
        "year": 2000,
        "price": 123,
        "addedDateTime": "2020-01-22 14:10:53 -0600"
Posted in Software Engineering | Comments Off on GraphQL with curl examples

AMD Ryzen Build

Some facts on the CPU: it is currently #135 on PassMark [16,9499]

The video card was the least expensive PCI available at Micro Center.

This build is intended for two distinct purposes. The first is a general-purpose, install linux, install multiple linux, experiment machine. The second is as a Windows gaming machine, which will require a “real” GPU and Windows 10 install.

Note: The AMD CPU came with a coupon code for a free game. However, redeeming the code requires “AMD Verification Tool”, and that tool is windows-only. So, don’t fall for AMD if you want to use linux. Probably, just don’t fall for AMD, period.

Item Product Cost
CPU AMD Ryzen 7 2700x 3.7Ghz 8 Core, 16 threads AM4 (4.3Ghz turbo) AM4, 105W $159
Cooler Wraith Prism Cooler
RAM G.Skill Aegis 16GB (2 x 8GB) DDR4 2400 (PC4 19200) CL17 F4-2400C17D-16G – Black/Red $68
Motherboard Gigabyte B450M Aorus AM4 mATX Motherboard $40
Power Supply Apea 500W reuse
Video EVGA GeForce 8400GS 1GB Single-Fan DDR3 $42
Video Gigabyte Radeon RX 570 4GG 256-bit GDDR4 x16 $115
Case NZXT GAMMA Classic ATX Mid Tower reuse $31
M.2 Drive Intel 660p Series 1TB – NVMe 3.0 x4 3D2, QLC SSD (SSDPEKNW010T8X1) $83
SS Drive None
HD Drive None
4-to-8 pin adapter Coboc EPSADLP48-6 6in ATX 12V P4 4-Pin LP4 to EPS 12V 8-pin $2
OS Ubuntu to start, then Windows 10 Pro 64-bit OEM
Total $394
Tax 7.529% $24
Grand Total $418

tax newegg 7.526%
tax microcenter 7.529%

Posted in Computer Builds | Comments Off on AMD Ryzen Build

ZFS replace drive before it fails

The main Virtual Machine Server was seeing hardware failures and ZFS “scrub not zero bytes”. One of the (cheap) Hitachi Ultrastar 2TB disks was starting to fail, after only 1.5 years. Smartctl was showing 53 recent errors.

  pool: vmstorage
 state: ONLINE
  scan: scrub repaired 1.75M in 5h12m with 0 errors on Tue Oct  1 07:12:22 2019

        NAME                                            STATE     READ WRITE CKSUM
        vmstorage                                       ONLINE       0     0     0
          mirror-0                                      ONLINE       0     0     0
            ata-Hitachi_HUA723020ALA641_YFG31Y3A-part2  ONLINE       0     0     0
            ata-Hitachi_HUA723020ALA641_YFG4GJ8A-part2  ONLINE       0     0     0

/var/log/messages was showing stuff like this, over and over:

Sep 15 03:03:42 dellt3600 smartd[19459]: Device: /dev/sdc [SAT], 831 Currently unreadable (pending) sectors

Looking in /dev/disk/by-id/*, the failing drive has a serial number of YFG4GJ8A.

So, the setup for these commands became:

export DISK_GOOD=/dev/disk/by-id/ata-Hitachi_HUA723020ALA641_YFG31Y3A-part2
export DISK_BAD=/dev/disk/by-id/ata-Hitachi_HUA723020ALA641_YFG4GJ8A-part2
export DISK_REPLACE=/dev/disk/by-id/ata-Hitachi_HUA723020ALA641_YGJ0JSYA-part2

The command to remove the failing, but not yet failed, drive from the mirror:

zpool detach vmstorage $DISK_BAD

(At this point, I shutdown the machine, and had to swap disks since there was only room for two 3.5″ HDDs).

After reboot, the command to add the new disk into the mirror:

zpool attach vmstorage $DISK_GOOD $DISK_REPLACE

Resilvering 1.25TB took 4h56m.

Note: if you “pre-partition” your ZFS disks (like I do), then you also need the “root” disk to run parted:

export DISK_REPLACE_ROOT=/dev/disk/by-id/ata-Hitachi_HUA723020ALA641_YGJ0JSYA

Use ‘unit s’ to create the partitions with exactly the same sector counts as the drive being replaced.

Just recording the replacement drive: $42 – HGST/Hitachi Ultrastar 7K3000 2TB 7200RPM Enterprise Grade Sata III For the record – the drive arrived new, with 0 hours power-on time. Vendor was DBSKY.

Posted in ZFS | Comments Off on ZFS replace drive before it fails

Core i7-9700K Build

Some facts on the CPU: it is currently #67 on PassMark [17,329, single thread 2,909] It was launched 2018 October. It used 14mm architecture and does not have hyperthreading. Since the “i7-9700”, i.e. non-K, does not exist (and may never exist) this overclock-capable CPU was the choice, and it will never see overclocking. This also meant that the first time in 20+ years, a separate cooler was needed. Gone are the days of buying top-50 CPUs. The #2 i9-9940X comes in with a 28,121 score. The AMD Ryzen Threadripper and Xeon E5 make up the bulk of the CPUs higher on the list.

The video card was a Micro Center refurbished item, and kicked off this entire build. GPU prices and RAM prices have “kind of” recovered at this point – so the system finally got built, after a 10 month delay. The price was $380, plus a $50 2-year protection plan. Currently, new GTX 1080 Founders Edition sells for $620 from 3rd party (“Quick-Ship”), and a GTX 1080 MSI sells for $575 fulfilled by Amazon. Neither Amazon nor Newegg are directly selling 1080s now. The 1080s have been available for ~3 years, the RTX2070s for ~2 months. 1080s sell for $500, 2070s sell for $510, 2080s sell for $750. One benchmark shows the 2080 at #6, the 2070 at #8 , and the 1080 at #12. CUDA cores: 1080=2560, 2070=2304, 2080=2944.

The “old style” Windows Experience Index, with its maximum of 7.9, is not available in Windows 10. The “winstat” application reported a systemscore=9.2, memoryscore=9.3, cpuscore=9.3, videoencodescore=9.9, diskscore=9.2.

To honor the fastest/most expensive video card in the house, a new monitor was purchased. The Dell 1600×1200 Ultrasharp monitors needed a bit of an upgrade. The new Dell 27″ G-Sync cost a bit more money (it looks like about $100 more) because of the G-Sync. It has a 1ms response time, and 170 degree horizontal viewing, with a resolution of 2560×1440. The first purchase from Micro Center had to be returned after 18 days, because 25 thin horizontal white stripes appeared on the right-hand side, from the top to almost the bottom. The second purchase from Best Buy included a 4-year Geek Squad product replacement plan.

All product links are from the actual vendor.

Item Product Cost
CPU Intel Core i7 9700K 3.6Ghz (4.9GHz Turbo) Socket 1151 95W Eight-Core Desktop Eight threads $400
Cooler Cooler Master Hyper 212 EVO Universal CPU Cooler $33
RAM CORSAIR Vengeance LPX 32GB (2 x 16GB) 288-Pin DDR4 SDRAM DDR4 2666 (PC4 21300) Memory Kit Model CMK32GX4M2A2666C16 $240
Motherboard Gigabyte Z390 Aorus Pro WiFi LGA 1151 ATX Intel Motherboard $160
Power Supply Corsair CX Series 650 Watt 80 Plus Bronze Certified Non-Modular Power Supply (CP-9020122-NA) CX650 $60
Video NVIDIA Founders Edition GeForce GTX 1080 Single-Fan 8GB GDDR5X PCIe Video Card Refurbished $430
Case Corsair Carbide 300R Mid Tower Case (Black) CC-9011014-WW $48
M.2 Drive Samsung 970 EVO 250GB – NVMe PCIe M.2 2280 SSD (MZ-V7E250BW) $78
SS Drive Samsung 860 EVO 1TB 2.5 Inch SATA III Internal SSD (MZ-76E1T0B/AM) $128
HD Drive None
BD/DVD/CD Samsung Optical Drive SH-224DB/BEBE $21
OS Windows 10 Pro 64-bit – OEM $150
Total $1748
Tax 7.529% $131
Monitor Dell S2716DGR 27″ WQHD 2560×1440, 144Hz, 1ms response, HDMI and DP, G-Sync Gaming LED Monitor $508
Grand Total $2387

Returned monitor:
Dell S2716DGR 27″ WQHD 144Hz HDMI DP G-Sync Gaming LED Monitor, $430+$33

tax newegg 7.526%
tax microcenter 7.529%

For additional game storage:

SS Drive, 2021 Sept, Samsung 860 EVO 1TB SATA III Internal SSD (MZ-77E1T0B/AM) $138

Posted in Computer Builds, Core-i7 | Comments Off on Core i7-9700K Build

CPU Comparison

The table includes both “as built” and “dreaming only” CPUs.

CPU PassMark Benchmark
2 i9-7908XE@2.6 27,786
36 i9-9900K@3.6 20,156
38 i7-7900X@3.3 20,046
67 i7-9700k@3.6 17,329
96 i7-8700k@3.7 15,960
264 i7-4770k@3.5 10,091
401 i7-2600k@3.4 8,444
510 i5-6500@3.2 7,240
Posted in Core-i5, Core-i7 | Comments Off on CPU Comparison


Current and historical record of Mice and Keyboards.

Item Product Year Cost
Mouse Logitech G403 Wired Optical Gaming Mouse with RGB Lighting 2018 Nov $35
Mouse Logitech G Pro Gaming FPS Mouse with Advanced Gaming Sensor 2016 Dec $52
Mouse Corsair Raptor M30 Gaming Mouse 2014 Mar $50
Mouse Logitech G400s 910-003589 Optical Gaming Mouse 2013 Sep $45
Mouse Logitech M510 Wireless Computer Mouse 2011 Dec $25
Mouse Logitech MX 581 Gaming-Grade Optical Mouse 2011 Feb $37
Mouse Logitech G5 2-Tone 6 Buttons Wired 2000 dpi 2007 Mar $46
Mouse Redragon M601 Wired Gaming Mouse, 3200 dpi 2018 May $12
Mouse Corsair Dark Core RGB Wired or Wireless Gaming Mouse, 16,000 dpi 2018 Dec


Mouse Razer DeathAdder Essential Gaming Mouse, 6,400 dpi (replaces Corsair Dark Core, above) 2021 Nov


Keyboard + Mouse Amazon Basics Wired Keyboard and Wired Mouse 2013 Nov $15
Keyboard Logitech G213 Prodigy Gaming Keyboard. With Mech-Dome keys that are full height, deliver a full 4mm travel distance, 50g actuation force, and a quiet sound operation. 2021 Sep $45
Keyboard Rosewill Mechanical Gaming Keyboard with Cherry MX Brown Switches (RK-9000V2 BR). 2021 Feb $90
Keyboard Corsair Vengeance K70 Mechanical Keyboard – Red LED – Cherry MX Brown Switches 2014 Oct $130
Keyboard Corsair Raptor K30 Gaming Keyboard 2014 Sep $33
Keyboard Logitech K400 2.4GHz Wireless Touch Keyboard 2012 Mar


SSD External Samsung T5 Portable SSD 1TB 2022 Jan


Posted in Computer Builds | Comments Off on Peripherals

Java Command Line Parsers Taxonomy

Ever wanted to see a complete list of Java command-line parser libraries? Here is the project for you.

The git project javacommandlineparsers contains a list of Java command-line parser libraries, in JSON format, and transformed to markup, csv and html.

It also shows how to perform groovy templating with gradle. The file is generated from a template using SimpleTemplateEngine, and a .csv and .html files are generated with groovy.text.markup.MarkupTemplateEngine.

The raw .json content is available here.

Posted in Software Engineering, Software Project | Comments Off on Java Command Line Parsers Taxonomy