Vue.js carousel State Fair

This is the announcment page for Minnesota State Fair Space Tower.

This was the “next level of difficulty” for Vue.js. It involved using vue-cli to create the webpack-simple basis for the project, then learning how to incorporate components into the project (in this project, that was Vue Carousel), and learning the npm run build and how to export that to a static web page.

Because of the (ridiculously) heavy-weight build process, it took a while to get going. And, it still has a “root” (aka “leading slash”) problem in the final build.js. It also has JavaScript that has the beginnings of a “cache-bust” technique, aka src=”dist/IMG_3107.jpg?1974a0f53e964bb24495a619408dbaf3″, but the dist directory itself only has the short-name “IMG_3107.jpg”.

Overall, still pretty simple, and holds lots of promise as an AngularJS replacement.

Posted in Uncategorized | Comments Off on Vue.js carousel State Fair

The future of AngularJS and Angular 2

It has been fun developing AngularJS applications. It was the first complete framework that was both a higher level of abstraction than jQuery and easy enough (not “easy”, per se) to learn and use. You can even play with my Tic-Tac-Toe AngularJS applcation.

Then, around September, 2014, Angular 2.0 was announced, and 2.0.0 was released September, 2016. After 20 minutes of using Angular 2.0, it became clear that “drastically different” might have been an understatement.

It made me sad. Sad like when the Java people lent their name to JavaScript. A lot of confusion resulted, and Java took a hit that took a decade to recover.

“Angular” isn’t going to recover.

AngularJS was awesome, but Angular 2 was a classic case of “second system syndrome“. Angular 2 froze people out of AngularJS, but totally lost ground to the other libraries and frameworks. Angular 4 is out as of March, 2017, and the biggest thing they are bragging about is “Semantic Versioning“. They don’t seem to have much else to brag about, so why not? Oh yeah, ngIf now has an “else”. Yeah, that’s what is what is keeping Angular 2/4 down…

If there is ever a greenfield project in my future, it will use React with Redux and webpack.

Another likely alternative is Vue.js

[If the tone of this post seems weird, it is because this is a “record my current understanding and prediction” more than anything else.]

A seriously messed up post on Angular and React: here. It has the opposite prediction – that “Angular” is going to be great. Another messed up post is here. This post confuses AngularJS with Angular 2 (like, “Angular 2, created in 2010”)

Aurelia is another viable competitor in this field.

Posted in Software Engineering | Comments Off on The future of AngularJS and Angular 2

Should REST and microservice APIs be Versioned?

This is a very lively topic.

The starting point for these questions must be: in our world (enterprise grade computer science), what isn’t versioned? The answer to that is: everything is versioned.

Therefore, the actual question splits into two parts:
1) What makes REST and micro-services so special that they don’t get versioned? The answer here is easy – nothing. They must be versioned, just like everything else.

One post (here) tries to convince others of “versioning only as a last resort”. This book tries to convince others that “just monitoring the logs” is a sufficient solution to the problem. This is all amateur-hour stuff. Just ignore them.

Now that we know it must be versioned, the actual question arises:
2) How should the versioning be implemented in REST and microservices?

The choices seem to be:
1) URL: (e.g. http://yourapp/v1.2.3/person…)
2) URL parameter: (e.g. http://yourapp/person?Version=1.2.3)
3) Header – custom (e.g. “X-Api-Version: v1.2.3”)
4) Header – Accept (e.g. “Accept: application/yourapp.v1.2.3”)
5) Payload (e.g. inside the JSON { “version” : “v1.2.3”, “name” : “joe”, … } )
6) Hostname: (e.g. http://v1.yourapp/person)

People argue against #1 with strange arguments like a URL “represents a person” or “the idea of a person”. It is strange because “person” is not, and can not, be a universally workable concept. (Quick test if you are not convinced – write client side code that receives person JSON data, and computes the total of that person’s age and height. It can’t be done, for multiple reasons: you don’t know the keys and you don’t know the units. What if “age” is actually “bornAt” as milliseconds since the epoch?) The genius Roy T. Fielding simply got it wrong for the enterprise – a resource is a universal constructs only in a university classroom. And, “Controls have to be learned on the fly” may eventually work once machines take over, but for now, even “age plus height” can’t be learned dynamically.

So – there just isn’t A concept of a person. There must be some implementation behind it, and that implementation needs to be versioned. Because in the enterprise world, something always changes.

The argument against #3 and #4 is a variation of the above: the URL “http://yourapp/api/person” makes it look like “person” is universal. But it is not universal. So the URL is a lie. It must be versioned.

The argument against #5 is the “surprise!” factor. Your client application had link to a person, but the data came back 3 versions in the future, and your client application has no mechanism available to “correct” the request, unless #1-#4 is provided. It’s the “box of chocolates” style programming – you never know what you’re going to get.

Since #6 is basically #1 with the “v1” moved to the left so much that it ends up in the DNS of the hostname, is shares the avantages. However, it proposes an uncomfortable use of DNS. And it ties the concept to a particular hostname (or pattern of hostnames, like “v1.stage.app.com”, “v1.prod.app.com”, etc.) In the end it seems like a personal preference that a REST server should respond to requests without needing to know its DNS entry. (Or worse, creating option 4a: “Look for a version in the ‘host:’ header of the request”).

One clever variation is just: do more than one! It seems like extra work, and you have to worry about wconflicts (like 3 different versions in the same request). But, it does ensure that everybody yells at you equally 🙂

There are plenty of enterprise-grade APIs that use #1. They do differ on /v1/ versus /v1.0.0/, with the majority using the simpler /v1/.
* Dropbox – #1 (“/2/”), and is too embarrassed to document it
* Amazon – #2 (“/foo?Version=2012/01/02”)
* Twitter – #1 (“/1.1/”), also too embarrassed to document it
* Best Buy Developer API – #1 (“/v1”), also too embarrassed to document it
* Facebook – #1 (“/v2.5/”) with #3 in return (“facebook-api-version:v2.0”)

More links on versioning:

Quote from the book:

Posted in Software Engineering | Comments Off on Should REST and microservice APIs be Versioned?

Is Spring RestTemplate Thread Safe

Is Spring RestTemplate Thread Safe?

The only correct answer to this question is: No

A quick look at the source (the link is tied to a specific version so the line number is always correct) on line 230 shows the method “setMessageConverters” which mutates the field messageConverters without proper thread safe access or publishing.

There are multiple incorrect answers to this question found on the web:

All of these incorrect answers are actually answering the question: “Is there a way to use an instance of RestTemplate that is thread safe?” or just quoting other incorrect posts. (Especially disappointing is the blog post on spring.io. But, it was written in 2009, so maybe it was thread safe back then. Since at least 2012 (line 171) it has not been thread safe.) Or, they answer “Yes” then say “But don’t add a MessageConverter, because that is not thread safe”.

“Is there a way to use an instance of RestTemplate that is thread safe?” is a different question, and like all Java objects, the answer of course is: Yes.

If your code avoids the non-thread-safe methods, then you can claim thread safety.
If your code simply doesn’t call any methods on the instance, you can claim thread safety.
If your code only has one thread, you can claim thread safety.

But the actual code of RestTemplate.java shows that it is not thread safe.
And the typical use is to create a singleton instance that is used by multiple threads.
It is a “cross your fingers and hope nothing weird happens” kind of thread safe.

Which, of course, is not thread safe at all.

Posted in Software Engineering | Comments Off on Is Spring RestTemplate Thread Safe

Hadoop Terasort VirtualBox benchmarks

Setup: 3-node hadoop (hdp v2.6.1), installed via Ambari 2.5.1 and running in VirtualBox 5.1 on one physical machine.

The terasort is traditionally run on 1 terabyte of random data (SIZE=10,000,000,000). In 2008, Yahoo! ran terasort on 1TB of data in 209 seconds, on a cluster of 900 nodes.

However, this benchmark is for significantly less data (SIZE=10,000), or 1/1,000,000th the amount of data and significantly fewer nodes (1 physical machine).

teragen: 340 seconds
terasort: 352 seconds
teravalidate: 6 seconds

Each (virtual) node:
8GB RAM, 2CPU, 80GB separate /hadoop/hdfs/data mount, Ubuntu 14.04.2 LTS

Physical machine:
32 GB RAM, 8-core AMD FX-8350 4.0GHz, 1TB WD Black HDD, Ubuntu 16.04 LTS xenial

Posted in Storage | Comments Off on Hadoop Terasort VirtualBox benchmarks

Secret Share 1.4.3 on Maven Central

Secret Share in Java on Maven Central

Just completed a release of the Secret Share in Java project to Maven Central.

Search for it using search.maven.org.
Or, go to SecretShare-1.4.3 directly.

GroupId: com.tiemens
ArtifactId: secretshare
Version: 1.4.3

This release adds a “-paranoid” flag to the combine operation. This is useful if you have more shares than you need and you are not sure if you have typographical errors in the values. The “-paranoid” will try subsets of the shares to recreate the secret, then keep track of which secrets were seen most often.

Sonatype Notes:

The Sonatype instructions were updated. See ReleaseProcess.txt.

Posted in Software Project | Comments Off on Secret Share 1.4.3 on Maven Central

Real World Debugging Exercise

Just completed the Real World Debugging Exercise.

It is a static page on this site (just like the Lot Area Calculator).

Posted in Uncategorized | Comments Off on Real World Debugging Exercise

VirtualBox USB Flash for ZFS

To set up your USB Flash drives as pass-through in VirtualBox, here is what you can do.

Next, plug in your flash drive and record its device. For me, it was /dev/sde. Look in /var/log/syslog, or dmesg output, for the “[sde] Attached SCSI removable disk” line. My device looked like this:

ls -l /dev/sde
brw-rw---- 1 root disk 8, 64 Nov 24 15:22 /dev/sde

If you don’t run virtualbox as root (and you should not), then you’ll need to use the group to access /dev/sde. Above, you can see it is “disk”.
Now, make sure the user that is running virtualbox is in the “disk” group (e.g. ‘tim’ user):

sudo usermod -a G disk tim

After this, you may or may not have to reboot. Run “groups” as your user. Make sure you see “disk” listed. If not, reboot (or maybe just logout and login).

Then, for each USB Flash drive, run this VBoxManage command:
(Note: this command just creates a *.vmdk file that “links” to a raw device. It does not touch or alter the device, but it does make sure your current user has permissions to the device):

VBoxManage internalcommands createrawvmdk -filename usbflash1.vmdk -rawdisk /dev/sde1
chown tim:tim usbflash1.vmdk

Then, in the virtualbox GUI, under settings, choose Storage, then “add hard disk”, then “Choose existing disk”, then select the usbflash1.vmdk file created above.

See the ZFS Fkash Drives on how to create your ZFS volume from the USB Flash drive(s).

Next up: plugging the USB Flash drives into a different ZFS instance.
(In case it is not clear, the trade-off here is between:
1) Two separate USB flash drives, which can be used by just about any computer, but must be manually kept in sync, and manually compared looking for differences, versus
2) Two flash drives as a single ZFS mirror volume, where ZFS makes sure the contents are the same and scrubs them to make sure they stay that way. However, you need a ZFS system to mount and use them.)

The contents of usbflash1.vmdk:

# Disk DescriptorFile
version=1
CID=1d3a8b59
parentCID=ffffffff
createType="fullDevice"

# Extent description
RW 15130017 FLAT "/dev/sde1" 0

# The disk Data Base 
#DDB

ddb.virtualHWVersion = "4"
ddb.adapterType="ide"
ddb.geometry.cylinders="15009"
ddb.geometry.heads="16"
ddb.geometry.sectors="63"
ddb.uuid.image="33234751-eccf-4188-8e3d-a903b8a4ad74"
ddb.uuid.parent="00000000-0000-0000-0000-000000000000"
ddb.uuid.modification="ee7f3673-388e-42ab-9fa6-698cd80fb5bd"
ddb.uuid.parentmodification="00000000-0000-0000-0000-000000000000"
ddb.geometry.biosCylinders="941"
ddb.geometry.biosHeads="255"
ddb.geometry.biosSectors="63"
Posted in ZFS | Comments Off on VirtualBox USB Flash for ZFS

ZFS Flash Drives

As an experiment, I created a small (8GB) ZFS volume in a mirror configuration using two 8GB USB Flash drives.

The purpose was to put “small, but really important things” on this storage system.

It actually ended up being two discrete pieces of work. First, setting up a VirtualBox FreeNas 9 image with “USB passthrough”. Second, running the FreeNas/BSD commands to create the volume by hand, since FreeNas insists on using 2G (fixed) per disk when it creates volumes. 2GB of 1000GB isn’t even a drip; 2GB of 8GB is significant.

So, second things first: creating the partition and volume and getting FreeNas to acknowledge it.

Once you know the two devices in FreeNas (for me it was /dev/ada1 and /dev/ada2), you can run these commands:

gpart create -s gpt /dev/ada1

gpart add -i 1 -t freebsd-zfs /dev/ada1
# now run ls -l /dev/gptid, make note of the new entry,
#   e.g.  d50a7a3b-aeb0-11e6-86a2-080027bed2b8

gpart add -i 1 -t freebsd-zfs /dev/ada2
# make note of that gptid too, 
#    e.g.  e9c860e9-aeb0-11e6-86a2-080027bed2b8

Since my two USB drives were not exactly the same (7.4G versus 7.2G), the zpool create required the “-f” flag:

zpool create -f  usb8 mirror /dev/gptid/d50a7a3b-aeb0-11e6-86a2-080027bed2b8 /dev/gptid/e9c860e9-aeb0-11e6-86a2-080027bed2b8

zpool status
  pool: usb8
 state: ONLINE      
  scan: none requested 
config:   
        NAME                                            STATE     READ WRITE CKSUM  
        usb8                                            ONLINE       0     0     0 
          mirror-0                                      ONLINE       0     0     0    
            gptid/d50a7a3b-aeb0-11e6-86a2-080027bed2b8  ONLINE       0     0     0 
            gptid/e9c860e9-aeb0-11e6-86a2-080027bed2b8  ONLINE       0     0     0  

Then, to get FreeNas to acknowledge it:

zpool export usb8

Finally, in the FreeNas GUI, run “Import Volume”, no encryption, selecting the volume name in the drop-down list. Under “Disks”, you should see two “7.xGiB” entries, not the “5.xGiB” the the 2GB swap partition would have given you.

If you didn’t make note of the gptid as you created the partitions, you can run

gpart list

And look for the “Name: ada1p1” entry to get the “rawuuid” value to use with /dev/gptid/hxhxhx….

Using the gptid/rawuuid, instead of /dev/adaX, is a good idea in general. In this particular case, remember that the device is actually a VirtualBox-created “Disk”, with a VirtualBox serial number, and a VirtualBox gptid.

See the next post on how the “pass-through” was created in VirtualBox.

Posted in ZFS | Comments Off on ZFS Flash Drives

iptables flush and ssh

Fun lesson from today: don’t blindly clear/flush your iptables.

If you are remotely managing a system, then make sure you run this:

sudo iptables -P INPUT ACCEPT
sudo iptables -P FORWARD ACCEPT
sudo iptables -P OUTPUT ACCEPT

before you run this:

sudo iptables -F

It turns out there are people who will write default block (“policy DROP”) style rules (instead of the using a default “policy ACCEPT” and ending with REJECTs/DROPs at the end that match everything). If the default policy is not ACCEPT, then clearing the rules will immediately remove your ssh access to the machine.

The lack of a default “policy REJECT” is one reason you will see more “policy ACCEPT” – because then, at the end, you can choose between REJECT or DROP. If you write your rules “in the other direction”, you loose the flexibility to choose – you must DROP. There are numerous posts on why DROP is not necessarily better than REJECT. (“Attackers don’t really feel any pain with DROP, but your legitimate users sure do”.)

Posted in Software Engineering, Ubuntu | Comments Off on iptables flush and ssh