Tim On Computer Science

After working with AngularJS for a couple of months now, I can finally express a concise answer to “What is AngularJS?”

It is:

1. MVC where the model is on the client side
2. MVC where the view is a template based in the .html, and is rendered on the client side
3. MVC where the controller is “live” – changes to the model reflect in the template immediately

The key: “on the client side”. No more complicated mappings inside your .jsp from fields to Java objects, no more complicated mappings from “post actions” to specialized controllers that track the application state. No more painting the initial page one way with .jsp and then updates with AJAX. It replaces your .jsp template with more more natural .html with embedded template variables and controls, and keeps everything straight.

AngularJS throws in a couple of “neat tricks” – dependency injection, testability, separation of client-server, scope. But AngularJS’s two tag lines: “HTML enhanced for web apps” and “AngularJS — Superheroic JavaScript MVW Framework” — don’t provide much neither of which is very helpful.

AngularJS (or some other library that does MVC-client-side better, now that the secret is out of the bag) is the wave of the future. The productivity gains are incredible. It is literally easier to re-write your .jsp and implement that one new feature than it is just to extend your .jsp.

If you have received this error message:

java.lang.IllegalStateException: Unsupported cookie spec: default

It is because Amazon made their SDK dependency look like this:
+— com.amazonaws:aws-java-sdk
| +— org.apache.httpcomponents:httpclient:[4.1, 5.0) -> 4.4-beta1
| | +— org.apache.httpcomponents:httpcore:4.4-beta1

i.e. they made an open-ended statement that their SDK would work will all 4.x releases of httpclient.

As of 4.4-beta1, their statement became false. Somewhere down in the guts of httpclient, “default” is no longer a valid cookie specification, and now parts of the AWS SDK do not work.

See 2014/06/28/computer-science-hard-things/ ‎for a full essay on the problems with “dependency resolution”. In this case, Amazon just messed up – there is no way any particular aws-java-sdk release can claim compatibility with an entire 4.x line of the httpclient library.

The fix (at least in gradle; it should be similar in all build systems) is to exclude httpcomponents on the aws-java-sdk line, and then add a specific httpcomponents (e.g. 4.1 worked nicely for me, since presumably Amazon actually tested with that release before their release). Since Amazon was “fuzzy” about their actual dependency requirements, you may have to try 4.2, 4.3, etc. to make sure you get an actually-compatible-with-aws version of httpclient.

This post was inspired by The Training Wheels Came Off by Aslak Hellesøy, author of The Cucumber Book.

TL;DR – Use a Domain Specific (testing) API to implement your Domain Specific Language

That article describes the motivation behind removing web_steps.rb — in a nutshell, they were removed because these step definitions are not at the correct level of abstraction for a properly defined Cucumber .feature file. The direct quote on the subject: “Cucumber was designed to help developers with TDD at a higher level”.

The basic idea is that your .feature file should not be written like this:

Scenario: Successful login
  Given a user "Aslak" with password "xyz"
  And I am on the login page
  And I fill in "User name" with "Aslak"
  And I fill in "Password" with "xyz"
  When I press "Log in"
  Then the http status should be 200
  Then the http session cookie should not be empty

Instead, your .feature file should look like this:

Scenario: Successful login
  Given log in succeeds with a user "Aslak" with password "xyz"

Notice at this level, there is no mention of http, http status 200, cookies, buttons or button names, etc. It describes only the high-level test.

In his article, he codes to the idea in this post, but never names explicitly says it. The idea: keep your .feature definitions high-level, and implement your step definitions using a set of intermediate helper methods. This intermediate level is what I call the Domain Specific API (DSA) from my title. It looks like this:

  @Given("^log in succeeds with a user \"([^\"]*)\" with password \"([^\"]*)\"$")
  public void log_in_succeeds(String user, String password) {
     dsa.actionLogin(user, password);
     dsa.checkStatus(200);
     dsa.checkLoginCookieSet();
  }

In essence, the approach leads to four levels of test:

1. .feature file
2. step definitions implementations
3. DSA implementations
4. technology library (e.g. HttpClient)

The extra “Domain Specific Api” layer allows you to dive into the implementation-specific details without “polluting” your main .feature files with too many details.

Reference Links:
For Java, see Cucumber-JVM.
For Ruby, see Capybara.

There is a popular saying about Computer Science (see here and here):

There are only two hard things in Computer Science: cache invalidation and naming things.

— Phil Karlton

There is a funny variation that makes it “There are only two hard problems in Computer Science: cache invalidation, naming things, and off-by-one errors.”

I propose there are actually three hard things:

1. Naming things
2. Cache invalidation
3. Dependency resolution

My criteria for being a “hard thing”:

1. Must be applicable to multiple scopes
2. Must not be fully solved

Examined this way, it is interesting to see why these are the three deserve to be on the list:

1. Naming things
   1. Applicable to every area in computer science – variable names, class names, machine names, network names, security policy names, URIs, etc. It even applies to this list: think of the difference between naming the first item “cache invalidation” versus just “caching”.
   2. Not at all solved. You can barely say we have good heuristics for this.
2. Cache invalidation
   1. Applicable to multiple layers of computer hierarchy: CPU registers, L1, L2, L3, etc., disk caches, network resource caches, DNS caches, etc.
   2. Solved in the sense we know it is a balancing act between efficiency and correctness. Not solved for the general case, however. If there even is a “general case” at all.
3. Dependency resolution
   1. Applicable to multiple domains: run-time (think Dependency Injection), build time (think Apache Ivy and Maven), hardware-software, distributed systems, and probably more
   2. Solved in the sense we know about topological sorting to help with transitive dependencies.
      For run-time, the entire sub-field of dependency injection has multiple solutions: Spring Framework, Guice, PicoContainer. Does anybody remember DLL Hell? That shows that “API definition” (which is a candidate for its own “Hard Thing” entry) is a sub-problem of dependency resolution.
      For build time, the better build systems make it easy to specify your dependencies and add global exclusions to get you out of transitive dependency issues.
      For hardware-software, think about the hardware requirements for running a particular application or installing a particular driver.
      For distributed systems, think about (for example) your application requires which version of which database. For provisioning, has been partially solved by Chef and Puppet and others. For detection, still very much roll-your-own.

So, did I create any converts? Do you agree there are 3 Hard Things in Computer Science?

Starting with a quotation I saw at a rest stop while on vacation:

Dryland farming works best when in a wet year.

This was on a placard explaining that Dryland farming had a string of successful years when it was wet. But when it got dry, the same techniques failed.

I’ll summarize that as: if you have a problem, and implement a fix, and fixes the problem, you still might not know how much your fix actually worked.

Which brings me to how Agile has “fixed” waterfall development. And whether these are just the “wet years” for Agile.

Getting back to the title – an Evolutionary Stable Strategy (ESS) describes a strategy, that if adopted by a population in a given environment, cannot be invaded by any alternative strategy that is initially rare. The first thing to note is that waterfall development is positively a non evolutionary stable strategy – Agile started out rare, but has effectively invaded (subjective judgement only – statistics are hard to find. Most of the statistics like to spout “Agile is 3 times more likely to succeed” – as if a 3% chance versus 1% chance is worth bragging about…) It is also true is that waterfall itself had previously been invaded by “hybrid waterfall” long before Agile, making it doubly non-ESS, if that is even possible. Being non-ESS is no big deal in itself.

Of interest here is why I’m claiming that Agile is non-ESS. After all, Agile is still on its upswing (again, hard to find concrete statistics here). And quite possibly, no development process is stable because of the inherent fickleness of management and their desire to chase the new fad. So, if no process is stable, then it isn’t saying very much to claim that Agile is also non-ESS.

My value-add is: I think I know the reason Agile will be successfully invaded and replaced.

I believe Agile’s replacement will come about as a result of Tim’s Rule on Agile (only highly experienced developers can make Agile work) and Choose Your Path Wisely (after years of choosing not to learn, you no longer have the option of learning). My assertion: Agile does not create developers that are sufficiently capable in executing Agile successfully. My assertion is based on observing developers with multiple years of “successful” Agile development experience, yet at the same time lacking in critical software engineering skills. Why does it matter, you ask? If they are on a successful Agile project, and have “succeeded” without those skills, then aren’t those skills by definition not needed? My answer: the skills are needed, and they are being provided by developers (or scrum masters, or business owners, etc.) with extensive non-Agile experience. And once that pool of people is gone, or stretched too thin, the Agile-only generation, who choose the Agile path, will be unable to step up and provide those critical skills (not “unwilling”, just literally “unable”). There will be much pain and frustration as the Agile consulting industry struggles to figure out what has gone wrong. And lots more finger-pointing (“You’re doing Agile wrong!”). In the end, I’m claiming that Agile is only working now because it is the “wet” years. And the “dry” years are coming.

This article was inspired by a strangely acting Gradle build script. (Unimportant details: it worked fine locally, and it worked in Jenkins the first time, but when I renamed the job, it mysteriously started failing with a compile error every time after that.)

As I was troubleshooting that Jenkins job, I had a thought – I’ve known engineers and one manager who would have “just fixed it” – they would have changed the job name back to what worked and moved on. In the case of the (ex) manager, he would have made a decree to just change the name back, and would have been angry if anybody dared question his command.

I used to be annoyed by that manager – until I realized two things: (1) He does not have any real choice at this stage in his career. He has chosen the easy path for so long, he literally never developed the skills needed to understand and solve difficult problems. And since he doesn’t have the skills, he assumes nobody else does either, or gets jealous when they demonstrate those skills, and (2) I have chosen a different path. I think of it as “no magic in my world”. As in, until I figured it out, it was “magic” why one Jenkins job worked yet the other failed, and “magic” is not allowed as an explanation in my world. (I also sometimes think of this as “never be afraid of the system you are developing”.) And yes, I figured it out, and learned another piece of the Gradle system. There is neither “magic” nor fear left in that particular area for me.

A thought-provoking question: which Path is more efficient? I can’t say for sure – because when I can rattle off an answer to the current problem in 2 seconds, that feels good, and feels productive, but it took years to learn where to put the X. Either pay gradually over time or gradually over time become afraid.

I do know one thing though: when push comes to shove, I’m capable of acting like (and I have numerous times acted like) I’ve followed the “head in the sand” path. The opposite is not true, however.

For those of you on the “no magic” path – the “X” is: Do not use project.name as part of your generated sources. Any time the directory name changes (as in Jenkins, which creates the workspace directory name from the Jenkins job name), the value of Gradle’s project.name will change too.

For those of you on the other path – if you are reading this, then there is still hope for you yet.