Tim On Computer Science

There is a popular saying about Computer Science (see here and here):

There are only two hard things in Computer Science: cache invalidation and naming things.

— Phil Karlton

There is a funny variation that makes it “There are only two hard problems in Computer Science: cache invalidation, naming things, and off-by-one errors.”

I propose there are actually three hard things:

1. Naming things
2. Cache invalidation
3. Dependency resolution

My criteria for being a “hard thing”:

1. Must be applicable to multiple scopes
2. Must not be fully solved

Examined this way, it is interesting to see why these are the three deserve to be on the list:

1. Naming things
   1. Applicable to every area in computer science – variable names, class names, machine names, network names, security policy names, URIs, etc. It even applies to this list: think of the difference between naming the first item “cache invalidation” versus just “caching”.
   2. Not at all solved. You can barely say we have good heuristics for this.
2. Cache invalidation
   1. Applicable to multiple layers of computer hierarchy: CPU registers, L1, L2, L3, etc., disk caches, network resource caches, DNS caches, etc.
   2. Solved in the sense we know it is a balancing act between efficiency and correctness. Not solved for the general case, however. If there even is a “general case” at all.
3. Dependency resolution
   1. Applicable to multiple domains: run-time (think Dependency Injection), build time (think Apache Ivy and Maven), hardware-software, distributed systems, and probably more
   2. Solved in the sense we know about topological sorting to help with transitive dependencies.
      For run-time, the entire sub-field of dependency injection has multiple solutions: Spring Framework, Guice, PicoContainer. Does anybody remember DLL Hell? That shows that “API definition” (which is a candidate for its own “Hard Thing” entry) is a sub-problem of dependency resolution.
      For build time, the better build systems make it easy to specify your dependencies and add global exclusions to get you out of transitive dependency issues.
      For hardware-software, think about the hardware requirements for running a particular application or installing a particular driver.
      For distributed systems, think about (for example) your application requires which version of which database. For provisioning, has been partially solved by Chef and Puppet and others. For detection, still very much roll-your-own.

So, did I create any converts? Do you agree there are 3 Hard Things in Computer Science?

Starting with a quotation I saw at a rest stop while on vacation:

Dryland farming works best when in a wet year.

This was on a placard explaining that Dryland farming had a string of successful years when it was wet. But when it got dry, the same techniques failed.

I’ll summarize that as: if you have a problem, and implement a fix, and fixes the problem, you still might not know how much your fix actually worked.

Which brings me to how Agile has “fixed” waterfall development. And whether these are just the “wet years” for Agile.

Getting back to the title – an Evolutionary Stable Strategy (ESS) describes a strategy, that if adopted by a population in a given environment, cannot be invaded by any alternative strategy that is initially rare. The first thing to note is that waterfall development is positively a non evolutionary stable strategy – Agile started out rare, but has effectively invaded (subjective judgement only – statistics are hard to find. Most of the statistics like to spout “Agile is 3 times more likely to succeed” – as if a 3% chance versus 1% chance is worth bragging about…) It is also true is that waterfall itself had previously been invaded by “hybrid waterfall” long before Agile, making it doubly non-ESS, if that is even possible. Being non-ESS is no big deal in itself.

Of interest here is why I’m claiming that Agile is non-ESS. After all, Agile is still on its upswing (again, hard to find concrete statistics here). And quite possibly, no development process is stable because of the inherent fickleness of management and their desire to chase the new fad. So, if no process is stable, then it isn’t saying very much to claim that Agile is also non-ESS.

My value-add is: I think I know the reason Agile will be successfully invaded and replaced.

I believe Agile’s replacement will come about as a result of Tim’s Rule on Agile (only highly experienced developers can make Agile work) and Choose Your Path Wisely (after years of choosing not to learn, you no longer have the option of learning). My assertion: Agile does not create developers that are sufficiently capable in executing Agile successfully. My assertion is based on observing developers with multiple years of “successful” Agile development experience, yet at the same time lacking in critical software engineering skills. Why does it matter, you ask? If they are on a successful Agile project, and have “succeeded” without those skills, then aren’t those skills by definition not needed? My answer: the skills are needed, and they are being provided by developers (or scrum masters, or business owners, etc.) with extensive non-Agile experience. And once that pool of people is gone, or stretched too thin, the Agile-only generation, who choose the Agile path, will be unable to step up and provide those critical skills (not “unwilling”, just literally “unable”). There will be much pain and frustration as the Agile consulting industry struggles to figure out what has gone wrong. And lots more finger-pointing (“You’re doing Agile wrong!”). In the end, I’m claiming that Agile is only working now because it is the “wet” years. And the “dry” years are coming.

This article was inspired by a strangely acting Gradle build script. (Unimportant details: it worked fine locally, and it worked in Jenkins the first time, but when I renamed the job, it mysteriously started failing with a compile error every time after that.)

As I was troubleshooting that Jenkins job, I had a thought – I’ve known engineers and one manager who would have “just fixed it” – they would have changed the job name back to what worked and moved on. In the case of the (ex) manager, he would have made a decree to just change the name back, and would have been angry if anybody dared question his command.

I used to be annoyed by that manager – until I realized two things: (1) He does not have any real choice at this stage in his career. He has chosen the easy path for so long, he literally never developed the skills needed to understand and solve difficult problems. And since he doesn’t have the skills, he assumes nobody else does either, or gets jealous when they demonstrate those skills, and (2) I have chosen a different path. I think of it as “no magic in my world”. As in, until I figured it out, it was “magic” why one Jenkins job worked yet the other failed, and “magic” is not allowed as an explanation in my world. (I also sometimes think of this as “never be afraid of the system you are developing”.) And yes, I figured it out, and learned another piece of the Gradle system. There is neither “magic” nor fear left in that particular area for me.

A thought-provoking question: which Path is more efficient? I can’t say for sure – because when I can rattle off an answer to the current problem in 2 seconds, that feels good, and feels productive, but it took years to learn where to put the X. Either pay gradually over time or gradually over time become afraid.

I do know one thing though: when push comes to shove, I’m capable of acting like (and I have numerous times acted like) I’ve followed the “head in the sand” path. The opposite is not true, however.

For those of you on the “no magic” path – the “X” is: Do not use project.name as part of your generated sources. Any time the directory name changes (as in Jenkins, which creates the workspace directory name from the Jenkins job name), the value of Gradle’s project.name will change too.

For those of you on the other path – if you are reading this, then there is still hope for you yet.

This shows the way to get gradle to use a local Archiva server to publish your project artifacts.

IMPORTANT: You must configure your “guest” user in Archiva to have the correct permissions (roles) to publish to the Archiva server. To do this, in the Archiva WebUI (the default is http://localhost:8080), on the left side under “USERS”, click “Manage”, then click the blue pencil next to “guest”, then click the “Edit Roles” (next to the blue “Edit” button), then checkbox select “Global Repository Manager” and “Global Repository Observer” and press Update.

Interesting fact: Archiva will accept “SNAPSHOT” artifacts into its “internal” repository (i.e. the upload will succeed, and Archiva will store the artifacts in its “/repositories/internal/” directory). BUT it will not serve these artifacts, saying instead “managed repo is configured for release only”. If you accidentally publish SNAPSHOT artifacts to the “internal” repository, then you’ll have to clean your “/repositories/internal/” directory by hand – the WebUI won’t let you.

So, given that interesting fact, the “if” logic (below) looks for “SNAPSHOT” and sets the url to the correct location. Note that the single “maven { }” entry is different from how it was configured for resolving, where there were two entries. (See Gradle with Local Archiva).

This shows the correct ‘publishing’ section of your build.gradle file for publishing to your local Archiva server. This also shows how to publish your “-sources” artifact. Feel free to substitute “localhost” for the actual machine name or IP address of your Archiva server.

(2014/6/10 update: added ext.isReleaseVersion and added checks before using the .publishBaseUrl property before using it.)

Versions:
archiva: 2.0.1
gradle: 1.11

// Used in publishing - the new plugin:
apply plugin: 'maven-publish'

// Used in publishing - pom information: 
group =  'com.tiemens'
version =  '0.1-SNAPSHOT'
project.ext.isReleaseVersion = !version.endsWith("SNAPSHOT")

// Used in publishing - source artifact: 
task sourceJar(type: Jar) {
    from sourceSets.main.allJava
}

publishing {
    publications {
        mavenJava(MavenPublication) {

            from components.java

            artifact sourceJar {
                classifier "sources"
            }
        }
    }

    repositories {
        maven {
            if (project.hasProperty('publishBaseUrl')) {
                if (! project.ext.isReleaseVersion) {
                    url project.publishBaseUrl + "/snapshots"
                } else {
                    url project.publishBaseUrl + "/internal"
                }
            } else {
                    // this is a notice that 'publish' requires .publishBaseUrl
                    url "http://you.must.configure.project.publishBaseUrl"
            }
        }
    }

}

Just for documentation, this shows the error message you get if you don’t set up your guest user with the proper roles.

$ gradle --debug publish
... snip ...
17:09:38.693 [DEBUG] [sun.net.www.protocol.http.HttpURLConnection] sun.net.www.MessageHeader@19507226 pairs: {PUT /repository/snapshots/com/tiemens/CardWar/0.5-SNAPSHOT/CardWar-0.5-20140421.220938-1.jar HTTP/1.1: null}{User-Agent: maven-artifact/3.0.4 (Java 1.7.0_45; Linux 3.2.0-31-generic-pae)}{Host: sitearchiva:8080}{Accept: text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2}{Connection: keep-alive}{Content-Length: 21915}
17:09:38.703 [DEBUG] [sun.net.www.protocol.http.HttpURLConnection] sun.net.www.MessageHeader@10775136 pairs: {null: HTTP/1.1 401 Unauthorized}{Date: Mon, 21 Apr 2014 22:09:38 GMT}{Set-Cookie: JSESSIONID=11ohtcvqtpqu3s9r3iis2a7c1;Path=/}{WWW-Authenticate: Basic realm="Repository Archiva Managed snapshots Repository"}{Content-Length: 0}{Server: Jetty(8.1.14.v20131031)}
17:09:38.737 [INFO] [org.gradle.api.internal.project.ant.AntLoggingAdapter] [ant:null] An error has occurred while processing the Maven artifact tasks.
Diagnosis:

Error deploying artifact 'com.tiemens:CardWar:jar': Error deploying artifact: Failed to transfer file: http://sitearchiva:8080/repository/snapshots/com/tiemens/CardWar/0.5-SNAPSHOT/CardWar-0.5-20140421.220938-1.jar. Return code is: 401
...snip...


More documentation links:

Shows example using authentication, instead of using “guest” with publish role:
http://forums.gradle.org/gradle/topics/maven_publish_and_setting_snapshotrepository_and_releaserepository

This shows the way to get gradle to use a local Archiva server to resolve dependencies (where the Archiva server in turn resolves to maven central).

(Side comment: documentation for gradle is sparse, incomplete and out-of-date – e.g. ‘mavenRepo’ shows up in many searches.)

So, if you’re interested in how to use a local Apache Archiva server in your environment, then this shows the correct ‘repositories’ section of your build.gradle file. In an upcoming post, I’ll document how to publish to your local Archiva server. Feel free to substitute “localhost” for the actual machine name or IP address of your Archiva server.

Versions:
archiva: 2.0.1
gradle: 1.10

repositories {
   maven {
     url 'http://localhost:8080/repository/internal'
   }
   maven {
     url 'http://localhost:8080/repository/snapshots'
   }

  // if your Archiva is set up correctly, 
  // then you don't need mavenCentral() here:
  //  mavenCentral()
}

Just for documentation, this shows the “dependencies” section of the build.gradle file. If your Archiva is set up correctly, these 3 dependencies will actually download/store 8 .jar files (because of transitive dependencies).

dependencies {
  testCompile group: 'info.cukes',   name: 'cucumber-java',    version: '1.1.5'
  testCompile group: 'info.cukes',   name: 'cucumber-junit',   version: '1.1.5'
  testCompile group: 'junit',        name: 'junit',            version: '4+'
}

Files that end up in your apache-archiva/repositories directory (showing just the .jar files):

apache-archiva-2.0.1/repositories/internal/junit/junit/4.11/junit-4.11.jar
apache-archiva-2.0.1/repositories/internal/org/hamcrest/hamcrest-core/1.3/hamcrest-core-1.3.jar
apache-archiva-2.0.1/repositories/internal/info/cukes/cucumber-html/0.2.3/cucumber-html-0.2.3.jar
apache-archiva-2.0.1/repositories/internal/info/cukes/cucumber-java/1.1.5/cucumber-java-1.1.5.jar
apache-archiva-2.0.1/repositories/internal/info/cukes/gherkin/2.12.1/gherkin-2.12.1.jar
apache-archiva-2.0.1/repositories/internal/info/cukes/cucumber-junit/1.1.5/cucumber-junit-1.1.5.jar
apache-archiva-2.0.1/repositories/internal/info/cukes/cucumber-jvm-deps/1.0.3/cucumber-jvm-deps-1.0.3.jar
apache-archiva-2.0.1/repositories/internal/info/cukes/cucumber-core/1.1.5/cucumber-core-1.1.5.jar

Documentation – The good stuff:
http://www.gradle.org/docs/1.8-rc-1/release-notes.html — shows the change from mavenRepo to maven.
http://gradleproject.wordpress.com/2013/02/14/multiple-maven-repositories-in-gradle/ – now NOT to set up your repositories entry.

More Documentation – The list of things that don’t work:
http://gradle.1045684.n5.nabble.com/using-Archiva-maven-repo-w-Gradle-td4579298.html
http://joshdiehl.com/2011/07/11/using-apache-archiva-with-gradle/