Thursday, September 25, 2014

Agile Is Overripe

Haters Welcome

I wrote a blog post criticizing Scrum, and a bunch of people read it. A lot of people seemed to be talking about it too. I started regularly seeing 50+ notifications when I signed into Twitter, which was a lot for me.

There weren't a lot of people defending Scrum. Most of the tweets looked like this:

Of the tweets which defended Scrum, they mostly looked like this example of the No True Scotsman fallacy:

I've seen this from people who are old enough to know better, including one Agile Manifesto co-author, so it's entirely possible there's a little war afoot in the world of Scrum, over how exactly to define the term. Sorry, Scrum hipsters, but if there is indeed such a war, you either are losing it, or (more probably) you already lost it, years ago. I'm going to use the term as it's commonly understood; if you have an issue with the default understanding of the term, I recommend you take it up with Google, Wikipedia,,, and so on and so forth. I don't care enough to differentiate between Scrum Lite and Scrum Classic, because they both taste like battery acid to me.

However, I did get one person - literally only one person - telling me that Scrum actually works, and that includes planning poker:

(As it happens, it's someone I know personally, and respect. Everyone should watch his 2009 CUSEC presentation, because it's deep and brilliant.)

Another critic ultimately led me to this blog post by Martin Fowler, written in 2006:

Drifting around the web I've heard a few comments about agile methods being imposed on a development team by upper management. Imposing a process on a team is completely opposed to the principles of agile software, and has been since its inception...

a team should choose its own process - one that suits the people and context in which they work. Imposing an agile process from the outside strips the team of the self-determination which is at the heart of agile thinking.

I'm hoping to find out more, later, about what it's like when you're on a Scrum team and it actually works. To be fair, not every Scrum experience I've had has been a nightmare of dysfunction; I just think the successes owe more to the teams involved than to the process. And regarding Fowler's blog post, a lot of the people who endorsed my post seemed to do so angrily. So I would guess that many, many of these "fuck yeah" tweets came from people who had Scrum imposed on them, rather than choosing it. And therefore I think both of these areas of criticism are worth listening to.

However, of all the criticisms of my blog post that I saw, literally every single one overlooked what is, in my opinion, my most important criticism of Scrum: that its worst aspects stem from flaws in the Agile Manifesto itself.

Quoting the original post:

I don't think highly of Scrum, but the problem here goes deeper. The Agile Manifesto is flawed too. Consider this core principle of Agile development: "business people and developers must work together."

Why are we supposed to think developers are not business people?


The Agile Manifesto might also be to blame for the Scrum standup. It states that "the most efficient and effective method of conveying information to and within a development team is face-to-face conversation." In fairness to the manifesto's authors, it was written in 2001, and at that time git log did not yet exist. However, in light of today's toolset for distributed collaboration, it's another completely implausible assertion...

In addition to defying logic and available evidence, both these Agile Manifesto principles encourage a kind of babysitting mentality.

Sorry, Agile, I am in fact both a business person, and a developer, at the same time. Since my business involves computers, being competent to use them only supercharges my business mojo. This is how I achieve the state of MAXIMUM OVERBUSINESS.

More seriously, I recently started a new job at a company called Panda Strike; our CEO convinced me that the real value in the Agile Manifesto was that it facilitated a change in business culture which was actually inevitable due to a technological shift which happened first.

Moore's Law Created Agile

Agile development replaced waterfall development, an era of big design up front. In waterfall development, you gather requirements, write a spec, get approval on the spec, build your software to match that spec, then throw it over a wall to QA, and only show it to your users once you're done. It's important to realize that big design up front powered a ton of incredible success stories, including putting astronauts on the moon, plus nearly everything in software before the late 80s or early 90s, with the possible exception of the Lisp machine.

I don't want to bring back that era, but to be fair, we lost some things in this paradigm shift. And I think it's pretty easy to imagine how rapid prototyping, iterative development, and YAGNI might all be inappropriate for putting astronauts on the moon. That kind of project wouldn't fit a "design as you go" mentality. It would look like something out of The Muppet Show, except people would die.

In the very early days of computing, you'd spend a lot of time working out your algorithm before turning it into a stack of punch cards, because you wouldn't get a lot of chances to run your code; any error was very expensive.

Big design up front made an enormous amount of sense when the machinery of computing was itself enormous also. But that machinery isn't enormous any more, and hasn't been enormous for a long time. According to someone who's done the math:

a tweaked Motorola Droid is capable of scoring 52 Mflop/s which is over 15 times faster than the 1979 Cray 1 CPU. Put another way, if you transported that mobile phone back to 1987 then it would be on par with the processors in one of the fastest computers in the world of the time, the ETA 10-E, and [those] had to be cooled by liquid nitrogen.

Like all benchmarks, however, you need to take this one with a pinch of salt... the underlying processors of our mobile phones are probably faster than these Java based tests imply.

In between the day of the Cray supercomputer and the modern landscape of mobile phones which can run synthesizers good enough for high-profile album releases and live performances, there was the dawn of the personal computer. As the technology got smaller, faster, and cheaper, Moore's Law rendered a whole lot of management practices obsolete. Development cycles of two entire years were common at the time, but new teams using new technology could churn out solutions in months rather than years, and PowerBuilder developers launched a revolution underneath COBOL devs, starting around 1991, in the same way Rails developers later dethroned Java, starting around 2005, after it became possible to build simple web apps in minutes, rather than months.

In our lifetimes, it may become possible for software-generating software to churn out new apps in seconds, rather than minutes, and if/when that occurs, the culture of the tech industry (which, by then, may be equal to the set of all industries) will need to change again. It's hard to see that far with accuracy, but as far I know, there are basically just two ways a business culture can transform: evolution and persuasion. Evolution is where every business which ignores the new reality just fucking dies.

Persuasion is where you come up with a way to sell a new idea to your boss. This is pretty much what the Agile Manifesto was for. In the early days of Agile, the idea that your boss would force it on you was a contradiction in terms. Either you forced it on your boss, or it just didn't happen at all.

Obviously, times have changed. Quoting Dave Thomas, one of the Agile Manifesto's original authors:

The word "agile" has been subverted to the point where it is effectively meaningless, and what passes for an agile community seems to be largely an arena for consultants and vendors to hawk services and products.

So I think it is time to retire the word "Agile."

Epic Tangent: Ontology Is Overrated

One of the best tech talks I've ever heard, "Ontology Is Overrated" by Clay Shirky, covers a related topic. It's ancient in web terms, hailing from all the way back in 2005, when Flickr and del.ici.ous were discovering the incredible power of tagging, something we now take for granted. The talk includes an interpretation of why Google crushed Yahoo, during the early days of Web search engines. A sea change in technology brought with it a philosophical sea change, which Yahoo ignored - even going so far as to re-establish obsolete limitations - and which Google exploited.

I'll summarize the talk, since text versions don't appear to be online any more. You can still read a summary, however, or download the original audio, which I definitely recommend. It's a talk which stuck with me for almost ten years, and I've heard and given many other talks during that time.

When you look at the Dewey decimal system, which librarians use for storing books on shelves, it looks like a top-down map of all ideas. But it fails very badly as a map of all ideas. Its late 19th-century roots often become visible.

Consider how the Dewey decimal system categorizes books on religion, in 2014:
  • 200 Religion
  • 210 Natural theology
  • 220 Bible
  • 230 Christian theology
  • 240 Christian moral & devotional theology
  • 250 Christian orders & local church
  • 260 Christian social theology
  • 270 Christian church history
  • 280 Christian denominations & sects
  • 290 Other & comparative religions
Asian religions get the number 299, and they have to share it with every tribal and/or indigeneous religion in Australia, Africa, and the Americas, as well as Satanism, Discordianism, Rastafarianism, and Pastafarianism. Buddhism, however, shares the number 294 with every other religion which originated in India. So at best that's a number and a half, out of 100 available, for Asian religion, and all associated topics. Asia contains about 60% of the world's population.

As a map of all the ideas about religion, this is horribly distorted, but it's not actually a map of ideas about religion. It's really just a list of categories of physical books in the collections of American libraries.

Before Google existed, Yahoo first arose as a collection of links, and soon grew large enough to be unwieldy - at which point, Yahoo hired an ontologist and categorized its links into 14 top-level categories, creating in effect a Dewey decimal system for the web. But Yahoo innovated a little, bringing in an element of Unix. If you clicked on the top-level category "Entertainment," you'd get a "Books@" link, where the little "@" suffix served to indicate a symlink. Clicking that would land you in "Books and Literature," a subcategory of "Arts," because according to Yahoo, "Books" were not really a subcategory of "Entertainment."

Librarians use a similar workaround in their systems, namely the fractional decimals which indicate subcategories, so you can say (for example) that a book is about Asia, and about religion. These workarounds are inevitable, because (for example) books can be both literature and entertainment. Or, to be more general, categories are social fictions, and to put a book about Asian religion in the Asia category, rather than the religion category, is to say that its Asian-ness is more important than its religion-ness. The hierarchical nature of ontology means it always imposes the priorities of whichever authority or authorities created the hierarchy in the first place. But with a library, you have an excuse, because a physical book can only be in one place at a time. With web links, there's no excuse.

So, rather than applying this legacy physical-shelf-locating paradigm to a set of web pages, Google allowed you to simply search the entire web. You could never expect librarians to pre-construct a subcategory called "books which contain the words 'Minnesota' and 'obstreperous,'" but Google users in 2005 could work with exactly that subcategory any time they wanted. Flickr and took these ideas much further, creating ad hoc quasi-ontologies by allowing users to tag things however they wanted, and then aggregating these tags, and deriving insight from them.

(Today, unfortunately, you might not get results containing both "Minnesota" and "obstreperous" if you searched Google for those words. Google's lost a tremendous amount of signal through its use of latent semantic indexing to detect synonyms, and to other, similar compromises. This diminishes the Google praise factor in Shirky's talk, but doesn't harm his overall argument in any important way. What does suggest a possible need for revision is the emergence of filter bubbles, where companies try to pre-emptively derive user-generated categories, and then confine you to them, based on what category of user they estimate you to be. Filter bubbles thus impose a new kind of crowd-sourced ontology, which holds serious dangers for democracy.)

Anyway, although this was a fantastic talk, the main point I want to make is that Google defeated Yahoo here by recognizing the whole concept of ontology for the unnecessary, inessential historical relic that it was. Google even briefly used the DMOZ project, an open-source categorization of everything on the web - yes, this actually existed, and it started life with the name Gnuhoo, because of course it did - but dumped DMOZ because nobody even used it when they could just search instead. Ontology is overrated, and Yahoo's failure to recognize that cost them an enormous market.

The Agile Manifesto existed because developers and consultants had begun to recognize that many ideas in tech management were unnecessary, inessential historical relics. Although it opposed these ideas, it didn't even argue that they should be thrown out entirely, just that they were overrated.

Remember, waterfall development reigned supreme. The Agile Manifesto did a great thing in improving working conditions for a lot of programmers, and in achieving new success stories that would have been impossible under the old paradigm. But I can't praise the Agile Manifesto for tearing down the status quo without also acknowledging that over time, it has become the new status quo, and we will probably have to tear it down too.

Synchrony Is The New Ontology

The most obvious flaw in the Agile Manifesto is the claim that face-to-face conversation is the best way for developers to communicate. It's just not true. There's a reason we write code onto screens, rather than transmitting it as oral history like an epic poem from before the invention of the written word. Face-to-face communication has a lot of virtues, and there are certainly times when it's necessary, but it's not designed to facilitate extremely detailed changes in extremely large code bases, and tools which are designed for that purpose are often superior for the task.

Likewise, I don't want to valorize a tired and harmful stereotype here, but there's a lot of development work where you can go days without needing to talk to anyone else for more than a few moments.

In many industries, companies just do not need to have synchrony or co-location any longer. This is an incredible development which will change the world forever. Do not expect the world of work to look the same in 20 years. It will not.

It's not just programming. Overpriced gourmet taco restaurants no longer need locations.

In 2001, when the Agile Manifesto was written, Linux was already a massive success story for remote work and asynchronous development. But it was just one such story, and somewhat anomalous. In 2014, nobody on the web is building a business without open source. Because of that fact, and because of the fact that just about every open source project runs on remote work and asynchronous development, we can also say that there are very, very few new technology companies today which do not already depend on the effectiveness of remote work and asynchronous dev, because these businesses would fall apart without their open source foundations, and those foundations were built with remote work and async dev.

The bizarre thing about most companies in this category, however, is that although they absolutely depend on the success of remote work and async dev, and although they absolutely and literally could not exist without the effectiveness of remote work and async dev, they nonetheless require their employees to all work in the same place at the same time.

Consider that GitHub's a distributed company where a lot of people work remote. Consider also that a lot of startups run development entirely through GitHub. This means a lot of CTOs will happily bet their companies on libraries and frameworks developed remotely, and a product which was developed remotely, yet they don't do remote dev when it comes to running their own companies.

Yahoo put ontology onto its web links simply because it never questioned the common assumption that if you want to navigate a collection of information, you do so by organizing that information into a hierarchy.

Why do tech companies have offices?

In this case, the Agile Manifesto just went stale. It's just a question of the passage of time. The apps, utilities, and devices we have for remote collaboration today are straight-up Star Trek shit by 2001 standards.

In 2001, when the Manifesto was written, you could argue against Linux as a model for development in general. Subversion was still new. Java (developed at a corporation, inside office buildings) was arguably superior to Perl, which was probably the best open source alternative at the time. There weren't profitable, successful companies built this way. You could call Linux a fluke. But we have profitable, successful, remote-oriented companies today, and legions of successful open source projects have validated the model as well.

A software development process that doesn't acknowledge this technological reality is just silly.

Two-year development cycles and big design up front were to 1990s programming as ontology was to 1990s web directories. They were ideas that had to die and Agile was right to clear them away. But that's what synchrony and co-location are today, and the Agile Manifesto advocates in favor of both.

And this synchrony thing isn't the only problem in the Agile Manifesto. I may blog in future about the other, deeper problems in the Manifesto; I already covered the "businesspeople vs. developers" problem in the Scrum post.