Agile development is a big ol’ turd which only serves two purposes: to keep academics busy and to promote useless people to agile coaches so they don’t accidentally break anything valuable.
Agile seems alluring: don’t do any planning, split everything into sprints of 1-3 weeks, make sure the code is production ready at all times, save time and money by reducing overhead and give the power back to developers.
Except it is none of that. Agile is trying to sit between two chairs, and not only does it plant its fat ass solidly on the floor, it also tips over that expensive vase and also the floor is sharks. It is the worst of all worlds in one terrible packet.
Let me tell you why agile is a bunch of garbage.
Agile Leads to Under-generalization
In agile, everybody has to just think in terms of one sprint. That means that we should not try to anticipate future needs, but just implement the simplest that works.
That leads to situations where a feature is first implemented in one way, only to have so many patches it becomes not only the ship of Theseus, but instead a horseboat clobbered together using rocket parts with a finely masked web for the bottom.
Let me give an example. In a project, we send confirmation mails to customers. Ok, put in a template with some HTML and junk and a bit of generated content based on the order and name. Easy-peasy, medium t-shirt including connecting to the mail server.
Now, we also want to send mails for other actions, like creating an account or resetting a password. Refactor the templates, so it is possible to change visual styles centrally and set up two new templates. Small t-shirt since we already have the mail infrastructure.
Now, the customer wants to send mails in different languages. Refactor and move the messages to language-specific files. The template is a bit of a mess with fragments of sentences to allow customising the text in multiple languages with differing grammatical structures while still generating some content. Mails now comprise three templates and two language files. Small t-shirt since we already have almost all of the infrastructure for mails and basically just have to move the messages out.
Next, mails should have slight variations; a welcome greeting for new customers and a welcome back for returning customers. One TOC if they purchase electronic goods only and one if they buy physical goods as well, as electronic goods have different return policies. These mails are really just the same, so instead of copying the templates, we use simple conditions in the template to add or remove a few sentences. Small t-shirt since this is really just a tweak to templates and a few more texts.
Now we have 8-10 different mails – nobody really knows the exact number because the variations are subtle and needs to be gleaned by inspecting several templates in rather intricate detail. The customer gets message back from the design bureau and we need to update the visual style, but that’s a small t-shirt because we already have the templates, and when, two weeks later, the final copy is ready, it is another small to update the texts.
To support “easy” changes in the future, we implement logic so templates can be embedded in the application or loaded from an external location. That way the application can run without the external location and mails can be updated without updating the application. Everybody is happy.
No decision made at any point is wrong. Unironically. Every decision made is sound and well-reasoned at the time. But the end result? Holy shit that is a mess. To adjust a single mail, we have three templates, two language files with multiple fragments each, all split over 1-2 locations on 3 or so different environments. This is a configuration hell.
The customer assumes, and rightfully so, really, that changing the text or even layout of a message should be simple, but it requires cross-referencing a handful of files, mentally tracing thru all the permutations of mails to make sure than any combination makes sense in multiple languages. And long back-and-forths to try and establish all variations of mails. Add to that that the external location has been disabled because that just made everything an untestable mess, so everything has to churn thru a 1 hour CI/CD pipeline towards the acceptance environment.
And this example is a simplified version of the real case.
The mail system should have been implemented in one of two ways: templates should be put in a versioned database so customers can change them themselves. Templates should either be completely flat – one mail, one template, no conditionals or shit – or have a layout template with header and footer and a few well-defined blocks (like an order list). One template per language. That would lead to some duplication, but that complexity would reflect the complexity of the setup and hopefully lead to fewer permutations of mails.
Any agile coach will uselessly tell you that you just have to refactor the code, which is all good and well. But when does the boiling frog jump out? Is simple textual changes suddenly a size large because we want to fix the mess we got ourselves in? Because that is a hard sell after doing the design as a small change or even being the hero implementing 4 variations as a small change.
Then do the refactoring piece-meal the agile coach says while pretending the sharks circling are really just Left Shark and not about to devour them whole (and rightfully so). Yeah, good luck with that during a two week sprint, including retrospectives and sprint-planning and planning poker and a dozen more dumb things that are definitely not overhead but integral to the agile experience. And you have to end with a ship-ready product after those two weeks (really 1.5 weeks due to not-overhead, and really-really 1 week because the testers need time too and that sprint-demo is not going to plan itself or be fun if somebody commits an exciting untested change the day before), and everything goes to master every evening because no breaking merges near wrap-up of the sprint, so good luck doing a large restructuring.
Yeah, you can refactor some things some of the time, but good luck getting all of the things fixed all of the time. The original mail templates were not general enough for the future changes, and the agile approach encouraged changing the initial design to a point where decisions made turned out to be wrong down the line. Somehow, a boat built out of rocket parts turns out to be a bad horse.
Agile Leads to Over-generalization
Oooh, that’s so clever: first I have a headline saying under-generalization, then one saying over-generalization.
Naturally, developers try to compensate for the under-generalization inherent to agile approaches. They do that by creating parts that are more general to anticipate future changes. Just the part that I said that agile tells you not to do in the previous section.
Again, the rationale is sound: by making something more generic now, we end up with a more coherent design, hopefully avoid building a terrible patchwork, and save some time overall by “getting things right the first time.”
The problem is that an agile approach invites getting started and sort out the details later. The customer is invited to make changes to the design as development progresses. That means that the stream-lined engine-room you prepared for your boat makes your horse’s ass look big and also is of very little use. At the same time, you didn’t anticipate a boat needing a bridle and the saddle is really just a badly repurposed chair from the bridge of the boat. I did not think this analogy thru, so I’m learning so much about horses and boats. Also, in the next section I’ll switch to another analogy (two even!) without even acknowledging that fact.
In a project, we needed to get some prices from a teller system. The plans for the system were grand at the time, so the system was set up to support products with different pricing for different kinds of customers and also depending on locations (think special prices for members and differentiated pricing based on eat-in or take-away). At the end of the day, the system does pretty much nothing with all of these prices and product details, and only really needs roughly a dozen product prices and not the hundreds we synchronise. That means the system now has a big highly structured product database strapped on, and 2-3 different ways of getting the price you actually need, because the way this was done changed throughout the project. The horseboat made of rocket parts has an unused machine room and a saddle made by sawing legs off a bar stool.
Again, I do not think anybody involved did anything wrong at the time. Experience teaches that there will be changes, so it may be prudent to plan ahead. Some of the generality really did turn out to be an advantage and some not so much. Had it been done less generic, we’d have ended up with more patchwork and fewer machine rooms and it is not obvious which is better (it’s all bad).
Agile Leads to Gold-plated Turds
Agile projects start with a vision and markedly little planning. Everybody has their own vision, and the agile coach goes on about how this is an experience we all have to go thru and development is a process. Development starts with whatever seems most important that day.
Then one day, the first release or demo hits. Now, there’s something concrete to talk about. All the color enthusiasts come out and have all the opinions about the color of the bikeshed. There are still 20 sprints ahead and the customer is king, so there is all the time to change the color of the bikeshed and move the sun beds around on the Titanic.
The bikeshed goes thru 5-8 sprints of iteration, and becomes very, very pretty. The sun beds are prefectly aligned and are located just where they are needed. The first features look great, but suddenly we are at sprint 10 out of 25, and we have really only iterated over the first 3-5 sprints. We have had preciously little time to pick up new things, because our 1 week of actual work time per sprint has been reduced to 2 days since we had to deal with customer wishes. What was urgent and important took precedence over what is not urgent but still important, and we have now spent 40% of the budget on 15% of the work.
The best case is that ambitions get lowered, the average case is that ambitions are kept but corners are cut, and the worst case is that ambitions stay the same and effort is now spent on getting the walkway to the bikeshed looking perfect or getting the parasols aligned perfectly for the sun beds. With another 15 sprints to go, it is still possible to get a positive result, and it is also possible to tell yourself this is just a small delay and we’ll make it up later.
What you can be certain of, is that the first sprints to get cut are the ones without new features but with only refactoring, only exacerbating the problem from the first section. With everybody getting stressed it can be harder to pick up the complex tasks, like building the nuclear reactor the bike shed is just a minor accessory to, instead of getting the quick wins that show repeated progress with minimal effort like raising the tent for the opening reception. At the same time, there is an expectation management challenge in convincing the customer not to change the color of everything 7 times to see which one is nicer, even though they have been trained to do just that, throughout the past 5 sprints.
At the end of the day, the project will have spent too much time on some things, and there is a very real risk that the things spent time on are not the most complex ones that really should have had the most attention. This is again not really anybody’s fault. It is natural the developers and project leads want to cater to customer requests. The first color change is no big deal, and if you’ve done four already, what makes the fifth change unreasonable? The customer likely doesn’t do a lot of projects and are learning as the project goes on: they do as they are taught, even if the lesson is implicit and not what anybody wants.
Agile Leads to Delays
By far, the biggest problem in any IT project is third party integration. Doubly so if it involves changing hardware and/or network connectivity.
I was working on a project for a customer, where we were 3 IT suppliers working on 3 different applications that needed to work together in one landscape. The landscape was hosted in two different data centers and a third data center needed to connect to parts of the system, as did the customer location. For the project, we also needed new hardware, and the application were all introducing new endpoints, some internal and some towards the other applications.
I had already been part of a similar but smaller project with the same parties where we went thru the fun 2 months management journey of “connecting two DCs” and the one month “agreeing what this WSDL means”-celebration. To connect our office to the data center of one of the other parties, I needed to communicate my wishes to somebody from another team (the network team) within my company. He would then consult some network diagrams and take contact to the two (both out-sourced) data centers involved in the connection on our side, as well as somebody from the other company to set up their side of the connection. It was not a trivial connection, opening around 10 ports on around 5 environments with 3-4 different connection paths, involving at least 3 VPNs and 6 teams from 4 companies, but it was also not exactly rocket surgery. Anyway, it kept the poor PM busy because he had to ask for status and escalate daily or more for weeks near the end. Similarly, we had some issues with a software interface, which was only prolonged due to long release cycles.
Anyway, with that experience in hand, when the customer asked for an agile approach, I nodded, smiled, and said yes and largely ignored it. I made a classic implementation plan focusing on the big hurdles first: the new network and software connections. I strongly suggested that the first couple of sprints be dedicated solely to get the connections working, and in our plan extended our application with the new interftaces and a stub implementation. That way, the other parties could immediately exercise the connections, test the network was open, and check that the requests and responses looked the way they expected, and the testing guys could start devising tests and debug them against a kind-of live system. The other parties did not; they even threw stupid agile mantras at me like “it has to be a functioning product” and “we should make business deliveries, not technical deliveries” etc.
I happily ignored that and instead went on my way implementing technical services towards others in our system, before building internal services on top of that, and finally building external functionality. The user-interface was developed in parallel, typically a sprint or two behind the backend services, and the tests were done another 1-2 sprints later. Such a staged approach is allowed by some agile persons, but not by purists and definitely not agile coaches.
It may be too subtle, especially for agile coaches and other idiots, but take a guess who finished on time in this project? Nobody! We finished around 2 months before time, but to be fair we also had less work to do than the others. The other parties finished 3 months late on a 6 months project. They found out that network connection were not set up in the “small t-shirt” at the end of the penultimate sprint and that “making sure that two systems both adhering to the specification can talk to one another” is also a t-shirt and probably of a size that would fit a gamer (and not the e-girl kind).
The staged approach allows full usage of a sprint, as testers test what was developed in a previous sprint, so the sprint does not have to end early for the developer to allow the tester time with the system.
We would probably also have finished on time had we adhered to the agile manifest and other bad ideas, but only because we had 4 months of work and 6 months to do it.
If you adhere to an agile approach, you will get delays because you cannot move technical high-risk parts to the front because they have no business value. You are therefore at a real risk of only being able to properly test third party integration near the end because both parties need to have decided that their end of the connection is more important than coloring the horseboat green before it can be tested.
Concluding Remarks
While agile may seem like a good idea, all of the above issues are solved by basically being less agile. Traditional methods, like waterfall, also have issues; it is a very real problem that a developer and a customer may misunderstand each other and it is a problem that a developer delivers a very good boat if the customer really wanted a horse. A proper analysis and planning phase can take some of that away, but there will always be issues that only become apparent when a system is there.
The problem is: that is a hard problem. Agile claims to solve it, but only does so by hiding it away in color of the bikeshed discussions, where people discuss, come to an agreement, and feel better, but doesn’t actually resolve the hard problems. Waterfall “solves” this by making changes impossible or expensive. Neither is perfect, but I am leaning towards the solution of waterfall to be less bad. It is ok to not pretend that hard problems aren’t hard, so making a hard change to the system should be met with resistance. Repainting the bike shed from lavender to lilac seems a lot less important if it is immediately met with a quote for €10000.
To be absolutely clear, I don’t think acted wrongly in any of the above cases. I would have, have, and likely will do something similar in the future. Not all code can be the Euclidic ideal and business functionality does take precedence over code quality, and sometimes you need to do a quick fix. It is just very bad when your entire methodology encourages the wrong focus and hides problems by quick fix after quick fix. Because your horseboat of Theseus needs to be maintained for years after building, and no matter how expensive it was to build, it will be more expensive to maintain for the entire lifetime; doubly (or rather a couple of orders of magnitude) so if it has a lot of technical debt left over from features all implemented as 10 small t-shirts worth of effort rather than one or two large ones.
