Hofstadter's Law: It always takes longer than you expect, even when you take into account Hofstadter's Law. — Douglas Hofstadter, Gödel, Escher, Bach
Developers and their managers both want scope and delivery time to match, but despite working for a common goal they rarely do. Often the gap is so large that deadlines are missed, products become irrelevant, and teams can face burn out.
There are three major schools of thought on why Hofstadter’s Law exists:
- Software is Art
- Software vs Entropy
- Productivity Declines
Software is art. This school of thought describes software development as a new field (we don't have centuries behind us) that is inherently prone to underestimating challenges and describing exactly how long something will take. There are debatable claims made here, however, they correctly identify the industry isn't doing a great job at measuring work, which is a fundamental requirement to measure success ( Software is Art, We're Gardens, Not Engineers ).
Software vs Entropy. This school of thought describes that more moving parts in software makes it harder to grow, maintain, reason about, on-board new developers, and ship quickly ( Mythical Man-Month, Working software is only half the job ).
Productivity Declines. This describes software slowing down as a result of talent, process and environment. You may have great developers with processes that don't fit your needs, developers who don't meet your needs, or simply not giving developers the kind of space to dedicate focus to their mentally challenging work (10x developers, Context-Switching, Small Batches, Flow ).
Really, all three have an element of truth to them.
As a quick and dirty way to gauge the problem, we conducted an informal survey of 50 people who self-identify as professional programmers and managers (those who write code or manage ‘every day’). While the sample size is small, this nonetheless serves as a signpost for the types of challenges software professionals find themselves running into every day.
The goal wasn't to diagnose or find the reason, but rather understand perceptions and mindsets for different stakeholders and the way teams work.
Key data points were:
- How often does software get released (bugfix, features, everything)
- Are you a manager, developer, or both?
- How often does one of these slow you down:
3a. Technical debt or underestimated challenges
3b. Product requirements change while work is in progress
3c. Business direction changes while work is in progress
3d. Developers are deprived of focus
- What can improve things?
- What can slow things down?
What do we believe slows development teams down?
Overall, all groups had some level of issue with each category, but when aggregated by group there were notable differences in perception. Possibly even more interesting is that developers who are also managers perceive all issues as being more frequent than their peers who only fill one of these two roles.
It's widely understood that the least important thing gets the least focus, so if we adjust for "perception" by ranking the lowest score as zero and highest as 100 we can get an idea of what is more important than the other, based on respondent type.
Here's a look at that projection:
The headline here is that Developers and Dev Managers are not having the same conversation that Managers are. Anecdotally, we know developers are always asking for time to rewrite things. This could be due to one of two reasons:
- A communication breakdown about technology and product
- Faulty memory of events, or bias towards familiar tasks.
And lastly, breaking these challenges down by how often software goes out looks like this.
The key insights:
- All groups believe interruption / lack of focus is a problem, and it outranks most problems (So why are open offices so popular?).
- Teams that ship every month or beyond (including 3 months and yearly) see product requirements changing while work is in progress as a huge challenge, and have less problems with technical debt or underestimates than their peers.
- Teams that ship every day have very few problems with the design team changing work, but they frequently have troubles with technology (more on this later).
- Teams that ship every week get no love in any category :(
According to Managers and Developers, Agile was a huge driving force in helping teams ship faster. While there were a few who suggested it slowed their teams down, it was overwhelmingly supported. Specific shutouts went to Kanban.
Developers and developer-managers both suggested using new frameworks would improve their productivity.
A few developers were excited by new design patterns they could use to construct code, and this is a really healthy stance.
A few developing managers were excited about automating more work (especially with regards to UI testing), and performance improvements in build, test, and deploy speeds.
Some managers shared that constantly changing priorities, a failure to communicate, constant meetings, and helping others were the biggest restraints to their progress. Developer-Managers attributed slow automated processes, a bad developer culture, fire fighting, and helping others as their largest restraints.
Developers were the most vocal. Slow automated process, context switching (focus), changing priorities, too many meetings, high tech debt, overloaded sprints, adding requirements to existing sprints, lack of motivation, and unclear process were on their list of restraints.
What does this mean?
The largest obstacle seems to be a disconnect between perceptions of managers and developers.
As software moves through its life-cycle it starts with a need, gets evaluated against other priorities, scoped out and designed, developed, released, and people experience using it. Your process probably looks close enough to this that you can agree this is generally what happens, but obviously this is not perfect and universal.
There is a theory (called "The Theory of Constraints") that actually describes two schools of thought that are used when trying to improve speed and quantity of work shipped ("throughput").
- Cost Accounting: Focus on individual steps in a process being at full capacity at all times.
- Throughput Accounting: Focus on total throughput, individual steps don't need to be at full capacity.
They're radically different, and cost accounting feels so obvious and intuitive it can be a huge struggle to get your organization to abandon it. It is likely that individual managers are given incentives based on their specific group's performance.
This may work on battle fields, but perhaps not in software development, and we know for a fact that this mindset caused many manufacturing plants to close forever. The flaw is that individual steps become competitive instead of collaborative.
When considering human biases, it is clearly more than pure coincidence that managers think the bottlenecks are more severe on the business side, and developers think it is more severe on the tech side. Both observers are attributing the problem to the thing they are most exposed to. Software isn't the only place this happens: this is a human flaw.
Without data, someone who can correctly identify a problem must be able to (a) correctly identify artifacts, (b) correctly remember the relevant facts, (c) recall facts perfectly, (d) has correct reasoning and interpretation, (e) was not influenced by peer suggestion, (f) was not influenced by social trends, and (g) self-image/confidence was not a factor.
One of the single largest human flaws in the planning process is called "Loss Aversion". Put in simple terms, we all know that the average car driver feels a strong emotional pain when the price of gasoline goes increases 5%, but more of a mundane smile when it decreases 5%.
Put into software development terms, it can feel like we're gaining something when we don't take quality control into account because we can ship faster. Or perhaps we don't put in the work up front designing and scoping out the project because it would take too much time from the developers writing code. NASA released a report that describes how this sort of process compounds the price paid to resolve a defect [pdf].
Individual perspectives are too narrow to correctly identify the issue. Broader perspectives can identify true bottlenecks and flawed processes. The only way to fix a flaw is to know where it is.
Developers who ship daily indicated their build process was in their way. Is it likely that these safeguards and processes slow down total business throughput, or is it likely that these processes prevent bugs in production? Well, without real data, you'll never actually know.
Easy-to-grab metrics in software are either unfair or just vanity metrics. But that doesn’t mean we should just throw up our hands and give up.
Meaningful data can and should be used to answer questions such as:
- Did our process changes make things better?
- Are we polishing a feature to death at the cost of shipping it?
- How do Wednesday all-hands meetings affect productivity?
- Which engineers made the biggest impact last month?
- How much burn went toward paying down technical debt?
Meaningful data cuts through office politics and subjectivity, and takes the guesswork out of success.
Data + Culture Community
Data is part of the solution, the part that demonstrates an answer is not a guess. The missing part is something that will create a development culture that focused on throughput, without the heavy processes, and permits individuals to really reach that trifecta of autonomy, mastery, and purpose.
Processes can be useful, but it is a double edged sword. The heaviest process find themselves in places where teams don't know who they have to communicate with, or alignment towards a common goal is either disconnected or not pragmatic.
There is now a news letter sharing ideas in this topic, and you can subscribe right here:
Creating the technology can often be a challenge, however about creating the context for the people in your organization can be even harder. Put in as much time thinking about your team as humans as you do planning the technology.
1. Competing Priorities and Projects Behind Schedule
One of the greatest books I have read on this topic is "The Goal", which has been adapted to an IT services context in a book you may have already read "The Phoenix Project". The author of The Phoenix Project has recently released a handbook that can help you do a number of really valuable things in your organization such as: Prioritize based on value-streams, de-risk and automate processes, how to organize your teams, how to fit in time for personal development, and other useful solutions to your challenges.
I should also mention the The Mythical Man-Month, which has been the long-standing go-to-guide on how to think about projects in software development.
Lastly, there's a very useful article on Priceeconomics which explains Why Are Projects Always Behind Schedule.
2. Technical challenges
If I had to recommend one book to be on every developer's desk it would be The Pragmatic Programmer. This book describes techniques to help developers reason about ways to write software that gets the job done without over or under building. It will help you avoid some of the common pitfalls in thinking that can make new frameworks and rewrites seem like the perfect solution. It also offers a number of techniques you can use to increase productivity by looking at problems in a different way.
I haven also found a lot of technical challenges from from spoken language and communication. I found Eliezer Yudkowsky's thinking on this topic to be extremely useful in re-aligning how I think about words, and simply communication in general. I would recommend starting with 37 Ways Words Can Be Wrong, and slowly reading the entire set of articles in A Human's Guide To Words.