Did we Take the Agile Manifesto Too Far for System Design?
Avoiding extremes: striking a balance between BDUF and agile in system design.
👋 Hi, this is Thomas. Welcome to a new edition of Beyond Runtime, where I dive into the messy, fascinating world of distributed systems, debugging, AI, and system design. All through the lens of a CTO with 20+ years in the backend trenches.
QUOTE OF THE WEEK:
“A complex system that works is invariably found to have evolved from a simple system that worked. A complex system designed from scratch never works and cannot be patched up to make it work. You have to start over with a working simple system.” - Gall’s Law
In many organizations, the term 'Agile' has evolved far beyond its original intent, leading to a proliferation of processes, frameworks, and tools.
These Agile methodologies are often embraced with the promise that strict adherence will yield better software at a faster pace. Yet, this rigorous adherence frequently morphs into cumbersome bureaucracy, resulting in software laden with both intentional and unintentional technical debt that strays from the core philosophies of Agile.
Particularly in system design, it seems that the true essence of the Agile Manifesto has been overshadowed by an extreme interpretation. The Manifesto advocates for adaptive planning and evolutionary development but does not discard the need for thoughtful design.
Yet, some teams, in their zeal to become 'more Agile,' may have overlooked this balance, embracing rapid change at the expense of sustainable system architecture.
BDUF vs Agile
In the past, system design was dominated by the Big Design Up Front (BDUF) approach, especially when software and websites were extremely expensive to build.
This method, closely aligned with waterfall development practices, required that every aspect of a project be meticulously planned and finalized before any actual development began. The rationale was straightforward: minimize costly mistakes by extensively planning ahead.
However, this often led to projects that were consistently over budget, overdue, and overly complex, with engineers feeling disconnected and uninspired by their work.
Enter Agile, a methodology designed as a direct counter to the rigidity of BDUF. Rather than exhaustive upfront planning, Agile embraces adaptability through iterative development, rapid prototyping, and frequent releases.
The core of the Agile Manifesto is to build software in a way that is more in tune with the realities of changing business requirements, emerging technologies, and market dynamics. It advocates for a process that learns and evolves through ongoing feedback rather than one that claims to predict every need from the start.
This philosophical shift influenced not just software development but also the broader business practices within tech. Influential practices like Lean Startup and accelerators such as Y Combinator have championed the principles of moving quickly, iterating based on real user feedback, and minimizing upfront commitments. These practices underscore a fundamental shift from planning to experimentation, acknowledging that in the fast-paced tech world, adaptability often trumps prediction.
What the Agile Manifesto Says vs What we Think it Says
The Agile Manifesto advocates a preference for “[…] responding to change over following a plan”. However, this emphasis on flexibility is often misconstrued.
The manifesto explicitly values planning; however, it prioritizes adaptability higher. This distinction is crucial and often overlooked, as the creators of the manifesto themselves have clarified that “while there is value in the items on the right [following a plan], we value the items on the left [responding to change] more”.
Furthermore, the Agile principles underscore the importance of continuous design, stating, “Continuous attention to technical excellence and good design enhances agility.”
This highlights that effective system design is not antithetical to Agile methodologies but a pivotal aspect of it.
Despite these clarifications, many Agile teams have misinterpreted the manifesto's emphasis on “Individuals and interactions over processes and tools” and “The most efficient and effective method of conveying information to and within a development team is face-to-face conversation.”, taking them to mean that documentation and diagrams are unnecessary.
In today's software development, where systems often consist of hundreds or thousands of components, dependencies, and APIs, relying solely on memory and verbal communication is impractical.
Visual tools like diagrams are not just supplementary; they are essential. They help prevent the misunderstandings and delays that can arise from purely verbal communication by aligning mental models across the team.
Maintaining updated documentation and diagrams is critical in managing the complexity of modern software systems. They ensure that discussions are grounded in a clear, shared understanding of the system's current state and history, making these conversations more about moving forward than rehashing past decisions.
Implementing an Agile Framework vs. Being Agile
Over the past two decades, numerous processes, frameworks, and tools have emerged to help engineering teams adopt Agile. However, there's a profound difference between merely implementing an Agile framework and truly embodying an Agile mindset.
Adopting an Agile Framework often translates into rigid adherence to specific rules and rituals in many large organizations. This can lead to a plethora of ineffective meetings, products developed based on guesswork, bloated tasks, and oppressive deadlines. Such rigid implementations can stifle the very flexibility and responsiveness that Agile aims to promote.
A study by Engprax suggests that Agile projects have a significantly higher failure rate—268%. While this figure should be approached with skepticism due to criticisms of the study’s methodology and potential biases, it nonetheless points to common pitfalls in Agile implementations.
This misalignment often stems from the clash between Agile's ideals and organizational demands for stringent processes and reporting. This conflict can lead to endless cycles of hastily patched code, unfinished features, and dreaded, unproductive stand-ups. Such environments betray the original intentions of the Agile Manifesto, which emphasizes adaptability, team collaboration, and customer-focused iterative development.
System Design: Bridging Agile Purists and Waterfall Advocates
While it's clear that Big Design Up Front (BDUF) is not feasible—predicting the future with precision is impossible—it's also impractical to forego all planning in favor of pure agility. We cannot simply dive into development without any overarching vision; a balance must be struck to maintain coherence and avoid costly errors.
As Dave Farley aptly suggests in his perspective on what effective software architecture should entail, it involves establishing "a collection of constraints that we agree to adopt and maintain, which will provide us some protection from making dumb mistakes". This approach means that while initial designs will not be perfect and will require evolution as business needs change, having a structured yet flexible architectural plan is vital.
In essence, successful system design in the modern context lies in foreseeing potential issues without being rigidly tied to initial assumptions. It's about crafting a resilient framework that supports growth and adaptation while providing a clear path forward, thereby marrying the foresight of traditional practices with the adaptiveness of Agile methodologies.
💜 This newsletter is sponsored by Multiplayer.app.
Full stack session recording. End-to-end visibility in a single click.
I originally wrote about this topic in this article:
Grokking the System Design Review - everything your team should know
I explored these topics:
Big Design Up Front vs Agile Design
The Importance of “Some” Upfront System Design
How Much System Design Is Enough?
Beyond “Upfront” System Design
📚 Interesting Articles & Resources
“Learning about distributed systems: where to start?” - Murat Demirbas
Understanding distributed systems is more than just coding; it requires a foundational approach. This article outlines key areas to focus on, including predicate logic, impossibility results, and consensus algorithms, to truly grasp the complexities and unique challenges of distributed systems.
“A Crash Course on Distributed Systems” - ByteByteGo
Distributed systems consist of multiple computers (nodes) that work together to perform tasks, often spanning geographical boundaries. They offer flexibility, scalability, and reliability but come with unique challenges like coordination, consistency, and handling failures. This crash course explores the benefits and strategies to manage these complexities effectively.
“Distributed Systems Are Hard” - Anne Currie
Distributed systems are notoriously difficult to design and manage due to issues like network partitions, consistency challenges, and unpredictable failures. This article explores why these systems are so complex and offers insights into the inherent trade-offs involved.