Incremental Software Delivery
Delivering working software in small increments every sprint requires a different way of working. To reduce overhead, teams will often look to technical practices to enhance their ability to deliver. In this blog, we explain Incremental Software delivery.
Choosing the right technical practices will increase our team's agility by giving them the confidence that what they are delivering is well-designed, tested, and meets expectations. By improving the team's confidence, we will speed up our ability to deliver.
In this blog, we'll look at some of those technical practices and how they work with incremental software delivery approaches.
Building the thing right versus building the right thing
Continuous Integration/Deployment/Delivery and the DevOps culture
Building the thing right versus building the right thing
The practices that we use are often known as "the intangibles" of software delivery because from an outsider's point of view, they aren't visible as part of the features we deliver but they have the potential to help us build a much better product.
"Building the thing right" means a focus on crafting our software.
"Building the right thing" means focusing on getting good outcomes for the customer.
Unfortunately, sometimes the bias can be towards the latter, the pressure to deliver often being greater than the desire for quality.
Poor quality manifests itself in a few ways: poor performance (slow), doesn't work as expected, takes forever for the team to make requested enhancements, and so on. As a customer, this may only be something that you become aware of after being on the receiving end of poor-quality software.
An incremental approach should help relieve the pressure on the team to deliver. A customer will likely be less nervous about what their money is being spent on if they can see and give feedback on the software. If the customer can use the increments provided so far, it's a win-win.
And although it may seem counter-intuitive, focusing on quality and adopting practices that build it into our software from the beginning will speed up delivery. Why? Because we avoid accumulating something called technical debt.
The Software Industry and the Agile Manifesto, when discussing the Agile principle continuous attention to technical excellence and good design enhances agility. We defined it as follows:
Technical debt is a term first coined by Ward Cunningham; it describes the accumulation of poor design that crops up in code when decisions have been made to implement something quickly.
Ward described it as technical debt because if you don't pay it back in time, it starts to accumulate. As it grows, subsequent changes to the software get harder and harder. What should be a simple change suddenly becomes a significant refactor/rewrite to implement.
When you first started out writing software it was fast and easy, just like when you first started mowing the field. We didn't always take the best or straightest path, and in our haste to get the job done, we probably neglected consistency. But when the field was green, we didn't need to think about things like that.
In his blog post, Ron visualizes something like the following:
Over time, areas of our code get little neglected and small bushes start to form. One day, we find that we don't have time to remove the shrubs, so we begin to mow around them. This is similar to parts of our code that we find hard to maintain.
Instead of tackling the problem head-on and fixing the code, which would slow us down, we go around it, for example by creating a similar code that does a slightly different job. The field starts to look like this:
Soon, it gets harder and harder to navigate the area as the bushes get bigger and more numerous. It takes longer and longer to mow the field; at some point, we may even give up trying to mow it at all, only going into that area when we have to. We start to think of this code as "legacy" and consider ways we can replace it. Sometimes, it's just easier to blow it all away and start again:
All of the practices that we describe in this section are aimed at preventing those thickets and bushes springing up in our software. If we maintain a disciplined approach to building software using these techniques, it will ultimately speed up our delivery, increase our agility and our product's medium to long-term viability.
In a nutshell, refactoring is the art of small and continual improvements to the design of our code while preserving its behavior. The intention is to create behavior-preserving transformations to our system which ultimately make it more maintainable.
Each time we change parts of our software, we purposely refactor parts of the code that is in our path. To ensure we preserve current behavior, we use automated tests which tell us if our code is still working as we refactor.
Using Ron's analogy of fields, thickets, and bushes from the previous section, instead of mowing around the bushes, with refactoring, we cut a path through each bush we encounter. It looks a little like the following:
Coding standards are hygiene factors which should be possible to automate using tools such as SonarQube—an open source tool for automating code reviews and the static analysis of code.
One of the principal causes of bugs in our software is complexity, primarily because it makes our code hard to read. If the code is hard to comprehend, it causes developers to make mistakes because they misinterpret how it functions.
Over time, it will even affect the developer who wrote it, and they may struggle to maintain their original intentions.
We should always be thinking about how we can make things more easily readable. Sound naming conventions, fewer lines per method, fewer paths, and loose coupling of code are just a few examples.
Remember, it's relative. Some areas may need more attention than others. Definite red flags requiring attention are units of code with many lines; aim to decompose these into more manageable chunks.
Also, look for places that have high cyclomatic complexity. These are hotspots for attention because the code in these locations has multiple paths running through it, often obscuring some or all of its purpose.
You can refactor without automated tests, but you have to do it carefully. You will need to manually test a lot and use tools that you trust, for example, there are automated refactoring tools for Java and C#.
For instance, most modern IDEs have simple refactoring support, so when changing a method signature, the IDE will help locate and modify it throughout the codebase.
Some tools are purpose built and more sophisticated than those provided by an IDE. Re-sharper by JetBrains is one example which plugs into Visual Studio.
How does this keep us Agile?
It reduces the thickets and bushes that start to spring up in our software; it prevents them from taking firm root in our code. This reduces the time it takes for us to introduce additional functionality or enhance and maintain existing features, making us much more reactive to our customers' changing needs.
We can apply this to our code. If we always leave it in a better state than when we found it, over time it will maintain its usefulness. There will be less opportunity for those thickets and bushes to grow.
Remember, refactoring should be treated as making small, continuous improvements to our software.
How does this keep us Agile?
Let's just state up front that the topic of Test-Driven Development can and will cause polarizing responses among software Development Teams. People either love it or hate it.
I believe much of the controversy is caused because we fail to see TDD for what it is: a specification-driven test harness for designing and writing more straightforward software. TDD packs so much punch in the Agile community because it encourages a mindset of building what is necessary and nothing more.
Simple software is easier to maintain, more robust, easier to scale, and lacks technical debt or feature bloat. If Scrum is a set of training wheels for better software delivery, TDD is the training wheels for better (simpler) software design.
The many benefits of using TDD include the following:
It's a specification-focused approach that reduces complexity because we're less likely to write software that we don't need It makes our design simpler and our code clearer
Refactoring refines our design and makes our code more maintainable, a step often haphazardly undertaken if done without TDD's specification-driven framework
Writing tests as specifications upfront improves the quality of our specifications and makes them more understandable to others
The automated test suite serves as documentation for our software—the tests act as the specification, and it's easier for new team members to understand what the code does
Having a suite of readily-repeatable tests gives us the confidence to release our software
The resulting tests support refactoring, ensuring that we can make the code simpler while maintaining its intended behavior
Some argue that TDD approaches take longer to write code, and this is true in the literal sense because we're designing and writing specifications at the same time.
However, when you consider all the other activities involved in software development, other than just "writing" code, this argument breaks down. Plus, the medium to long-term effects on our code base will result in significant cost savings. Plainly put, in my experience, TDD software has far less technical debt and far fewer bugs.
In fact, you should probably see TDD as a way of improving and validating your software design. It's a much more holistic perspective. It's testing++.
Things to try
Select a TDD champion, the person or people who most support the practice within the team. Have them set up the framework and create the initial approach.
Choose one User Story to implement TDD on. Once the User Story is done, have the champion/champions workshop their findings back to the team and teach them how they can implement TDD as well.
The team should then select the next User Story they'd like to TDD and then pick a team member to pair program the solution with the TDD champion.
Alternatively, if the TDD champion is already a confident TDDer, they could introduce the whole team to the Test-Driven Development approach using workshops, or pair or mob programming.
Whichever approach you take, you should work in that configuration as a team until the User Story is "done," that is, delivered into the hands of your customer.
A full SDLC end-to-end experiment is necessary if we're to understand the wide-ranging benefits this particular practice has. Remember, building software isn't just about writing code; our work isn't complete until it's in the hands of our customer.
Before starting, discuss pairing etiquette:
Agree on a start and finish time for the session. Ensure you schedule time for regular breaks.
Discuss how to avoid distractions. For example, turn off an email and instant messenger notifications. Also, have mobile phones turned to silent or preferably off.
Decide how to manage outside interruptions. For example, explain to the person or persons interrupting that you'll be on a break soon, tell them when that break is and that you can talk to them then.
Determine who will drive first and how often you'll exchange the keyboard. Make sure you swap the keyboard regularly. Do not allow one person to drive exclusively.
Accept that pairing is not silent, but like any new skill, we will need to learn how to describe what we're doing while coding. It's often odd at first.
Remember, don't just pair program the "hard bits," pair program for an entire story from start to delivery.
To keep things interesting, try pairing with different members of the team. Also, bear in mind that you both don't have to be software developers to make a pair.
For example, try pairing with a Product Owner, in particular, if the current work involves creating a tangible aspect of the system such as part of the user interface or another form of output. In these situations, they will be able to offer immediate feedback on the direction you're taking.
Pair programming ping pong
Pair programming ping pong combines the two practices of TDD and pair programming and turns them into a fun, collaborative game.
Two software developers, one computer
The usual pair programming setup
It starts in the usual pairing way, with the two developers using one computer, one acting as the driver, the other as the navigator.
We play ping pong in the following way:
The first developer writes a new test. It fails because there is no code to fulfill it yet. The keyboard is passed to the second developer.
The second developer implements the code needed to fulfill the test.
The second developer then writes the next test and sees that it fails. The keyboard is passed back to the first developer.
The first developer implements the code needed to fulfill the test and so on.
In the usual TDD manner, refactoring happens after the code to fulfill the test is written.
At the beginning of this blog, we looked at the birth of Agile and what provoked this movement. In the above section, The Software Industry and the Agile Manifesto, we discussed how, in an attempt to correct our inability to bring projects in on time and budget, we sought to get more precise in our predictions.
It was felt that if other engineering disciplines such as civil engineering were able to be more precise with their process, and software engineering should be more in line with it.
However, the reality is that we don't build software like we construct bridges or buildings or hardware. With all of those physical things, by the time we get to the actual construction phase, we've already created the design schematics. The construction phase is an endeavor in logistics based on assembling the different components in the right order.
As Neal Ford, software architect at Thoughtworks, describes it, with software, the code itself is our design. Our build phase in software is when we compile/interpret the program code so it can be run.
Our program's concrete outputs are the actual results of the design, in the form of pixels illuminated on the screen or messages sent over the ether. That is the tangible result of "building" the design we've written in our software's code.
So, if we're to get better at producing the right output, we have to get better at "designing" our code. Practices we've discussed so far in this blog will certainly help us evolve our design safely and efficiently.
For example, TDD creates a test harness based on our software's specifications. This allows us to design our software to the specifications prescribed in our requirements (User Stories, their associated acceptance criteria, and possible test scenarios).
TDD's red/green/refactor approach to software development helps us ensure that intended behavior will continue to work as we begin to make changes to our software's underlying structure, for example, improving its scalability.
How does this keep us Agile?
Software design isn't something we do before we start writing code, it's something we do as we write it. When patterns emerge, we can begin to take advantage of them, creating abstractions for reuse.
This helps us avoid being caught by the You Ain't Gonna Need It (YAGNI) principle of software design. YAGNI happens when we design software features for some perceived future need and not the actual needs we have now.
It doesn't mean that we just start programming. Having some understanding of the problem and formulating a strategy regarding how we're going to solve the problem is necessary to get us started on the right path. It does mean that we shouldn't try to solve problems we don't have yet.
Activity – emergent design discussion
Hold a time-boxed discussion with the team about software design.
What you'll need: The team, a table big enough for the team to sit around, a whiteboard if you want to record salient parts of the conversation, a timer. Setup: A roundtable discussion with a designated scribe if taking notes.
Compare and contrast software to civil engineering. Take a look at how they design, test, and build versus how we design, test, and build.
Discuss the statement, specifications are our design, the code is our design, testing is our design validation, build/compile is our construction phase, and the output we see on-screen is our building.
How does this keep us Agile?
CI has spread beyond the XP community to other Agile practitioners because it reduces the likelihood of integration issues. It also means we receive feedback earlier regarding how our code performs with that of other developers.
CI has significant benefits over source control strategies, such as feature branching, which creates the tendency to refine features until they are ready to release. Leaving the feature branch open for extended periods of time without committing back to the trunk increases the risk of collision.
Continuous Delivery (CD) is an extension of our CI practice and involves setting up push-button deployment processes so that our software can be deployed on request to the production environment. This requires an extra level of discipline on top of CI and means we will need to ensure the following:
Our software is always kept in a deployable state.
Our software's environment configuration is automated, which means we can deploy to any environment on-demand.
It often requires our software's deployment to be outage-less, meaning that new features are delivered seamlessly to our customer.
A deployment pipeline is set up which automatically ensures various steps are taken during each deployment.
Our team takes ownership and responsibility for the deployment and management of their software in the production environment. This means they are no longer reliant on another team and don't have to wait in line.
Setting up a Continuous Delivery process requires close collaboration between our development and operations teams. The cross-pollination of skillsets creates more understanding and ownership of our product in its operational environment within both teams. This is often referred to as a DevOps culture.
This level of automation often involves problem detection during deployment; if any issues are detected, the deployment can be stopped and rolled back immediately. This introduces resilience to our release process.
The benefits that CD brings us to include:
By deploying to production early and often, we significantly reduce the risk that something might go wrong during that later stage of our delivery cycle. We increase resilience in our product, and its overall stability because we can quickly roll forward and back in our production environment seamlessly.
Making changes to production daily irons out any potential problems early and makes deployment to our production environment an insignificant event, and less costly.
Deployment is treated as part of our development cycle; this solves two problems:
Our Development Team has ownership of end-to-end delivery through our deployment pipeline. We no longer have to wait for another team to do work for us.
Our business doesn't have to wait to receive the benefits of our good work.
We close all-important feedback loops by getting our software into our customers' hands as soon as possible.
Things to try
To set up a CD on top of the three stages already outlined in the CI section, we'll need an additional stage:
Implement a push-button outage-less deployment of our software to our production environment
Confirm success or highlight failure of the release to our human deployment technician
Implement automatic rollback functionality should a problem be detected during the rollout phase
Make errors visible to our human deployment technician
Once you have CI and CD in place, the final step is to remove the need for the push button and fully automate deployment to our production environment.
For a variety of business reasons, it sometimes makes sense not to switch on new features immediately. We can use strategies such as feature toggles to hide or show features once they've been deployed.
Feature toggles enable us to deploy a feature to our production environment behind a toggle or switch, which can be turned on/off at any time without the need to redeploy code. This has a couple of benefits:
It's rare for a software team to have a testing environment exactly the same as our production environment; it's too expensive. With a feature toggle, however, we are able to selectively turn on the feature and test it.
This will help us ascertain if our software works well with production-only configurations such as load balancing, caching, replication tiers, and so on.
It gives us a more controllable release strategy. With a feature toggle, we can selectively turn on features for certain user groups, for example, in an alpha testing situation where we want our new features to be seen by a select group.
How does this keep us Agile?
The benefits on top of the CD include:
Every change goes into production directly; each increment goes as soon as it is committed.
The feedback loop is as short as possible.
By using strategies such as feature toggles, versioning, or branch by abstraction, it is possible to deploy to our production environment without necessarily using the new functionality.
This means that our code, even though it isn't finished yet, is already deployed to production, which gives us valuable feedback on the integration and deployment of our new software.
Implement an automated smoke test during production, triggered by a deployment
Implement a notification system which immediately alerts us of success or failure in production due to a rollout
Automate the outage-less deployment during production, triggered by a successful build/test on our CI server
We've looked at a few different practices that specifically target increasing our confidence when using an incremental delivery approach.
Refactoring helps us keep our software in a healthy and easy to maintain state. Using the analogy of the field, it's essential that we keep the weeds down because before we know it, we may be dealing with thickets or bushes. To do this, we regularly garden our code as we enhance or add to existing areas of functionality.
We can think of Test-Driven Design (TDD) or specification-driven approach because it changes our thought processes regarding how we write software compared to a test-after pattern.
Refactoring and TDD support an emergent approach to designing our software. So, although we still require some architectural design upfront, we require less big-design thinking overall.
Also, the resulting TDD automated test suite helps us verify and validate our software is still working as intended throughout its development life cycle.
Finally, CI, Continuous Delivery, and Continuous Deployment allow us to avoid the significant integration and deployment issues that plague waterfall projects.
All of these practices for building software keep our focus on delivering small increments of working software. They help us avoid the perils of Water-Scrum-Fall.
Tightening Feedback Loops in the Software Development Life Cycle
Now that we've got you through the foundations of setting up your Agile team, this blog is where we start to look at the "secret sauce" of Agile.
The adage "people don't know what they want until they see it" is just as true in the software industry as any other. The sooner we can deliver something useful to our customer, the earlier they can use it in a real-world environment, and the sooner we will be able to gather feedback on whether we're on the right track.
People often talk about Agile "speeding up" delivery, which it usually does. But this doesn't necessarily mean we deliver the same set of requirements at a faster pace just by working overtime.
Working longer hours is an option, but it is not sustainable over an extended period. Instead, delivery in an Agile context means we become smarter in terms of how to deliver the same set of requirements; this is how we speed up.