DevOps and Agile principles
This blog explains how DevOps and Agile principles utilize in the organization to economically release smaller batches of code on a more frequent basis.
One of the biggest potential advantages of software products is that once a new feature is created, the marginal cost to manufacture and distribute it should ideally be next to nothing.
If your development processes don’t allow you to economically release small batches of new functionality, then you are not able to realize these full benefits of the software.
You also lose the benefits of receiving rapid feedback from your customers or market to see if the new capabilities are meeting expectations and understand outcomes to inform next steps.
This fundamental Agile principle of releasing frequently tends to get overlooked or ignored by organizations that approach Agile transformations by scaling teams. It has been so overlooked by these organizations that new practices called DevOps and Continuous Delivery (CD) have begun to emerge to address this gap.
In DevOps, the objective is to blur the lines between Development and Operations so that new capabilities flow easier from Development into Production.
On a small scale, blurring the lines between Development and Operations at the team level improves the flow. In large organizations, this tends to require more structured approaches like a Continuous Delivery.
Applying these concepts at scale is typically the source of the biggest breakthroughs in improving the efficiency and effectiveness of software development in large organizations, and it should be a key focus of any large-scale transformation and is a big part of this blog.
In this blog, we purposefully blur the line between the technical solutions like Continuous Delivery and the cultural changes associated with DevOps under the concept of applying DevOps principles at scale, because you really can’t do one without the other.
DevOps and Continuous Delivery are concepts that are gaining a lot of momentum in the industry because they are addressing the aforementioned hole in the delivery process. That said, since these ideas are so new, not everyone agrees on their definitions.
Continuous Delivery tends to cover all the technical approaches for improving code releases and DevOps tends to be focused on the cultural changes. From our perspective, you really can’t make the technical changes without cultural shifts.
Therefore, for the proposes of this blog, we will define DevOps as processes and approaches for improving the efficiency of taking the newly created code out of development and to your customers.
This includes all the technical capabilities like Continuous Delivery and the cultural changes associated with Development and Operations groups working together better.
There are five main objectives that are helpful for executives to keep in mind when transforming this part of the development process so they can track progress and have a framework for prioritizing the work.
1. Improve the quality and speed of feedback for developers
Developers believe they have written good code that meets its objectives and feel they have done a good job until they get feedback telling them otherwise. If this feedback takes days or weeks to get to them, it is of limited value to the developers’ learning.
If you approach a developer weeks after they have written the code and ask them why they wrote it that way or tell them that it broke these other things, they are likely to say, “What code?,” “When?,” or “Are you sure it was me?”
If instead, the system was able to provide good feedback to the developer within a few hours or less, they will more likely think about their coding approach and will learn from the mistake.
The objective here is to change the feedback process so that rather than beating up the developer for making mistakes they don’t even remember, there is a real-time process that helps them improve.
Additionally, you want to move this feedback from simply validating the code to making sure it will work efficiently in production so you can get everyone focused on delivering value all the way to the customer.
Therefore, as much as possible you want to ensure the feedback is coming from testing in an environment that is as much like production as possible. This helps to start the cultural transformation across Development and Operations by aligning them on a common objective.
The Operations team can ensure their concerns are addressed by starting to add their release criteria to these test environments.
The Development teams then start to learn about and correct issues that would occur in production because they are getting this feedback on a daily basis when it is easy to fix.
Executives must ensure that both Development and Operations make the cultural shift of using the same tools and automation to build, test, and deploy if the transformation is going to be successful.
2. Reduce the time and resources required to go from functionality complete or release branching to production
The next objective is reducing, as much as possible, the time and resources required to go from functionality complete or release branching to production.
For large, traditional organizations, this can be a very lengthy and labor-intensive process that doesn’t add any value and makes it impossible to release code economically and on a more frequent basis.
The work in this phase of the program is focused on finding and fixing defects to bring the code base up to release quality.
Reducing this time requires automating your entire regression suite and implementing all-new testing so that it can be run every day during the development phase to provide rapid feedback to the developers.
It also requires teaching your Development organization to add new code without breaking existing functionality, such that the main code branch is always much closer to release quality.
Once you have daily full-regression testing in place, the time from functionality complete or branch cut to production can go down dramatically because the historic effort of manually running the entire regression suite and finding the defects after development is complete has been eliminated.
Ideally, for very mature organizations, this step enables you to keep trunk quality very close to production quality, such that you can use continuous deployment techniques to deploy into production with multiple check-ins a day.
This goal of a production-level trunk is pretty lofty for most traditional organizations, and lots of businesses customers would not accept overly frequent releases.
Working towards this goal, though, enables you to support delivery of the highest-priority features on a regular cadence defined by the business instead of one defined by the development process capabilities.
Additionally, if the developers are working on a development trunk that is very unstable and full of defects, the likely reaction to a test failure is “that’s not my fault, I’m sure that defect was already there.”
On the other hand, if the trunk is stable and of high quality, they are much more likely to realize that a new test failure may, in fact, be the result of the code they just checked in.
With this realization, you will see the Development community begin to take ownership of the quality of the code they commit each day.
3. Improve the repeatability of the build, deploy, and test process
In most large, traditional organizations, the repeatability of the entire build, test, and deploy process can be a huge source of inefficiencies. For small organizations with independent applications, a few small scrum teams working together can easily accomplish this process improvement.
For large organizations that have large groups of engineers working together on a leveraged code base or lots of different applications that need to work together, this is a very different story.
Designing a deployment pipeline for how you build up and test these systems is important. It needs to be a structured, repeatable process, or you are going to end up wasting lots of time and resources chasing issues you can’t find and/or trying to localize the offending code in a large, complex system. The objective here is to make sure you have a well-designed, repeatable process.
4. Develop an automated deployment process that will enable you to quickly and efficiently find any deployment or environment issues
Depending on the type of application, the deployment process may be as simple as FTP a file to a printer or as complex as deploying and debugging code to hundreds or thousands of servers.
If the application requires deploying to lots of servers, debugging the deployment process can be as complicated as finding code defects in a large system.
Additionally, it can complicate the process of finding code issues because the system test failures can be either be code or deployment related. Therefore, it is important to create a deployment process that can quickly identify and isolate any deployment issues before starting system testing to find code issues.
5. Remove the duplication of work that comes from supporting multiple branches of similar code
Another important objective at this stage of the transformation is taking duplication out of the process. The goal is to have your marginal costs for manufacturing and deploying software to be almost zero.
This starts to break down when you are working with more than one branch of similar code.
Every time you find a code issue, you need to make sure it is correctly ported and working on every associated branch. It breaks down, even more, when your qualification requires any manual testing, which is expensive and time-consuming on these different branches.
There are lots of different reasons you will hear for needing different branches. Some reasons are customer driven.
Other reasons for different branches include the belief that you need branches at different levels of stability or that you will have to bring in architectural changes and different branches make that easier.
All these issues have been solved by organizations that have learned how to develop on one trunk and have realized the efficiencies of this approach.
It will take time and will be an organizational change management challenge, but the duplicate work associated with multiple branches is a huge inefficiency in most large software development organizations.
Every time you see a branch that lasts more than a couple of days, you should think of it as duplicate costs in the system impacting the inherent economic benefits of your software.
Repeat this phrase until you become comfortable with it: “Branches are evil; branches are evil; branches are evil.” If you are working in a branch-heavy organization, this may take some time to address, but every time you see a branch you should ask why it’s there and look for process changes that will address the same need without creating branches.
Applying DevOps principles in the enterprise has huge advantages but can take several months to years, depending on the size of the organization. Therefore, you need a clear set of business objectives to help define priorities and track progress to ensure you implement the most valuable changes first.
Because it is not a physical device that has to be manufactured, the software can deliver new capabilities with very low manufacturing and distribution costs.
Most large, traditional organizations, though, have trouble taking advantage of software’s flexibility because their development processes do not allow them to economically release small batches of new capabilities.
Applying DevOps principles at scale is all about evolving the development process to make it easy to support frequent releases of new capabilities.
CREATING A CULTURE OF TRUNK DEVELOPMENT
While applying DevOps principles at scale requires good technical solutions, it is mostly about changing how people work. The technical solutions will require some finesse, but by far the biggest challenges will be around organizational change management and changing the culture.
If executives can’t get people to embrace the changes then it doesn’t make sense to invest in the technical solutions. One of the most important cultural changes for aligning the work across the organization is having all the teams continually integrating and testing their code in a production-like environment.
Developing on Trunk
Getting a team to apply DevOps principles at scale is a challenge. In traditional organizations when you describe the vision and direction of the large-scale Continuous Delivery on the trunk to the engineers, they immediately will tell you why it won’t work and how it will break when bringing in large changes.
Most large, traditional organizations have been working so far away from Continuous Delivery for so long they can’t imagine how it could possibly work.
Therefore, the executive team is going to have to be very active in leading and driving this change. This happens by sharing the strategy/vision and showing how other traditional organizations have successfully transformed their development processes.
Executives need to understand that driving trunk to a stable state on a day-to-day basis in the enterprise is going to be a big change management challenge but this is probably the most important thing they can do to help coordinate the work across teams and improve the effectiveness of the organization.
Once engineers have worked in an environment like this they can’t imagine having worked any other way. Before they have experienced it, though, they can’t imagine how it could ever work.
Executives have to help lead this cultural transformation by starting with an achievable goal that includes a small set of automated tests, then increasing the minimal level of stability allowed in the system over time.
From a technical perspective, the team will have to learn development practices like versioning services, rearchitecture through abstraction, feature flags, and evolutionary database design techniques.
In versioning services, you don’t modify a service if it is going to break the existing code. Instead, you create a new version of that service with the new capability. The application code is then written to ensure it calls the version of the service it is expecting.
Then over time as all the application code is updated to the new service the old version of the service can then be deprecated. This approach enables always-releasable code and ensures the application layer and services layers of the code can move independently.
Rearchitecture through abstraction is a technique that allows you to refactor major parts of the code without breaking existing functionality. In this case, you find an interface in the code where you can start the refactoring.
This interface should have a nice set of automated tests so you can ensure the new and old code will behave the same way with the rest of the system and you can test both the old and new code paths through this interface to make sure they are working.
The old interface is used to keep the code base working until the newly refactored code is ready. Then once the refactoring is complete, the old code is deprecated and the new code takes over running with the broader system.
Feature flags are another technique that enables developers to write new code directly on trunk but not exposes it to the rest of the system by using feature flags to turn it off until it is ready.
Finally, the evolutionary database is a technique like versioned services that enables you to make database schema changes without breaking existing functionality.
Similar to versioning services, instead of modifying the existing data, you add new versions with the schema changes and then deprecate the old versions when all the applications are ready.
These are not really technically challenging changes, but they are different ways of working that will require coordination and process changes across the development groups.
Leadership Team Responsibilities
If you are working to keep trunk stable with green builds in a large organization, making that “job 1” for everyone is not an easy task, and it is an important cultural shift that executives need to help facilitate.
All it takes is a few developers not buying into the change to create a train wreck with long periods of broken builds and an inefficient process.
At HP we started with rules that were strongly enforced by the leadership team. The first was that if you were going to commit code, you were required to stick around until the build was complete with the automated acceptance tests passing (green). Failing to do so was considered a “lab felony.”
In four years this only happened twice, and in each case, it led to the director having an expectation-setting conversation with the individual and their management chain. The second rule was making sure people were not continuing to commit code into a broken (red) build.
This was considered a “lab misdemeanor” because it was potentially piling more bad code onto a broken build that people were working to get fixed.
We had this problem because we started with a traditional continuous integration process that required everyone swarming a red build because this train wreck was blocking all the rest of the code from progressing down the deployment pipeline.
Setting these rules and driving the changes in behavior required a fair amount of focus and energy by the leadership team. We found that it was taking a lot of our change management capacity to keep the builds green.
Therefore, over time we learned how to have our tools automatically enforce the behavior we wanted. We did this by creating gated commits or auto-revert.
Gated commits enabled us to constantly have green builds and avoid train wrecks of broken builds without using the management team’s change management capabilities.
If a code commit or group of commits did not result in a green build, instead of having everyone swarm to fix the issue on the trunk, we gated the changes that caused the build to go red so they never made it onto trunk in the source code management (SCM) tool.
Instead of having a train wreck blocking code progressions on the tracks, the offending code was moved off the tracks, debugged, and fixed so the next builds could keep moving through the system.
No one could commit on a red build because the commits ahead of you that caused the previous build to go red were simply pulled off to the side allowing those who made the commits to resolve the issues.
The SCM contained the versions of code that resulted in the most recent green build and not the versions of code that resulted in a red build. The system picked up the next set of commits in the pipeline and kicked off the next build.
Instead of lab management walking around and asking engineers why they committed on a red build, we created a process that eliminated the problem. For those developers that had their code auto-reverted, the system would send them an e-mail saying their code was rejected because a certain set of tests had failed.
They were then responsible for fixing the issue and recommitting the code. This gave the developers time to fix the issue correctly without the rest of the organization breathing down their necks while the defect blocked the entire deployment pipeline.
This process change and toolset had a few very big advantages. First, it kept the build from ever becoming blocked (red) so people could always commit code and have it flow to the trunk.
Second, it focused on the impact of the failure of the smallest group of people possible. Now instead of the whole organization focusing on keeping the build green, the tools automatically enforced the change in behavior.
As you start working with your teams to begin implementing these DevOps principles at the scale you can either start with your leadership using its change management capabilities to drive the change or invest in process changes and the tooling to force the correct behavior.
Both work, but we would recommend taking the time to setup auto-revert or gated commits to drive this change. It will take a little more time to get started, but the rollout will go so much more smoothly that it is worth it.
When we set out the vision of one main branch for all current and future products using continuous integration at HP, most of the engineers thought we had lost our minds. They would avoid making eye contact when we walked down the hall and were just hoping at some point we would snap back to reality.
It was especially challenging for our technical lead of the build, test, and release process. He really wanted to engage us in the branching discussion.
Every time he did we would say “we aren’t going to branch.” It got to the point where he probably wore out a thesaurus trying to find ways to talk about branching without using the B-word.
We like this lesson for two reasons: First and foremost it is our personal belief that this is the biggest opportunity for improving development productivity in the enterprise. Second, it shows the big shift in mindset that needs to occur.
The lead went from the biggest nonbeliever to the biggest supporter of the approach. Until engineers have worked in this type of environment, they will never believe it can work.
Once they have developed in this type of environment, however, they can’t imagine ever going back. Over the years we have had several different engineers that, after leaving HP, called to talk about how backward development is in their new companies.
They complain they are not nearly as effective because the feedback is non-existent and it is very hard to release code into production.
Transforming software development processes in a large organization is a big change management challenge. While technology can help, if the executives are not willing to lead and drive cultural changes like developing on the trunk, then no amount of technology is going provide the necessary breakthroughs.
Developers need to know they can develop stable code on the trunk and should take responsibility for ensuring the code base is always close to release quality by keeping the builds green.
They also need to understand the objective of the organization is to optimize the delivery of new capabilities all the way out to the customer not just on their desktops.
Ensuring each developer is responsible for making sure every check-in works in this production-like environment will start the cultural shift of aligning Development and Operations on a common objective.
Keeping builds green can be done solely by process change and leadership enforcement. Ideally, though, you should design your continuous integration process to help drive the required changes in behavior.
This is probably one of the biggest things you can do to improve the productivity of software development in large organizations. Most engineers, however, will argue initially it can’t be done.
Only by working in this type of environment will they understand how much more productive it is.
Then they can’t imagine working any other way. The challenge for executives is leading people on the journey from nonbelievers to champions of the new way of working.
It is important for executives to understand early on if the organization is embracing this cultural change because if it doesn’t, all the investments in technical changes will be a waste of time.
Agile Practice Deep Dive
If Agile is the engine that helps you achieve escape velocity and overcome the laws of organizational gravity, the retrospective is the valve that stops that engine from overheating and burning out.
A retrospective is a meeting at the end of a sprint or the completion of a project during which a team reflects on the way they work together and identifies changes they can make for the next sprint or project.
Note that the goal of a retrospective is explicitly not to critique the actual work that was just completed, but rather to reflect on how the team completed that work.
The retrospective constitutes an important opportunity for teams to build a shared sense of purpose and ownership around the way they work. It is a chance to first ask the question, “Is the way we’re working helping us live up to our principles and achieve our goals,” and then, “What are we going to do about it next time?”
For many teams I’ve worked with, especially those outside of product and engineering, speaking openly about what is and is not working can be an uncomfortable prospect.
People often assume that the way they currently work must have been put into place for a very good reason and that questioning it might amount to undermining somebody else’s experience or authority.
But I have been truly amazed at how many times a particular practice or artifact—like a regularly scheduled meeting that provides no real value or a campaign planning template that demands way too much information—turns out to be a simple historical accident that nobody has felt empowered to reevaluate.
I have participated in many retrospectives, for example, in which somebody sheepishly suggests that a long-standing practice is no longer serving the team, only for everybody else in the room—including that team’s leader—to react in swift and violent agreement.
Moments like this can have an immediate and incredibly positive effect on a team’s morale and productivity, but they simply do not occur unless you create space for them.
One of the fantastic things about retrospectives is that you can run them at the end of any project, regardless of whether that project involves any other Agile practices.
Adding a retrospective to any regularly scheduled work such as a monthly newsletter or quarterly planning meeting can open up a new kind of space for teams to discuss how they work together and why.
Most issues can be solved with communication, and a lot of companies miss the human side of things. They think that Agile is going to make people faster, but people are not robots! And there are a lot of silly issues and miscommunications that can become big problems unless you bring people together for an open dialogue.
Sometimes, just the safety net of being able to express yourself in front of your team is enough to get started—and once you have that, you start to get a team into the state of flow. And for me, that starts with the retrospective.
Each Agile ceremony has a reason. But to me, the retrospective is the key to continuous improvement. Without that, a strict and regimented framework is all you’ve got.
We’ve talked to a lot of Scrum masters who have trouble communicating the value of Agile to their teams. And if you have that problem, you need to start from the beginning. You need to open up that dialogue and give everybody on the team the opportunity to say what they really feel.
There will almost certainly be conflict, but that conflict is good—it’s what drives you to the next level. Without bringing that conflict into the open, a lot of teams are stuck in that first level, unable to work through the very first challenges that they find in a newly formed team.
Retrospectives are often challenging for the very reason that they are ultimately so valuable: they open up space where people can voice their doubts, questions, and uncertainties about the way that they work together.
These conversations are rarely comfortable, and they rarely yield easy or certain answers. But the simple fact of the retrospective is often enough to begin addressing the unspoken beliefs and assumptions that can leave teams feeling stuck and disempowered. Here are a few tips for keeping your retrospective on track:
Model vulnerability and uncertainty
If you are the person bringing Agile practices to your team, your teammates look to you to understand how they are “supposed” to behave during a retrospective.
This might leave you feeling like you need to have all the right answers or to defend the Agile practices you’ve introduced.
But the best thing you can do for your team is to model the kind of openness and honesty that will enable you to continuously adapt and improve. If somebody asks about a particular practice and you don’t have a good answer, feel free to say, “I don’t really know. What does everybody else think?”
Stay focused on what you are going to do next time
In many organizations, taking time to reflect on the way a team works only happens when there is a major problem. As a result, retrospectives can devolve into evasiveness and finger-pointing.
The engineering teams at Etsy solved this problem by holding what they call “blameless postmortems,” in which participants can openly reflect on mistakes they made without fearing retribution.
But another step you can take to avoid the blame game is to shift the conversation toward what you will do differently for the next sprint or project. For example, “Regardless of who was responsible for what happened last time, what can we all do together next time to make things better?”
Treat future changes as experiments, not mandates
Regularly scheduled retrospectives provide your team with the opportunity to adjust course after each sprint or project. This means that there is very little risk associated with trying a new practice or approach; after all, if it isn’t working, you will have the opportunity to change it or roll it back during your next retrospective.
Remind your team that every change you make is essentially an experiment; you won’t know whether it works until you try it, and you are prepared to adjust course based on what you learn.
Keep your principles in focus
I’ve often found it helpful to have my team or organization’s guiding Agile principles physically present during a retrospective. This helps to keep the team focused not just on how they are working together, but also on why they are changing the way they work in the first place.
These principles can also serve as a powerful mediator when there is a disagreement, enabling you to ask, “Which of these approaches is most aligned with our guiding principles?” rather than,
“Which of these approaches do we like more?”
Hold space for what is working
Although running a retrospective is a critical way to adjust course when things are not working well for your team, it is also a critical way to acknowledge the things that are working well for your team.
One approach I’ve found helpful is to ask each team member to quickly write down three things that worked well and three things they think could be changed for the next sprint or project. This gives equal time to the things that are worth protecting and the things that stand to be refined.
Break the ice
A team’s first couple of retrospectives can be particularly awkward. Sometimes, it is helpful to bring a sense of fun and ease to a retrospective by starting with an icebreaker or framing the retrospective itself as a kind of game.
Mindful Team, whose co-founder Emma Obanye provided the insightful commentary about retrospectives earlier in this blog, offers a card game called The Retrospective Game that can provide a great first step for teams that are not used to sharing reflections in a group.
Finally, and perhaps most importantly, resist the temptation to cancel your retrospectives if you have a lot of work to do. For teams and organizations that see
Agile just as a means to increase velocity, the retrospective can seem like a waste of productive time—after all, you aren’t actually making anything, just sitting around and talking about how you make things.
But making time for a team to reflect on how it works is a non-negotiable if you want that team to embrace any new ways of working.
In the absence of a retrospective, it is inevitable that any set of Agile practices will fail to achieve their full potential, as the team implementing those practices comes to see them as just another form of “business as usual” that they are powerless to question or adapt.
You are regularly killing off projects that are not creating value for your customers
In most organizations, abandoning or killing off a project feels like a failure. The person responsible for that project runs the risk of losing status, resources, and in some cases even their job.
But, in organizations that have truly embraced uncertainty, killing off a project is a sign of success.
It means that you are open to the possibility of your customers and your market changing, and will not sink additional resources into something that you have learned is no longer likely to succeed.
If you’ve reached the point where a project lead is comfortable saying, “I think we should kill this idea and put our resources elsewhere,” you are well on your way to building a truly Agile organization.
To keep the momentum going around this, you might want to:
Acknowledge and celebrate team and project leads who are brave enough to adjust course and substantially redirect a project already in progress.
Make sure that the success of every project is measured not just by operational metrics such as “on time” and “on a budget,” but also by the value, it creates for your customers.
Keep a record of what projects were abandoned and for what reason so as to avoid these projects being accidentally revisited without this context.
When specific Agile practices are not working for your team, you work together to change them
It might feel like the ultimate end goal of an Agile journey is to have your entire organization 100% onboard to a single, stable, and consistent set of practices and rituals. But this goal runs deeply counter to the very first statement of the Agile Manifesto: “Individuals and interactions over processes and tools.”
As the individuals in your organization and the individuals you serve change, so too must your Agile practices. If the way you work is evolving to meet the needs of your team, this is a sign that you are being true to the principles of Agile, not a sign that you have failed at implementing the practices of Agile.
To keep the momentum going around this, you might want to:
Share information about your evolving Agile practices beyond your team. Some teams I’ve worked with, for example, send out memos stating what change they made, what effect they hoped this change would have, what effect it actually had, and what they are going to do about it moving forward.
Have open and candid conversations with your team, at both a group and one-on-one level, about what value they are deriving from Agile practices.
Put a name to the set of practices that are working for you and your teams, such as the Spotify model or Enterprise Design Thinking, to foster a sense of collective ownership of the practices that you are utilizing.
You are withholding important information until the next year planning or budgeting meeting
One of the dangers with regular organizational cadences is that they can compel people to withhold important information until what they consider to be the officially sanctioned time and place.
This can happen when engineers don’t share critical blockers until the next daily stand-up, or when marketers shelve customer insights until the next campaign planning cycle. This self-imposed delay can result in wasted resources, missed opportunities, and crippling bottlenecks when that regular cadence does come around.
If this is happening, you might want to:
Schedule more frequent “check-ins” between infrequent meetings to track progress and share new information.
Have a discussion with your team about what type of information constitutes an out-of-cadence “emergency” and should be communicated immediately, and to whom it should be communicated.
Create a formal template or practice for handling “emergency” information that comes in from customers, clients, or executives. This helps you retain some sense of structure and procedure while still accounting for the reality that important new information could arise at any time.
You are working a certain way “because it’s Agile”—and that’s it
Just because a particular practice is Agile does not mean that it’s a good fit for your organization or that it will help you deliver more value to your customers in a faster and more flexible way.
If the best justification you can think of for working a certain way is “because it’s Agile,” then it is very unlikely that you and your teammates are cultivating a shared sense of purpose and ownership over the way that you work together.
If this is happening, you might want to:
Communicate clearly that whatever practices you implemented initially are only a starting point. Set clear expectations that the way your team or organization is working six months from now should and must be different from what that framework or methodology looks like on paper.
Push back on absolute quantitative metrics for Agile adoption (such as “20% of our projects must be Agile within the next five years”) because they can easily turn the entire universe of Agile practices and principles into a meaningless checkbox.
Run a deprivation test; cease all Agile rituals and ceremonies for a week, and see what happens. Afterward, run a retrospective with your team and use this as an opportunity to “hit reset” and initiate a frank conversation about what is and is not working.
The three guiding principles we have covered so far capture three concepts that are at the very heart of what makes Agile such a powerful movement: customer centricity, collaboration, and openness to change.
Committing to any one of these three guiding principles can make an immediate difference for any team or organization looking to work with, not against, the realities of a fast-changing world.
But the real magic happens when these three principles are applied together to create a harmonious cycle of learning, collaborating, and delivering.
As this cycle gains momentum, so too does the belief that real change is possible. The alchemical union of principles and practices at the heart of Agile provides teams not just with a new way of working, but with the permission to ask why they are working a certain way—often for the first time.
As teams take more ownership over the way that they work, they become more comfortable challenging the fundamental beliefs and expectations that have allowed “business as usual” to persist through prior attempts at organizational change.
Creating space for meaningful, sustainable, and ongoing change means accepting the fact that there is no single framework or set of practices that will lead every organization to guaranteed success.
Agile reminds us that organizations are not operational puzzles to be solved, but rather collections of individuals working together to meet the fast-changing needs of their customers. This means that every individual has a role to play in making their organization fast, flexible, and customer-first.
In this sense, “Agile for Everybody” is not just a statement about the broad applicability of Agile principles, but also a reminder that Agile is at its most potent and transformative when everybody in an organization, regardless of their level, their team, or their role, apply those principles to their day-to-day work.
Leadership in the Agile Organization
Many of my conversations with Agile practitioners took a swift and immediate turn toward the topic of leadership. A 2006 Harvard Business Review article titled “Embracing Agile” speaks to how many Agile initiatives wind up being undermined by the very organizational leaders who often demand them:
When we ask executives what they know about agile, the response is usually an uneasy smile and a quip such as “Just enough to be dangerous.” They may throw around agile-related terms (“sprints,” “time boxes”) and claim that their companies are becoming more and more nimble. But because they haven’t gone through training, they don’t really understand the approach.
Consequently, they unwittingly continue to manage in ways that run counter to agile principles and practices, undermining the effectiveness of agile teams in units that report to them.
For what it’s worth, I believe that training is only part of the solution here. I know many people who have been through hours and hours of Agile training and still have no clear idea of what is expected of them and why.
The bigger challenge, as I learned from my first experience with Agile, is that the underlying values of Agile—the substantive ideas lurking behind all that shiny jargon—are often at odds with the behaviors and expectations that organizational leaders have developed through years of successfully navigating “business as usual.”
Indeed, each of our three laws of organizational gravity also exerts a force on our organization’s leaders.
The Three Laws of Organizational Gravity often create situations in which the people feel that it is against their best interest to bring to the attention of their managers any information about the way they work or the people they serve that might be perceived as “bad news.”
This adds up to a situation in which leaders have a very difficult time gaining an accurate understanding of the challenges facing both their colleagues and their customers.
And that, in turn, can leave employees feeling misunderstood and powerless—how are their leaders supposed to make things better if they don’t even understand what’s going on?
When organizational leaders have largely been insulated from the on-the-ground challenges facing both their employees and their customers, they are often left with the impression that “business as usual” is working just fine. It is no surprise, then, that their initial interest in Agile often privileges incremental operational improvements over substantive cultural change.
When enlisting team and organizational leaders in support of any Agile initiative, it is important to start from a place of empathy, not an accusation. Do not assume that leaders are being willfully dismissive or difficult if they don’t seem to understand what these new ideas mean, or if they seem hung up on superficial improvements.
Break the cycle of strategically withholding and sugar-coating information by being open, honest, and candid, and giving the leaders with whom you are speaking a chance to respond in kind.
We’re trying to help people really understand this, on both a personal and a corporate level. It doesn’t really matter what tools you use—Agile, Scrum, whatever.
The thing we all have in common is either we’re adding value as defined by the marketplace or we’re adding waste. And, in order to continue adding value and eliminating waste, we need to be constantly improving. What a lot of leaders don’t anticipate is that “continuous improvement” means continuously admitting to and addressing your own mistakes.
Indeed, while “continuous improvement” is an easy enough concept to agree to in theory, organizational leaders are often ill-prepared for the emotional labor that goes into admitting that their best efforts may still have fallen short, or that the needs of their organization may have outpaced their own vision or experience.
This is particularly important when we are working with the people who often feel like they have the most to lose in a transition toward Agile practices: middle managers.
Many of these middle managers have spent their entire careers carefully managing the flow of information upward and downward—something that actually runs quite counter to the mandates for transparency and collaboration that come with Agile.
As IBM CMO Michelle Peluso pointed out, implementing the practices and principles of Agile often means reshaping the role of middle management:
In huge companies, there are often a lot of people who sit at the middle management layer and spend all day moving information up and down.
They gather information from their direct reports, they send it up the ranks, they get information from farther up the ranks, and they break it down and share it with their direct reports. If you are truly Agile, you don’t need this kind of hub-and-spoke model.
The teams have to solve things themselves. If you’re going on an Agile journey, you need to embrace the idea that you’re going to get rid of some middle management.
The good news is, a lot of great middle managers can be repurposed to do more impactful things. While there is less need for shuttling information up and down, there is a much greater need for things like Agile coaching and cross-functional guild leadership. That can open up exciting new career paths for people.
But it can be very hard for people who are used to measuring their success by the number of direct reports they have, and suddenly find themselves in this cross-functional world.
There’s a real sense of, “We earned our way here, we spent years getting to this point.” It’s very frustrating and very emotionally difficult for people, for very real reasons. You need to approach it with empathy and tackle it head-on.
I do think that oftentimes when you think about describing Agile to a team, it is framed up in terms of why it’s good for the organization at large. But it’s also important to ask, “Why are YOU going to be a better leader if you go Agile?” For starters, having a successful Agile transformation on your résumé can make you a hot commodity.
But beyond that, embracing Agile means that you get to work side by side with data scientists, creatives, engineers—you get to really see and understand how they do their work. It creates an extraordinary learning environment.
It’s important to be very clear about what people can expect individually; there are some really personal things about the journey to Agile that I think need to be understood and reinforced.
Respecting this personal dimension gives us a way to approach Agile that holds true to its founding values. It gives us a chance to build stronger and more transparent relationships between and among organizational leaders.
And it gives us a chance to approach colleagues who may be fearful or resistant to change with openness, empathy, and curiosity that will ultimately help us implement Agile principles and practices in a more impactful and inclusive way.