DevOps and Agile principles
This blog explains how DevOps and Agile principles utilize in the organization to economically.
This fundamental Agile principle of releasing frequently tends to get overlooked or ignored by organizations that approach Agile transformations by scaling teams. It has been so overlooked by these organizations that new practices called DevOps and Continuous Delivery (CD) have begun to emerge to address this gap.
There are five main objectives that are helpful for executives to keep in mind when transforming this part of the development process so they can track progress and have a framework for prioritizing the work.
1. Improve the quality and speed of feedback for developers
Developers believe they have written good code that meets its objectives and feel they have done a good job until they get feedback telling them otherwise. If this feedback takes days or weeks to get to them, it is of limited value to the developers’ learning.
If you approach a developer weeks after they have written the code and ask them why they wrote it that way or tell them that it broke these other things, they are likely to say, “What code?,” “When?,” or “Are you sure it was me?”
If instead, the system was able to provide good feedback to the developer within a few hours or less, they will more likely think about their coding approach and will learn from the mistake.
The objective here is to change the feedback process so that rather than beating up the developer for making mistakes they don’t even remember, there is a real-time process that helps them improve.
Additionally, you want to move this feedback from simply validating the code to making sure it will work efficiently in production so you can get everyone focused on delivering value all the way to the customer.
Therefore, as much as possible you want to ensure the feedback is coming from testing in an environment that is as much like production as possible. This helps to start the cultural transformation across Development and Operations by aligning them on a common objective.
The Operations team can ensure their concerns are addressed by starting to add their release criteria to these test environments.
The Development teams then start to learn about and correct issues that would occur in production because they are getting this feedback on a daily basis when it is easy to fix.
2. Reduce the time and resources required to go from functionality complete or release branching to production
Once you have daily full-regression testing in place, the time from functionality complete or branch cut to production can go down dramatically because the historic effort of manually running the entire regression suite and finding the defects after development is complete has been eliminated.
Ideally, for very mature organizations, this step enables you to keep trunk quality very close to production quality, such that you can use continuous deployment techniques to deploy into production with multiple check-ins a day.
This goal of a production-level trunk is pretty lofty for most traditional organizations, and lots of businesses customers would not accept overly frequent releases.
Working towards this goal, though, enables you to support the delivery of the highest-priority features on a regular cadence defined by the business instead of one defined by the development process capabilities.
Additionally, if the developers are working on a development trunk that is very unstable and full of defects, the likely reaction to a test failure is “that’s not my fault, I’m sure that defect was already there.”
On the other hand, if the trunk is stable and of high quality, they are much more likely to realize that a new test failure may, in fact, be the result of the code they just checked in.
With this realization, you will see the Development community begin to take ownership of the quality of the code they commit each day.
3. Improve the repeatability of the build, deploy, and test process
Depending on the type of application, the deployment process may be as simple as FTP a file to a printer or as complex as deploying and debugging code to hundreds or thousands of servers.
If the application requires deploying to lots of servers, debugging the deployment process can be as complicated as finding code defects in a large system.
Additionally, it can complicate the process of finding code issues because the system test failures can be either be code or deployment related. Therefore, it is important to create a deployment process that can quickly identify and isolate any deployment issues before starting system testing to find code issues.
Applying DevOps principles in the enterprise has huge advantages but can take several months to years, depending on the size of the organization. Therefore, you need a clear set of business objectives to help define priorities and track progress to ensure you implement the most valuable changes first.
Because it is not a physical device that has to be manufactured, the software can deliver new capabilities with very low manufacturing and distribution costs.
Most large, traditional organizations, though, have trouble taking advantage of software’s flexibility because their development processes do not allow them to economically release small batches of new capabilities.
Applying DevOps principles at scale is all about evolving the development process to make it easy to support frequent releases of new capabilities.
Developing on Trunk
Getting a team to apply DevOps principles at scale is a challenge. In traditional organizations when you describe the vision and direction of the large-scale Continuous Delivery on the trunk to the engineers, they immediately will tell you why it won’t work and how it will break when bringing in large changes.
Most large, traditional organizations have been working so far away from Continuous Delivery for so long they can’t imagine how it could possibly work.
Therefore, the executive team is going to have to be very active in leading and driving this change. This happens by sharing the strategy/vision and showing how other traditional organizations have successfully transformed their development processes.
Executives need to understand that driving trunk to a stable state on a day-to-day basis in the enterprise is going to be a big change management challenge but this is probably the most important thing they can do to help coordinate the work across teams and improve the effectiveness of the organization.
Once engineers have worked in an environment like this they can’t imagine having worked any other way. Before they have experienced it, though, they can’t imagine how it could ever work.
Executives have to help lead this cultural transformation by starting with an achievable goal that includes a small set of automated tests, then increasing the minimal level of stability allowed in the system over time.
From a technical perspective, the team will have to learn development practices like versioning services, rearchitecture through abstraction, feature flags, and evolutionary database design techniques.
In versioning services, you don’t modify a service if it is going to break the existing code. Instead, you create a new version of that service with the new capability. The application code is then written to ensure it calls the version of the service it is expecting.
Then over time as all the application code is updated to the new service the old version of the service can then be deprecated. This approach enables always-releasable code and ensures the application layer and services layers of the code can move independently.
Rearchitecture through abstraction is a technique that allows you to refactor major parts of the code without breaking existing functionality. In this case, you find an interface in the code where you can start the refactoring.
This interface should have a nice set of automated tests so you can ensure the new and old code will behave the same way with the rest of the system and you can test both the old and new code paths through this interface to make sure they are working.
The old interface is used to keep the code base working until the newly refactored code is ready. Then once the refactoring is complete, the old code is deprecated and the new code takes over running with the broader system.
Feature flags are another technique that enables developers to write new code directly on the trunk but not exposes it to the rest of the system by using feature flags to turn it off until it is ready.
Finally, the evolutionary database is a technique like versioned services that enables you to make database schema changes without breaking existing functionality.
Similar to versioning services, instead of modifying the existing data, you add new versions with the schema changes and then deprecate the old versions when all the applications are ready.
These are not really technically challenging changes, but they are different ways of working that will require coordination and process changes across the development groups.
When we set out the vision of one main branch for all current and future products using continuous integration at HP, most of the engineers thought we had lost our minds. They would avoid making eye contact when we walked down the hall and were just hoping at some point we would snap back to reality.
It was especially challenging for our technical lead of the build, test, and release process. He really wanted to engage us in the branching discussion.
Every time he did we would say “we aren’t going to branch.” It got to the point where he probably wore out a thesaurus trying to find ways to talk about branching without using the B-word.
We like this lesson for two reasons: First and foremost it is our personal belief that this is the biggest opportunity for improving development productivity in the enterprise. Second, it shows the big shift in mindset that needs to occur.
The lead went from the biggest nonbeliever to the biggest supporter of the approach. Until engineers have worked in this type of environment, they will never believe it can work.
Once they have developed in this type of environment, however, they can’t imagine ever going back. Over the years we have had several different engineers that, after leaving HP, called to talk about how backward development is in their new companies.
They complain they are not nearly as effective because the feedback is non-existent and it is very hard to release code into production.
Transforming software development processes in a large organization is a big change management challenge. While technology can help, if the executives are not willing to lead and drive cultural changes like developing on the trunk, then no amount of technology is going provide the necessary breakthroughs.
Developers need to know they can develop stable code on the trunk and should take responsibility for ensuring the code base is always close to release quality by keeping the builds green.
They also need to understand the objective of the organization is to optimize the delivery of new capabilities all the way out to the customer not just on their desktops.
Ensuring each developer is responsible for making sure every check-in works in this production-like environment will start the cultural shift of aligning Development and Operations on a common objective.
Keeping builds green can be done solely by process change and leadership enforcement. Ideally, though, you should design your continuous integration process to help drive the required changes in behaviour.
This is probably one of the biggest things you can do to improve the productivity of software development in large organizations. Most engineers, however, will argue initially it can’t be done.
Only by working in this type of environment will they understand how much more productive it is.
Then they can’t imagine working any other way. The challenge for executives is leading people on the journey from nonbelievers to champions of the new way of working.
It is important for executives to understand early on if the organization is embracing this cultural change because if it doesn’t, all the investments in technical changes will be a waste of time.
Agile Practice Deep Dive
If Agile is the engine that helps you achieve escape velocity and overcome the laws of organizational gravity, the retrospective is the valve that stops that engine from overheating and burning out.
A retrospective is a meeting at the end of a sprint or the completion of a project during which a team reflects on the way they work together and identifies changes they can make for the next sprint or project.
Note that the goal of a retrospective is explicitly not to critique the actual work that was just completed, but rather to reflect on how the team completed that work.
The retrospective constitutes an important opportunity for teams to build a shared sense of purpose and ownership around the way they work. It is a chance to first ask the question, “Is the way we’re working helping us live up to our principles and achieve our goals,” and then, “What are we going to do about it next time?”
For many teams I’ve worked with, especially those outside of product and engineering, speaking openly about what is and is not working can be an uncomfortable prospect.
People often assume that the way they currently work must have been put into place for a very good reason and that questioning it might amount to undermining somebody else’s experience or authority.
But I have been truly amazed at how many times a particular practice or artefact—like a regularly scheduled meeting that provides no real value or a campaign planning template that demands way too much information—turns out to be a simple historical accident that nobody has felt empowered to reevaluate.
I have participated in many retrospectives, for example, in which somebody sheepishly suggests that a long-standing practice is no longer serving the team, only for everybody else in the room—including that team’s leader—to react in swift and violent agreement.
Moments like this can have an immediate and incredibly positive effect on a team’s morale and productivity, but they simply do not occur unless you create space for them.
One of the fantastic things about retrospectives is that you can run them at the end of any project, regardless of whether that project involves any other Agile practices.
Adding a retrospective to any regularly scheduled work such as a monthly newsletter or quarterly planning meeting can open up a new kind of space for teams to discuss how they work together and why.
Most issues can be solved with communication, and a lot of companies miss the human side of things. They think that Agile is going to make people faster, but people are not robots! And there are a lot of silly issues and miscommunications that can become big problems unless you bring people together for an open dialogue.
Sometimes, just the safety net of being able to express yourself in front of your team is enough to get started—and once you have that, you start to get a team into the state of flow. And for me, that starts with the retrospective.
Each Agile ceremony has a reason. But to me, the retrospective is the key to continuous improvement. Without that, a strict and regimented framework is all you’ve got.
We’ve talked to a lot of Scrum masters who have trouble communicating the value of Agile to their teams. And if you have that problem, you need to start from the beginning. You need to open up that dialogue and give everybody on the team the opportunity to say what they really feel.
There will almost certainly be conflict, but that conflict is good—it’s what drives you to the next level. Without bringing that conflict into the open, a lot of teams are stuck in that first level, unable to work through the very first challenges that they find in a newly formed team.
Retrospectives are often challenging for the very reason that they are ultimately so valuable: they open up space where people can voice their doubts, questions, and uncertainties about the way that they work together.
These conversations are rarely comfortable, and they rarely yield easy or certain answers. But the simple fact of the retrospective is often enough to begin addressing the unspoken beliefs and assumptions that can leave teams feeling stuck and disempowered. Here are a few tips for keeping your retrospective on track:
Model vulnerability and uncertainty
If you are the person bringing Agile practices to your team, your teammates look to you to understand how they are “supposed” to behave during a retrospective.
This might leave you feeling like you need to have all the right answers or to defend the Agile practices you’ve introduced.
But the best thing you can do for your team is to model the kind of openness and honesty that will enable you to continuously adapt and improve. If somebody asks about a particular practice and you don’t have a good answer, feel free to say, “I don’t really know. What does everybody else think?”
Stay focused on what you are going to do next time
In many organizations, taking time to reflect on the way a team works only happens when there is a major problem. As a result, retrospectives can devolve into evasiveness and finger-pointing.
The engineering teams at Etsy solved this problem by holding what they call “blameless postmortems,” in which participants can openly reflect on mistakes they made without fearing retribution.
But another step you can take to avoid the blame game is to shift the conversation toward what you will do differently for the next sprint or project. For example, “Regardless of who was responsible for what happened last time, what can we all do together next time to make things better?”
Treat future changes as experiments, not mandates
Regularly scheduled retrospectives provide your team with the opportunity to adjust course after each sprint or project. This means that there is very little risk associated with trying a new practice or approach; after all, if it isn’t working, you will have the opportunity to change it or roll it back during your next retrospective.
Remind your team that every change you make is essentially an experiment; you won’t know whether it works until you try it, and you are prepared to adjust course based on what you learn.
Keep your principles in focus
I’ve often found it helpful to have my team or organization’s guiding Agile principles physically present during a retrospective. This helps to keep the team focused not just on how they are working together, but also on why they are changing the way they work in the first place.
These principles can also serve as a powerful mediator when there is a disagreement, enabling you to ask, “Which of these approaches is most aligned with our guiding principles?” rather than,
“Which of these approaches do we like more?”
Hold space for what is working
Although running a retrospective is a critical way to adjust course when things are not working well for your team, it is also a critical way to acknowledge the things that are working well for your team.
One approach I’ve found helpful is to ask each team member to quickly write down three things that worked well and three things they think could be changed for the next sprint or project. This gives equal time to the things that are worth protecting and the things that stand to be refined.
Break the ice
A team’s first couple of retrospectives can be particularly awkward. Sometimes, it is helpful to bring a sense of fun and ease to a retrospective by starting with an icebreaker or framing the retrospective itself as a kind of game.
Mindful Team, whose co-founder Emma Obanye provided the insightful commentary about retrospectives earlier in this blog, offers a card game called The Retrospective Game that can provide a great first step for teams that are not used to sharing reflections in a group.
Finally, and perhaps most importantly, resist the temptation to cancel your retrospectives if you have a lot of work to do. For teams and organizations that see
Agile just as a means to increase velocity, the retrospective can seem like a waste of productive time—after all, you aren’t actually making anything, just sitting around and talking about how you make things.
But making time for a team to reflect on how it works is a non-negotiable if you want that team to embrace any new ways of working.
In the absence of a retrospective, it is inevitable that any set of Agile practices will fail to achieve their full potential, as the team implementing those practices comes to see them as just another form of “business as usual” that they are powerless to question or adapt.
You are regularly killing off projects that are not creating value for your customers
In most organizations, abandoning or killing off a project feels like a failure. The person responsible for that project runs the risk of losing status, resources, and in some cases even their job.
But, in organizations that have truly embraced uncertainty, killing off a project is a sign of success.
It means that you are open to the possibility of your customers and your market changing, and will not sink additional resources into something that you have learned is no longer likely to succeed.
If you’ve reached the point where a project lead is comfortable saying, “I think we should kill this idea and put our resources elsewhere,” you are well on your way to building a truly Agile organization.
To keep the momentum going around this, you might want to:
Acknowledge and celebrate team and project leads who are brave enough to adjust course and substantially redirect a project already in progress.
Make sure that the success of every project is measured not just by operational metrics such as “on time” and “on a budget,” but also by the value, it creates for your customers.
Keep a record of what projects were abandoned and for what reason so as to avoid these projects being accidentally revisited without this context.
When specific Agile practices are not working for your team, you work together to change them
It might feel like the ultimate end goal of an Agile journey is to have your entire organization 100% onboard to a single, stable, and consistent set of practices and rituals. But this goal runs deeply counter to the very first statement of the Agile Manifesto: “Individuals and interactions over processes and tools.”
As the individuals in your organization and the individuals you serve change, so too must your Agile practices. If the way you work is evolving to meet the needs of your team, this is a sign that you are being true to the principles of Agile, not a sign that you have failed at implementing the practices of Agile.
To keep the momentum going around this, you might want to:
Share information about your evolving Agile practices beyond your team. Some teams I’ve worked with, for example, send out memos stating what change they made, what effect they hoped this change would have, what effect it actually had, and what they are going to do about it moving forward.
Have open and candid conversations with your team, at both a group and one-on-one level, about what value they are deriving from Agile practices.
Put a name to the set of practices that are working for you and your teams, such as the Spotify model or Enterprise Design Thinking, to foster a sense of collective ownership of the practices that you are utilizing.
You are withholding important information until the next year planning or budgeting meeting
One of the dangers with regular organizational cadences is that they can compel people to withhold important information until what they consider to be the officially sanctioned time and place.
This can happen when engineers don’t share critical blockers until the next daily stand-up, or when marketers shelve customer insights until the next campaign planning cycle. This self-imposed delay can result in wasted resources, missed opportunities, and crippling bottlenecks when that regular cadence does come around.
If this is happening, you might want to:
Schedule more frequent “check-ins” between infrequent meetings to track progress and share new information.
Have a discussion with your team about what type of information constitutes an out-of-cadence “emergency” and should be communicated immediately, and to whom it should be communicated.
Create a formal template or practice for handling “emergency” information that comes in from customers, clients, or executives. This helps you retain some sense of structure and procedure while still accounting for the reality that important new information could arise at any time.
You are working a certain way “because it’s Agile”—and that’s it
Just because a particular practice is Agile does not mean that it’s a good fit for your organization or that it will help you deliver more value to your customers in a faster and more flexible way.
If the best justification you can think of for working a certain way is “because it’s Agile,” then it is very unlikely that you and your teammates are cultivating a shared sense of purpose and ownership over the way that you work together.
If this is happening, you might want to:
Communicate clearly that whatever practices you implemented initially are only a starting point. Set clear expectations that the way your team or organization is working six months from now should and must be different from what that framework or methodology looks like on paper.
Push back on absolute quantitative metrics for Agile adoption (such as “20% of our projects must be Agile within the next five years”) because they can easily turn the entire universe of Agile practices and principles into a meaningless checkbox.
Run a deprivation test; cease all Agile rituals and ceremonies for a week, and see what happens. Afterwards, run a retrospective with your team and use this as an opportunity to “hit reset” and initiate a frank conversation about what is and is not working.
Agile reminds us that organizations are not operational puzzles to be solved, but rather collections of individuals working together to meet the fast-changing needs of their customers. This means that every individual has a role to play in making their organization fast, flexible, and customer-first.
In this sense, “Agile for Everybody” is not just a statement about the broad applicability of Agile principles, but also a reminder that Agile is at its most potent and transformative when everybody in an organization, regardless of their level, their team, or their role, apply those principles to their day-to-day work.
Leadership in the Agile Organization
Many of my conversations with Agile practitioners took a swift and immediate turn toward the topic of leadership. A 2006 Harvard Business Review article titled “Embracing Agile” speaks to how many Agile initiatives wind up being undermined by the very organizational leaders who often demand them:
When we ask executives what they know about agile, the response is usually an uneasy smile and a quip such as “Just enough to be dangerous.” They may throw around agile-related terms (“sprints,” “time boxes”) and claim that their companies are becoming more and more nimble. But because they haven’t gone through training, they don’t really understand the approach.
Consequently, they unwittingly continue to manage in ways that run counter to agile principles and practices, undermining the effectiveness of agile teams in units that report to them.
For what it’s worth, I believe that training is only part of the solution here. I know many people who have been through hours and hours of Agile training and still have no clear idea of what is expected of them and why.
The bigger challenge, as I learned from my first experience with Agile, is that the underlying values of Agile—the substantive ideas lurking behind all that shiny jargon—are often at odds with the behaviors and expectations that organizational leaders have developed through years of successfully navigating “business as usual.”
Indeed, each of our three laws of organizational gravity also exerts a force on our organization’s leaders.
The Three Laws of Organizational Gravity often create situations in which the people feel that it is against their best interest to bring to the attention of their managers any information about the way they work or the people they serve that might be perceived as “bad news.”
This adds up to a situation in which leaders have a very difficult time gaining an accurate understanding of the challenges facing both their colleagues and their customers.
And that, in turn, can leave employees feeling misunderstood and powerless—how are their leaders supposed to make things better if they don’t even understand what’s going on?
When organizational leaders have largely been insulated from the on-the-ground challenges facing both their employees and their customers, they are often left with the impression that “business as usual” is working just fine. It is no surprise, then, that their initial interest in Agile often privileges incremental operational improvements over substantive cultural change.
When enlisting team and organizational leaders in support of any Agile initiative, it is important to start from a place of empathy, not an accusation. Do not assume that leaders are being willfully dismissive or difficult if they don’t seem to understand what these new ideas mean, or if they seem hung up on superficial improvements.
Break the cycle of strategically withholding and sugar-coating information by being open, honest, and candid, and giving the leaders with whom you are speaking a chance to respond in kind.
We’re trying to help people really understand this, on both a personal and a corporate level. It doesn’t really matter what tools you use—Agile, Scrum, whatever.
The thing we all have in common is either we’re adding value as defined by the marketplace or we’re adding waste. And, in order to continue adding value and eliminating waste, we need to be constantly improving. What a lot of leaders don’t anticipate is that “continuous improvement” means continuously admitting to and addressing your own mistakes.
Indeed, while “continuous improvement” is an easy enough concept to agree to in theory, organizational leaders are often ill-prepared for the emotional labour that goes into admitting that their best efforts may still have fallen short, or that the needs of their organization may have outpaced their own vision or experience.
This is particularly important when we are working with the people who often feel like they have the most to lose in a transition toward Agile practices: middle managers.
Many of these middle managers have spent their entire careers carefully managing the flow of information upward and downward—something that actually runs quite counter to the mandates for transparency and collaboration that come with Agile.
As IBM CMO Michelle Peluso pointed out, implementing the practices and principles of Agile often means reshaping the role of middle management:
In huge companies, there are often a lot of people who sit at the middle management layer and spend all day moving information up and down.
They gather information from their direct reports, they send it up to the ranks, they get information from farther up the ranks, and they break it down and share it with their direct reports. If you are truly Agile, you don’t need this kind of hub-and-spoke model.
The teams have to solve things themselves. If you’re going on an Agile journey, you need to embrace the idea that you’re going to get rid of some middle management.
The good news is, a lot of great middle managers can be repurposed to do more impactful things. While there is less need for shuttling information up and down, there is a much greater need for things like Agile coaching and cross-functional guild leadership. That can open up exciting new career paths for people.
But it can be very hard for people who are used to measuring their success by the number of direct reports they have, and suddenly find themselves in this cross-functional world.
There’s a real sense of, “We earned our way here, we spent years getting to this point.” It’s very frustrating and very emotionally difficult for people, for very real reasons. You need to approach it with empathy and tackle it head-on.
I do think that oftentimes when you think about describing Agile to a team, it is framed up in terms of why it’s good for the organization at large. But it’s also important to ask, “Why are YOU going to be a better leader if you go Agile?” For starters, having a successful Agile transformation on your résumé can make you a hot commodity.
But beyond that, embracing Agile means that you get to work side by side with data scientists, creatives, engineers—you get to really see and understand how they do their work. It creates an extraordinary learning environment.
It’s important to be very clear about what people can expect individually; there are some really personal things about the journey to Agile that I think need to be understood and reinforced.
Respecting this personal dimension gives us a way to approach Agile that holds true to its founding values. It gives us a chance to build stronger and more transparent relationships between and among organizational leaders.
And it gives us a chance to approach colleagues who may be fearful or resistant to change with openness, empathy, and curiosity that will ultimately help us implement Agile principles and practices in a more impactful and inclusive way.