July 10, 2019
By Carla Rudder, writer and content manager, The Enterprisers Project
Doing DevOps and doing it at scale are two very different things. Here’s how to push past barriers to wider success
For many organizations, the first steps on the journey to DevOps don’t involve much pain. Small, passionate teams form, new processes take the place of old ones, and people achieve early wins.
But these are mere flashes of brilliance or illusions of progress, says Ben Grinnell, managing director and global head of technology and digital for North Highland. While encouraging, such early wins won’t help organizations achieve what they ultimately want: DevOps at scale.
It’s easy to see how you end up with an ‘us and them’ culture.
“So often, companies run these ‘trailblazer’ projects without thinking about the fact that if you want to scale, these projects must leave a trail that others can and want to follow,” says Grinnell. “Trailblazing teams are usually staffed by confident people new to the organization who have done it before elsewhere. They are encouraged to disrupt and break the rules that others are supposed to follow. It’s easy to see how you end up with an ‘us and them’ culture, which inhibits the transfer of skills and knowledge.”
This culture challenge is just one reason scaling DevOps proves to be hard. Teams must also grapple with the complexity that is par for the course for fast-moving, technology-driven organizations, says Steve Newman, founder and chairman for Scalyr.
“In the modern world, services change as often as necessary. While constantly building and pushing new features is exciting, it becomes a headache to coordinate and troubleshoot when issues arise,” says Newman. “In hyper-growth organizations, engineering teams struggle to keep track of what’s being changed across a larger group and the cascade effect it has on the dependencies. Moreover, teams become frustrated when it’s harder to communicate what problems resulted when they no longer have visibility into the changes.”
[ Some common DevOps wisdom falls flat. Read 7 pieces of contrarian DevOps advice. ]
How are determined teams beating these challenges and scaling DevOps across larger organizations? We turned to the experts to get advice for people and teams who are stuck on their DevOps journey. It all starts with patience, even though you’re chasing a speedier software development cycle and faster business processes.
1. Remember, culture change takes time
Jayne Groll, CEO, DevOps Institute: “To my mind, expanding DevOps should be as incremental and iterative as agile software development itself (and equally as cultural). Agile and DevOps both advocate for small teams. But as small teams integrate with other small teams, the net result is more teams that are exercising a new way of working, and cultural transformation at scale begins to emerge.”
[ Read our related story: DevOps culture: 3 ways to strengthen yours in 2019 ]
2. Take your time planning and choosing frameworks
Eran Kinsbruner, lead technical evangelist, Perfecto: “To scale, DevOps teams need to match traditional process, skills, and tools, and then grow each of the DevOps phases slowly and stabilize them. It starts with planning the user stories and value streams properly, then going through software building and version control using practices like trunk-based development to handle multiple code branches and merges.
“Then comes the integration and testing phase, which requires a scalable platform for automation. For this, teams need to have proper frameworks that match their skills and project objectives. The deployment to production phase should be completely automated using orchestration tools and containers. Throughout, it’s important to virtualize phased environments (staging, QA, production) and ensure the latest test data is being used to draw relevant insights. The analysis must be smart and capable of handling big data with fast, actionable feedback.”
Learn about how to become certified in DevOps Foundation
3. Change the nature of accountability
“Being accountable” need not equate to “having caused some disaster.”
Gordon Haff, Technology Evangelist, Red Hat: “Setting up a system and environment that allows and encourages experiments enables successful failure in agile software development. It doesn’t mean that no one is accountable for failures. In fact, it makes accountability easier because ‘being accountable’ needn’t equate to ‘having caused some disaster.’ In this respect, it changes the nature of accountability. Four factors will prove crucial: Scope, approach, workflow, and incentives.” (For more detail on these factors, read the full article by Gordon Haff: DevOps lessons: 4 aspects of healthy experiments.)
4. Create a clear path forward
Ben Grinnell, managing director and global head of technology & digital, North Highland: “To scale, I recommend that leaders invest in a ‘clear the trail’ program alongside trailblazing DevOps efforts. The purpose of this program is to clear the debris from the disruption of the DevOps journey – rules that have been broken, etc. – and make sure the path forward is clear.
“Create organizational support and momentum through communications that are bigger than the trailblazers by celebrating the successes of the new ways of working. Coach the people in the next wave of projects who are nervously trying DevOps for the first time and recognize that these followers are very different from the trailblazers.”
5. Democratize DevOps tools
Steve Newman, founder and chairman, Scalyr: “Democratize the tools. All tools must be accessible and approachable to anyone willing to spend the time to learn the basics of tooling. For example, if the ability to query log files is limited to the three people who are ‘certified’ on a tool, you’ll always be limited to a team of three who can only solve that problem – no matter how big your environment is. It creates a bottleneck that can result in serious (business) consequences.”
6. Optimize for happy teams
Tom Clark, head of common platform, ITV plc: “You can do anything, but not everything – so think big, start small, and iterate quickly. Over time, you’ll build a reputation for success, which will tempt more groups to adopt your practices. Finally, don’t get distracted trying to build high-performing teams; instead, optimize them for happiness and you’ll eventually get high performance as a side effect.”
7. Remember Conway’s Law and Kanban boards
Logan Daigle, director of DevOps strategy and delivery, CollabNet VersionOne: “We must understand the implication(s) of Conway’s Law on the enterprise. Conway’s Law states (in my own words) that the products we build and the processes we use to build them (including DevOps) will be structured like the organization.”
If the organization is highly siloed and has many handoffs for planning, building, and releasing software, the success of scaling will be non-existent or short-lived. If the organization chooses to build and operate cross-functional teams around products that are funded to be market-oriented, there’s a much better chance of scaling to success.
“The other important aspect of successful scaling is to make all of the work that is in progress (WIP) visible on Kanban boards. If the people in the enterprise have a place to go to see this, it enables a lot of collaboration that is good for scaling.”
8. Look for organizational scar tissue
Manuel Pais, DevOps consultant and co-author of “Team Topologies“: “A suboptimal approach when trying to expand DevOps practices beyond Dev and Ops is to try to directly apply the practices to other functions. While that can undoubtedly bring some benefits (e.g., for automating manual controls), we can uncover orders of magnitude higher gains if we start by understanding the delivery and feedback processes.”
If we identify organizational scar tissue, meaning procedures and controls that have been implemented in response to past incidents but that no longer make sense (due to changes in product, technology, or processes), then we should remove or simplify them rather than automate an inefficient or unnecessary process.”
9. Don’t do 31 flavors of DevOps
Antony Edwards, COO, Eggplant: “DevOps is a very vague term, and so when people roll out DevOps, it’s a custom version. The biggest anti-pattern in organizations is that they roll out 20 different versions of DevOps that don’t work well together. You can’t have three different dev teams with different interfaces between product management and dev, and you can’t have different expectations of what gets moved into staging how to process feedback. If you try to operate in this manner, DevOps will never scale.”
10. Evangelize the value to the business
MORE ON DEVOPS
Newman: “Spread ‘recognition of value.’ Learn to and focus on effectively communicating the value of what you do. DevOps practices provide incredible time-saving and cost-saving values (think: less downtime, quick Mean Time To Restore), and teams need to highlight (and evangelize) how these initiatives are critical to the success of the business. This will help with the adoption of DevOps practices to larger involved communities and create a larger impact across the organization.”