This blog was co-authored by myself and Simon Darken. Once a year, PagerDuty’s SREs get together for a three-day, in-person offsite. With the team spread...by Dave Bresci
December 5, 2018
Here at PagerDuty, we’re pretty focused on being involved in the DevOps community by providing perspectives on where we’ve been, where we are and where we’re headed as a community — and of course hearing from the community as well! And, if you follow this blog you probably saw an earlier post recapping our Predictions and Trends in DevOps webinar, which brought together four DevOps thought leaders to give us their perspective on what’s happening in 2017 and beyond. If you haven’t already watched it, you can still catch it on-demand!
In the first half of the discussion, we discussed the mainstreaming of DevOps and its adoption in the enterprise. For the second part of our conversation, we looked at two other key questions, specifically:
Let’s dive in!
Ilan Rabinovich of Datadog finds it strange that the idea of security is something “new” that needs to be added to DevOps culture. He points out that security is part of good software hygiene and should be part of every product. In the same way that developers have learned to test early and often with Continuous Integration and Continuous Delivery, as well as to automate more QA work, DevOps teams need to do the same with security, learning to integrate security at every commit. He points out that good teams have been incorporating security for years. The DevOps world needs to follow best practices in the same way that top-performing development organizations do, and start the security conversation earlier.
Chris Gervais of Threat Stack feels that security HAS to be part of the operational model in 2017. While he notes that there can be some dependencies when it comes to implementing security, it can be done. DevOps teams must seize the moment and integrate security into the overall process, even if that isn’t part of the current DevOps model in their organizations. Gervais suggests that security be built into the software and product development lifecycle, such that everyone on the team is thinking about security. Instead of seeing security as “the team that says no,” security should be incorporated into best practices and automation, and if a separate security team already exists, it is best to get them involved in DevOps. Pull them in “quick and early” so that they are not at the end of transactions. When their influence is baked in, it’s far more likely that the typical negotiations and compromise between Engineering and Ops and Security are all accounted for, and that the resulting work products can be rooted in organizational goals around agility and velocity as well as security.
To that end, in terms of security collaboration, Gervais suggests that upfront discussions can be had with the underlying premise that the organization is going down the DevOps path together. As such, DevOps must help set the stage and create ways to work collaboratively. He warns that not involving the security team up front can lead to brick walls down the road and will needlessly slow down development projects. Security should become part of the DevOps fabric and should no longer be the domain of the few, but the “domain of the many.”
When asked about security as part of DevOps, John Rakowski of AppDynamics, rather cheekily replied, “Isn’t security already part of good IT and IT for the enterprise?”
He went on to say that if so, security is also surely part of DevOps — knowing of course that there can be something of a disconnect between groups in IT. Rakowski points out that when people say “DevOps,” it is really about Dev and Ops coming together — but that the larger trend is more because technology is now so important to enterprises and overall business success, such that it really touches every stakeholder in IT. This importance is signified by new terms, like “BizDevOps” that go beyond Dev and Ops. There are similar new phrases coming into the marketplace dialogue that he doesn’t want to hear — DevSecOps for example — because it confuses the market even more. Rakowski’s eloquent position is that security IS part of DevOps and proactively integrating other critical considerations such as business and security requirements is part of DevOps culture, or at least should be. He sees it as being about the entire software delivery lifecycle and delivering what the business needs.
Arup Chakrabarti of PagerDuty sees a future union of security and DevOps emerging via new tools. He points to Continuous Integration style tools that can offer a “green checkmark” for security in software as code is written, as well as the ability to leverage static analysis tools (such as code inspection solutions) that will help drive best practices for software testing, and as a result drive convergence with security testing. He sees efficiencies being driven by greater levels of automation in the security realm, as more tooling is being launched to automate and gather security data. Arup is encouraged by the fact that security issues are being addressed earlier in the software lifecycle. He also cautions that testing after the fact can be too late, “the testing game can be moved earlier and earlier into the lifecycle, with better results.”
“Yes, we have to get closer to what each other [is and] are doing,” commented Datadog’s Rabinovich. Rabinovich sees Dev moving closer to infrastructure, and infrastructure people moving closer to the application stack in 2017, for faster issue resolution. Again, he stressed collaboration: “If we don’t understand each other’s needs, we’re not going to meet each other’s expectations.” He suggests that conversations with users and product managers will reveal necessary features and functionality that meet customer needs, and that Ops being closer to the code will help avoid “building the wrong thing.”
Rabinovich suggests that if you’re in an organization where Ops is a centralized infrastructure services team and doesn’t necessarily touch the application, you will need to know how to build APIs, tools and a platform that applications are deployed on top of — and to make sure that it runs effectively. He underscores the need for robust software deployment services, such that code can be deployed easily and scale. He says it isn’t a matter of just employing a turnkey solution; rather, it’s about understanding the needs of the product development team and at the same time getting rooted in understanding developers’ needs. Rabinovich also notes that in an organization where product engineers have to hand over code to infrastructure or operations engineers, the operations team need to know how to resolve or mitigate issues and operate the production environment while waiting for the next upgrade or patch, because they’re the ones being paged in the middle of the night!
Chris Gervais of Threat Stack sees this issue pretty clearly, stating, “They will, and they have to – and the nature of what can be shared among Dev and Ops will get more declarative.” He cited teams that have adopted microservices and then externalized configuration parameters, as examples of those that have done a great job giving Ops teams much more to work with. He suggests that Ops “might not have to know the discrete lines of code,” but if they’re using configuration management tools to lay bits down, provision and get them running, they can get more insight into how systems operate. This helps them to identify problems more quickly.
Gervais explained that if Ops has more insight into how various applications operate, they can better understand any needs for adjustment. He also sees adjacency between Dev and Ops as enabling changes via a cookbook and making quick adjustments, as well as a future where Ops can work with engineering to run health and performance checks that are linked to tools like runbooks, such that they can go in and make fixes more efficiently and autonomously.
He further explains that in an environment where contracts [between Dev and Ops] around sharing information have been made, teams can take advantage of externalized configurations and service discovery so that Ops teams can change the characteristics of the software and adjust performance as needed. He sees it all as part of pulling teams together to ensure they communicate and democratizing access to information that was once siloed or stovepiped. Because, as Gervais says, “in the middle of the night, having a playbook points the team to doing the right things initially,” and empowering Ops to make code changes leads to faster resolution.
John Rakowski of AppDynamics thinks that moving Ops closer to the application codebase “seems to be a hot topic at the moment” and that “simply, the answer is yes.” According to Rakowski, Ops teams will need to be closer to, and better versed, in the code base they support in terms of delivering applications and services. He also sees that in a similar way, Dev teams will need to be closer to Ops — because it isn’t just about writing code, but how the products associated with that code drives the business in terms of revenue, customer experience, and loyalty.
He adds that one of the central themes of DevOps is about breaking down silos between Dev and Ops. As such, the coming together of these two disciplines is essential. Even in an era where the term “NoOps” has occasionally cropped up, there will always be a need for Ops. He sees that Ops’ focus will be on ensuring quality, such as performance in production. In the next 3-5 years, Dev and Ops teams will transform, blending the requisite IT and Ops skill sets to drive speed, with a focus on automation and reliability.
PagerDuty’s Arup Chakrabarti also explored the somewhat controversial notion of “NoOps,” commenting that there was a “hullabaloo when the term was coined.” He does see agreement around the intent and mindset to automate parts of operational work. NoOps has potential to free up time to focus on bigger and more critical tasks (such as providing value during major business-impacting issues), and Arup commented that good Ops engineers and Systems admins focus on automating manual tasks and that the best ones take an approach that enables them to “get time back.”
We covered a lot of DevOps ground in this broad-ranging discussion. Our four experts gave us some meaningful insights, plus a few more that we’re planning to share. At PagerDuty, we’re excited to see where the rest of the year takes the DevOps space, and we’ll most certainly be back with more thoughts as we see how the predictions played out this year, and into 2018 and beyond. In the meantime, we welcome your comments, and encourage you to watch our Predictions and Trends in DevOps webinar!