DevSecOps for US Federal agencies (FIPs, STIGs, auditors, AOs, and all that)

Episode 187 · 30:12 · SRT · archive.org · all Tanzu Talk

Well, hey, this is another interview that I first did as part of the videos that we do nowadays over at Tanzu.tv, you know, in my little show, Tanzu Talk. But I think it's a great overview of what it looks like to think about and put a lot of security into the build tool chain that you would have, kind of the way you think about how you're doing DevOps, as people are calling it nowadays, DevSecOps, which is a delightful term. But it's based on not just sort of like some theoretics and things, but an actual little case study that Chris Willis, who works here at VMware Tanzu, is doing with one of his US federal customers. And I think he really does a great job going over how they thought about what the problems are for securing a build tool chain and the runtime environment, you know, all the Kubernetes and Tanzu build service, all the stuff that we do over here in VMware Tanzu land, and kind of starts to get you thinking about how you would worry about that secure tool chain yourself and make sure that you're doing things in as he was doing with his customer in a kind of highly regulated, very secure environment. And it's a fun conversation. Also, you know, before I start, there's a lot of videos that I and other people put up. If you go to Tanzu.tv, you can find tons of them. And I try to put several of them here in the podcast feed, but there's more than we can handle. And I've got a lot of little tiny videos over there as well. So just go over to Tanzu.tv and you can look around there, you know, as the kids like to say, like and subscribe. Enjoy the interview. So why don't you introduce yourself? Yeah, my name is Chris Willis, and I am what's called a solutions engineer for VMware. So you're, I don't know, it probably is few. One of the few people around who has encountered the idea in practice applied the notion of DevSecOps. So first of all, like what is DevSecOps? And whenever I see a phrase like that, what I always think is, does that mean that DevOps wasn't secure already?

Yeah, I didn't really follow the whole rise of DevSecOps as a buzzword. I'll offer my definition. I haven't gotten a secure DevSecOps pipeline working and process working at my client. I'll say this, it's a lot about unifying all the three teams. So DevSec and ops around a secure and consistent path to product. From the developer's keyboard to production, what are you doing to make sure that process is not only secure, but it's repeatable and it fits into the business as a whole. Everyone is rallied around this process and accepts it as the way to do business. And if you want to make changes to that DevSecOps pipeline, you're doing it through the GitOps model, through PRs, things like that. But the big thing is consistency, repeatability, and using a lot of good ingredients. If you use a cooking analogy, using a lot of good ingredients, meaning you're not pulling things straight from Docker Hub, you're maybe getting them from a place, your base images and container images from a place, from a known good source. So sometimes you can find yourself on Stack Overflow and someone says, hey, use this container image to do a thing. A lot of times you're not double checking to see what is that container image doing. And if it were to make it to production, it could have severe consequences. So you want to make sure that that supply chain is good. It's really baking in security to your CICD process in a holistic manner, really embedding your security engineers and having them be not just people who are checking requirements or yes, you're vulnerable to this CVE or you're not, but someone who can also contribute and add to the pipeline in a meaningful way. But not boiling the ocean, you can get into analysis paralysis with all of these tools, these security tools you have now, right? It's I want the CICD process, it's going to have this and that and I'm going to bolt this on here. And you can quickly lose focus in the business

process and that unification of the three teams, which is really what I think DevSecOps is getting at, can fall away as you continue to just have this cowboy attitude of let me add this cool tool and this cool tool, right? You want to make sure it's a unified front between all three parties. Let's talk about that, that boiling the ocean problem. When you're solving a problem like this or any problem, there's a certain amount of not being perfect that you have to find acceptable, which in a security context is difficult to be comfortable with. In reading about DevSecOps and like over the years, it seems like one of the key things that people have to figure out is like the software has to be released. So eventually, and actually very frequently, maybe weekly or monthly, we have to release software. And the most secure thing to do would be not to release software. Like maybe we can boil a lake, but we can't boil the ocean. I think Fed makes this pretty clear cut in certain ways. There's the use of FIPS compliant cryptographic libraries. Your encryption has to be using known good algorithms or algorithms that at least in the government space, they deem as secure. There's also this idea of STIGs. So these are vast lists of if you have a Red Hat base image, or you have a Ubuntu base image, it's got to have this file tweaked this way, this file tweaked this way, blah, blah, blah, all the way down. So it's like the configuration of various things that needs to be configured this way. Okay. So I think your encryption has to be, you have to know you're using good encryption libraries, which is typically the case, I would say. And then the last thing I'll say is the source of everything. So where everything's coming from. So if you have bad, a bad supply chain, if you have someone injecting malicious code in that supply chain somewhere, that really matters too. Those are

like the three things that you really want to be paying attention to. Before the fed space, we have something called an authorizing official. And they're ultimately the one that kind of like you're saying they accept that risk. They're the one that says, yes, this can go to production or no, it can't. This is the process I'm over. If you follow this process, I'm comfortable with things going to production repeatedly through. You could never have a situation where every release and this old way is every release goes to the authorizing official or the local what's called an ism or security person. That's part of the team and say, Hey, is this okay? Okay. How about this? If you're releasing on a cadence that Silicon Valley releases on and getting things to users quickly, that's just not possible. That's what we did with Tanzu. And what really helps is that that consistency and that supply chain problem specifically is huge. The reason that we have dev and ops next to security in dev sec ops is that we want to have very rapid automated releases. And it seems like a lot of security stuff is not easy to automate as it currently exists. So it seems like that is in a very high level way. The kind of problem statement of dev sec ops is that we can automate a tremendous amount of getting that software to production, but we can't really automate the security stuff. I think a lot of that has to do with just like the systems and the policies just aren't to use another funny phrase automatable. There's not APIs into it. So that seems like one of the core things in the last few years trying to be solved for is how you automate those things. So you listed, well, specific to the, and I should say, when we say fed, we mean US United States fed. I'm sure it's similar in other governments, but sorry, there are other federal governments. Like there are three things specifically in the US fed space that you had to pay attention to. And that is you need to have certified is the

wrong word, but you need to have the official, the sanctioned encryption that you're using. And they're for various components in your stack, whether it's an operating system or whatever else. Does it go up? Does the configuration, what did you call it? The stig that where you've got. Yeah. Yeah. Yeah. And so the other thing is you have, as you were saying, the stig where you have, again, authorized configuration of components from the operating system. And does that go up to like middleware? Like how far up does that configuration go? Yeah. It can go to like Tomcat would have a stig or yeah, something like that. So it, it does go pretty high up there and that's where things can get tricky. Yeah. Yeah. I imagine. And then the third thing is, and help me word this correctly, but it is being able to demonstrate and trust, being able to demonstrate that the chain of ownership or the chain of custody or the provenance or whatever, like where did all this stuff come from and what version is it? And then obviously it's, I'm being overly explicit. It's worth saying the second part and also trusting the report that you get, like generating reports, one thing, but trusting it is a whole separate thing. You, you have to open up visibility into the product because it's okay. Why should I trust like you VMware? Yeah, sure. Okay. You're, you're a us-based company and sure, blah, blah, blah. But like, why should I trust this? Right. And so a lot of it is opening up process and getting the developers in contact, the developers of the secure supply chain in contact with the developers of Tanzu application catalog and Tanzu build service in these products and, and having them walk them through things that aren't open on the website, right? They're not on the Tanzu website. It's like, Hey, this is how we, an open source release is made to Apache Tomcat. This is the process that it goes through in Bitnami's now they're part of VMware, but in Bitnami's process to secure it, make sure everything is aligned and you should trust that. So in some cases, it's not just me as the, in the

convincing business saying, Hey, you should totally trust us. It's, Hey, we'll link you up with the engineers of the team. They'll walk you through the process and then you can decide whether or not that sounds like something you could trust. So it's, there are, there is an art form to it. We have those three things. Now, I think what would be useful is to, since you have worked on a POC and projects and stuff around this, can you walk us through a more concrete example of how this all comes together? What we're talking about really, and the proof of concept that we put in front of the AO, the authorizing official was something where the developer commits code, right? Goes into GitHub. We're all used to that process, but then there's a pipeline putting this together. And by the way, we've put together a reference implementation for other people to use. If they're using concourse, they can reuse this. But what it does is it picks up and says, Oh, Hey, I see a new git commit. I'm going to run this through a tool called sonar queue, which is what we're using. And ultimately we'll say this is a pass or fail, right? This has too many code smells, or this is under the threshold that you've told me is good or bad and gives it a rating. And it's looking for all sorts of things, vulnerabilities, but in the code itself. And then you have a OWASP dependency checker that's going and saying, Hey, the libraries that this developer used to write this code, are those good? Is there anything wrong there? And in some ways we double tap on that dependency front later when we do the container repository scan, but it's good to have the upfront sort of quick feedback loop. So if the developer commits code and sonar cube or OWASP dependency checker right away says, no, there's a problem here. The pipeline will fail, kick it back to them and they can improve and do whatever to get things moving again. When you automate it, does it take a long time or a short time or? No, it can be, it can be within the order of minutes to get feedback. I mean, I guess it could depend on how much code you're, you're scanning, but typically with like microservices and smaller

things. Yeah. You're talking a few minutes. The next part. So let's say you pass the gate, right? Cause if you don't pass the gate, it gets kicked back. You have to make the change, but let's say you pass the gate into kind of that third phase. This is where Tanzu build service. So this is much more part of the supply chain piece of it that I was talking about. Like that third thing we talked about where the Tanzu build service is taking the code using a regularly updated and maintained build packs and what are called stacks, which are just like the base operating system layer of the container. It's using those foundational pieces again, regularly updated and maintained by VMware to actually build the container. That's going to run the code. So that goes and builds the container, pushes it to Harbor, which is what we're using for a container repository. And then from there, we're doing again that, like I said, that double tap kind of scanning, right? So we're scanning the container using Trivi and that is generating its own set of findings that saying this container has two high vulnerabilities, three mediums and 20 low vulnerabilities. And then it depends based on the level of risk that the authorizing officials able to, or willing to accept. A lot of them will just say, yeah, make sure your highs are taken care of. And any mediums, again, we're referring to CVEs, right? Vulnerabilities that can be found on cve.mitre.org or anywhere else for the specific operating system or whatever. But that, that scanning is taking place and you're generating an artifact that ultimately the local security team can review. So you're gathering this body of evidence, this, this group of things that the security team is going to review, which is much better than how it was done previously, where you're actually looking through code or you're taking the developer's word for it. You're looking at hard published artifacts that were generated by the pipeline. And again, this is why your security team has to be bought in. Because if they say those

aren't good enough, then you got to go back to the drawing board. But the last thing we do, and this is to really complete the cycle, is we deploy the container, right? On Kubernetes, we deploy it with this contrast agent. And what contrast does is it will actually do like that real time scanning and feedback about in in runtime, is there an issue with the application and that library that you used in the container 20 days ago, was there a zero day just introduced? So that way, the security team can log into their contrast dashboard, they can look and say, Oh, hey, it looks like this application has a vulnerability. And the reason why Tanzu build service is so critical is typically that vulnerability, oftentimes those vulnerabilities are in libraries, or they're in the base operating system of the container. So what you would do in that case is not really go back to the app developer, which is what was done previously, the app developer, patch your stuff, or go back to the operator and say, Hey, patch this thing that can spin out of control as far as like cycle time, because now the operator or developer has to say, Oh, okay, did they release a fix for it yet? Let me go find it. Does my code work with it? What's going on? But in this model, what you do is the operators would update the stack and build packs, right on Tanzu build service. So those base OS layer, and then the build packs that include the libraries and say, Hey, can you push that again? And we'll see if that vulnerability pops up. And many times what will happen is the vulnerabilities patched at the speed of the developer pushing new code. So that's pretty much all they have to do to patch their application with Tanzu build services is push the new code. And it'll build using the latest build pack and stack And you're off to the races that whole chain is built around when we become knowledgeable of a bad piece of code. We need to see if we have it and fix it right when a CV comes out and identifies this little

wing ding version 1.2 a B seven beta, whatever has a vulnerability in it. We need to know where it is and basically fix it as soon as possible. What DevOps can bring to that is what you're describing at the end is that what DevOps with containers, let's put it that way, because of the way that containers are layered from the operating system to the frameworks that you need to whatever your custom code is, right? Like those three layers are don't have to be treated as just one giant monolithic thing that developer that's a black box. And so all throughout the build process, you can treat each of those three layers separately and scan them all the time. And then more importantly, when you find out a problem in one of those three layers, and I guess there's also the layers of configuration and networking, there's a whole bunch of other stuff, but whatever. All of these things are atomic units that you can independently manage and update as needed. We'll go back to an example we were using earlier. I don't know if this is true. You'd have to tell me technically, but like we find there's a problem in Tomcat and because of the way in our build process and because we're doing things containerized because the way that Tomcat is layered in to the application that we have and because of the way configuration is done separately. Like Tomcat doesn't become this big ball of configuration that only the developers can unwind. And therefore, when something, some bad chunk of code is out there, it's a lot faster because we have this automated pipeline to ask the development team to put this new version of Tomcat in and make sure it doesn't break your application really quickly. And then let's deploy that thing quickly. And we don't have to worry about doing a major rev of the entire project. We can address this, this one little layer essentially. Yeah. Small changes. And those layers, those container image layers are critical because you're in many times you're just swapping out one layer with, with the new version of open SSL or whatever. And at scale, that can be really important. You think you have a thousand applications times when the next heart bleed happens. Yeah. You got to figure out a way to patch all of those.

Whereas in this model, you would just say, Hey, we updated the new stack or build pack that addresses that, that issue. We push your code. We'll go through and do our testing again, all of your CI processes. But ultimately we'll rebuild the container with the new patched container of open or library of open SSL. So it moves at the speed of the developer. The developer doesn't need to rely on anyone else. What role does standardization of all of this play in a large organization? This pipeline is meant to take in all code, but I think, yeah, the one part around standardization is you really do need to be bought into the idea of build packs in order for this to succeed. A lot of our pivotal customers and Tanzu application service customers, they are very familiar with build packs, but that's the standardization layer, right? That's the level of automation and standardization that needs to be bought into in order for a lot of this to work. Because you're basically committing to, yes, I will write my code in such a way that I'm used to using build packs. Developers can get started and experiment with that on KPAC. Tanzu build service is really just like a wraparound KPAC. So that's something to check out. Yeah. Build packs are a big part of that standardization. What is it about a build pack that enables all of this? Let's take node, for example, right? So when your code comes in, the first thing the build pack is going to do is it's going to scan and say, uh, what language is this? Is this a language that I even, you know, support, right? And it's going to see the package.json file or something and say, oh, okay, this is a node application. All right. I need to pull in these libraries, right? So it's going to go ahead and pull in all these libraries. And then from there, it's going to, it's going to go in and actually build the container that level of, of automation. But there are certain flags you need to set in certain instances. Sometimes you need, uh,

custom things added, but build packs, uh, and custom build packs provide a way to repeatedly inject things. So you're not just using, I think a lot of developers listening to this will, will probably be used to the idea of Docker files. Docker files are good when they're small, but as applications get more complex, they grow in size and they become really hard to maintain. And you can get a lot of really snowflakey builds. And that's really where the, our authorizing official becomes uncomfortable because Docker files, you could run anything. You could say, Hey, go grab this random thing on the internet and put it in the container. Whereas with Tanzu build service, you don't, your operators are more in control of that. And with build packs, you're more in control of what gets put into the container. From the standpoint of an operations or a security person, if you have a more opaque image that just the developers just gave you an image and they're like, run this, but you rely on them to give you a new image when it needs to be fixed. Whereas if you're relying more on a build pack, we'll assemble everything for you. You can just take that one, the operating system, for example, and upgrade it and rebuild everything without necessarily having to worry about the developers having to redo everything for you. I think you're hitting upon a good point is that the way a lot of organizations are adapting to this challenge of having developers pulling things in through Docker files, through Docker hub and other internet resources, they're adapting to this with a golden image. So they'll say, this is the golden Ubuntu container image. Thou shalt use this container image. The thing that build packs really bring and Tanzu build service brings is that ability to be dynamic, but then create things that are processes that are repeatable through custom build packs. So that you're with the golden image standpoint, you have to, it creates the old DevOps issue, right? It's, Hey, I'm a developer. I want to do this thing. Hey, your golden image doesn't work, or it doesn't have X. The operator adds X. Oh, it's not the right version of X, blah, blah, blah, back and forth. Whereas with build packs, a lot of times you're tweaking those and fine tuning those, you're able to set environment variables that say, Hey, I want this version of this. I want this version of that. And that can be really powerful for the developer as they embrace build packs as a way to build their containers.

Rather than writing these huge Docker files or using these golden images, you have a more dynamic way to unblock developers and not have this back, this traditional, we always come back to this traditional DevOps issue of the developer. Ops tries to rein everything in by saying, okay, use this. And then developers say, okay, I use that doesn't work. Add this. And that can, that back and forth can go back, back and forth forever. Another thing just to connect things together is another thing to notice in the flow, so to speak, that you're describing is most everything is not done by a human, right? Like it's all automated and therefore like logged and you can like check on things and there's extensive records of what happened and what went where, right? Like for example, with a build pack, you can, if you wanted to, when it gets assembled and put in a registry somewhere, you can track everything that happened and who authorized it and all this sort of stuff. So you almost have potentially, I would imagine too much information for auditors, right? Like more than they're used to. And however, more positively, like what I've always, people have told me I've heard about is you can take that big pile of information and start to like generate really trustworthy reports about what's been going on and the, the chain of ownership and just everything that's been happening, which is a lot more handy than the worst case scenario is like the, the auditor's chief tool is Microsoft office basically. So again, it seems like that's something that playing with the three little phrases in there. That's something that DevOps brings to security is we can automate a tremendous amount of the security work that you're doing because really no one wants to like manually fill out a document or even worse, a spreadsheet, right? Like you would like it to be automated for you and then focus on the more interesting human relatable things rather than just, they use this version of the framework, copy, paste. They use this framework, copy, paste.

Yeah. Yeah. And I think there's a natural evolution named. So when I went through that big, long monologue about the proof of concept we put in front of the authorizing official and later got approved. Okay. So in that example, I'm a security person. I want to see release X of, of this app is going through. I want to see, am I comfortable with it? They're looking at a few tools, right? Unfortunately, yeah, you do have to look at the trivia scan. You do have to look at the sonar cube report. You do have to look at contrast to see. So you do have a few things you have to look at, but it is a starting point. And all of those things do have APIs such that eventually maybe we can get this all wrapped into one report. And it's just a single report. The security person gets a PDF. I hate to say it probably will be a PDF or some sort of document, but they pull it up and they say, okay, let me read from top to bottom. Okay. There's no critical vulnerabilities. There's this isn't an issue. This isn't an issue. And then they can make a decision. And that's really what we're after is I don't think in the Fed space, we will ever get to humans not being in the loop. I think it's just because of the nature of crossing that line into software, crossing a line into a disconnected classified environment that will always require a human to be like, that's good. But we really need to take, you know, baby steps towards how do we optimize that process and make it one thing that they look through and say, yeah, this is secure enough. This meets my, you know, benchmark, which ultimately, yeah, it's hard to automate around a person, how a person gets a warm and fuzzy about your release to production. I always have a lot of conversations with people trying to improve the way they do software and governance and security always come up. And a lot of what you're saying is, is reminding me of a conversation that I try to have with them, which is sometimes governance and security is really annoying and is a blocker.

And you can fix that, right? Like, it might just be like, there's things that you can do, and it's actually not that big of a deal. But be careful, because a lot of the times, while it may seem like annoying and onerous and time consuming, like, governance and security is like a core feature of your software. And so like, you would spend a lot of time making sure that the UIs and the screens and whatever else you're doing works well, like it's worth your time to spend time on that. But oftentimes, especially in the Fed space, right, it being very secure is a feature. And so it's not a bad thing to spend a lot of time on that. Not a lot of time every single cycle, but to always be optimizing the tool chain to use the way you think about it. And really think about in the same way that like, nowadays, when we think about software development, the way that we do configuration and release management and automate builds and automate tests, like, developers aren't really, oh, man, what a terrible way to spend my time. That's part of what to be a developer and do application development is. And I think as you've been going over, like, if you take a more programmatic, automated way, a programmer's way of solving security problems and making security run better, then you can start to really just not in a dismissive way to use the word just, but you can just have it be part of the feature of your application, which is totally doable. And I don't know, maybe it's still takes some time. But like, again, it's a feature, right? Like, it's worth spending time on it as part of your application, not just like some annoying other people to talk to. Yeah, I think you're highlighting the, listen, we couldn't have, it wasn't just the proof of concept I put together that ultimately made this customer successful. It was a lot of labs as well. And so labs has expanded what they've offered. And to your point, they do have folks who deal with this sort of thing, not only platform as a product, but really what you're saying, which is like secure supply chain as a product, right?

Because it does need to, like you're saying, it does need to be a product. We not only have had huge strides using Pivotal Labs in the platform as a product space, but in the security tool chain as a product, that has been critical because those are the people navigating. You have a PM, we're looking at getting a designer, but those are the people who are navigating in the Fed space. There's often like a security approver. And so they're navigating those waters, building those relationships because anyone watching this that's dealt with security and Fed relationships are huge, right? You don't get to production without good relationships, open and honest conversations. Yeah, I can't stress enough, like you're saying, that product mindset around secure supply chain and building out that automation. All right. Thanks for going over that. The next time I have a conversation about DevSecOps, I'll be a little less snarky than I usually am. It's easy to get snarky. I was at first too. Yeah, yeah. But I get snarky about it. Yeah. Yeah. Thank you so much. Thanks for having me on.

Tanzu Talk · 🍕 Get more pizza.