1
00:00:00,000 --> 00:00:05,260
Well, hey, this is another interview that I first did as part of the videos that we do nowadays

2
00:00:05,260 --> 00:00:12,040
over at Tanzu.tv, you know, in my little show, Tanzu Talk. But I think it's a great overview of

3
00:00:12,040 --> 00:00:18,240
what it looks like to think about and put a lot of security into the build tool chain that you

4
00:00:18,240 --> 00:00:22,980
would have, kind of the way you think about how you're doing DevOps, as people are calling it

5
00:00:22,980 --> 00:00:29,900
nowadays, DevSecOps, which is a delightful term. But it's based on not just sort of like some

6
00:00:29,900 --> 00:00:35,400
theoretics and things, but an actual little case study that Chris Willis, who works here at VMware

7
00:00:35,400 --> 00:00:39,480
Tanzu, is doing with one of his US federal customers. And I think he really does a great

8
00:00:39,480 --> 00:00:45,700
job going over how they thought about what the problems are for securing a build tool chain and

9
00:00:45,700 --> 00:00:52,560
the runtime environment, you know, all the Kubernetes and Tanzu build service, all the stuff that we do

10
00:00:52,560 --> 00:00:59,580
over here in VMware Tanzu land, and kind of starts to get you thinking about how you would worry about

11
00:00:59,580 --> 00:01:04,860
that secure tool chain yourself and make sure that you're doing things in as he was doing with his

12
00:01:04,860 --> 00:01:11,680
customer in a kind of highly regulated, very secure environment. And it's a fun conversation.

13
00:01:12,100 --> 00:01:17,360
Also, you know, before I start, there's a lot of videos that I and other people put up. If you go to

14
00:01:17,360 --> 00:01:22,360
Tanzu.tv, you can find tons of them. And I try to put several of them here in the

15
00:01:22,360 --> 00:01:27,940
podcast feed, but there's more than we can handle. And I've got a lot of little tiny videos over there

16
00:01:27,940 --> 00:01:33,940
as well. So just go over to Tanzu.tv and you can look around there, you know, as the kids like to

17
00:01:33,940 --> 00:01:40,580
say, like and subscribe. Enjoy the interview. So why don't you introduce yourself? Yeah, my name is

18
00:01:40,580 --> 00:01:46,640
Chris Willis, and I am what's called a solutions engineer for VMware. So you're, I don't know,

19
00:01:47,200 --> 00:01:52,780
it probably is few. One of the few people around who has encountered the idea in practice applied the

20
00:01:52,780 --> 00:01:59,900
notion of DevSecOps. So first of all, like what is DevSecOps? And whenever I see a phrase like that,

21
00:01:59,960 --> 00:02:03,120
what I always think is, does that mean that DevOps wasn't secure already?

22
00:02:03,540 --> 00:02:10,060
Yeah, I didn't really follow the whole rise of DevSecOps as a buzzword. I'll offer my definition. I haven't

23
00:02:10,060 --> 00:02:17,600
gotten a secure DevSecOps pipeline working and process working at my client. I'll say this,

24
00:02:17,720 --> 00:02:25,140
it's a lot about unifying all the three teams. So DevSec and ops around a secure and consistent

25
00:02:25,140 --> 00:02:31,800
path to product. From the developer's keyboard to production, what are you doing to make sure that

26
00:02:31,800 --> 00:02:39,300
process is not only secure, but it's repeatable and it fits into the business as a whole. Everyone is rallied

27
00:02:39,300 --> 00:02:45,820
around this process and accepts it as the way to do business. And if you want to make changes to that

28
00:02:45,820 --> 00:02:51,640
DevSecOps pipeline, you're doing it through the GitOps model, through PRs, things like that. But the big thing

29
00:02:51,640 --> 00:02:59,140
is consistency, repeatability, and using a lot of good ingredients. If you use a cooking analogy, using a lot of

30
00:02:59,140 --> 00:03:05,340
good ingredients, meaning you're not pulling things straight from Docker Hub, you're maybe getting

31
00:03:05,340 --> 00:03:11,680
them from a place, your base images and container images from a place, from a known good source. So

32
00:03:11,680 --> 00:03:17,960
sometimes you can find yourself on Stack Overflow and someone says, hey, use this container image to do

33
00:03:17,960 --> 00:03:22,960
a thing. A lot of times you're not double checking to see what is that container image doing. And if it

34
00:03:22,960 --> 00:03:27,960
were to make it to production, it could have severe consequences. So you want to make sure that that

35
00:03:27,960 --> 00:03:37,960
supply chain is good. It's really baking in security to your CICD process in a holistic manner, really

36
00:03:37,960 --> 00:03:44,800
embedding your security engineers and having them be not just people who are checking requirements or

37
00:03:44,800 --> 00:03:51,240
yes, you're vulnerable to this CVE or you're not, but someone who can also contribute and add to the

38
00:03:51,240 --> 00:03:57,100
pipeline in a meaningful way. But not boiling the ocean, you can get into analysis paralysis with all

39
00:03:57,100 --> 00:04:02,180
of these tools, these security tools you have now, right? It's I want the CICD process, it's going to

40
00:04:02,180 --> 00:04:08,300
have this and that and I'm going to bolt this on here. And you can quickly lose focus in the business

41
00:04:08,300 --> 00:04:14,160
process and that unification of the three teams, which is really what I think DevSecOps is getting at,

42
00:04:14,240 --> 00:04:20,020
can fall away as you continue to just have this cowboy attitude of let me add this cool tool and

43
00:04:20,020 --> 00:04:25,220
this cool tool, right? You want to make sure it's a unified front between all three parties.

44
00:04:25,560 --> 00:04:30,140
Let's talk about that, that boiling the ocean problem. When you're solving a problem like this

45
00:04:30,140 --> 00:04:36,980
or any problem, there's a certain amount of not being perfect that you have to find acceptable,

46
00:04:37,240 --> 00:04:44,000
which in a security context is difficult to be comfortable with. In reading about DevSecOps and

47
00:04:44,000 --> 00:04:49,380
like over the years, it seems like one of the key things that people have to figure out is like

48
00:04:49,380 --> 00:04:56,000
the software has to be released. So eventually, and actually very frequently, maybe weekly or monthly,

49
00:04:56,220 --> 00:05:01,240
we have to release software. And the most secure thing to do would be not to release software.

50
00:05:01,240 --> 00:05:05,700
Like maybe we can boil a lake, but we can't boil the ocean.

51
00:05:06,000 --> 00:05:12,960
I think Fed makes this pretty clear cut in certain ways. There's the use of FIPS compliant

52
00:05:12,960 --> 00:05:20,160
cryptographic libraries. Your encryption has to be using known good algorithms or algorithms that at

53
00:05:20,160 --> 00:05:27,480
least in the government space, they deem as secure. There's also this idea of STIGs. So these are

54
00:05:28,080 --> 00:05:35,260
vast lists of if you have a Red Hat base image, or you have a Ubuntu base image, it's got to have this file

55
00:05:35,260 --> 00:05:38,740
tweaked this way, this file tweaked this way, blah, blah, blah, all the way down.

56
00:05:38,820 --> 00:05:42,860
So it's like the configuration of various things that needs to be configured this way.

57
00:05:43,020 --> 00:05:43,100
Okay.

58
00:05:43,160 --> 00:05:49,240
So I think your encryption has to be, you have to know you're using good encryption libraries,

59
00:05:49,340 --> 00:05:55,980
which is typically the case, I would say. And then the last thing I'll say is the source of

60
00:05:55,980 --> 00:06:02,180
everything. So where everything's coming from. So if you have bad, a bad supply chain, if you have

61
00:06:02,180 --> 00:06:08,460
someone injecting malicious code in that supply chain somewhere, that really matters too. Those are

62
00:06:08,460 --> 00:06:15,580
like the three things that you really want to be paying attention to. Before the fed space, we have

63
00:06:15,580 --> 00:06:20,260
something called an authorizing official. And they're ultimately the one that kind of like you're

64
00:06:20,260 --> 00:06:25,500
saying they accept that risk. They're the one that says, yes, this can go to production or no,

65
00:06:25,540 --> 00:06:30,480
it can't. This is the process I'm over. If you follow this process, I'm comfortable with things

66
00:06:30,480 --> 00:06:36,360
going to production repeatedly through. You could never have a situation where every release and this

67
00:06:36,360 --> 00:06:44,140
old way is every release goes to the authorizing official or the local what's called an ism or security

68
00:06:44,140 --> 00:06:49,500
person. That's part of the team and say, Hey, is this okay? Okay. How about this? If you're releasing

69
00:06:49,500 --> 00:06:55,980
on a cadence that Silicon Valley releases on and getting things to users quickly, that's just not

70
00:06:55,980 --> 00:07:01,740
possible. That's what we did with Tanzu. And what really helps is that that consistency and that

71
00:07:01,740 --> 00:07:07,300
supply chain problem specifically is huge. The reason that we have dev and ops next to security

72
00:07:07,300 --> 00:07:15,680
in dev sec ops is that we want to have very rapid automated releases. And it seems like a lot of

73
00:07:15,680 --> 00:07:24,040
security stuff is not easy to automate as it currently exists. So it seems like that is in a very high

74
00:07:24,040 --> 00:07:31,220
level way. The kind of problem statement of dev sec ops is that we can automate a tremendous amount of

75
00:07:31,220 --> 00:07:36,360
getting that software to production, but we can't really automate the security stuff. I think a lot of that

76
00:07:36,360 --> 00:07:41,080
has to do with just like the systems and the policies just aren't to use another funny phrase

77
00:07:41,080 --> 00:07:48,180
automatable. There's not APIs into it. So that seems like one of the core things in the last few years

78
00:07:48,180 --> 00:07:55,400
trying to be solved for is how you automate those things. So you listed, well, specific to the,

79
00:07:55,620 --> 00:08:00,240
and I should say, when we say fed, we mean US United States fed. I'm sure it's similar in other

80
00:08:00,240 --> 00:08:04,920
governments, but sorry, there are other federal governments. Like there are three things specifically

81
00:08:04,920 --> 00:08:10,140
in the US fed space that you had to pay attention to. And that is you need to have certified is the

82
00:08:10,140 --> 00:08:13,740
wrong word, but you need to have the official, the sanctioned encryption that you're using.

83
00:08:13,740 --> 00:08:20,180
And they're for various components in your stack, whether it's an operating system or whatever else.

84
00:08:20,400 --> 00:08:25,460
Does it go up? Does the configuration, what did you call it? The stig that where you've got.

85
00:08:25,620 --> 00:08:30,880
Yeah. Yeah. Yeah. And so the other thing is you have, as you were saying, the stig where you have,

86
00:08:30,960 --> 00:08:37,120
again, authorized configuration of components from the operating system. And does that go up to like

87
00:08:37,120 --> 00:08:40,820
middleware? Like how far up does that configuration go?

88
00:08:42,020 --> 00:08:47,140
Yeah. It can go to like Tomcat would have a stig or yeah, something like that. So it,

89
00:08:47,200 --> 00:08:52,580
it does go pretty high up there and that's where things can get tricky.

90
00:08:53,020 --> 00:08:58,780
Yeah. Yeah. I imagine. And then the third thing is, and help me word this correctly, but it is

91
00:08:58,780 --> 00:09:04,780
being able to demonstrate and trust, being able to demonstrate that the chain of ownership or the

92
00:09:04,780 --> 00:09:09,220
chain of custody or the provenance or whatever, like where did all this stuff come from and what

93
00:09:09,220 --> 00:09:13,880
version is it? And then obviously it's, I'm being overly explicit. It's worth saying the second part

94
00:09:13,880 --> 00:09:19,160
and also trusting the report that you get, like generating reports, one thing, but trusting it

95
00:09:19,160 --> 00:09:20,940
is a whole separate thing.

96
00:09:20,940 --> 00:09:26,720
You, you have to open up visibility into the product because it's okay. Why should I trust

97
00:09:26,720 --> 00:09:31,920
like you VMware? Yeah, sure. Okay. You're, you're a us-based company and sure, blah, blah, blah. But

98
00:09:31,920 --> 00:09:38,160
like, why should I trust this? Right. And so a lot of it is opening up process and getting the

99
00:09:38,160 --> 00:09:45,060
developers in contact, the developers of the secure supply chain in contact with the developers of

100
00:09:45,060 --> 00:09:50,240
Tanzu application catalog and Tanzu build service in these products and, and having them walk them

101
00:09:50,240 --> 00:09:54,560
through things that aren't open on the website, right? They're not on the Tanzu website. It's like,

102
00:09:54,560 --> 00:10:01,360
Hey, this is how we, an open source release is made to Apache Tomcat. This is the process that it goes

103
00:10:01,360 --> 00:10:07,960
through in Bitnami's now they're part of VMware, but in Bitnami's process to secure it, make sure

104
00:10:07,960 --> 00:10:16,820
everything is aligned and you should trust that. So in some cases, it's not just me as the, in the

105
00:10:16,820 --> 00:10:21,340
convincing business saying, Hey, you should totally trust us. It's, Hey, we'll link you up with the

106
00:10:21,340 --> 00:10:25,880
engineers of the team. They'll walk you through the process and then you can decide whether or not

107
00:10:25,880 --> 00:10:30,740
that sounds like something you could trust. So it's, there are, there is an art form to it.

108
00:10:30,880 --> 00:10:35,860
We have those three things. Now, I think what would be useful is to, since you have worked on a POC

109
00:10:35,860 --> 00:10:41,300
and projects and stuff around this, can you walk us through a more concrete example of how this all

110
00:10:41,300 --> 00:10:41,940
comes together?

111
00:10:41,940 --> 00:10:46,620
What we're talking about really, and the proof of concept that we put in front of the AO, the

112
00:10:46,620 --> 00:10:51,660
authorizing official was something where the developer commits code, right? Goes into GitHub.

113
00:10:51,880 --> 00:10:57,600
We're all used to that process, but then there's a pipeline putting this together. And by the way,

114
00:10:57,600 --> 00:11:02,780
we've put together a reference implementation for other people to use. If they're using concourse,

115
00:11:02,840 --> 00:11:08,200
they can reuse this. But what it does is it picks up and says, Oh, Hey, I see a new git commit.

116
00:11:08,600 --> 00:11:14,000
I'm going to run this through a tool called sonar queue, which is what we're using. And ultimately

117
00:11:14,000 --> 00:11:20,200
we'll say this is a pass or fail, right? This has too many code smells, or this is under the

118
00:11:20,200 --> 00:11:25,500
threshold that you've told me is good or bad and gives it a rating. And it's looking for all sorts

119
00:11:25,500 --> 00:11:31,880
of things, vulnerabilities, but in the code itself. And then you have a OWASP dependency checker that's

120
00:11:31,880 --> 00:11:38,060
going and saying, Hey, the libraries that this developer used to write this code, are those good?

121
00:11:38,260 --> 00:11:44,680
Is there anything wrong there? And in some ways we double tap on that dependency front later when we do the

122
00:11:44,680 --> 00:11:50,900
container repository scan, but it's good to have the upfront sort of quick feedback loop. So if the

123
00:11:50,900 --> 00:11:56,140
developer commits code and sonar cube or OWASP dependency checker right away says, no, there's a

124
00:11:56,140 --> 00:12:02,000
problem here. The pipeline will fail, kick it back to them and they can improve and do whatever to get

125
00:12:02,000 --> 00:12:02,940
things moving again.

126
00:12:02,940 --> 00:12:06,320
When you automate it, does it take a long time or a short time or?

127
00:12:06,480 --> 00:12:12,040
No, it can be, it can be within the order of minutes to get feedback. I mean, I guess it could

128
00:12:12,040 --> 00:12:17,420
depend on how much code you're, you're scanning, but typically with like microservices and smaller

129
00:12:17,420 --> 00:12:21,680
things. Yeah. You're talking a few minutes. The next part. So let's say you pass the gate,

130
00:12:21,780 --> 00:12:24,920
right? Cause if you don't pass the gate, it gets kicked back. You have to make the change,

131
00:12:24,920 --> 00:12:29,580
but let's say you pass the gate into kind of that third phase. This is where Tanzu build service.

132
00:12:29,780 --> 00:12:34,980
So this is much more part of the supply chain piece of it that I was talking about. Like that third

133
00:12:34,980 --> 00:12:42,220
thing we talked about where the Tanzu build service is taking the code using a regularly updated and

134
00:12:42,220 --> 00:12:47,700
maintained build packs and what are called stacks, which are just like the base operating system layer

135
00:12:47,700 --> 00:12:54,080
of the container. It's using those foundational pieces again, regularly updated and maintained by

136
00:12:54,080 --> 00:12:59,720
VMware to actually build the container. That's going to run the code. So that goes and builds the

137
00:12:59,720 --> 00:13:06,080
container, pushes it to Harbor, which is what we're using for a container repository. And then from

138
00:13:06,080 --> 00:13:11,320
there, we're doing again that, like I said, that double tap kind of scanning, right? So we're scanning

139
00:13:11,320 --> 00:13:20,500
the container using Trivi and that is generating its own set of findings that saying this container has

140
00:13:20,500 --> 00:13:29,520
two high vulnerabilities, three mediums and 20 low vulnerabilities. And then it depends based on

141
00:13:29,520 --> 00:13:36,160
the level of risk that the authorizing officials able to, or willing to accept. A lot of them will just

142
00:13:36,160 --> 00:13:41,200
say, yeah, make sure your highs are taken care of. And any mediums, again, we're referring to

143
00:13:41,200 --> 00:13:49,100
CVEs, right? Vulnerabilities that can be found on cve.mitre.org or anywhere else for the specific

144
00:13:49,100 --> 00:13:54,880
operating system or whatever. But that, that scanning is taking place and you're generating an

145
00:13:54,880 --> 00:14:00,740
artifact that ultimately the local security team can review. So you're gathering this body of evidence,

146
00:14:00,740 --> 00:14:06,260
this, this group of things that the security team is going to review, which is much better than

147
00:14:06,260 --> 00:14:10,840
how it was done previously, where you're actually looking through code or you're taking the

148
00:14:11,080 --> 00:14:15,940
developer's word for it. You're looking at hard published artifacts that were generated by the

149
00:14:15,940 --> 00:14:20,640
pipeline. And again, this is why your security team has to be bought in. Because if they say those

150
00:14:20,640 --> 00:14:25,100
aren't good enough, then you got to go back to the drawing board. But the last thing we do,

151
00:14:25,880 --> 00:14:31,280
and this is to really complete the cycle, is we deploy the container, right? On Kubernetes,

152
00:14:31,980 --> 00:14:39,660
we deploy it with this contrast agent. And what contrast does is it will actually do like that real

153
00:14:39,660 --> 00:14:46,440
time scanning and feedback about in in runtime, is there an issue with the application and that

154
00:14:46,440 --> 00:14:53,880
library that you used in the container 20 days ago, was there a zero day just introduced? So that way,

155
00:14:53,940 --> 00:15:00,120
the security team can log into their contrast dashboard, they can look and say, Oh, hey, it looks like this

156
00:15:00,120 --> 00:15:05,980
application has a vulnerability. And the reason why Tanzu build service is so critical is typically

157
00:15:05,980 --> 00:15:11,200
that vulnerability, oftentimes those vulnerabilities are in libraries, or they're in the base

158
00:15:11,200 --> 00:15:17,920
operating system of the container. So what you would do in that case is not really go back to the app

159
00:15:17,920 --> 00:15:22,840
developer, which is what was done previously, the app developer, patch your stuff, or go back to the

160
00:15:22,840 --> 00:15:28,660
operator and say, Hey, patch this thing that can spin out of control as far as like cycle time,

161
00:15:28,660 --> 00:15:34,820
because now the operator or developer has to say, Oh, okay, did they release a fix for it yet? Let me go

162
00:15:34,820 --> 00:15:40,600
find it. Does my code work with it? What's going on? But in this model, what you do is the operators

163
00:15:40,600 --> 00:15:48,400
would update the stack and build packs, right on Tanzu build service. So those base OS layer,

164
00:15:48,520 --> 00:15:53,840
and then the build packs that include the libraries and say, Hey, can you push that again? And we'll see if

165
00:15:53,840 --> 00:15:58,940
that vulnerability pops up. And many times what will happen is the vulnerabilities patched at the speed

166
00:15:58,940 --> 00:16:03,500
of the developer pushing new code. So that's pretty much all they have to do to patch their application

167
00:16:03,500 --> 00:16:09,800
with Tanzu build services is push the new code. And it'll build using the latest build pack and stack

168
00:16:09,800 --> 00:16:17,040
And you're off to the races that whole chain is built around when we become knowledgeable of a bad piece

169
00:16:17,040 --> 00:16:23,460
of code. We need to see if we have it and fix it right when a CV comes out and identifies this little

170
00:16:23,460 --> 00:16:31,560
wing ding version 1.2 a B seven beta, whatever has a vulnerability in it. We need to know where it is

171
00:16:31,560 --> 00:16:36,740
and basically fix it as soon as possible. What DevOps can bring to that is what you're describing

172
00:16:36,740 --> 00:16:41,740
at the end is that what DevOps with containers, let's put it that way, because of the way that

173
00:16:41,740 --> 00:16:47,720
containers are layered from the operating system to the frameworks that you need to whatever your custom

174
00:16:47,720 --> 00:16:53,360
code is, right? Like those three layers are don't have to be treated as just one giant monolithic

175
00:16:53,360 --> 00:16:59,360
thing that developer that's a black box. And so all throughout the build process, you can treat each of

176
00:16:59,360 --> 00:17:03,360
those three layers separately and scan them all the time. And then more importantly, when you find out

177
00:17:03,360 --> 00:17:07,380
a problem in one of those three layers, and I guess there's also the layers of configuration and

178
00:17:07,380 --> 00:17:11,760
networking, there's a whole bunch of other stuff, but whatever. All of these things are atomic units

179
00:17:11,760 --> 00:17:18,380
that you can independently manage and update as needed. We'll go back to an example we were using

180
00:17:18,380 --> 00:17:23,100
earlier. I don't know if this is true. You'd have to tell me technically, but like we find there's a

181
00:17:23,100 --> 00:17:27,840
problem in Tomcat and because of the way in our build process and because we're doing things

182
00:17:27,840 --> 00:17:33,460
containerized because the way that Tomcat is layered in to the application that we have and because of

183
00:17:33,460 --> 00:17:40,180
the way configuration is done separately. Like Tomcat doesn't become this big ball of configuration that

184
00:17:40,180 --> 00:17:45,760
only the developers can unwind. And therefore, when something, some bad chunk of code is out there,

185
00:17:45,840 --> 00:17:50,460
it's a lot faster because we have this automated pipeline to ask the development team to put this

186
00:17:50,460 --> 00:17:55,340
new version of Tomcat in and make sure it doesn't break your application really quickly. And then let's

187
00:17:55,340 --> 00:18:00,640
deploy that thing quickly. And we don't have to worry about doing a major rev of the entire project.

188
00:18:00,960 --> 00:18:03,580
We can address this, this one little layer essentially.

189
00:18:04,120 --> 00:18:09,060
Yeah. Small changes. And those layers, those container image layers are critical because

190
00:18:09,060 --> 00:18:14,420
you're in many times you're just swapping out one layer with, with the new version of open SSL or

191
00:18:14,420 --> 00:18:19,680
whatever. And at scale, that can be really important. You think you have a thousand applications

192
00:18:19,680 --> 00:18:25,140
times when the next heart bleed happens. Yeah. You got to figure out a way to patch all of those.

193
00:18:25,140 --> 00:18:31,280
Whereas in this model, you would just say, Hey, we updated the new stack or build pack that addresses

194
00:18:31,280 --> 00:18:37,000
that, that issue. We push your code. We'll go through and do our testing again, all of your CI

195
00:18:37,000 --> 00:18:43,500
processes. But ultimately we'll rebuild the container with the new patched container of open

196
00:18:43,500 --> 00:18:50,720
or library of open SSL. So it moves at the speed of the developer. The developer doesn't need to rely

197
00:18:50,720 --> 00:18:59,160
on anyone else. What role does standardization of all of this play in a large organization?

198
00:18:59,500 --> 00:19:07,760
This pipeline is meant to take in all code, but I think, yeah, the one part around standardization

199
00:19:07,760 --> 00:19:14,640
is you really do need to be bought into the idea of build packs in order for this to succeed. A lot of

200
00:19:14,640 --> 00:19:20,720
our pivotal customers and Tanzu application service customers, they are very familiar with build packs,

201
00:19:20,720 --> 00:19:26,660
but that's the standardization layer, right? That's the level of automation and standardization that

202
00:19:26,660 --> 00:19:32,100
needs to be bought into in order for a lot of this to work. Because you're basically committing to,

203
00:19:32,560 --> 00:19:37,800
yes, I will write my code in such a way that I'm used to using build packs. Developers can get

204
00:19:37,800 --> 00:19:42,880
started and experiment with that on KPAC. Tanzu build service is really just like a wraparound KPAC.

205
00:19:43,100 --> 00:19:47,080
So that's something to check out. Yeah. Build packs are a big part of that standardization.

206
00:19:47,440 --> 00:19:50,620
What is it about a build pack that enables all of this?

207
00:19:50,720 --> 00:19:55,740
Let's take node, for example, right? So when your code comes in, the first thing the build pack is

208
00:19:55,740 --> 00:20:01,020
going to do is it's going to scan and say, uh, what language is this? Is this a language that I even,

209
00:20:01,120 --> 00:20:06,940
you know, support, right? And it's going to see the package.json file or something and say,

210
00:20:07,140 --> 00:20:11,100
oh, okay, this is a node application. All right. I need to pull in these libraries,

211
00:20:11,100 --> 00:20:15,300
right? So it's going to go ahead and pull in all these libraries. And then from there,

212
00:20:15,300 --> 00:20:20,200
it's going to, it's going to go in and actually build the container that level of, of automation.

213
00:20:20,200 --> 00:20:25,600
But there are certain flags you need to set in certain instances. Sometimes you need, uh,

214
00:20:25,600 --> 00:20:30,940
custom things added, but build packs, uh, and custom build packs provide a way to

215
00:20:30,940 --> 00:20:36,520
repeatedly inject things. So you're not just using, I think a lot of developers listening to this will,

216
00:20:36,680 --> 00:20:41,440
will probably be used to the idea of Docker files. Docker files are good when they're small,

217
00:20:41,440 --> 00:20:48,680
but as applications get more complex, they grow in size and they become really hard to maintain.

218
00:20:48,680 --> 00:20:55,020
And you can get a lot of really snowflakey builds. And that's really where the, our authorizing

219
00:20:55,020 --> 00:20:59,740
official becomes uncomfortable because Docker files, you could run anything. You could say,

220
00:20:59,840 --> 00:21:04,280
Hey, go grab this random thing on the internet and put it in the container. Whereas with Tanzu build

221
00:21:04,280 --> 00:21:08,900
service, you don't, your operators are more in control of that. And with build packs, you're more

222
00:21:08,900 --> 00:21:13,940
in control of what gets put into the container. From the standpoint of an operations or a security

223
00:21:13,940 --> 00:21:20,060
person, if you have a more opaque image that just the developers just gave you an image and they're

224
00:21:20,060 --> 00:21:24,120
like, run this, but you rely on them to give you a new image when it needs to be fixed. Whereas if

225
00:21:24,120 --> 00:21:28,520
you're relying more on a build pack, we'll assemble everything for you. You can just take that one,

226
00:21:28,520 --> 00:21:33,880
the operating system, for example, and upgrade it and rebuild everything without necessarily having

227
00:21:33,880 --> 00:21:37,320
to worry about the developers having to redo everything for you.

228
00:21:37,500 --> 00:21:43,660
I think you're hitting upon a good point is that the way a lot of organizations are adapting to this

229
00:21:43,660 --> 00:21:49,800
challenge of having developers pulling things in through Docker files, through Docker hub and other

230
00:21:49,800 --> 00:21:54,880
internet resources, they're adapting to this with a golden image. So they'll say, this is the golden

231
00:21:54,880 --> 00:21:58,360
Ubuntu container image. Thou shalt use this container image.

232
00:21:58,360 --> 00:22:03,560
The thing that build packs really bring and Tanzu build service brings is that ability to be dynamic,

233
00:22:03,560 --> 00:22:08,380
but then create things that are processes that are repeatable through custom build packs. So that

234
00:22:08,380 --> 00:22:15,020
you're with the golden image standpoint, you have to, it creates the old DevOps issue, right? It's,

235
00:22:15,180 --> 00:22:19,420
Hey, I'm a developer. I want to do this thing. Hey, your golden image doesn't work, or it doesn't have X.

236
00:22:19,580 --> 00:22:40,860
The operator adds X. Oh, it's not the right version of X, blah, blah, blah, back and forth. Whereas with build packs, a lot of times you're tweaking those and fine tuning those, you're able to set environment variables that say, Hey, I want this version of this. I want this version of that. And that can be really powerful for the developer as they embrace build packs as a way to build their containers.

237
00:22:40,860 --> 00:22:56,020
Rather than writing these huge Docker files or using these golden images, you have a more dynamic way to unblock developers and not have this back, this traditional, we always come back to this traditional DevOps issue of the developer.

238
00:22:56,440 --> 00:23:03,560
Ops tries to rein everything in by saying, okay, use this. And then developers say, okay, I use that doesn't work. Add this.

239
00:23:04,060 --> 00:23:07,980
And that can, that back and forth can go back, back and forth forever.

240
00:23:08,340 --> 00:23:19,740
Another thing just to connect things together is another thing to notice in the flow, so to speak, that you're describing is most everything is not done by a human, right?

241
00:23:19,760 --> 00:23:28,620
Like it's all automated and therefore like logged and you can like check on things and there's extensive records of what happened and what went where, right?

242
00:23:28,620 --> 00:23:39,140
Like for example, with a build pack, you can, if you wanted to, when it gets assembled and put in a registry somewhere, you can track everything that happened and who authorized it and all this sort of stuff.

243
00:23:39,320 --> 00:23:44,960
So you almost have potentially, I would imagine too much information for auditors, right?

244
00:23:45,020 --> 00:23:46,240
Like more than they're used to.

245
00:23:46,400 --> 00:24:09,740
And however, more positively, like what I've always, people have told me I've heard about is you can take that big pile of information and start to like generate really trustworthy reports about what's been going on and the, the chain of ownership and just everything that's been happening, which is a lot more handy than the worst case scenario is like the, the auditor's chief tool is Microsoft office basically.

246
00:24:10,100 --> 00:24:14,800
So again, it seems like that's something that playing with the three little phrases in there.

247
00:24:14,860 --> 00:24:28,700
That's something that DevOps brings to security is we can automate a tremendous amount of the security work that you're doing because really no one wants to like manually fill out a document or even worse, a spreadsheet, right?

248
00:24:28,700 --> 00:24:38,440
Like you would like it to be automated for you and then focus on the more interesting human relatable things rather than just, they use this version of the framework, copy, paste.

249
00:24:38,540 --> 00:24:41,160
They use this framework, copy, paste.

250
00:24:41,520 --> 00:24:41,840
Yeah.

251
00:24:42,220 --> 00:24:42,700
Yeah.

252
00:24:42,800 --> 00:24:46,000
And I think there's a natural evolution named.

253
00:24:46,000 --> 00:24:54,380
So when I went through that big, long monologue about the proof of concept we put in front of the authorizing official and later got approved.

254
00:24:55,340 --> 00:24:55,740
Okay.

255
00:24:55,740 --> 00:24:58,420
So in that example, I'm a security person.

256
00:24:58,680 --> 00:25:02,500
I want to see release X of, of this app is going through.

257
00:25:02,620 --> 00:25:04,200
I want to see, am I comfortable with it?

258
00:25:04,300 --> 00:25:06,300
They're looking at a few tools, right?

259
00:25:06,400 --> 00:25:09,360
Unfortunately, yeah, you do have to look at the trivia scan.

260
00:25:09,520 --> 00:25:13,020
You do have to look at the sonar cube report.

261
00:25:13,220 --> 00:25:15,920
You do have to look at contrast to see.

262
00:25:16,080 --> 00:25:20,720
So you do have a few things you have to look at, but it is a starting point.

263
00:25:20,720 --> 00:25:29,400
And all of those things do have APIs such that eventually maybe we can get this all wrapped into one report.

264
00:25:29,400 --> 00:25:31,100
And it's just a single report.

265
00:25:31,240 --> 00:25:33,500
The security person gets a PDF.

266
00:25:33,820 --> 00:25:41,720
I hate to say it probably will be a PDF or some sort of document, but they pull it up and they say, okay, let me read from top to bottom.

267
00:25:41,940 --> 00:25:42,380
Okay.

268
00:25:42,380 --> 00:25:43,900
There's no critical vulnerabilities.

269
00:25:44,140 --> 00:25:45,620
There's this isn't an issue.

270
00:25:45,700 --> 00:25:46,500
This isn't an issue.

271
00:25:46,720 --> 00:25:48,080
And then they can make a decision.

272
00:25:48,320 --> 00:25:55,540
And that's really what we're after is I don't think in the Fed space, we will ever get to humans not being in the loop.

273
00:25:55,860 --> 00:26:07,760
I think it's just because of the nature of crossing that line into software, crossing a line into a disconnected classified environment that will always require a human to be like, that's good.

274
00:26:08,120 --> 00:26:20,980
But we really need to take, you know, baby steps towards how do we optimize that process and make it one thing that they look through and say, yeah, this is secure enough.

275
00:26:21,200 --> 00:26:33,600
This meets my, you know, benchmark, which ultimately, yeah, it's hard to automate around a person, how a person gets a warm and fuzzy about your release to production.

276
00:26:33,600 --> 00:26:40,380
I always have a lot of conversations with people trying to improve the way they do software and governance and security always come up.

277
00:26:40,600 --> 00:26:50,260
And a lot of what you're saying is, is reminding me of a conversation that I try to have with them, which is sometimes governance and security is really annoying and is a blocker.

278
00:26:50,480 --> 00:26:51,900
And you can fix that, right?

279
00:26:51,920 --> 00:26:56,080
Like, it might just be like, there's things that you can do, and it's actually not that big of a deal.

280
00:26:56,240 --> 00:27:06,780
But be careful, because a lot of the times, while it may seem like annoying and onerous and time consuming, like, governance and security is like a core feature of your software.

281
00:27:07,180 --> 00:27:15,260
And so like, you would spend a lot of time making sure that the UIs and the screens and whatever else you're doing works well, like it's worth your time to spend time on that.

282
00:27:15,260 --> 00:27:20,680
But oftentimes, especially in the Fed space, right, it being very secure is a feature.

283
00:27:21,480 --> 00:27:26,360
And so it's not a bad thing to spend a lot of time on that.

284
00:27:27,280 --> 00:27:33,080
Not a lot of time every single cycle, but to always be optimizing the tool chain to use the way you think about it.

285
00:27:33,080 --> 00:27:48,200
And really think about in the same way that like, nowadays, when we think about software development, the way that we do configuration and release management and automate builds and automate tests, like, developers aren't really, oh, man, what a terrible way to spend my time.

286
00:27:48,560 --> 00:27:52,400
That's part of what to be a developer and do application development is.

287
00:27:52,460 --> 00:28:12,320
And I think as you've been going over, like, if you take a more programmatic, automated way, a programmer's way of solving security problems and making security run better, then you can start to really just not in a dismissive way to use the word just, but you can just have it be part of the feature of your application, which is totally doable.

288
00:28:12,320 --> 00:28:14,160
And I don't know, maybe it's still takes some time.

289
00:28:14,160 --> 00:28:17,240
But like, again, it's a feature, right?

290
00:28:17,300 --> 00:28:23,580
Like, it's worth spending time on it as part of your application, not just like some annoying other people to talk to.

291
00:28:23,900 --> 00:28:32,880
Yeah, I think you're highlighting the, listen, we couldn't have, it wasn't just the proof of concept I put together that ultimately made this customer successful.

292
00:28:33,160 --> 00:28:36,320
It was a lot of labs as well.

293
00:28:36,320 --> 00:28:40,100
And so labs has expanded what they've offered.

294
00:28:40,380 --> 00:28:53,100
And to your point, they do have folks who deal with this sort of thing, not only platform as a product, but really what you're saying, which is like secure supply chain as a product, right?

295
00:28:53,160 --> 00:28:56,140
Because it does need to, like you're saying, it does need to be a product.

296
00:28:56,140 --> 00:29:09,240
We not only have had huge strides using Pivotal Labs in the platform as a product space, but in the security tool chain as a product, that has been critical because those are the people navigating.

297
00:29:09,540 --> 00:29:15,660
You have a PM, we're looking at getting a designer, but those are the people who are navigating in the Fed space.

298
00:29:15,780 --> 00:29:17,980
There's often like a security approver.

299
00:29:18,240 --> 00:29:27,320
And so they're navigating those waters, building those relationships because anyone watching this that's dealt with security and Fed relationships are huge, right?

300
00:29:27,440 --> 00:29:33,000
You don't get to production without good relationships, open and honest conversations.

301
00:29:33,000 --> 00:29:40,480
Yeah, I can't stress enough, like you're saying, that product mindset around secure supply chain and building out that automation.

302
00:29:40,740 --> 00:29:40,900
All right.

303
00:29:40,920 --> 00:29:41,900
Thanks for going over that.

304
00:29:41,980 --> 00:29:46,800
The next time I have a conversation about DevSecOps, I'll be a little less snarky than I usually am.

305
00:29:47,620 --> 00:29:48,940
It's easy to get snarky.

306
00:29:49,020 --> 00:29:50,240
I was at first too.

307
00:29:50,440 --> 00:29:50,800
Yeah, yeah.

308
00:29:50,800 --> 00:29:52,180
But I get snarky about it.

309
00:29:52,240 --> 00:29:52,400
Yeah.

310
00:29:52,760 --> 00:29:53,080
Yeah.

311
00:29:53,220 --> 00:29:53,840
Thank you so much.

312
00:29:53,920 --> 00:29:54,520
Thanks for having me on.