Security processes & culture, with Molly Crowther (Ep. 96)

Episode 81 · 51:39 · SRT · archive.org · all Tanzu Talk

Well, we were at sales kickoff last week. I'm sorry. I'm sorry. Field kickoff. FKO. And that was pretty fun. You gave a nice talk there, Richard. You mastered the art of the gigantic picture on one slide. Yeah, I tried not to go with the scary thing like a clown or something that would really mess people up. Although we did see clowns later in the week. Clowns in the woods. Yeah, we had a good crowd there. It's crazy to see how many folks are here at Pivotal and how fast we've grown over even these last few months from a sales perspective. It was nice to see you in person as well. We don't just have our digital experiences. That's right. Not just our digital experiences. I like that. Well, why don't you introduce yourself briefly, guest, before I regale the audience with more nonsense from my head? Hey, I'm Molly Crowther. I'm a senior technical program manager at Pivotal. I work mostly on security for Cloud Foundry. Fantastic. So here's what I wanted to ask you, Richard. Now, when you have the giant pictures in your presentation, I have found the following. And maybe you too, Molly, have had this problem. It's like, so you got a big picture. And the colors in the picture change. And then often they conflict with the text you want to use. And, you know, you can do the thing where, like, where, like, you got, like, a solid background. So you got that going on. Or maybe you can do, like, the thing where you mess with the transparency of it. Or, like, I don't even know. But, like, what is y'all's solution for the giant picture? And then you need to put a title on there. And then you can have it actually, like, be readable. Yeah, these are first world problems, aren't they? I either go, you know, it's tough to find pictures that always have, like, that one light area where you can put dark text on it. That's way too convenient. So I usually do end up, as you said, either I put, like, you know, dark box on top of it with a transparency and then put text on that. Or do, like, a black bar across the whole thing and put the text on that. So, yeah, there's just nothing worse than getting those slides up on the screen and everything's completely unreadable as you put, like, green text on a, you know, green background.

So try to be considerate to the speaker. You seem to do the same. When I see your talks, sometimes you use pictures and the like. And, yeah, it seems like you use a mix of things, no? Yeah, yeah. I'm always very unsatisfied with it. I don't know. So, Molly, I've seen several of your presentations now. And by several, I mean two. I actually did some preparation for this episode, which is just to blow Richard's mind. Oh, great. But thank you. But would you ever be so bold as to only have a picture on a slide? Like, is that something you would go for and trust that it works out? Yeah, yeah? Yeah. So what would be your criteria, your sort of setup for, were you like, I'm going just picture, no text? I mean, I use emoji a lot, like, to copy Ansi and Justin Smith a bit. So just, like, one emoji that you can transition into another. I think I've used my dog as a single picture on a slide. So, yeah, it just depends on whether it can convey all of the subtlety that I need it to. Yeah, yeah. Maybe I should try that. I think I've only used a big picture like that once. Yeah, I mean, Molly, even with that style, too, not, like, my slide with four words on it is any better. But we've really transitioned to this, like, there's no crutches anymore. You can't just sit there with your nine bullets and read things and prepare, like, an hour before. Like, now, like, I practiced a lot for my keynote last week because I had no notes. Like, I had no words on the slide. I couldn't just stare at a screen deer in a headlight. So do you find the same thing when you did, like, your SpringOne talk, which was a lot of emojis and pictures? Was that just a ton of prep, or did you just feel like you had a good flow and you had two or three notes in the, you know, in the slide notes? I actually did exactly what you said that we can't do, which is I wrote out a whole bunch of prose, like, an hour before, but just used it in the notes instead of on the slide. It would be magic. There you go. I used to find that really helpful that I would write, like, a little essay before a presentation, and then it would kind of, like, stick in my mind.

I guess I do that every now and then. I mean, I think, you know, I mean, one of the difficulties I have is another technique I've seen people use, and our buddy Andrew Clay Schaefer does this a lot, is you have, like, the single word or sentence on a slide, and you sort of, like, flip through it. But I figure, I think that would either just be reading the sentence, or, like, you would need an awesome amount of, ironically, for the format preparation to know that, like, what is the next sentence going to be. Because I find my problem is, like, I'll be on a slide, and then, as they say, I'll speak to the next three slides, and then I switch to them, and I'm always like, oh, I already covered that. Yeah, I know. It's tricky. I mean, last week, I knew the three points I wanted to say on each slide and just tried to make sure I memorized it and then forgot two of the three on at least half the slide. So it was perfect. That's right. Well, you had, you know, you had my favorite point in all presentations, which is Richard needs a raise. I think that was very, very thoroughly delivered. That was good. That is the only thing. The only thing everyone came up and asked. Like, nobody remembered the other nine and a half minutes of the slide except the one stupid joke slide about Richard needs a raise I had on the screen for 10 seconds. Like, that's the only thing anyone brought up. See, another successful marketing feat. You identified the most important stakeholder and prioritized their backlog, and everything else, just, you know, the other personas can wait in line. But that's good. The message got through. It's a subtle campaign of persuasion. Step one completed. So we have a few little news items, and we'll come back, or, you know, we'll talk with Molly about those presentations and more. But one of them, speaking of keynote-y things, somehow you dug up a fun – I like the format of this article because it uses, like, just sort of like this in-passing comment from Microsoft's Scott Guthrie, the red polo guy. And he's basically like, yeah, you know, in Azure, people are using blockchain, but it's – he doesn't say nothing burger, but it's basically a nothing burger.

And then, boom, there's a whole rest of the article that someone builds stuff around, as I'm sure, like yourself, Richard. I appreciate the art of, holy crap, I was supposed to email my editor something in three hours, and I have nothing. So that's a good dressing up the nothing burger in a full-on burger, I think, including, like, some goat cheese and caramelized onions, maybe. But I thought it was a fun article in the sense of, like, it is worth pointing out that, like, yeah, you got whatever a Bitcoin is, and all of this theoretic awesomeness of what you could do with the, I don't know, not only distributed nature of, but sort of like this public record of promises that people made or things that they said that are verified with each other. But thus far, there doesn't seem to be a lot of actual uses of it, aside from, you know, making 50 cent and Kim.com rich and things like that. Yeah, I mean, I included it in the news this week because I think it was – I mean, you pointed it out well. I just – we get really excited about stuff, and, like, what moves the needle for public cloud? I would still bet you that if we looked at Amazon, we looked at Microsoft and others, they're making the vast majority of their money still on just compute, like IaaS. And, yes, I mean, they have great growth in the databases and great growth in these other things. We get excited about Lambda or functions and blockchain, and, I mean, I would just be shocked if it was any more than a half percent of their actual revenue, if that. So just I think those reminders, we get excited, but the bread and butter for every cloud provider is probably scarily their commodity stuff, which is infrastructure. Yeah, yeah, it's kind of – yeah. Well, you know, we don't often get a – we're not lucky enough to have a security perspective all the time. What do the people in the security world make of all that blockchain stuff, Molly? Like, it seems like – I don't know. It seems like you all would be like, yeah, we got fingerprints and PGP stuff. I don't know what your problem is. Although maybe – I don't know. Maybe it's, like, some revolutionary awesomeness that you could do something with. I think a lot of people are pretty skeptical.

I mean, if you think about all the stuff that's been going on with, like, cryptocurrencies in the last couple weeks, like, the bubbles and the bursts and the – everybody just trying to make money from literally nothing at all. So, I haven't figured out yet how blockchain is actually going to help the vast majority of people other than, you know, maybe in some of the financial tech spaces. Yeah. Yeah. You know, all that weird tulip mania kind of puts a – did you say a pal? It puts a bad shadow over all of it, which is weird. That stuff's annoying. Anyhow. So, then, also, in exciting news, everyone's favorite thing that they've forgotten that they depend on every day, Dropbox, looks like it's going to IPO. And in that, in those things, like, I think on one of my other podcasts, we'll probably talk about it to, like, horse glue-inducing nausea of beating a horse. But, like, you know, one, they had, like, a $1.1 billion in revenue, which that's astounding, and lots of users. But then they also – like, who knew that so many normal people use Dropbox? I thought it was just a thing, like, nerds use to sync up their Markdown files, but it's apparently a big deal, which I guess you could know by the amount of apps that integrate with it on your phone land. But, anyways, they also said that they had moved all their infrastructure to their own infrastructure, out of public clouds, I assume. And then there was similar reports that Apple – the headlines were very deceptive here. I'm, like, cynical guy on this podcast today. But the headlines were, like, Apple confirms it uses Google Cloud. And then you read the article, and it's like – and they also use AWS. So, you know, I guess good redundancy is nice. And then I think also Spotify runs on Google Cloud, which makes sense because they compete with Amazon. So, just like the healthcare industry, retail, music, you wouldn't want to run on AWS.

But, so, Richard, does it matter that Dropbox runs on its own infrastructure? Does this mean everyone needs to start doing that? Gosh, it seems like there were two violent hot takes with that news coming out. First was, like, see, public cloud isn't for everybody. Like, Dropbox is fine. They did all their infrastructure. Maybe you should, too. And then there's the other violent hot take, which was, they're a complete unicorn. You're an idiot if you ever do anything on-premises. So, it's like there's probably, as always, in the middle of – yes, Dropbox, and I think it was mostly storage. Because, obviously, Dropbox is a storage company. They got some benefits from their own storage infrastructure. And, again, there's reasons they want to own what they're putting on here for depreciation purposes or even sovereignty. Who knows? So, yes, maybe this is just good for Dropbox and not for everybody else. But there is still value sometimes in controlling your own destiny, and that's not always a bad move. And, yes, you might not have the flexibility you get from public cloud, but maybe you like the more predictability you get from owning it yourself. So, I think there's multiple lessons in there. Probably none of them are the whiplash-oriented ones. Yeah, it needs some lukewarm takes. Leave a glass of water on the kitchen counter all day, and that's the temperature your take should be. I mean, assuming that you've got good climate control in your house and it's not either really hot or cold. I was also looking through some of their financials, and they were talking about how they had slashed expenses. You know, I think, whatever, everyone has a different thing for it, but they slashed administrative expenses by a huge amount. And I was thinking, like, man, that must have been a rough year for per diems and travel if they had to cinch up their expenses. Tragic. A lot of snacks going out the door. That's right. So, you know, my heart goes out to the previously pampered staff of Dropbox. They're not going to have those thick, fluffy, white carpets at their conference booth anymore. Or, you remember, have you ever seen their booths? They've got, like, wood paneling, and they've got, like, really clean t-shirts on.

Oh, I don't know if it's mahogany. I mean, you know, they're no Google or Facebook, but they have really nice booths. And they would have, like, those fake little square rectangular boxes of fake grass. It's very calming and nice. They probably still have a big booth coming up at South by Southwest. I'll have to check that out. So, over in y'all's land, we also, like, refreshed our website. Big industry-shattering news, as I recall. And it's very nice. Like, as I was telling someone, I didn't, like, load it ten times to see if it randomly does this or not. But there's, like, five of our customer stories, I think, like, right off the bat, which is good, I think. Yeah. Yeah, we spent some time, I mean, as always, trying to simplify messaging. Because it gets way too easy with every tech company to end up getting stuck in transformation hell and all these sort of, you know, digital business garbage stuff. Where you're just, like, what do you actually do? Like, what do you sell? And so, all of this is always an effort to get down to, like, hey, Pivotal sells software. We sell a methodology and, like, practices. And, hey, the software is a platform you run apps on and a bunch of cool framework-y stuff. Yeah. That's true. Because that's what people are looking for, not a manifesto about how you're transforming everything. And for everybody. Yeah. They want to know what it looks like. It reminds me. You ever read that, speaking of the hell of transformation, you ever read that Neil Gaiman short story where this guy, like, all of a sudden pops into this room. And there's sort of, like, this smooth-skinned demon in there who, like, tortures him for an eternity. And then he turns into the smooth-skinned demon and tortures another person who comes in. It's a regular, like, oh, I see what's going on here. I think that's what a lot of digital transformation marketing is like. It's just an endless loop with no point unless you enjoy the process. That's right. Yeah, that's good. That's good to know. That's right. Next time you need to inspire your staff to come up with some good content, Richard, you can use. I'll read that story to them.

I need a strong intro for my next white paper. I think I found it. That's right. And then finally, also in Pivotal News, we have, like, a new sort of, like, PCF or Cloud Foundry on your desktop thing coming out. Now, I haven't gotten a chance to read up on this, but what's the story with it? Yeah, we've got, you know, if you want to do Cloud Foundry development, your choices are either pushed to public, which is how most public clouds work. They try to do some local simulators and things. But a lot of times you end up developing in the public cloud itself. And Pivotal customers have had, or all customers, have been able to use PCFDev, which is, like, the single virtual machine with all of PCF stuffed in there. And it's fine. It runs on your machine. It's cool. It has some other dependencies, though, that were a bit of a pain. So there's been an effort to continue to break things apart. And so CFDev is super early alpha, but people seem excited about it. And this just runs kind of native virtualization on your Mac or on your PC and others. And just brings all the Cloud Foundry components down. So as a developer, I can just see a push, and it's all just running in containers on my desktop. And then, hey, things are awesome. Now let's go push it to public. So there's going to be a few different tools here. The CFDev is just open source Cloud Foundry. We'll do some PCF kind of extensions over time. So just kind of early stuff. We're messing around with this. But I've already played with it. It works well. It seems like a bunch of people were excited. And the whole point is how are we making development a little easier when you are disconnected or you don't want to deal with the latency or dealing with public cloud? Or even if you don't have an environment yet publicly, your company hasn't set up your Cloud Foundry environment yet, are you frozen? Or can you actually do some simulation locally? Well, that sounds useful. So speaking of usefulness, it's another masterful transition. Well done. So security is useful. And you've given a couple of talks on that recently, right, Molly? What are those talks that you went over in the sense of how are they representative of what you work on around here?

What do you spend your time doing? Yeah, so the first talk that I did last year was at Cloud Foundry Summit in Basel, Switzerland, back in October, I think. And that was mostly about how we do vulnerability management within Cloud Foundry, which is pretty interesting just because as a platform, it allows you to update your operating system images really fast and really easily. So we had, I think there was a lot of content coming out after Meltdown Inspector talking about how, like, Pivotal Web Services, for instance, updated all of their images in about a day. Where I'm sure we have, there's other people in the industry who are still struggling to update after Meltdown Inspector. So that's one of the things that I work on day to day is just vulnerability management for the community. And then the talk at SpringOne was more about trying to bring healthy security culture into your organization as a whole and the way that we've done that at Cloud Foundry and a couple of different ways that people can try to improve the security culture of their own organizations. It's interesting going through your stuff and realizing that basically, like, there's sort of, like, all the flashy, high-profile stuff of security of, like, you know, I found this chink in the armor of something and, ooh, a bug and I got a fancy name. Like, I forget the name of the one that had been in, like, the kernel for, like, 11 months, like some dog or elephant or something. Or I got a bleeding heart and all this. And then there's almost, like, the bigger effect that happens is, like, well, now what? Now we need to, like, have a process of, like, distributing that and notifying it and integrating it into here. It's almost, like, the difference between, I mean, it's sort of, like, just scouting something out and finding it versus, like, well, now we need to build an entire city over here.

And a lot of what I thought was interesting about what you're going over from that perspective was how do we basically build up that process from not much process? And how do we track these things and make sure that they're integrated? And then the part that I thought was really interesting was it turns out it's really hard to get people to apply these patches in the sense that, and I'm kind of reverse engineering that, that, like, in the sense of how much effort y'all put in to notify people they need to update these things, it would seem to indicate that they don't just sort of, like, gleefully do it. And, in fact, like, it might be easier to get people to, like, update the apps on their phone than to, like, patch the stuff that they need to in production, it would seem like. I don't know if that's a bad generalization. No, I think that's definitely true. And it's just a different philosophy than people have been used to over the past, you know, 30 to 40 years in tech. And Justin Smith talks about this a lot, but kind of the old way of thinking before people were beginning to be cloud native was, you know, we have our infrastructure, and it needs to stay the same, because we know, you know, what we're getting ourselves into. And anytime it changes, that's when we're unsafe, where really, once we're getting into the situation where you're in a public cloud, or you're pushing apps all the time, it's better for you if your infrastructure is constantly changing. So, as, you know, you're tearing down your servers and building them back up with Bosch, and it's really easy to do, like, that's conceivably kicking out anybody who made their way into your network that maybe you won't know about until years later, even. So, we're definitely running into the situation where customers have long certification cycles, where, you know, we're trying to push updates to them, and they're like, well, we need to understand the security posture of this and everything.

And that takes, you know, can take months, where it's like, we have this, we have the update ready for you, like, you can just take it, you'll be more secure if you just take it. And all of these certification processes are actually making you less secure in the meantime. So, you know, along those lines, I mean, I guess technically they're something of a competitor now, but whatever, like, how did, like, all the security community respond to, like, the, before they pivoted to Kubernetes, like, the core OS sort of theory of things, which, correct me if I'm wrong, but if I remember when I talked to Alex a long time ago, their CEO there, he was like, isn't it awesome how Chrome just auto-updates? And, like, why don't our servers do that? And then they had a bunch of auto-updating for the servers. Like, is that cool? Or is it, like, the worst idea ever that you would have your OSs updating in production without you really knowing about it? No, I think that's great. Like, that's exactly where we're trying to get to. Yeah, yeah. So, I know, yeah, there's been a lot of work with, like, PCF pipelines in the pivotal space, but also, like, there's been plenty of work. There's been plenty of talk at Cloud Foundry Summit on doing this in the open source as well. Like, whenever a new stem cell is available, just pick it up and roll it out as widely as you can. Because we're doing that kind of smoke testing for you. And the changes that actually come through, like, with a new stem cell upgrade, for instance, are usually pretty small. It's just packages getting updated. So, the more we can get people feeling comfortable in saying, like, I know I'm going to be more secure if I'm constantly updating. That's where we want our customers to be. Interesting. So, one thing you mentioned earlier, Molly, was kind of vulnerabilities in that area that you deal with. Can you – I don't feel like I'm super smart on how we handle all this. Could you kind of walk through maybe an incident or a day in the life of, like, where do vulnerabilities get reported from?

Do we discover them? Do we hear them outside? What's a CVE? Do we submit those somewhere? What does that look like? How does that make its way into software? Could you kind of explain what it means to discover, report, and then patch a vulnerability, at least in Pivotal's world? Yeah, definitely. So, if we think of vulnerabilities that are discovered, like, within Cloud Foundry rather than within our third-party dependencies, so, like, something that we're actually responsible for fixing within our own code, generally, those reports come from either Pivotal customers, other customers that use different distributions of Cloud Foundry, or kind of trusted partners that we have in the community. And those generally get reported to us either through – we have two email addresses. There's security at pivotal.io and security at Cloud Foundry.org that basically end up in the same place. And we have a security triage team that's a few pairs of engineers in San Francisco paired with a support person from Pivotal. And they work to first make sure that they can reproduce the issue and that it actually makes sense to start working with the product teams to potentially fix it. And then they do some coordination with the product teams to make sure that any changes that they make are actually – actually fix the whole issue and don't introduce any other regressions. Because the security triage team is kind of tasked with understanding the actual security implications of the bug. And the product teams are fixing it, but also making sure it doesn't impact usability or basically create a breaking change for customers. And then kind of the next part, once we know that there is a vulnerability, we'll reserve a CVE number, which stands for Common Vulnerabilities and Exposures.

There's like a national database that handles all of the CVE disclosures. And we'll actually notify the Cloud Foundry Foundation if there's a problem in open source code. And then some notification goes out to the different companies that have certified distributions of Cloud Foundry. And then from there, it's like basically just working as quickly as we can to get the bug actually fixed and pushed out to the open source and to CF deployment. And then to make its way back into, you know, pivotal closed source code as well. Yeah. So does everyone use that CVE registry? It sounds like no. That obviously gets up to discretion. So, I mean, how does someone choose to participate in that? What if they're not using that? Are they just distributing patches other ways or communicating other ways? Kind of how did we decide to go in with that? Yeah, I think it's an industry best practice. But there definitely are companies that either don't publish their CVEs publicly or they publish them kind of ad hoc to, like, various mailing lists. We've tried to be really, really good about both publishing our CVEs publicly as soon as possible so that customers know that there's something that they need to pick up. But also getting that information back to the national database so that other kind of alerting software that's based on that national database will let our customers know that there's a CVE. So it just kind of closes the feedback loop to make sure that everybody knows that there's something that needs to be fixed. And that's one of the ways that we've kind of built this culture at Cloud Foundry is that when you find a CVE, we obviously want it to be reported because it's usually in open source code and other people could conceivably find it. So we want teams, if they find things, to also report them to us, like, within their own product, which they've gotten really good at doing because teams know that they're not going to get in trouble if they find a security issue.

Like, it's going to be another reason to encourage customers to upgrade. So there's that kind of, like, positive feedback loop when issues actually get reported to us. It makes sense. Remind me, how long have you been at Pivotal? Almost two years. Two years as of yesterday, actually. Nice. Almost like me. So a question for you is, on one hand, you know, from the outside looking in, someone could say, like, gosh, what are you advertising the fact that you have vulnerabilities? Isn't that kind of lousy PR that you're bragging about? Hey, look at all these vulnerabilities we have and that we've patched them. So, I mean, is there a cultural thing that you ever noticed that we had to get through? Or was it kind of built in from default? Like, no, this is transparency. These are things that need to be fixed. And regardless of where blame lies, whether that's in an open source thing that we have nothing to do with or whether it's our own code, like, this is the responsible thing to do. And we should celebrate that. Do you think there's ever any angst about advertising this sort of thing? And do you even see enterprises sometimes being skeptical about racist awareness because people will be blamed for something? Or do we gotten past that and say, no, this is just responsible computing? I think one thing that really helped when we first started kind of the really big push for more security in Cloud Foundry, which is about almost two years ago, I guess, was Justin Smith, who's kind of our head of security, said to me when we were getting started, like, I hope we find as many things as possible. Like, we can't set up any sort of cultural compensation around not finding things. Like, we should be happy anytime someone reports anything to us. And it doesn't matter kind of how big or how small. And I think that really helps set the tone for what our security program is going to look like. And that's a really important thing for organizations to think about. Like, how are you culturally encouraging people to report issues?

There's definitely been a lot of philosophical discussion about how open should we be and how quickly. Because we definitely do have customers. Cloud Foundry has used a lot of big banks. And they've said, you know, I wish that we got to hear about this vulnerability before the public did, for instance. But we've been able to have conversations with them and explain that, like, secrecy only really helps the bad guys. It doesn't help the good people actually find out what's wrong and patch quickly. So I think we have built that in from the start. And that's been really important, like, as we try to make Cloud Foundry more secure. So just as you're doing, that's another thing you've spoken about quite a bit is the, I don't know. Like, you could call it, like, the culture or, like, you should have developers stop writing insecure code. Like, all of those kinds of ways of thinking about it. But you went through some actual, like, tactics you had to not be, like, the scary security person coming in and telling people how to live their life. And so, like, in your experience around here and elsewhere, like, how do you introduce that sort of, like, security hygiene? And in the sense that, like, you would like it to actually stick and not just be annoying for people. Yeah, I mean, one of the things that we've been doing lately, and I'm actually in Portland this week doing this, is security workshops with kind of as many product teams as we can get our hands on. And those workshops have been developed with the cognitive science pair that actually works at Pivotal. It's pretty cool that we have some scientists in residence that are helping us with this. And the idea is when people are scared about security or they don't feel knowledgeable about security, you obviously want to teach them.

But we don't think it's super effective to either, like, sit them in front of a computer and make them watch a video for an hour or sit them in a meeting room as a team and just talk at them, which is something that we've tried previously, and we didn't find it very effective. So what we've been doing has been going around to product teams and doing exercises with individual pairs to help them just become a little bit more skeptical about their product. It's what we're trying to do. And then we actually teach members of the team to facilitate the same workshop with other people on their team, so that when we leave a location, that knowledge can kind of grow and scale more virally than having to fly people around and the only knowledge sharing happens when we're there. Yeah, I love that. So, I mean, I'm trying to map that to, you know, specific. I think I've watched enough videos in my life where it was like, hey, quick, take this SOP or training on security. Great, now you're secure. Get back to work as a developer. But, I mean, so let's, you know, you talk to enterprise customers or you see the work they're doing. How is security supposed to fit into even, we'll call it agile delivery of projects, but really projects in general. It's not scalable for security pros to be embedded in every team full time. That doesn't seem like there's enough of them across the enterprise. So, is it this sort of kind of pairing education sort of mode and then you cycle through? Is it hopefully not the code review at the end where everyone's breathing down your neck because you just have to say yes before it goes to prod? Like, what is the right place for security in a project delivery cycle, especially nowadays when it can't be an afterthought? Yeah, I think we think about it kind of the same way as design. Probably, you know, 10 years ago design was like pretty colors that got thrown in at the end after all the code was already built, right?

But now we're trying to build usability into every story. So, as you write a story, you're trying to figure out how is this feature actually going to be used? What does it look like on screen to someone? How do they interact with it? And with the pairing exercises, we're trying to get the same parity as usability or high availability get, like, within Pivotal and Cloud Foundry teams. So that when you're sitting down and writing a story or working on a feature, you're thinking, what are the actual security implications here? Let's think about not just valuable information that someone might want to steal, but usable information that someone could get out of this component if they've already exploited a system and are trying to, like, move laterally through your network. Yeah, do you do something then after that? So, before a team ships, I mean, heck, we have another PCF release coming up here in a couple of weeks. I mean, is security part of pipelines? Is security a check that also happens at the end somewhere? Is that unnecessary if you've been doing it throughout the way? What happens after you feel like you've kind of coached a team up and they're in good shape? When else do you intersect with them? I personally don't. A lot of the different teams have their own ways that they go about this, either, you know, compliance pipelines that are helping them figure out whether they're PCI compliant or, you know, HIPAA compliant, things like that. A lot of it is done through actual customer deployments rather than on the product side. But at the moment, it's definitely up to individual teams to understand what makes sense for their product. And we found that that could be... So, like code analysis? Yeah. Things like that. That would be up to them. Yeah, there are... I know various teams are doing, like, static code analysis, for example. We're also doing vulnerability scanning for the platform as a whole and helping to integrate into, like, their current workflows.

Like, everything that we're doing is just to try to make sure that the teams have ownership of it and to fit into their existing workflows so that security isn't something that's preventing you from shipping or slowing you down at the end. Like, the farther forward we can build it into the process, the less disruptive it is. Yeah. Can you give any examples of what you would tell a pair or even just in general, like, what should devs be looking out for? Like, I think it's really lame. I guess I did this on stage at Field Kickoff last week, too. But just saying, like, be more secure, like, that's not even useful anymore. So, like, what are some of those things specifically around whatever it is, coding techniques or practices around how you run or at least privilege? Like, what do you encourage teams to do to be responsible? Well, so there's kind of two parts of it. It's one part, as like I said about the workshops, it's just teaching teams to be more skeptical. So, one of the things about Pivotal and Cloud Foundry in general is that our development teams are very empathetic to our users and to people who want to do the right thing while using the system. And they just don't have as much experience thinking about people who don't have the best intentions or who actually want to break their product. So, just kind of switching that mindset, we found has been really helpful and ends up with teams, like, once they've been trained, actually reporting more vulnerabilities to us. Anecdotally, we don't have, like, data on that at the moment. But the other thing is just to help the teams understand that the security triage team especially is there to help them. If there's any, anytime they have a question about something, like, that's what, that's the sort of behavior that we want to encourage. Like, we want to help teams know that if they do have a question, there's someone to go to so that there's kind of that psychological safety.

Because you don't want a pair to be working on a story and say, oh, I think if we do this feature this way, you know, somebody's social security number is going to get leaked. But no one told us to care about that, so let's just ignore that for right now and we'll just do this story and we'll deliver it. Like, that healthy skepticism is what we're trying to build in because we're not going to have a security person looking over the shoulder of every single pair who's developing a product. So you've covered this to some extent here and there, but I wonder, like, I don't even know the terminology for it, but, like, a fair amount of kind of what we talked about. And I think what y'all's team does is sort of, like, securing Cloud Foundry itself. And, like, a question that I often get is, so if we have these product teams and I've heard about, like, you know, DevOps and balance teams and all of this stuff. Like, do I actually have, like, a dedicated security person on that team? And then if I don't, am I doing the same thing that I'm doing now where I've got, like, security audits? Or, like, what do you, like, what do you do in that case? Like, what are the controls? I think that's a security term, right? Like, what are the controls that people have if they're just going to let these people, like, deploy multiple times a week instead of having to audit things? I'm assuming, you mean, like, on the customer side? Yeah, yeah. The actual application that people are working on, not, like, the platform that people are using. Yeah, I think it varies customer to customer and industry to industry. So, those, like, different customers have different regulatory things that are set up in place already. I'm just, like, less on the application development side. So, I don't know as much about how they're securing their applications. But there are obviously plenty of add-ons for Cloud Foundry that help with that.

And plenty of work on certifications. So, for instance, John Field has gotten numerous Cloud Foundry customers through, like, PCI compliance for their foundations. And I'm assuming that ends up touching the development side as well. Right. But, yeah, it's all about, you know, vulnerability scanning the languages that you use. Hopefully, you're not using languages that let you get buffer overflows and things like that really quickly. But, yeah, I mean, third-party libraries and dependencies are just a huge part of keeping application development secure in kind of a cloud-native way. Yeah, I guess there's a layer of trusting yourselves and then also trusting, as you say, the third-party stuff you bring it in. And it sounds like, I mean, I get a lot of these questions, you know, to invoke our smooth-skinned demon friend of transformation, right? Like, sort of like the, well, what about this thing and what about that? And oftentimes, you know, people will – they'll give a lengthy answer of, like, why this stuff doesn't work for them, like all these dependencies they have. And maybe, as you're kind of alluding to, they have, like, very stringent security needs. And I feel like frequently the answer is, like, well, you should do that stuff. Like, there's no secret sauce of removing that kind of toil from what you have to do. So just put it into the mixture of other things that you've improved instead, which I guess is a little depressing of an answer because you always want to, like, remove toil. But then again, maybe it's a relief to know that you're not missing something. You didn't, like, skip those five pages in the Lean Enterprise book that solved your problem of how to integrate with, like, six different off-the-shelf systems and make sure that you pass your five-capital-letter security standards from, you know, the RAND Corporation or whatever you're freaking out about. Is RAND still around or have I been watching too many old movies? No, I think you're fine. All right. Thank you.

I was wondering, Molly, because, you know, we talk a little bit about devs on the enterprise side. But if we go back to the Pivotal teams, I think you do help with a patch day we do internally. Am I remembering that right? Yeah. So what's that? I mean, how do we keep our own teams? And when there's a security crisis and everyone goes, hey, my backlog is set. And you say, haha, just kidding. Here's something we all have to take care of right now. What's a patch day like or a patch week? How does that work at Pivotal? We've definitely iterated on this process a lot. So patch days slash weeks usually focus on the Ubuntu stem cells that Bosch provides that are the base image of all your VMs if you're running Ubuntu VMs in Cloud Foundry. So the company that provides Ubuntu, which is Canonical, they periodically release security updates for Ubuntu. And those are classified by severity. So you'll have, like, lows, mediums, highs, and criticals. And if something is high or critical that gets released by them, we try to push stem cells and have all of the pivotal Cloud Foundry teams patch with that as quickly as possible. But if, for instance, we haven't had a high severity issue, there are a lot of lows and mediums that kind of sit around waiting for us to pick them up. So we'll do a scheduled patch week where Bosch will provide the stem cells and all of the pivotal teams will cut their tiles to use the new stem cell. That's changed, you know, because recently Bosch decided that they were going to start just cutting stem cells every two weeks. Because we have a lot of different customers that have different SLAs around CBEs that come out.

So some will say, like, if a low CBE has been in the wild for 20 days and we don't fix it, we're out of compliance. Or other customers say, you know, two weeks. Or, you know, every other Friday. Or you have to patch on a full moon. And, like, all of the different requirements are often in conflict with each other because some customers want to patch CBEs very quickly. And some want to patch them very slowly because there's obviously risk every time you make an update. So the patch week is just a way to coordinate all of the pivotal teams that create tiles for Cloud Foundry to have them all patch the stem cells at the same time. Yeah, that's great. Hopefully, I mean, I would think some of our customers could benefit from that stuff, too, just because, I don't know, sometimes it just can't be like, hey, we're bored right now. Let's do security stuff. There's probably got to be a constant rhythm and somebody keeping an eye on it and making it somewhat interesting and beneficial, not just, crap, it's toil time. It's time to go patch stuff. So the last thing I had for you was, you know, as we look at ourselves, our customers, whatever, where do you see the kind of separation between what a developer is responsible for security-wise, an operator, what they should depend on platforms for? I mean, what's the ideal here? Because maybe the answer is everyone's responsible for everything, or maybe there's a take of, hey, developers have a certain responsibility at their app, operators kind of keeping a platform healthy, a platform should try to do, you know, secure by default computing, whatever it is. In your mind, is there a separation there? Are there clear things that all of those three parties are responsible for? I guess from two years of working on Cloud Foundry, the first thing is definitely anyone building their own platform these days who don't have, you know, hundreds of engineers working on it. I would say to definitely rethink that when it comes to security, just from all the interesting bugs that we've found in our own platform, I could imagine other people doing the exact same thing.

And then you're not benefiting from, you know, a community building a secure platform for you. And then as you step it up to, like, actual operation of the platform, secure by default is definitely our philosophy. And I hope that we're trying to impart that to our customers as well, because there are definitely ways to configure any platform that are insecure that you have to worry about. So, from, like, the field teams to marketing to just everybody working on Cloud Foundry, I think everyone is trying to encourage customers to set up their own networks and platforms in as secure a way as possible. And I definitely think the platform can compensate for a lot of things that application developers might try to do. So, for instance, we have all sorts of networking layers built in that can help application front ends and back ends talk to each other without letting anyone else in, obviously. So, application developers have to be careful, but we have to assume that people in any system are going to make mistakes that could compromise your security. So, you want to take the actual individual judgment of a person out wherever possible. I think we've definitely seen lately that security incidents tend to happen when things are one person's responsibility, either to update something or to make a split-second decision. So, we want to take people and all their fragilities out of the security posture of the system as much as possible. Yeah, great answer. Well, I just have one more question. It's more of an advice thing. Now, if I figured out one thing about you, it's that you really like your dog, which is fantastic.

I'm all about liking dogs. But now I've got a dog. Very, very different build, but I think similar size. And this dog, all she's interested in doing is basically, like, licking my face constantly. Now, is this something that I can work on with her or I just need to accept it and, like, have that be a thing that happens and just celebrate having my face licked by a dog a lot? What's your opinion? Yeah, that's tough. My dog, who's Rory, who shows up in my presentations as often as possible, she'll do that, but not forever. Like, it'll just be a nice, like... Just momentary. Like, you're welcome home. I'm going to kiss your face a little bit. Yeah, that's definitely a tough one. That might be just something you have to live with. Yeah, I mean, basically, I think the arrangement that we have is she has... You know that zone that, like, if a fire's in your house, you're supposed to, like, crawl underneath the smoke so you don't asphyxiate? I think that zone level is her zone. And so if you go into that zone, she can do whatever she wants to you, which means you can basically never sit on the floor if you don't want to be covered in slobber. And, I don't know, maybe that's the truce that will work out, and then I can just, you know, control when it happens. But, man, she's very excitable with the licking, which I should ask some vet people, not that I know any, but, like, what the deal with that is. Because, I mean, you know, dogs don't, like, taste things as a primary mechanism. They're supposed to, like, smell stuff, right? I don't know. Well, that's good advice. But they taste stuff, right? They're not snakes. Well, I mean... Their tongue isn't for smelling stuff. I mean, like, in the sense of, like, it's a primary functional sense. Like, you wouldn't say that us humans go around and, like, taste things to figure out the world, right? You can speak for yourself, but okay. That's right. Exactly. Yeah. Whereas, you know, we're more visual and tactile and, I don't know. I'm no human biologist person.

Well, anyhow, thanks for being on. In addition to figuring out how to cope with my dog, we found out a few interesting things about security. But it's really that dog advice that I was here for. But if people want to follow up or see the presentations we were talking about or anything like that, where would you send them? So the SpringOne, 2017 website is great. It has all the talks. If they want to see it again, I'm actually going to be speaking in Austin on April 30th at the Deliver Agile conference that Richard so nicely invited me to. And then I'll be speaking at Cloud Foundry Summit Boston in April as well. Oh, exciting. All over the place. Well, great. Well, as always, this has been Pivotal Conversations. Thanks for listening. If you want to go find all the old episodes and find the RSS to subscribe to if you don't know how to do that on your own, you can go to soundcloud.com slash pivotalconversations, and it'll all be there. And about every Thursday or so, we post the full show notes, such as they are, the formal show notes, over at pivotal.io slash podcast. And then finally, just as one recommendation, you know, frequent podcast guest Jared, he had this post about three white papers that we had now. Don't stop listening yet, just because I said white paper. But three white papers that we had published. One of them is indeed on security and compliance. And there's probably some PCI talk in there, which would be interesting. And there's another one I read when I was flying back from the sales kickoff about, like, thinking about how value stream maps fit into all of this and how, like, there's an interesting emphasis on, like, waste removal from a lean perspective in the software development cycle that's neat. And then, you know, for those who are, like, really in the weeds of IT service management, there's a whole paper on chargebacks, which I think, you know, that's thrilling reading for people like myself. But it's definitely selective taste. But, you know, with that, we'll see everyone next time.

Bye-bye.

Tanzu Talk · 🍕 Get more pizza.