Operations Has Plenty To Do In A Cloud Native Enterprise

Episode 1 · 2015-06-10 · 57:20 · SRT · archive.org · all Tanzu Talk

So I was watching one of your old talks, and you made a reference to a, what was it, a Solaris storage admin job in the suburbs. And I was wondering, did you just make that up, or did you actually know a nice storage Solaris admin in the suburbs who was just happily going along reallocating space? Oh, hi, Coté. No, I actually got a recruiter contacting me in the last couple of years for a Solaris storage admin job out in some far-flung exurb that I'm pretty sure I would have to buy a car and then drive the car. And these are just things that are not going to happen. Yeah, yeah. You can't use your, what is it, your transit, oh my God, business to get out there. You'd have to buy a car. It's terrible. You know, this is one of the major criteria people in the tech world seem to have, is the transport to and from my job. Is it long? Does it require a car? It's very systems-type thinking that I think they get into. Honestly, I prefer a job that requires me to go to the airport from time to time rather than one that requires me to go to a specific place where I use my computer. Right. Yeah, yeah. You know, I didn't realize it until I was looking, but you have a lot of talks. You've been out and about. How did that start up? Well, let's see. I was the one-woman ops team at an ad tech startup called Eighth Bridge in Minneapolis, where I did go ride my bike the seven blocks to the office, to go into the office every day. And my boss at the time, Jamie Thingolstad, who was involved with this local unconference thing, said, you should go talk about our MongoDB there. And I thought, I could talk about MongoDB, but it's a very dev-focused conference. So I'm going to bring one of my dev co-workers to, you know, Mike Hobbs, to talk about, like, kind of the dev side. Sort of like your conference shield. Right. You know, I was like, there's not even an ops track at this conference. I felt a little weird. But it was actually a great experience. I got to talk about how MongoDB is sadness as a service, which I think everybody pretty much agrees with now.

And it was, I realized that even if you're, you don't have all the answers or you're not an expert TM on something, it's still valuable to share your experiences with other people. Right. Now, as a little parenthetical, so what is the deal with MongoDB? It's sort of like, I haven't used all these newfangled tools that the kids are up to nowadays, but it seems like everyone loathes MongoDB and yet it's really popular. Or is it sort of like just making fun of, like, people wearing, like, the wrong type of pants or something? Like, it's sort of like an elitist thing to do? Or is it actually like a loathsome thing? What's up with it? So MongoDB is actually great if you want to prototype something as a developer and you just want to write some JSON and just store some documents and the interface is super simple and you don't have to write any SQL. It is great for that. Unfortunately, it made a lot of marketing claims as to being, you know, quote unquote, web scale, but to hilarious results. But also tended to have, you know, data losing bugs with every release. Oh, right. That's sort of not what you want with a database. One might argue it's the core feature. Right. So it kind of got a reputation for being kind of a pain for people who are, you know, trying to operate it. I opened a lot of bugs with them when they were first called Tension about things like the way their connection pool lazily checked connections such that if you've changed your master route, you would expect that all of the open connections would not be valid anymore. But no, you've got to handle that on the application side or every single one of the ones open in the connection pool will be tried and fail. And it's just like, what? So there were a lot of things like that that made it difficult from an operational perspective. But if you're just trying it out and don't actually expect to operate it in production or at any kind of scale, I'm sure it's perfectly fine.

It's probably. Yeah, yeah, yeah. You know, MySQL aside, I always wonder if, like, the 2000s would have been different if Oracle just lowered their prices. If Oracle just stopped trying to sue everyone? Yeah. I mean, if the closed source alternative to all this stuff was people more gleefully used it for whatever reasons, they may not like it. But it is databases or a strange space because it seems like there's so many different ones of them nowadays. And they're all kind of, like, highly specialized to a task. And it'll be interesting if things consolidate down to just a few general purpose ones. But it doesn't really seem like that's the way stuff's going. Well, I think that there's definitely a lot of room for data stores for unstructured data, right? So, like, at 8thBridge that same couple of jobs ago now, I also ran an HBase cluster. And that was specifically because we had a firehose of data coming out of Facebook back when the OpenGraph API would let you harvest not only the data from people who had authed your app, but also some of their friends' likes. Right. So, we had a great deal of data coming in from Facebook. And it was not necessarily structured in a way that we could predict. So, having schema-less data being dropped into a wide-column data store is pretty handy, especially if you plan on mostly just running MapReduce jobs on it. Right. So, like, I think that there's a lot of room for data, you know, a variety of different kinds of data stores. And I don't want to malign MongoDB. Document stores have their place. I just think a lot of times you end up realizing that there is structure to your data. So, that's sort of the sort of thing where I think I've made jokes before about how you don't make your important technology decisions by running a Markov bot against the front page of Hacker News. Like, you don't necessarily want to choose a data store just because it's the hot-turning new data store that all the kids these days are using. Like, got to look at your problem space first and see what you're trying to accomplish before choosing a data store.

Yeah, you know, and that brings up something I was interested to hear your take on because I encounter this quite a lot when we go out in our Pivotal world and tell people they need to do all this crazy, like, I don't know, cloud stuff and think about DevOps and things like that. And given your background as an operations person, I am not an operations person. I'm the – I used to be the bad developer who would choose easy things and then have other people suffer for it. That was my form of entertainment, I guess. But, you know, a lot of operations people, like, I go over and, you know, talk about, like, DevOps and cloud with them. And basically the question they ask is, like, so where's the part in your chart where I get fired? And there's – the other way of asking that is, is there – in a more genuine way or helpful way is, is there curious, like, what an operation person does beyond the obvious of, like, working with the developers? And, you know, the reason this popped up in my mind just now is it seems like – so if you have lots of different types of databases, well, someone's got to figure out how to run all those weird things, right? And so there's almost, like – it almost plays into the old Jevon's paradox or Jevon or however you say it of, well, now we have an unlimited amount of types of middleware and services we can run. So someone needs to work on that, which is not going to be the developers. Nice. How many things with VIA in the front of them do you want to run today? That's right. But, you know, I mean, the more general question is, like, as – I mean, you're well-traveled in the scene where people would ask this. But, like, how do you sort of, like – what do you tell ops people who are, like, freaking out? Right. So I would tell them that I totally am a system of the old school as they possibly are. Like, I poured one out on the curb for SunOS 413, okay? Like, I was sad about Solaris because, come on, really? We're going to go System 5? Like, even if we've been doing this – excuse me.

Even if we've been doing this for a while, there's still always new stuff to learn. So, yes, of course, if you would like to, you know, just use Smitty or whatever the same way as you've done for the last 20 years, it's possible you're going to have a bad time. But as long as you're interested in learning new things – Or you could work in the suburbs, apparently. Well, I mean, I think that there's a very long tail of adoption, right? So there's probably always going to be places that you can do the thing you did 15 years ago. But if you are interested in learning new things, there's an infinite amount of new things to learn. And there's – I like to say that if there's – if it's possible to automate away some portion of your job, that was the boring part. And you should definitely automate that part so that you can do all of the more interesting, you know, scaling problems or all of the more, you know, all of the more interesting solutions you can provide that can bring actual business value. Because the part where you just ran, like, some script again and again, probably not the most interesting part of your job anyway. Yeah, I always think of that for as terrible as it ended up, that, you know, that wonderful part of Lost where they just had to push a button all the time. Right. Seems like something that needed to be automated. I mean, I guess it did have great value. If I'm not pressing the button, something terrible will happen. But it turns out not. But, like, what are – you know, what are some examples of, like, fun, interesting things to do instead of pushing the button that you've come across? Sure, exactly. So, for example, if you would like to migrate some of your workload to the cloud, which, by the way, I'll say that the cloud is kind of BS because the cloud is just someone else's computer. Like, in some cases, perhaps that computer is in U.S. East 1 in Virginia, and you're renting time on it. So, it's not this mystical, magical thing. It's just a series of computers somewhere. And if you're going to migrate some of your workload to, say, AWS, for example, there's a lot of things you can learn about how to scale, how to perhaps auto-scale up and down, how to connect their VPCs with, you know, internal VPNs you may already have.

Right. How you can have better, like, data locality so that you're not constantly incurring the cost and also, you know, time sink of moving data around, depending on where you're processing it. That kind of comes in when you're talking about, like, you know, running massive jobs on some amount of data. Well, can you localize the data to some sort of place where you're actually running those jobs? Like, there's a lot of ways that you can add value that aren't necessarily repetitive action. So, I would discourage people from wanting to pursue repetitive action if they're interested in computers. Because I think computers are really good at performing repetitive actions the same many, many times in a row. Humans, less so. I think we should stick to our strengths, which is, like, creative problem solving. Yeah, no, and it's almost like the, I don't know, tell me if you think this is fair, but it seems like in the ops world, you can divide up into at least two categories of, you know, if we take out, like, I don't know, security and network admin and other kind of, not niches and small, but, like, specialized tasks. And there's basically sort of, like, here's off-the-shelf sort of software and stuff, and also desktops, and someone has to make sure that stuff operates correctly. And then there's another category of stuff of, like, here's uniquely combined together IT that, like, we have caused to come into existence, and we have to constantly make sure that it's operating properly. And so it's almost like a unique application or system that we've done in-house, and it might have our own, like, software we've written in or whatever. And that's where, like, probably the bulk of the interesting work is in maintaining your custom stuff versus, like, time to add more hard drives to the exchange server. And, you know, like, keeping your ERP system happy and things like that. So I've actually had an opportunity to talk to some individuals at some large enterprises where they are worried about things like their ERP system, but then also how people operate, you know, interoperate with it.

And I think that there's probably interesting problems in all of the realms. Like, you don't have to be, like, a website, you know, public-facing, open API, whatever, in order to have interesting problems to solve. I do think that perhaps running an exchange server yourself in-house these days is possibly better not done. Like, I would say that you're probably going to add more value to your business by just going and getting one of the numerous cloud solutions that's available for that and having people actually help, I don't know, make the bring-your-own-device solutions work better for your enterprise or something. Yeah, yeah. And exactly, there's almost, like, even in that area of not your customized software, there's deciding what all that stuff is going to look like and then making sure it's wired up properly. Like, making sure that if I'm working from home remotely, if my kids watch Netflix, it doesn't ruin the day's work. That would be an interesting problem to solve for me. What, is your Netflix profile, like, something that is also affecting your work, or can you not set aside the kid profile? Oh, yeah, yeah, yeah. No, it's just a matter of, like, sucking up the, you know, it's got this annoying, sucking up the bandwidth. It has this annoying thing of, like, my profile is now stuck in kids' profile, and it won't move over. And I don't know what's wrong. It's misclassified me. It's terrible. Well, it kind of sounds like for your internal routing in your house that you need some sort of QoS. Sure. I don't know what any of that is. I'm busy filling up a Mongo database with a bunch of unstructured data. I have no idea about that. But this does raise another small cultural point. Now, since you've been working here, I've been learning a lot about your buddy Joe there. I know who wears Red Wing boots. I've got to look those up. But do you guys use different, do you pay for different services on your, like, Netflix and Spotify and things like that? Or do you use one account? That's a funny question. I think that's probably one of those philosophical questions in any relationship.

And the answer is we don't actually share any accounts. However, the person with the strongest opinions for whatever it is is the only one who usually bothers to have an account. Like, for example, I don't have a Netflix account at all. I never have. I see. Because I have very few opinions about what we actually watch movie-wise, but I have a lot of veto power. So he has a very long Netflix queue, and he watches a lot of it when I'm out of town. But then isn't he stressed out that you're going to, like, ruin the robot? Like, things you pick, like, if he doesn't like it, it's going to feed into the robot, and next thing you know, he's going to be recommended the wrong things. I'm not sure that he puts that much stock in the recommendation engines inside Netflix versus, like, I think he reads, you know, the Onion AV Club and talks to friends and what have you. Yeah. See, this is a crisis that's looming, is you have all these individuals, and you're going to be customizing stuff to the individuals, but then you have families. And if they intermix their stuff together, like, I don't know, it's going to pull families apart. They're going to realize how different they are. We've got to obscure and hide this. Or they can just learn to accept some of what each other likes. I mean, it took probably 12, 13 years into our relationship for me to actually start watching the Green Bay Packers. Oh, right. I just wasn't a fan of American rules football at all because I grew up as a hockey fan. Which, by the way, as a Texan, even though you're not from Dallas, you have some explaining to do about how Dallas stole our hockey team when I was a kid. That was unacceptable. I don't know. I'm sure money was involved. But, yeah, I eventually started, you know, I don't know, having sort of a Stockholm Syndrome appreciation for American rules football, and now I'm actually a Packer fan. We both have Packer stock. He got one in the original offering, and I got one for my birthday a couple years ago. We got them framed together. You know, some people put, like, a ketubah frame, you know, on the wall. We have, like, a Packer stock framed together. Hmm. Unifying. So, I guess your counseling is right. It's just like, well, this is an opportunity for you to get closer with your family by, you know, you've got your finely tuned music and Spotify, and every now and then, some, like, crazy 90s punk stuff is going to pop up in your list.

And you're like, where did that come from? Are you telling me you have a problem with Fugazi? Well, I like that one where they're not talking. That's a good album. Oh, my goodness. Anyhow, that stuff's popular up in the Midwest, as I remember, so I should watch myself. But, you know, so it's changing topics drastically. Speaking of sysadminning, I was, so you do that Arrested Development podcast, which is good stuff. And I was listening to one of them recently. It was the one, I think it was episode 40 with Charity and that other guy. And she's pretty, pretty unescapable because she, like, giggles all the time. So it's a very, very known voice there. Charity's great. Yeah, yeah. And we had MCR and Patrick from Etsy on that episode as well. Yeah, there you go. And you guys started talking a lot about the usual sort of thing of, what's the part that I started keying into? The thing that I thought was really interesting was the point you guys were making about when everyone needs to understand that when you go to management, that's not a promotion. Right? Oh, right. Like it's a career change. We had Lindsay Holmwood on the podcast recently with Courtney Nash, actually, and talking about cognitive neuroscience. But he wrote a really good blog post that, if we have show notes for this, we should put it in the show notes, about how management's not a promotion. It's a different job. Exactly. I have actually done that different job. I did management from 2005 to 2012 before I decided, you know, stuff it. I want to be an individual contributor again. Yeah, yeah, yeah. No, and there was a good discussion of that kind of divide, which I think the – my lack of being able to talk about it quickly is the whole point. It's like an odd topic of like – for many people, management kind of blows, right? And yet, it's – counterintuitive is the wrong word, but it's almost like not the common sense of how you would go about thinking about your job, right?

Like your goal is to get into management to – because you're sort of like excelling and, you know, it's a – you get the call from your parent and they're like, are you taking care of yourself? How's your job doing? Have you been promoted to management yet, right? They're checking in on you. And it struck me in listening to that that a lot of like what we talk about in the DevOps space is kind of like trying to emphasize that the simplistic common sense thing is what you should do instead of like what's normal to do. So like normally, like as we were joking about earlier, the developers write some stuff and put it in MongoDB and hand it off to ops people to take care of. And that doesn't work out. And a lot of stuff that we talk about in the DevOps world over and over again is like, hey, look, that doesn't work. So we should stop doing it. And possibly we should talk to each other about our needs before we choose a data store. I mean, in that particular case, I blame no one because I walked into a MongoDB already in progress. I made the best of it. But – and that is the sort of conversation that say I were in ops and working someplace where people wanted to put MongoDB in production. I would say, hey, let's talk about this. Yeah, indeed. And so like I used to – and I still do. Like I always get a little upset when it's sort of like, oh, you should just do the smart thing. But, you know, given the various – you know, you have an interesting job history of having been at basically like a – I don't know, not basically, a big ISP and then a university. Very normal, large, institutionalized places. And then several startups, you know, where they're doing things. And I wonder like – I don't know. I mean, what's the deal with people getting stuck in a rut of not just following the common sense? Like how do you – especially since you've managed, like how do you move them over to like doing something that's obviously better? Which the broader question is sort of like how do you do change management? You know, like cultural change management of moving stuff in because it's – that seems to be the part that everyone always bulldozes up against is people just keep doing the same bad stuff over and over again.

And we can't convince them to do otherwise. I think in many cases – and I'm not going to say I was great at this when I was in management – but I think in many cases people are motivated by, you know, by local optimizations, as Andrew maybe would put it. So, they're thinking about what their day-to-day – or they're thinking about whether or not they get the thing that they want, if the thing that they want is, you know, the exciting, shiny new framework that they've tried at home and was fun or whatever. And they don't necessarily feel motivated to see either what's better for the organization or what everyone can agree on. So, like trying to get people to talk amongst themselves, like honestly, about what they need, what they want, what benefits they think this particular technology choice will bring. Like, I think those are the hard conversations. Like, a friend of mine tweeted just the other day that, you know, who was it? Something along the lines of, what was it this – what was this thing that they told us about how, you know, you don't need communication skills to write software? Software is mostly communication. Yeah, yeah, no, and that's a good example. Like, I remember seeing that, and I remember, do you ever read that book, Microsurfs? There's some episode – there's an episode in it where one of the programmers goes off into his room and has a flat food diet because the only food they can give him are things that can be slid under the door. Oh, gosh. And, you know, the flat food diet mentality doesn't really work out too well in that case. But it is like, you know – But to your point, I've worked in some of these really large organizations, and you end up really just optimizing for your tiny unit, however small you define that unit as being, right? So, like, when I worked at US West and then we got bought by Quest, they laid off a bunch of managers, including mine.

I reported to a voice on a speakerphone in Denver. I never met him. And that was not what I would consider to be the sort of experience that really motivates humans, right? Like, I'm actually going to London next week and – or going to London tonight, speaking next week at a conference where I'm going to be talking about distributed systems and teams. And I think a big part of working successfully on a distributed team is feeling that human connection with the people you're actually working with, even if you don't see them every day. Right. No, I think that's definitely the case. And it is – so you've been at Pivotal, like, how long now? A month. A month, right. Yeah, yeah, and both of us work remotely. So it's – I haven't worked remotely at such a large company in a long time. And it's interesting seeing the different remote working patterns play out and not play out in a company like this. But no, I think you're right. That is – that's a good way of really drastically boiling it down is, like, especially with remote working, you need to make sure that you're not just a disembodied head somewhere. Because it's easy for each employee to kind of just slip back into, well, I'll do nothing helpful today, essentially. Right, and then in terms of also, like, whether you're remote or in person, isolation from the larger picture. So I think Pivotal does a pretty good job, from what I've seen so far, in spreading information amongst – you know, between teams, making sure that people on one team might understand some of the goals, some of the needs of other teams. And if you work someplace where that's not so much the case or there's a lot of entrenched battle lines and fiefdoms and people trying to hang on to their little area of control, that – I think in large organizations, that can be endemic. And it's a huge problem, right? Because that's the siloing that we all talk about. But, you know, farm metaphors aside, it's basically if you're only optimizing for, you know, local rationality and you don't even look beyond that, you're going to have a bad time.

As an organization. So then, no doubt, you've put together this talk already and rehearsed it countless times, right? So you probably know it forward and backwards. If I put together, you mean I have some slides and keynote? Sure. So, I mean, if you were to highlight, like, what you think some of the major things to get wrong that are fixable with remote working and distributed teams are, like, what would be those things? Like, what are the important things to get right? I really – I think that it mostly boils down to communication. Because if people feel like their management or their coworkers are distant from them, whether in terms of interest, in terms of connectivity, whether or not they're seeing them across a cubicle wall is almost immaterial. The important part is engagement. The important part is actually connecting with one another. So get that right and it doesn't necessarily matter where the people are since we're knowledge workers. We spend most of our days inside our own heads anyway. But being explicit, like yourself, being explicit about what you're doing and then also having everyone around you actually, from time to time, engage with what you're talking about, what you're saying. I mean, I think Chad is great for that, obviously. But any kind of collaborative work tools can help that. But people have to actually put in the effort to try to use them. Yeah. No, that's one thing I've noticed. And to a certain degree at Pivotal, but especially at other companies, is the idea of a information radiator in cyberspace is apparently very hard to get people to understand and do. Which is to say, exactly what you're saying is, like, it's somewhat easy if you're in person to sort of, like, just trust that information will disseminate. Just sort of, what do you call that? Face, not face to face, but word of mouth and mouth to mouth.

It's starting to sound a little weird there. But anyways. Hopefully we're not resuscitating anyone, but osmosis. Exactly. You have one of those funny little mouth jobs that the lifeguards all have nowadays to keep it nice and sanitary, to pass your information from brain to brain. But it does seem like, I'm always shocked, I still am, like, looking in the white-collar world, like, how little information sharing there is. Like, because, you know, usually in the development world, at least in most places, there's at least the information sharing of your version control system. And there's probably a ton more than that, right? But in general, like, walks of life, just as you were saying, being explicit about things and sharing information, it's oddly hard to get people to do. Which is, I don't really understand why. It depends on how they're incented, right? So, if information hiding can advantage you, or if, you know, being the one with the scoop and getting in on the ground floor first or whatever, you know, cut some kind of competitive scenario with other teams or, you know, with people outside your organization. Like, if that, if security through obscurity is actually providing you some sort of perverse incentive, you're going to do it. Like, people do what they're incented to do, and they do what they're measured on. And would you say that's, like, an optimistic or a cynical view on life? I mean, maybe I'm being cynical in saying that if you incent people to hide information and to undercut each other, that's what they'll do. Because humanity is, we're all flawed human beings. But, I mean, I would say the solution to that is not just to tell people to be better, but to incent people differently. So, I guess that's where management comes into it. Yeah, no, and, you know, I asked the question because it's easy to think that it's, like, a cynical view of things, right? That people will... Because it's realistic. Yeah, yeah. But it strikes me for, like, all the exciting, like, business, like, lore and, like, white-collar stuff that I read all the time.

It's very little spoken of, of, like, yeah, you need to set up incentives correctly, right? Like, this is a good example of one of those common sense things that we talk about in the DevOps world that people seem to, like, just not... It just doesn't really fit into their worldview, right? And it's sort of, like, if you set up a system where people are incented to do something, like, incented not to share information, then they won't share information. Like, I don't know what to tell you. I can give an example of something that I did wrong at the university, which is I tried very hard to keep our open tickets low. That did not necessarily translate to people being incented to make sure that those who sent in requests were happy with the result. It was more of a try to make them go away as quickly as possible. No, exactly. That's an example I've stolen from, I think it was a year ago, I saw a ThoughtWorks presentation. Or, no, it was at the beginning of this year. And they were presenting at a local cloud group, and they were complaining about traditional IT. And they were saying, well, the ticket people were paid on how often or how much, how many tickets they would close. So if you didn't fill out your ticket perfectly, they would close it. Right? Like, and so part of what, you know, we lost several months just learning how to write, learning the can't, if you will. The ticket can't, like, how to properly write the tickets. And it is, you know, I mean, it's another example of, like, studying the, as if you're in a management position or something like that. Studying the system and figuring out, like, what's operating correctly and not. And that's a huge part of what, you know, many of the conversations we have in the Pivotal world sort of start and revolve around. It's like, once you get this stack in here, now you need to change how you operate quite a bit. And I find myself, I mean, the reason I'm going on and on about this is talking about those issues pretty much all the time.

Because apparently they're hard. Yeah. Who would have thought? I mean, I also think that there's been a lot of talk lately about, like, you know, more structured solutions or how opinionated would you like your software to be or your platform to be and that sort of thing. And I think that that might be the wrong way to look at it. Because, ideally, I mean, many of us, we're builders, right? We like to tinker. We like to play with things. A whole bunch of software Legos and Lincoln Logs that we get to glue together look like fun to a lot of people like us. And, at the same time, letting people decide what's going to be best for their business based on what looks the most fun to assemble is maybe not necessarily the most productive conversation to have. Right. So, it's like going in and saying, like, hey, what are you actually trying to accomplish? And starting from there, like, I think drives a better tool selection process than saying, what looks the most fun? Let's see if we can find a project that will use that. Because I've totally done that. But is it, like, is it necessarily the most constructive use of time? I don't know. Yeah. Yeah. Yeah. It's the having – we even have a common way of putting it as having the right tool for the job. But there's even a rule before that, which is, like, do you need a tool? Or have you defined your, like, problem space? Exactly. And so, like, you go hang out at a lot of DevOps days and things like that. And you even help run the one up in Minneapolis or the Twin Cities, I guess. I'm never sure what I'm supposed to call it. But – We're technically – I mean, our DevOps days takes place in Minneapolis. And the Twin Cities are, in fact, the area. There you go. Do you know why the C on their baseball hat is, like, the – as you say, what do you call it? American rules football?

Like, why it's the same C that, like, is up in Chicago there? Like, what – how did that happen? I have no idea. Don't ask me how the sports ball people come up with logos. For someone – I mean, I assume they did that before – it probably happened before they started naming all the stadiums after various banks. But I have no idea. I mean, apparently you own shares in the Packers. So, you're committed now. You're on the vacant side of things. We'll have to look this up. Maybe someone listening knows. Anyways, so when it gets to this idea of, like, looking at the job and the value, like, how would you say in the DevOps community we are at kind of, like, I don't know, getting product managers involved? I don't really know how to say it, but, like, every now and then, like, you know, Damon Edwards always comes in and gives his talk about value stream mapping and things like that, right? But, like, do you think the maturity of the – maturity is the wrong word because it's always cyclical. But do you think at the moment in the DevOps days world, like, we talk about sort of, like, product definition enough? I think that that's – I hope that that's where things are moving. I mean, there's definitely been a lot of talk about tools. And I think tools are one of those visible ways that somebody can sell to their boss why they need to take two days off of work to go hang out with a bunch of their fellow nerds. Well, I'm going to learn about X, Y, and Z. Just, like, training, essentially. Right. But I think the larger value is when you start stepping away from a specific tool and seeing how people interact with the tools, how the tools can help us drive value. And so I have seen more and more talks being proposed and accepted for DevOps days that, you know, kind of like some of the talks you've been giving that talk about the larger businesses' needs. Andrew Cliff Schaefer was talking about this at DevOps days Pittsburgh back in 2014. And it's like, I think we have started going in that direction. I think there's still an awful lot of emphasis on tools. And then people kind of wonder, why is there all this squishy culture stuff? I want to learn about, you know, fill in my favorite tool here.

Yeah, yeah, yeah. No, no, that's true. Yeah. I mean, there's – I don't know what the percentage breakout is, but it kind of whacks back and forth over the years. It's – like, I think it was the goat can podcast where there was a bit of a diatribe about how, like, we're deep in a DevOps culture conversation cycle now, which a couple years ago we kind of escaped. And now that there's more and more people who are interested in DevOps, we sort of have swung back to talking about that, sort of like the – I don't know, the introductory stuff of DevOps, which is encouraging people to think culturally, which, to be fair, is a lot of what we've been talking about, right? Like, you need to start switching around your process and doing things, which, you know, is yet more softening of, like, it's actually okay to talk about culture stuff. We shouldn't get sick of that. I don't think we should get sick of that just because humanity is going to continue to have these issues. And also, especially when you're going with a – just from conference design, if you're going for a single-track conference and it's not super hyper-focused, like, say, Monodorama, then when you have a really tool-specific talk at a DevOps days, you do risk having people tune out. So I think it's pretty important that even your technology-specific talks have some takeaways in them for the wider audience who may not be using that specific tool. Right. And so that gets to another topic I wanted to talk to you about. So you not only, like, go speak at a lot of conferences, but you help organize several of them, as I was talking about earlier. And, like, what – like, in the space of, like, infrastructure software, cloudy stuff, DevOps, whatever, like, in your mind, what's the ideal setup for a conference, right? Like, what are the – like, let's say it's kind of, like, a general purpose one, not, like, a specialized one. Like, here's a conference dedicated to just monitoring.

But, like, how are you starting to see and, like, balancing conferences out? Like, like, is it all talks or all not talks? Or, like, what's your kind of ideal format? Open space, as we see at DevOps days, definitely has a lot of value. But if that doesn't fit into your specific kind of conference, like, say, there's no open space at Velocity. On the other hand, there are birds of a feather sessions. There are, like, you know, topical lunch tables. But more importantly, there's enough time for hallway track. Right. Like, I like single track conferences. I'm also fine with multi-track conferences. But the conferences that just have talk after talk after talk after talk are not only bad for people with small bladders, but are also sort of bad for letting people digest the information, talk to the speaker, talk to other attendees, even have time to get from one room to the next. So, I mean, I would say, like, if you're choosing topics, theming things in a room is okay. Spreading them out to make sure people do have time for those serendipitous encounters in the hallway is also a strategy. But making sure that there is enough time in between talks is, I mean, just kind of like passing time in the hallway in high school is actually important or people are going to have not, I'm not going to say a bad time, but they're going to have a better time if they have an opportunity for those serendipitous connections. Right. And then, so, I mean, if you were to give out advice for conference goers, like, what are some things you would throw out to them? Let's see. I would tell them, don't be afraid to sit in the first couple of rows at a talk. Right. Because speakers don't actually enjoy talking to empty chairs. They could probably do that at home. They probably own chairs. So, they're not going to, you know, hate on you, probably not call on you. There's usually not a the first three rows will get wet sort of sign.

We're not at SeaWorld, so. Or a Gallagher event. Right. Right. So, it's usually fine. Yeah, you're not usually at an improv show where if you sit up front, you risk getting pulled onto stage. So, it's probably fine to sit up where you can actually, you know, engage and interact with a speaker. If you're worried that you might have to bail, just kind of sit on one of the sides. And then, if you get paged or whatever, because hashtag OpsLife, you can sneak out. But then, I would also say, like, at lunch, go sit at an empty table. This is advice from Tom Duffield, actually, one of my DevOps Days Minneapolis and DevOps Minneapolis meetup co-organizers. He gave a really good Ignite about conference tips for introverts, where he says, sit at a table by yourself at lunch. There's a limited amount of tables. People will have to sit with you, and they will have to talk to you. Because at the very least, they'll say, hey, can I sit here? Oh, that's good. You know, I was just going to ask that, because I feel like, over the years, I have mastered the art of sitting at a full table and not making eye contact. If you go into the full table already, you can get away with that. Oh, yeah, because you can sort of hide in plain sight, as it were, right? And they probably have a conversation started, and you can just kind of, like, look at your food or look at your phone. But if you would like to interact with the other people at the conference, which I would say is often a goal, because we can all watch conference videos for free on YouTube. So, if we didn't want to interact with any other humans, there would be very little point going to the conference. Indeed. Yeah. And so, if you want to go to the conference and interact with other humans, I would say, feel free to actually go up and say something to the speakers. And feel free to actually talk to the person sitting next to you in the audience before things kick off. You might end up finding out that they're actually, because they're sitting in the same talk as you, there's a very good chance that they're also trying to learn similar things. Or that they have similar problems. Or maybe they work someplace super cool that you would like to work. Or maybe they have exactly the skill set that you're trying to hire for. I mean, if you don't talk to them, you'll never find out. And, like, you know you already have something in common with them because you're both sitting there at that talk.

So, then back when you were at the universities, I was reading through stuff you were up to there. You listed, like, Usenet as something that you monitored and set up. Like, what was that like? That was actually in ISPville. That was in the university, I would say. That's right. But even better. Like, so, that seems like it would be a lot of work. Like, what was involved in that wacky Usenet stuff? Honestly, the part that was the most work for running new servers in the early 2000s was the fact that we had this kind of ITIL-like process. Where, to make any changes to production, we were supposed to fill out this 42-page form and then wait two weeks for some people on the change control review board to decide whether or not we could do the thing. And so, when somebody would complain about missing parts for their binaries, we can draw a veil over exactly what it was they were trying to, you know, download. But when somebody would complain… Pictures of cats, probably. Cats. Probably. Probably pictures of cats, yes. That is what the internet is for. Or maybe cars. But when people would complain, the easiest solution was just to add some more news peers. And it was a completely non-invasive, simple change to production. And we did it all the time routinely. And every single time, I think the statute of limitations has expired on me admitting this, I would just do it and then I would file the emergency change control process, which was a much shorter form. Right. Because I was not going to wait two weeks for some people who had no idea what Usenet was to tell me it was okay for me to do a simple thing that we needed. And ideally, the sooner the better, so that the missing parts could trickle in. So, like, that's the kind of – that's an example, I think, of how heavy-handed IT processes can end up making even the simplest tasks unbearably painful and then also incent employees to lie. Yeah, and it also highlights the example of it's – to steal a metaphor from a famous book of that time, right? You've got to smell the cheese often to figure out if it's gone bad. And in the sense of, like, at some point, it was a good idea or there was a reason to set up this two-week process, right?

And instead of revisiting that and figuring out if it's still a good idea to have it, if it's satisfying the needs that you have, you don't revisit it. And, like, you know, it's yet another example of, like, well, that seems like common sense. But it's almost the – I mean, it's hard to go, you know, make up why something like that would happen. But often it seems like people just kind of slip into autopilot and they don't really look at their job as sort of, like, continual gardening of IT processes they have in place. And instead it's just, like, fortification of unchangingness. But what I really strongly believe is there is no such thing as standing still, right? Because if you choose to take no action, that's still a choice and the world's going to keep changing around you. Whether it's just the environment that your servers are operating in or BitRot or the way your users interact with it. Like, there's always going to be change. So trying to enact some sort of change freeze is just freezing your ability to react to the change. It's not going to prevent change from happening. Indeed. Yeah, yeah, yeah. You know, like you said, you're not going to be able to check your binaries. That's going to be terrible. Yeah, no, it was great. Like, I found that because, man, I often, Usenet is one of those things I forget about often. And then every now and then I'm reminded of it and it's like, what a crazy system that was. I mean, and not, like, really crazy, but how different the whole attitude and architecture and everything of Usenet was versus the way things are nowadays. Did you ever have to set up, like, gopher servers? Despite the fact that I worked at the University of Minnesota, I never had to admit a gopher server. I think Mark McCahill actually left the University of Minnesota, went to some other university where I think he had plans to, like, bring gopher back. I have no idea whatever came of that. Yeah, that one just lost out. It got creamed like everything else by the old web.

Hey, you know, Waze and Archery and Veronica were something, but we at least got to something that works okay most of the time. I mean, if you think of, like, the original, you know, RFCs that were, you look at, like, say, RFC 822, you think about how email started out. Well, obviously, they had to add on a lot of stuff like SPF and DKIM and all sorts of things later to try to deal with the abuse of the system. But it's like that constant iterating, you know, informed by conditions is a super important part of how the internet works. So you've been here a month, as you've been saying, and I watched, I forget where it was, but it was a pretty short presentation you gave where you're, like, in a very nicely wood-paneled room. And you're like, in a, you know, I'm going to go work in marketing. And the whole audience kind of erupts in laughter along with itself. Was that my adventure, excitement, and Jedi create these things? Yeah, yeah. There you go. There you go. So I hadn't thought about it, but ironically, it's sort of ironic. So what made you want to sign up for the adventure and excitement of marketing? What attracted you to this lot in life? I mean, because tell me if I'm wrong, but this is, like, the first non-practitioner job that you've had, essentially? Yeah. Yeah, it is. So I was perfectly happy working in ops at Drama Fever, and then my boss, Tim Gross, was leaving. And, you know, when important life changes, like your boss leaving, happen, you reevaluate what you want to be doing. Smell the cheese often. And so my reevaluation had me talk to Andrew Clay Schaefer, and he's a very convincing sort of man. So I had not, I had talked to a few other people, and it was just seriously about ops jobs. And then I talked to Andrew, and he made this very compelling argument over the course of a 90-minute phone call as to how I could still play with tech

and also talk to people about and educate people about and continue to learn about tech, and it would be fun. So I figured I'd give that a shot. So I hadn't really thought to myself, I want to work at Pivotal, or I want to work in marketing. It's more Andrew convinced me that this could actually be really interesting. And so far it is. And what do you, what would you call it? And I'll, as always, ask a question and talk a lot. But, like, it seems like there's this taboo nowadays of just calling it what we're used to, which is evangelism, which I don't know if that's good or bad. And so it's always a question I have of, like, as the job you described there, right? Like, how do you call that thing? What is that? You know, I really don't know. I mean, the problem, I think, with the term evangelism is that sort of sounds like you're just out there trying to promote your thing. And that's not really what this conversation is about, right? Because I'm interested in learning a lot about all facets of the ecosystem and trying all sorts of different stuff. And maybe collaborating, cooperating with people who are, you know, competitors or frenemies or whatever you want to call them. But, like, I don't have to be in a, strictly speaking, like, I am only going to talk about how great our thing is. Like, that's actually not my role. So I think maybe that's why evangelism kind of sounds like you're doing that. And this is more, you know, just kind of general technologist, I guess. Yeah. It's, I don't know enough about the security word, but I always think it's kind of funny that people will describe themselves as a security researcher. Which is, which usually seems to mean I hack into people's systems and tell them about it. And then that is a researcher. And to some extent, that's an interesting term for, like, what you're talking, like, what all of us, many of us on the team here do is, like, we're sort of, like, I always explain what I do is, like, I try to understand what problems are happening in industry. And explain how people are fixing them, right?

And sometimes I'll say something corny, like storytelling or something like that, because that just sounds, like, quaint. But to some extent, there is sort of, like, it is sort of, like, researching, right? Like, you're still looking at technologies, not even still. Like, you're figuring out how these technologies work and fit together and solve people's problem. And then in sort of, like, instead of publishing it in a classic research type of way, you're just, like, disseminating that information, which seems like it would be helpful. Right. So, like, you know, whether it's blog posts, conference talks, podcasts, whatever, contributing to the general store of information and knowledge out there, I think, is super valuable. I mean, I did a lot more, you know, case study type talks as a direct practitioner. And I think that moving forward, like, my talks look like they're going to be a little bit more of a synthesis. Like, still some actual practical stuff. Also, like, observing some possibilities, looking at some larger patterns. Researching. Perhaps. Yeah, yeah, yeah. Security researcher. Always a fun category. Well, so to that end, before we wrap up here, so what do you have planned out there, speaking of traveling out there into the world? As you humorously wrote somewhere, not all of the tech conferences are in the Twin City area yet. So, as you're going about, like, what are things you have in the books that people can look up to you and maybe make eye contact and talk to you at? Let's see. I'll be at operability.io in London next week. I will also be at, let's see, be at reInvent. I'm not speaking. I'll be at the Pivotal booth, talking to people about AWS and Pivotal stuff. Let's see. Velocity New York. I am speaking. At that one, I'm actually, one last hurrah of talking specifically about a drama fever case study, because that talk I submitted some months ago with a now former co-worker, Pete Shannon.

Velocity EU, my talk is actually going to be a little bit more of a general platform sort of talk. Not just, like, you know, the drama fever Docker and production story. Then, let's see. I think I'm on a panel at FutureStack. I'm giving another, the same talk from London about distributed systems and teams at Recon, and probably some other stuff that I'm not remembering at this very moment. Oh, we did submit, Casey and I, from Casey West, from our team and I did submit a joint talk to Cloud Foundry Summit in both Berlin and Shanghai. So, I guess it's possible, if that gets accepted, that people could chat with me at one of those. Well, what was the topic of the talks? Um, that one is about platforms that enable continuous delivery. Oh, right. And, um, the tooling and, uh, also, you know, organizational choices and, um, operability of your platform that can allow for, you know, a CI, CD sort of workflow. Right. Making sure your stuff works. Ideally, our stuff would work. That's great. That is something that almost everyone wants, I would imagine. And then, and then we've already mentioned the, uh, the Arrested DevOps. Is that, is that, is that like weekly? I lose track in my podcast listening. Like, I just listen to whatever's there, but how often does that come out? Sure. Arrested DevOps, um, it comes out, uh, usually twice a month is what we aim for. Oh, that's relaxing. Mm-hmm. So, uh, we're not quite as ambitious as a software-defined talk can be from time to time. Yeah, yeah. Well, that, that, we, uh, we slack on there all the time. Well, we also try to schedule guests, so it probably makes the scheduling a little bit more spaced out. Oh, scheduling guests. That's the worst. Scheduling. See, that, that is a problem that needs to be solved. Scheduling. I think that's, that's beyond all of our capabilities. We, we, we're, we're down here at the Moorlach levels, but, but someone, someone way up there needs to figure out scheduling. I don't know what the deal is.

Yeah. Calendaring, not a solved problem. Teleportation, also not a solved problem. Uh, this, uh, large array of, you know, various and sundry conferences on multiple continents is something that I think is going to require me to spend a lot more time on airplanes than I really prefer. Oh, yeah. I mean, I got me some of that TSA global entry and, you know, pre-check business, but it's, it's still a, you know, giant airport shaped hassle, so. Yes. Yes. Next thing you know, you'll be installing the Starbucks app. Oh, I don't even want to think about what kind of life choices would lead me to the Starbucks app. You just, you just, you just start preparing yourself and then when it happens, it'll be completely normal. It's fine. I already have the, I already have the set of life choices that makes it, um, cheaper to have the monthly go-go in-flight wifi subscription. All right. There you go. That's what we'll talk about that next time when you've had some, some, uh, some, some more travel, even more travel under your belt. We'll have travel tips. I'm sure there's a laundry list. Travel, travel tips for former practitioners. I think it's, uh, that's, that's a good bit. A good bet is there being a great deal of information that could be gotten there. Certainly from you, possibly from me at some point. So, so where, what you, you have, you have some main, uh, some main hub of information on the interwebs there, right? What's, what's your main place people can go to? Uh, probably the easiest way to keep up with me is to, um, either follow me on Twitter, uh, Bridget Krumhout on Twitter or, um, Bridget Krumhout.com. I tend to link an update to like any talks I'm doing, any podcasts that I've been on, possibly this one. I think we'll be out on the public internet at some point. Um, yeah, uh, if they, if they send me email, it will go to the bottom of the pile and it will take me a non-zero amount of time to reply to it. That's, that's my warning, my warning on email. Yeah. There's a lot of people who like to email me requests for my time and that is probably not the optimal way to get my attention. Yeah. Yeah. Yeah. I, I, I have, I have some emails sitting in my inbox from my, uh, financial planner of like some forms to fill out.

And it's just like, uh, that's never going to happen. I don't, I don't know. I don't know how, I don't know how to fix that problem, but filling out the forms, it's, it's too big of a batch. It's a good metaphor. And I need to have a small, if he asked me like two questions over the, every day over the series of like two weeks, it's probably more likely than if he sends me a big spreadsheet I have to fill out. Yeah. We had the, uh, we just actually just signed the contract for the venue for DevOps days, Minneapolis, 2016, which is going to be July 20th and 21st in Minneapolis in 2016. Mark your calendars now. Do not schedule conferences on top of it. That's right. I entreat you. Um, but we just signed that contract. But I think the thing that took me the longest was just, there is no change control or change tracking. There's no versioning of any sort when they're sending you hotel contracts. Right. So just going through and reading it. The red lining. Again and again. Yeah. Ugh. That stuff is a nightmare. We, yeah. This is another, another problem we'll put on this problem to solve, uh, introducing diff to the rest of the world. Right. And building that into the system. Getting people who send you, you know, banquet equipment orders to use version control would be a thing. Yes. Well, on that note, as always, this has been Pivotal Conversations. You can find, uh, the show notes that we've referenced that don't exist yet, but they will. Once you're listening to this at, uh, pivotal.io slash podcast with an S or without an S, depending on, on how you're feeling about, about it. And with that, we'll see everyone next time. Bye-bye.

Tanzu Talk · 🍕 Get more pizza.