1 00:00:00,000 --> 00:00:03,500 Welcome to a special episode of the Cloud Native in 15 Minutes podcast, 2 00:00:04,080 --> 00:00:07,380 where over the extended holiday season, we'll be sharing some of the highlights from the show's 3 00:00:07,380 --> 00:00:12,260 first six months. I'm Derek Harris from Pivotal, and if you follow the show on the Intersect site, 4 00:00:12,420 --> 00:00:17,500 that's pivotal.io slash intersect, you know I put together a blog post for every episode where I 5 00:00:17,500 --> 00:00:22,980 pull together some of the highlights in a text-based format. Well, this is like the audio-based format 6 00:00:22,980 --> 00:00:28,860 of that, except spanning several episodes. And while the first episode focused on highlights 7 00:00:28,860 --> 00:00:33,400 regarding cloud-native technologies and architectures, this episode focuses on some 8 00:00:33,400 --> 00:00:37,400 of the highlights regarding the business considerations of going cloud-native and 9 00:00:37,400 --> 00:00:44,920 modernizing your IT practices overall. We talk about SRE and AI and open source and enterprise 10 00:00:44,920 --> 00:00:50,740 startups with some of the smartest people in those spaces. And we're going to get to that in just a 11 00:00:50,740 --> 00:00:55,760 minute. But before that, a quick reminder that if you enjoy the podcast, please do share it with your 12 00:00:55,760 --> 00:01:00,140 friends or colleagues or on social media and rate it in the app or platform of your choice. 13 00:01:00,560 --> 00:01:05,380 Apple Podcasts is particularly helpful. So yes, here's the show. 14 00:01:06,760 --> 00:01:10,940 First, we have Andrew Wing, one of the godfathers of deep learning, a Stanford professor, 15 00:01:11,240 --> 00:01:18,880 serial entrepreneur, co-founder of Coursera, you name it, talking about the best way to get started 16 00:01:18,880 --> 00:01:24,000 with AI projects to wit by starting small and picking an easy win. 17 00:01:24,260 --> 00:01:29,440 I've seen a lot more companies fail by starting too big than by starting too small. So I recommend 18 00:01:29,440 --> 00:01:35,540 to most companies to start a few pilot projects, deliver a quick win in six months, and then use 19 00:01:35,540 --> 00:01:42,360 that to ratchet up to bigger and bigger projects. Even Google, when I was starting out the Google 20 00:01:42,360 --> 00:01:48,000 brain team, there was a lot of skepticism about deep learning at that time, both within Google as well 21 00:01:48,000 --> 00:01:53,460 as in the wider world, much as today, a lot of companies may still be uncertain how to use AI. 22 00:01:54,060 --> 00:01:59,480 So my first internal customer of the Google brain team was Google's speech recognition team. 23 00:02:00,320 --> 00:02:05,600 And speech recognition was not, maybe still is not Google's most important AI application. It's not web search 24 00:02:05,600 --> 00:02:11,260 or advertising. But by delivering a quick win there, it helped other teams within Google gain more fame 25 00:02:11,260 --> 00:02:16,720 than what my team and I could do. And then we got our second customer, which was to help Google Maps 26 00:02:16,720 --> 00:02:27,720 use AI to read house numbers, to more accurately geolocate houses and buildings on Google Maps. And only after delivering this second win to a second internal customer, 27 00:02:27,720 --> 00:02:49,280 did I then go on and start the more serious conversation with the advertising team. So I think the lesson to a lot of companies is start small and try to pick a project that can allow you to deliver a quick win because that often helps the organization learn how to do AI and opens the door to then doing bigger projects over time. 28 00:02:49,280 --> 00:03:01,240 And here's Andrew explaining why it's important for executives and other decision makers to have an understanding of what AI can and cannot do when they're trying to scope initial projects and figure out how it might be helpful to their organizations. 29 00:03:02,100 --> 00:03:12,880 Scoping the right AI projects is really hard. And there are a few gotchas on identifying the right thing to work on. One of them is just unrealistic expectations about what AI can and cannot do. 30 00:03:12,880 --> 00:03:29,340 For example, a few years ago, a lot of people thought chatbots could soon have fully general purpose conversations and talk about almost anything with anyone. That turned out not to be true. And companies that tried to build fully general purpose chatbots just did not succeed because the technology was not there. 31 00:03:29,340 --> 00:03:49,640 So I think it's important for executives to have enough of an understanding of what AI can do to scope the valuable and feasible projects. And then also to have the appropriate mechanisms for staffing them, tracking them, and helping set up a machine learning team for success. 32 00:03:49,640 --> 00:04:05,720 And finally, here is Andrew giving a quick and dirty explanation of deep learning, how it works and how it's useful. If you've been following AI over the past several years, you've no doubt heard this term. But if you think it's some sort of magic, Andrew is here to explain that it absolutely is not. 33 00:04:06,100 --> 00:04:15,660 Deep learning is maybe one of the hottest technologies in AI right now. And what we've found over the last several years is this is a wonderful technique for doing supervised learning. 34 00:04:16,180 --> 00:04:39,640 So, for example, if you're running a factory and you want an AI to look at the smartphones going down your manufacturing line and figure out if it is a scratch or not, you know, to do automatic defect inspection, then it turns out that deep learning is a very effective way to learn that input-output mapping, where the input would be, say, a picture of a smartphone, and the output will be, does the smartphone have a scratch on it or not? 35 00:04:39,640 --> 00:04:50,020 And a lot of recent rise of AI and supervised learning is driven by discoveries in deep learning, allowing us to make them go very accurate input-to-output mapping. 36 00:04:50,640 --> 00:05:07,740 And here we have Dave Rensen, a Senior Director of Engineering at Google and one of the company's original proponents and pundits of SRE, a.k.a. Site Reliability Engineering, explaining kind of the general principle behind Site Reliability Engineering, which is to reduce human toil. 37 00:05:07,740 --> 00:05:19,060 The way I like to think of it is this. You can live in one of two worlds. In the first world, a machine called a pager wakes you up at 3 in the morning because some other machine is having a hard time. 38 00:05:19,580 --> 00:05:27,280 In that world, you work for the computers. The world you want to live in is one where some system you're responsible for is having a problem. 39 00:05:27,660 --> 00:05:33,080 It sort of mitigates itself and then it writes a bunch of information out for you to debug the next morning after your morning coffee. 40 00:05:33,540 --> 00:05:38,900 That's a world where the machines work for you. SRE is about getting the machines to work for you. 41 00:05:39,240 --> 00:05:46,120 It's about performing all the operational things we have to do to keep our users happy, but getting the computers to perform as many of them as humanly possible. 42 00:05:47,040 --> 00:05:56,160 And that's the difference between sort of staring at a monitor and poking at a keyboard versus, you know, trying to write software or implement systems that fix themselves. 43 00:05:57,000 --> 00:06:08,600 SRE is often positioned as an alternative to DevOps, but here's Dave explaining why he doesn't think the two are mutually exclusive at all and in fact are about 99% the same thing. 44 00:06:09,120 --> 00:06:12,360 You know, this is just the silliest argument in the history. 45 00:06:12,460 --> 00:06:14,780 Well, maybe not in the history of tech, but certainly in modern tech. 46 00:06:14,780 --> 00:06:27,620 Okay. So here is the truth. SRE and DevOps develops mostly independently of one another, mostly at the same time in response to exactly the same set of problems. 47 00:06:27,780 --> 00:06:31,160 So unsurprisingly, they landed in really similar spaces. 48 00:06:31,440 --> 00:06:35,940 They share, you know, 99% of the same principles. 49 00:06:36,660 --> 00:06:40,320 So I don't, you know, I don't like to argue about like chronology and history and all that stuff. 50 00:06:40,400 --> 00:06:49,860 The way I like to think of it mentally is that SRE is a concrete opinionated, and it certainly is, implementation of DevOps principles. 51 00:06:50,200 --> 00:07:03,400 The thing I like about SRE is if you do SRE work at Google and then go work and then go to, I don't know, LinkedIn or Netflix or some other place with an SRE culture, you know, the, the activities will rhyme with one another. 52 00:07:03,400 --> 00:07:04,920 Like you will recognize the things. 53 00:07:05,380 --> 00:07:23,620 And here is Dave explaining one of these sometimes counterintuitive aspects of SRE, which is that if you're doing too well on your error budgets, that is, if you're, if you budget for X amount of downtime and significantly outperform that expectation, you may have over-engineered, which is not a good thing. 54 00:07:23,880 --> 00:07:26,820 There are actually two bad ways to blow an error budget. 55 00:07:27,180 --> 00:07:32,240 Obviously you can spend too much, meaning, you know, let's say, well, let's just use some math. 56 00:07:32,560 --> 00:07:40,060 Let's say that you have a system that you need where your SLO is 99.9% reliable, whatever reliable means in this context. 57 00:07:40,060 --> 00:07:46,520 It means you can have 43.2 bad minutes, or that's your error budget, every 30 days, right? 58 00:07:46,520 --> 00:07:54,440 So your system can be unreliable, however, we choose to measure that, for 43 minutes every 30 days and still meet its three nines target. 59 00:07:55,200 --> 00:08:02,400 Okay, what happens though when you blow that, when you're, you know, 50 minutes unreliable or 90 minutes or something in some 30-day period? 60 00:08:03,040 --> 00:08:05,400 Okay, that's an error budget policy, right? 61 00:08:06,200 --> 00:08:07,980 And there are two bad things that can happen. 62 00:08:08,060 --> 00:08:09,580 You can spend too much of the error budget. 63 00:08:10,380 --> 00:08:11,060 That's easy. 64 00:08:11,480 --> 00:08:13,640 You can also consistently spend too little. 65 00:08:14,340 --> 00:08:25,580 So if you determine that you can tolerate 43 minutes of error or bad minutes over 30 days, and you're consistently only spending, say, 10 minutes, that's not a victory. 66 00:08:26,000 --> 00:08:27,680 That means you've over-engineered reliability. 67 00:08:28,020 --> 00:08:36,160 You have more reliability than your users need, and that's time and resource and expense you could be applying to innovation or risk or some other thing. 68 00:08:36,240 --> 00:08:38,980 And so the question is, what do you do in that context? 69 00:08:39,520 --> 00:08:41,260 You know, how do you approach those things? 70 00:08:41,640 --> 00:08:45,340 And that's where you get to philosophical differences, you know, between companies and teams. 71 00:08:45,940 --> 00:08:50,440 You know, I, on my teams, I like to keep a pretty simple error budget policy. 72 00:08:50,960 --> 00:08:54,680 If we blow our error budget, I want a feature freeze, right? 73 00:08:54,760 --> 00:08:55,760 No new features. 74 00:08:55,760 --> 00:09:01,460 We are going to spend the time it takes to earn back our good minutes doing reliability-related things. 75 00:09:01,580 --> 00:09:05,520 But not every company has that or every team has that as a default policy. 76 00:09:05,720 --> 00:09:09,560 So that's the kind of a thing that might be a little different going from company to company. 77 00:09:10,280 --> 00:09:16,900 And finally, here is Dave Rensen explaining the importance of measuring the right things if you're going to undertake an SRE practice. 78 00:09:16,900 --> 00:09:28,020 That is specifically measuring the impact on users and the things that directly impact users, less so than measuring things such as CPU load or memory utilization. 79 00:09:28,820 --> 00:09:31,540 No one's users care about CPU load or memory pressure. 80 00:09:31,580 --> 00:09:33,160 Or disk fullness, right? 81 00:09:33,280 --> 00:09:34,460 You know, they don't care about that stuff. 82 00:09:34,520 --> 00:09:37,220 They care about how long did the thing I want to take? 83 00:09:38,480 --> 00:09:40,020 And did I get the correct answer? 84 00:09:40,720 --> 00:09:42,980 You know, or did the server throw an error or some other thing? 85 00:09:43,960 --> 00:09:46,600 You want to measure the things your users carry about. 86 00:09:46,780 --> 00:09:48,980 Carry, excuse me, care about. 87 00:09:49,520 --> 00:09:53,380 We like to say, you know, the important things are the symptoms, not the causes. 88 00:09:53,520 --> 00:09:56,500 I mean, the causes are important because you need the data to be able to debug and fix the thing. 89 00:09:56,760 --> 00:09:58,300 But your users care about the symptoms. 90 00:09:58,800 --> 00:10:02,540 And so those are the things not only that you want to measure, but you want to learn on. 91 00:10:04,380 --> 00:10:11,980 Steven O'Grady, the co-founder and principal analyst at RedMonk, writes a lot about open source and talks a lot about open source. 92 00:10:12,120 --> 00:10:17,300 And here he is explaining why, you know, open source is never really free these days. 93 00:10:17,300 --> 00:10:23,020 You're going to be paying for it one way or another, whether that's in paying for software or paying to develop that stuff in-house. 94 00:10:23,660 --> 00:10:27,180 Open source is not free as in beer, right? 95 00:10:27,600 --> 00:10:40,260 And that has been a sort of, much less so these days, but certainly years ago, people would sort of look at this and say, hey, I can sort of go get this off the shelf, you know, pick a category operating system database, whatever, and say, hey, I can use this for free. 96 00:10:40,500 --> 00:10:48,480 Well, at the end of the day, most businesses are not in the business of maintaining or developing software, and therefore they need help with that, right? 97 00:10:48,660 --> 00:10:55,340 And so at some point or another, you're going to be paying for it, even if you're not paying for it in the form of an upfront license, you know, sort of as you're used to. 98 00:10:55,860 --> 00:11:00,380 So, yeah, in other words, that's, you know, that's the most sort of important change. 99 00:11:01,080 --> 00:11:17,340 Now, as I mentioned before, you know, with open source, you are, you can theoretically enjoy lesser costs in the sense that in many cases, there are multiple suppliers versus something like a Windows, you know, for example, where you can only get that from Microsoft. 100 00:11:17,660 --> 00:11:26,660 And even in cases where, you know, there is, you know, I mentioned MySQL earlier, which was primarily developed by MySQL, other people can pick up and support that code, right? 101 00:11:26,760 --> 00:11:34,140 So if you don't like the support or service that you're getting from MySQL, theoretically, you could go to some third party, you know, supplier or Percona or somebody else. 102 00:11:34,640 --> 00:11:40,440 So, you know, that has a, you know, sort of has the ability to, you know, sort of somewhat decrease your costs. 103 00:11:40,900 --> 00:11:46,560 But, you know, at the end of the day, you know, like I said, the most important sort of takeaways is that this isn't, you know, this isn't free. 104 00:11:46,560 --> 00:11:52,620 You're going to have to pay for software to be developed and software to be supported and serviced over time, one way or another. 105 00:11:53,780 --> 00:12:02,920 One of the big considerations in IT today is whether to buy from a major cloud provider or to buy from maybe a startup or smaller company focused on a specific technology. 106 00:12:03,580 --> 00:12:11,420 And here is Stephen O'Grady again, this time talking about how to do the math on that equation, how to figure out when to buy from whom. 107 00:12:12,180 --> 00:12:19,280 Typically, what you find is, is that, you know, the company, you know, taking Elastic, you know, they are going to be in many, you know, they're the ones who primarily develop the software. 108 00:12:19,420 --> 00:12:23,000 They employ the vast majority of the engineers in the planet qualified to develop the software. 109 00:12:23,420 --> 00:12:29,200 So, for sure, there's going to be a, you know, sort of a technical distinction, right, in terms of sort of what they're able to offer. 110 00:12:29,800 --> 00:12:33,960 You know, so, you know, depending on what you need, you know, that may be sort of the right fit. 111 00:12:34,380 --> 00:12:42,740 In other cases, you know, it may be a case of, all right, well, the 20% or whatever the sort of actual technical delta is, you know, it's not super important to me. 112 00:12:42,780 --> 00:12:46,200 I just have some basic needs, in which case I'll go to a mega cloud provider, right? 113 00:12:46,220 --> 00:12:47,840 I'll go to one of the hyperscale folks. 114 00:12:48,420 --> 00:12:53,340 And, yes, this is not going to be bleeding edge, and I may lack features A, B, and C, but I don't care about those, right? 115 00:12:53,340 --> 00:13:01,000 So, again, it really comes down to what are you trying to do and how strategic, I guess, is the asset in question? 116 00:13:01,540 --> 00:13:17,280 And if the asset, you know, in question is strategic, then, you know, it's pretty reasonable at least to consider the people developing that asset as your first stop, you know, versus folks that are more in the business of, you know, essentially, hey, I run this and sort of then maintain this at scale. 117 00:13:17,480 --> 00:13:18,780 You know, those are meaningfully different things. 118 00:13:19,180 --> 00:13:36,380 And finally, we have Stephen O'Grady weighing in on the importance of contributing back to open source projects, both in terms of helping advance the projects, helping understand your understanding of the technology, and oftentimes to help your hiring and retention efforts. 119 00:13:36,380 --> 00:13:51,000 If you're an organization of sufficient size, you know, that you run some open source piece of infrastructure, and you have a patch, you have a fix, you have a, you know, sort of whatever, something, you know, to potentially contribute back to the project, you know, essentially, you have two options, right? 120 00:13:51,000 --> 00:14:02,880 Which is, you know, try to contribute that back upstream to the project, and have that become part of the core, or essentially, applying your unique patch every time a new release has come out. 121 00:14:02,960 --> 00:14:06,640 And in many cases, that'll void your support, your warranty, right? 122 00:14:06,800 --> 00:14:11,040 And that doesn't seem like a terribly complicated choice, you know, to most people. 123 00:14:11,500 --> 00:14:17,660 So that's why we're seeing sort of more, more thoughtful policies about contributing back to projects. 124 00:14:18,120 --> 00:14:28,380 And it can be difficult in some circumstances, because, you know, what you hear from, from enterprises is, I don't want to subsidize the development of, you know, you know, this vendor's product, and so on. 125 00:14:28,520 --> 00:14:37,580 But we're, we're sort of gradually getting people to the, to the point where it's like, look, this isn't, you're not paying to develop it, you're paying not to have to repatch this thing over and over, right? 126 00:14:37,660 --> 00:14:39,700 So, you know, progress, I think, is being made there. 127 00:14:40,100 --> 00:14:45,560 Launching their own open source projects, that is a more complicated question. 128 00:14:45,720 --> 00:14:47,020 It can be a very, very good thing. 129 00:14:47,280 --> 00:14:50,020 You know, Capital One has done this with, you know, things like Hygeia, and so on. 130 00:14:50,400 --> 00:14:55,620 It's harder, simply because, you know, basically, launching a project, you know, takes a fair amount of time and effort. 131 00:14:56,240 --> 00:15:01,640 And that may or may not be of, you know, sort of benefit, you know, to the parent organization. 132 00:15:01,940 --> 00:15:14,400 But in some cases, you know, if the, if the software itself is valuable, and, you know, sort of meaningfully, you know, received by the market, you can find, you know, homes, you know, in the form of foundations that will take on that code for you. 133 00:15:14,400 --> 00:15:21,720 And therefore, you get to offset some of the legal trademark IP validation, et cetera, overhead, you know, of maintaining these projects. 134 00:15:21,900 --> 00:15:22,940 So there are benefits. 135 00:15:23,060 --> 00:15:27,200 That's a, that's a more complicated question, though, than just sort of contributing fixes and patches back. 136 00:15:27,420 --> 00:15:31,000 And you can also recruit, you know, sort of, in many cases, better talent. 137 00:15:31,300 --> 00:15:39,240 Because, you know, if, you know, like, if I'm a talented engineer, and I can pick between two options, you know, one in which I write a whole bunch of code, nobody ever sees it. 138 00:15:39,440 --> 00:15:50,760 Or two, I write a whole bunch of code, you know, on behalf of an enterprise, and it makes its way into, you know, sort of these, these sort of public, open source projects, I'm more marketable, you know, sort of in the second example. 139 00:15:51,120 --> 00:15:59,100 So, you know, yeah, it certainly, you know, it helps with recruiting, it helps with, you know, sort of the talent that you can recruit, it helps for retention, in many cases. 140 00:16:00,180 --> 00:16:16,240 Here we have Jonathan Laird, general partner at venture capital firm Workbench, explaining how he used to go about, and his team used to go about during his tenure at Morgan Stanley, considering whether or not to look at a startup to solve a particular technology need. 141 00:16:16,740 --> 00:16:30,600 So, not to make this a total Morgan Stanley talk, but one of the ways that we viewed things internally was across really a couple quadrants, or four quadrants, rather, with these axes of, number one is the risk of the startup itself. 142 00:16:31,120 --> 00:16:33,520 And number two is the risk of the service internally. 143 00:16:34,280 --> 00:16:39,520 So, at the end of the day, most very early stage companies, Series A and B, would be in that high risk category. 144 00:16:39,520 --> 00:16:49,900 But the key is, can you map that to a service or a use case internally, where the hypothetical question of, if they disappear Monday, what happens, you're actually okay. 145 00:16:50,420 --> 00:16:55,360 Because you'd be surprised how many times, even in, let's use an example of like the database world. 146 00:16:55,780 --> 00:17:08,780 Back in the day, when we were looking at columnar databases and in-memory databases, when they were kind of a new thing, we had a whole team that was set up cross-functionally with different expertise to meet with some of the leading startups at the time. 147 00:17:09,340 --> 00:17:11,020 And this was a big undertaking. 148 00:17:11,200 --> 00:17:20,740 So, what we figured out was, what are use cases, whether it was like a fixed income team or another small pocket of the firm, where, look, we're not going to replace out IBM DB2 databases. 149 00:17:20,920 --> 00:17:22,640 We're not going to get rid of Oracle databases. 150 00:17:23,040 --> 00:17:25,000 They're not going away, and no one's going to touch them. 151 00:17:25,300 --> 00:17:37,000 But for these net new capabilities, could we actually generate revenue if we brought in a new analytic database that could compress the data, enable cost savings, and frankly, just give us faster calculations? 152 00:17:38,160 --> 00:17:45,480 So, part of it is really just, as a starting point, figuring out where can you actually explore in a safe, so to speak, way. 153 00:17:45,800 --> 00:18:00,480 And here is Jonathan again, this time with his co-founder and fellow general partner, Jessica Lin, discussing the importance of negotiation in software contracts, both from the enterprise side and the startup side, in terms of finding a middle ground. 154 00:18:00,480 --> 00:18:10,560 It has to start from the top, as we said, with these initiatives, but then there needs to be buying across legal, procurement, security, because the worst thing that we've seen is companies that are dying. 155 00:18:10,800 --> 00:18:14,740 And let's play off that chatbot example again, that say, hey, we have a use case. 156 00:18:14,840 --> 00:18:17,080 We want to share some data that's actually not sensitive. 157 00:18:17,200 --> 00:18:18,220 Maybe it's support tickets. 158 00:18:18,820 --> 00:18:22,420 And then they get stuck where they'll start engaging the startup community. 159 00:18:22,660 --> 00:18:23,780 They've gathered their requirements. 160 00:18:23,780 --> 00:18:28,940 And they'll meet with these companies and throw a 50-page kind of POC agreement in front of them. 161 00:18:29,220 --> 00:18:32,680 Or they'll put crazy, onerous security architecture reviews. 162 00:18:32,880 --> 00:18:35,000 Or things, as Jess is saying, like this code escrow. 163 00:18:35,160 --> 00:18:45,080 So one of the things that companies could do best is actually learn from some of these earlier adopters and more sophisticated buyers around what does procurement look like? 164 00:18:45,080 --> 00:18:55,540 Can we meet internally with our legal teams and come up with a way abridged version of an early test and ultimately support contract for this particular use case? 165 00:18:55,860 --> 00:19:00,100 And if you're buying from IBM, of course, that procurement cycle is made for them, right? 166 00:19:00,160 --> 00:19:03,020 But if you're an emerging startup, that's not going to be the best fit. 167 00:19:03,080 --> 00:19:10,440 And that's where, quite frankly, we spend all of our time at Workbench is coaching our startups to say, hey, what is actually non-negotiable in a procurement? 168 00:19:10,700 --> 00:19:17,700 But what are other things that, quite frankly, you need to give in on, whether it's an MSA or warranty of the product or indemnification, right? 169 00:19:17,720 --> 00:19:18,700 There's a lot of education. 170 00:19:18,700 --> 00:19:27,400 I think an opportunity for both the suit and the hoodie to meet in the middle in order to be able to accelerate the procurement process. 171 00:19:27,760 --> 00:19:32,980 One of the most important parts about engaging with a startup, especially if you're going to be using its technology for, 172 00:19:33,080 --> 00:19:37,280 any serious applications or workloads, is making sure that's a long-term viable company. 173 00:19:37,820 --> 00:19:48,000 And here's Jonathan Lehrer, again, explaining some of the questions he would ask at Morgan Stanley to vet startups and figure out whether they really are all that they claim to be. 174 00:19:48,440 --> 00:19:53,180 Part of my old job was to help with the financial viability assessments of the companies. 175 00:19:53,400 --> 00:20:02,800 And it was actually a lot of fun because what that entailed was getting on the phone with CEOs and CFOs of all of the startups that we did business with, which is actually a huge number. 176 00:20:03,440 --> 00:20:11,560 And reviewing their P&L, but also beyond just the pure numbers, because again, most of them were early, understanding things like who are your top 10 customers? 177 00:20:12,160 --> 00:20:14,520 What is the concentration risk associated with that revenue? 178 00:20:15,040 --> 00:20:17,560 Help us understand the background of your management team. 179 00:20:17,980 --> 00:20:20,780 Help us understand your net burn and then also your VCs. 180 00:20:21,300 --> 00:20:23,760 And does it look like they would fund you in the future? 181 00:20:23,940 --> 00:20:25,280 How much cash do you have left in the bank? 182 00:20:25,440 --> 00:20:35,280 And of course, a lot of startups are naturally reticent to share this information, but we actually did have a Chinese wall where it wasn't going to the procure of the house for the users. 183 00:20:35,280 --> 00:20:39,840 So they ultimately would share in an effort to try to win that Morgan Stanley contract. 184 00:20:39,960 --> 00:20:50,320 But the things that you would see in terms of some of these high-flying companies and candidly how small the revenue was, a lot of these companies that you see hyped in the news are pretty early. 185 00:20:50,880 --> 00:21:02,040 So the trick becomes who actually will get the benefit from that use case that they're pitching, so much so that they wouldn't use the AWS offering of it or the Microsoft Managed Service offering. 186 00:21:02,800 --> 00:21:09,040 And when a startup can actually solve a problem that they can bypass some of these big vendors, you know that you can be onto something huge. 187 00:21:09,580 --> 00:21:19,080 And finally, here's Jessica Lin one more time discussing the importance for startups as well as for enterprises of engaging with the right people inside the organization. 188 00:21:19,400 --> 00:21:27,900 Many times that's going to be a line of business user who has a budget to buy and has a need that needs to be filled ASAP. 189 00:21:28,320 --> 00:21:30,260 So there's a whole spectrum, right? 190 00:21:30,380 --> 00:21:34,040 And we've advised our companies, and John blogged about this. 191 00:21:34,100 --> 00:21:38,200 How do you find the ones that are much more sophisticated, much more early adopting? 192 00:21:38,820 --> 00:21:42,820 And the reality is that there's also, you know, people behind that curve. 193 00:21:42,820 --> 00:21:48,780 And the question is, with those folks, what tends to happen is they have these internal innovation teams. 194 00:21:49,080 --> 00:21:58,380 And the challenge is if they're not tied to actual business challenges or BU's, it can suck a lot of time out of startups. 195 00:21:58,380 --> 00:22:05,120 And they end up in these cycles with them, doing demos, showing them their product. 196 00:22:05,720 --> 00:22:11,320 And the end of the day doesn't end to actually anything meaningful, whether it's a POC or pilot. 197 00:22:11,920 --> 00:22:14,480 And for startups, you know, every day is burn. 198 00:22:14,760 --> 00:22:19,680 And so that can be really, really painful and actually harmful to the startups. 199 00:22:19,680 --> 00:22:30,140 And so for these large companies who are trying to, you know, make their KPIs, whether it's how many new startups they've met with, they don't realize that it can be really, really hurtful to startups. 200 00:22:30,320 --> 00:22:37,800 So we actually advise most companies to stay away from, quote unquote, innovation teams and as soon as possible, get into the line of business. 201 00:22:39,240 --> 00:22:42,680 And that's a wrap on this best of episode of Cloud Native in 15 minutes. 202 00:22:43,140 --> 00:22:49,660 If you enjoyed it and you learned something, please go ahead and rate us on the platform of your choice or wherever you're listening. 203 00:22:50,160 --> 00:23:02,040 And if you want more about digital transformation, you want to read some of the write-ups of this podcast, you want to just generally get smarter about cloud-native technologies, follow us on the web at pivotal.io slash intersect. 204 00:23:02,480 --> 00:23:03,760 Thank you so much for listening.