1 00:00:00,000 --> 00:00:12,320 Hi, everyone. Welcome to Pivotal Conversations. I'm Jeff Kelly. This week, Dormain and I chat 2 00:00:12,320 --> 00:00:17,400 with Farhan Tawar, who's the Vice President of Engineering at Helpful.com. Helpful.com 3 00:00:17,400 --> 00:00:21,080 provides a video and audio-based business communications platform to help business users 4 00:00:21,080 --> 00:00:25,220 collaborate more interactively. Farhan talks about the challenges of building a company 5 00:00:25,220 --> 00:00:29,200 from scratch, why it's important to apply smart architectural principles to empower 6 00:00:29,200 --> 00:00:33,140 development teams, and why he strives for boring infrastructure. Enjoy. 7 00:00:40,680 --> 00:00:48,880 I was thinking about this, Farhan, that it's almost four years to, I mean, here on recording 8 00:00:48,880 --> 00:00:56,400 day, perhaps when this goes live, it'll be four years, almost exactly, since you and I did a webinar 9 00:00:56,400 --> 00:01:01,260 on... Oh my God. The January, yeah, the January webinar. That's right. 10 00:01:02,460 --> 00:01:11,420 So, yeah, I was like, holy moly. It was fun, as always. And yeah, it's just been a long time, 11 00:01:11,500 --> 00:01:19,720 which is kind of crazy. So maybe you can catch some, catch me up, catch our audience up on what 12 00:01:19,720 --> 00:01:25,800 you've been up to. You were at Pivotal back then. That was four years ago. But you are at Helpful now. 13 00:01:26,720 --> 00:01:33,440 Yeah. So I can give you a little bit of the history. So I met, actually, Edward and Rob, me in 2009 14 00:01:33,440 --> 00:01:40,900 when I first got to Extreme Labs. And what was interesting about what Extreme Labs was doing was 15 00:01:40,900 --> 00:01:46,580 that Sonny, who was one of the founders of Extreme Labs, he was a Pivotal Labs client. And he was 16 00:01:46,580 --> 00:01:53,100 building a startup. And he got more excited about how they were building the stuff that they were 17 00:01:53,100 --> 00:01:57,280 building than his actual startup. So he was working with Pivotal Labs. And he's like, you know what, 18 00:01:57,340 --> 00:02:03,300 the way you're building my product is more interesting than my product. And so he decided to then, 19 00:02:03,660 --> 00:02:06,640 you know, that startup, I can't remember actually what happened to the startup, but he started a new 20 00:02:06,640 --> 00:02:11,920 company called Extreme Labs in 07, focused on two things that were different than Pivotal Labs. One 21 00:02:11,920 --> 00:02:16,060 was mobile. They're like, we're going to make a big bet on mobile. And the second thing was they 22 00:02:16,060 --> 00:02:21,960 wanted to do it in Canada. And so, you know, the, I think the, in the background, you know, Pivotal Labs 23 00:02:21,960 --> 00:02:28,260 North was kind of their idea on focus on mobile. And I joined in 2009 via the other co-founder who I 24 00:02:28,260 --> 00:02:32,020 went to school with, Amr. And right away, the first thing they did is they said, you got to go to San 25 00:02:32,020 --> 00:02:36,860 Francisco and sit down with Robby and Edward. And you have to sit down and understand what, 26 00:02:37,100 --> 00:02:39,820 how they are building software in a completely different way than anything you've ever done 27 00:02:39,820 --> 00:02:44,520 before. And so I wasn't like, you know, right out of school. I've been, you know, 2009, I graduated 28 00:02:44,520 --> 00:02:48,580 in 98. So I've been out of school for a while already, but I went down, I still kept that piece 29 00:02:48,580 --> 00:02:54,260 of paper of my notes from that meeting because it was mind blowing on how they were building software. 30 00:02:55,440 --> 00:02:58,440 And, you know, there's, as you know, there's lots of things that are popular, right? 31 00:02:58,440 --> 00:03:02,820 There were pair programming, tracker, sustainable pace, like all these things, but there was all 32 00:03:02,820 --> 00:03:08,800 these little minor notes too. And so we built Extreme Labs in their image, innovating on the mobile side 33 00:03:08,800 --> 00:03:15,180 and also focused on making changes to work with enterprise. Cause that was quite different than 34 00:03:15,180 --> 00:03:20,440 what Pivotal Labs was focused on. They were focused on startups and rails. And we were doing mobile and 35 00:03:20,440 --> 00:03:25,420 enterprise. And so I was, you know, modifying and adjusting the process, right? Robby calls us do the 36 00:03:25,420 --> 00:03:29,540 right thing. So you do the right thing, which is sometimes you got to make changes. And then 37 00:03:29,540 --> 00:03:36,540 surprisingly, or maybe unsurprisingly, Pivotal, Pivotal, Big Pivotal bought Extreme Labs in 2013, 38 00:03:37,440 --> 00:03:43,540 which was a, we were about the same size as Pivotal Labs at the time, 350 people. So we merged and became 39 00:03:43,540 --> 00:03:50,400 this giant force, which was the Pivotal Labs business inside of Pivotal. And then I was at Pivotal for 40 00:03:50,400 --> 00:03:56,240 two years doing two main things. One was, um, integration. So I became a VP engineering for the 41 00:03:56,240 --> 00:04:02,000 Northeast integration into Pivotal Labs business. And then, uh, I left the labs business to join 42 00:04:02,000 --> 00:04:08,580 Cloud Foundry and was the CTO for mobile and Cloud Foundry and worked with the team there to help bring, 43 00:04:08,580 --> 00:04:18,340 um, Cloud Foundry to folks who wanted to build, uh, scalable mobile infrastructure, which is, uh, a large 44 00:04:18,340 --> 00:04:23,180 part of where, uh, software is going. Um, so that's 2015. So there you kind of long winded way to get 45 00:04:23,180 --> 00:04:29,380 to 2015. And then I, uh, wanted to pursue my entrepreneurial route. So I actually never started 46 00:04:29,380 --> 00:04:35,140 a company. So 2015 left to start a company called Helpful with another serial entrepreneur, uh, in 47 00:04:35,140 --> 00:04:40,380 Toronto. So that's kind of the genesis of how I got to where I am now. So what does Helpful do? So 48 00:04:40,380 --> 00:04:48,580 in 2015, we had a few thesis, uh, theses, uh, thesis around, um, what, where we thought software was 49 00:04:48,580 --> 00:04:54,400 going. I was still interested in mobile, still interested in enterprise, um, and started getting 50 00:04:54,400 --> 00:05:02,280 even more interested in machine learning, machine intelligence. And so, um, we built a product in 2016, 51 00:05:02,280 --> 00:05:10,360 um, called the Helpful Directory, which was the AI powered employee directory for fast growing 52 00:05:10,360 --> 00:05:16,920 companies. We noticed that in many large, uh, fast growing companies, you end up building an employee 53 00:05:16,920 --> 00:05:23,160 directory because it was the thing that helped you figure out who knows who and who's doing what in 54 00:05:23,160 --> 00:05:30,100 the company. And, um, we did that for about a year. Um, we learned a lot in that process. And through that, 55 00:05:30,100 --> 00:05:38,580 one of our engineers, uh, in a hackathon project actually, um, built out a little video, uh, profile 56 00:05:38,580 --> 00:05:41,200 for herself. So he had a, you know, employee directory, had a little profile and she said, 57 00:05:41,220 --> 00:05:44,720 you know, it's much more compelling for me to put a little video on here than a little text profile. 58 00:05:45,500 --> 00:05:50,900 And so Patricia built a little video profile. And what we noticed was one, it was super compelling, 59 00:05:50,900 --> 00:05:56,220 you know, just like an about me and, um, what I'm working on my goals. Um, but we, people started 60 00:05:56,220 --> 00:05:59,880 hacking the video profile to talk to each other. So people would kind of use it as like a voicemail 61 00:05:59,880 --> 00:06:02,880 I would look at Patricia's profile and would tell me like what she's working on right now. 62 00:06:02,980 --> 00:06:07,620 And then I could leave a voicemail for her and people started using it very much like Snapchat. 63 00:06:07,780 --> 00:06:12,780 People use Snapchat in the consumer world. So we pivoted the product in 2017 to be focused 64 00:06:12,780 --> 00:06:20,420 on video. And so we built a Snapchat for work. It was enterprise ready, single sign on, um, 65 00:06:20,420 --> 00:06:27,960 AI transcriptions, stickers that were focused on, um, more around workplace stickers. So not like 66 00:06:27,960 --> 00:06:33,060 vomit comet and things like that, but more around, um, sales deal closed, you shipped a product, 67 00:06:33,060 --> 00:06:37,320 you recruited a new person, like those kinds of stickers. Uh, we deploy that solution. 68 00:06:37,940 --> 00:06:45,020 And then in 2018, we got really, really excited about the rise of audio and with podcasts that 69 00:06:45,020 --> 00:06:52,180 this is one of them and audio books being, uh, a new medium, we realized that there was a gap in the 70 00:06:52,180 --> 00:06:58,720 market in, um, live interactive audio. And so we built a tool called dialogue, which was focused on 71 00:06:58,720 --> 00:07:02,620 live interactive audio over the internet. So, you know, the example I always give people is when 72 00:07:02,620 --> 00:07:07,480 Donald Trump tweets, people wait for Nate Silver to respond from five 38. And it takes him about six 73 00:07:07,480 --> 00:07:13,460 hours to create an emergency podcast for that tweet. And with us, with dialogue, you can respond right 74 00:07:13,460 --> 00:07:18,980 away, reply, you know, start streaming live immediately and take callers and it's, you know, 75 00:07:19,040 --> 00:07:23,160 AM radio on the internet. Uh, and so that's what we're working on now. We have a bunch of, um, 76 00:07:23,600 --> 00:07:28,560 folks with pretty large audiences. Um, Ben Stiller uses it every week, Jake Tapper from CNN, 77 00:07:28,780 --> 00:07:33,740 Greta Ben Sustran, and a bunch of other folks who, uh, use it to connect with their fans, uh, 78 00:07:33,740 --> 00:07:37,940 every single week. So that's, that's what helpful is working on right now. Okay, great. That was a 79 00:07:37,940 --> 00:07:43,180 really good overview. So tell us about how you build software and how you're running the platform 80 00:07:43,180 --> 00:07:49,160 at helpful.com. Yeah. So let me, let me get into that. So I think, you know, the, the, the one 81 00:07:49,160 --> 00:07:53,920 testament that I can say about, you know, things from a previous, like, you know, coming from pivotal 82 00:07:53,920 --> 00:07:59,760 is that once I started my own company, which meant writing my own check, right. You can, you can 83 00:07:59,760 --> 00:08:03,700 basically say to yourself, okay, what are the things I still believe in that I have to 84 00:08:03,700 --> 00:08:10,340 now use writing with my own money? Right. So if you, you know, for example, um, let's say 85 00:08:10,340 --> 00:08:14,780 you worked at a restaurant and then, uh, you left the restaurant, do you still come there 86 00:08:14,780 --> 00:08:17,940 to eat? Because just like the way that they prepare the food and the food itself, you're 87 00:08:17,940 --> 00:08:21,100 like, it's still good. It's still, it still reminds you of like, Hey, this is really good 88 00:08:21,100 --> 00:08:25,140 craftsmanship. Right. And so when I left pivotal, I think a lot of people were looking to see 89 00:08:25,140 --> 00:08:30,260 like, what would I still do? Um, given that I left pivotal. And so I left pivotal and what 90 00:08:30,260 --> 00:08:34,580 do I still do? Well, with my own money and my own team, we pair program, we still use pivotal 91 00:08:34,580 --> 00:08:38,980 tracker. We still work nine to six, like all the things that, uh, I've been doing for the 92 00:08:38,980 --> 00:08:44,820 last 10 years in my career. And then got bolstered by learning from many of the people who invented 93 00:08:44,820 --> 00:08:48,740 the stuff. Um, I kept doing in my own company with my own money. So I think that's a good 94 00:08:48,740 --> 00:08:54,900 testament. And part of that is on the infrastructure side. So when we first started building helpful, 95 00:08:54,900 --> 00:09:00,260 um, the dialogue, uh, not dialogue to help the first product, which was the, uh, directory, 96 00:09:00,260 --> 00:09:06,420 we right away realized that of course we need some server infrastructure. And I gave a credit card 97 00:09:06,420 --> 00:09:12,260 to my first engineer. Um, and I said, Hey, go get an Amazon EC2. Right. Cause I was just like, 98 00:09:12,260 --> 00:09:15,380 you know, get some infrastructure. And he looked at me and he goes, you want me to manage servers now? 99 00:09:16,020 --> 00:09:20,020 Like he literally said that to me. And, you know, given that, you know, he was also from pivotal, 100 00:09:20,020 --> 00:09:25,380 he was reminding me of like, Hey, we just came from an environment where we tell everybody 101 00:09:25,380 --> 00:09:29,380 that don't do the undifferentiated have, don't do the undifferentiated heavy lifting. 102 00:09:29,380 --> 00:09:33,220 Why are you then going to Amazon EC2 and like setting up a server that you have to maintain? 103 00:09:33,220 --> 00:09:37,700 And so I looked at him like, you're right. We should be using PWS. And the reason we should 104 00:09:37,700 --> 00:09:42,100 be doing that is because it's going to allow us to containerize our application logic and only 105 00:09:42,100 --> 00:09:48,100 have to worry about the application. And so, um, PWS at that time, much less mature than it is now, 106 00:09:48,100 --> 00:09:54,500 but was, uh, did have production workloads on it. And so we were able to right away, 107 00:09:54,500 --> 00:10:00,340 bring up a PWS infrastructure, um, build an application, uh, deploy it, and then not have 108 00:10:00,340 --> 00:10:04,660 to worry about all of the runtime things that you have to worry about in managing a server. 109 00:10:04,660 --> 00:10:11,940 And that, um, since that day in early 2016, we've been on PWS the whole way. So we have, 110 00:10:11,940 --> 00:10:15,940 you know, we have a very, you know, when we're doing the video infrastructure, a pretty significant 111 00:10:15,940 --> 00:10:22,660 workload, um, we've had to push and pull PWS into different ways to make the things that we wanted 112 00:10:22,660 --> 00:10:29,700 to run as performant as they had to run. Um, you know, in that, in that time period, but it was, 113 00:10:29,700 --> 00:10:33,460 uh, it's, it's been a very, it's been a differentiator for us on the infrastructure 114 00:10:33,460 --> 00:10:38,340 side because we don't have to manage security updates. We don't have to manage, um, wondering 115 00:10:38,340 --> 00:10:42,020 if our things are running because they're, uh, that's automatically managed by the platform. 116 00:10:42,580 --> 00:10:47,140 It's automatically updated. Like all the things that we care about, um, happen for us through PWS. 117 00:10:47,140 --> 00:10:53,460 And then on the second side of it is, is support infrastructure, right? So we are very, uh, ingrained 118 00:10:53,460 --> 00:10:59,060 with the support team there. Things are going wrong. Um, we can message them. We can find out what's 119 00:10:59,060 --> 00:11:03,860 happening. We watch the incident reports. We have, you know, the PWS Twitter feed right into our Slack 120 00:11:03,860 --> 00:11:07,860 channels. We can see all the things that are happening and it allows us, you know, high confidence 121 00:11:07,860 --> 00:11:13,780 that our infrastructure works. And, you know, all goes back to that first comment from my first 122 00:11:13,780 --> 00:11:18,100 engineer who said, you want me to manage servers now? Like literally like what's the, what's the value 123 00:11:18,100 --> 00:11:25,700 in that? There is no value. Exactly. So now you've got, uh, you're able to put your team, you're able 124 00:11:25,700 --> 00:11:29,620 to focus your team on the things that do differentiate you, which of course is the application and the 125 00:11:29,620 --> 00:11:33,620 features and the capabilities and listen to your customers and iterating and those kinds of things. 126 00:11:33,620 --> 00:11:38,820 Right. And I think it's a, it's, it's not a minor point because what I think some people 127 00:11:38,820 --> 00:11:43,620 maybe take the wrong way is they feel like this sort of infrastructure is only for big companies, 128 00:11:44,420 --> 00:11:48,180 right? Like, oh, if we're, if we're a big company, we're a bank and we have thousands of 129 00:11:48,180 --> 00:11:53,060 applications and thousands or thousands of microservices, we might need this level of, uh, 130 00:11:53,060 --> 00:11:58,660 orchestration, auto automation. That's false. Even in, you know, when we started, we were only two 131 00:11:58,660 --> 00:12:04,660 engineers, right? So with two engineers, um, you know, everything we were pair programming. We were 132 00:12:04,660 --> 00:12:09,700 using tracker. Like we were, we were, you, we were using PWS. So it's not the case that you have to 133 00:12:09,700 --> 00:12:13,460 wait until, oh, you know what? There's this like breaking point where I, then I have to start thinking 134 00:12:13,460 --> 00:12:17,860 about automation in my infrastructure. It's, it's, you have very limited time. So if you have very 135 00:12:17,860 --> 00:12:23,620 limited time to react and build things for customers, why would you not want to spend all that 136 00:12:23,620 --> 00:12:28,900 time on the differentiation of your company, not on setting up a server and doing security and 137 00:12:28,900 --> 00:12:35,460 upgrading, like, like all the stuff that is like non essential. Um, so I'm, I'm glad we did it right 138 00:12:35,460 --> 00:12:41,220 away. And I would, I would encourage folks to think in that way early on rather than later. I don't think 139 00:12:41,220 --> 00:12:46,980 it was, you know, at any point like overkill. Right. And in fact, I mean, if you, you could make 140 00:12:46,980 --> 00:12:51,300 the arguments, it's even maybe more important if you're a smaller organization, if you've got two 141 00:12:51,300 --> 00:12:54,980 people, you don't want them, you want them spending all of their time on the product and, 142 00:12:54,980 --> 00:12:59,860 and as little time as possible on those things that aren't going to differentiate you, uh, like 143 00:12:59,860 --> 00:13:06,660 managing servers. So, all right. So that's great. So you're using PWS, which is essentially a hosted 144 00:13:06,660 --> 00:13:13,060 version of Cloud Foundry. Yes. So talk about some of the things you're doing in terms of the CICD 145 00:13:13,060 --> 00:13:17,300 and continuous deployment. Talk about how the platform enables you to do some of those things. 146 00:13:17,300 --> 00:13:23,700 Yep. Sure. So, I mean, maybe no surprise. We use Concourse, right? So we, we use Concourse CI. 147 00:13:24,260 --> 00:13:28,500 We have a hosted for, uh, we have a version, I think hosted for, uh, we heard that we host, 148 00:13:28,500 --> 00:13:34,580 I think on EC2, I gotta, I gotta check that. Um, but what that again allowed us to do was make sure 149 00:13:34,580 --> 00:13:38,820 that we have, you know, visible, uh, build monitors. We were actually part of the team that, 150 00:13:38,820 --> 00:13:42,740 uh, the Concourse CI team came here to see how we were using it. I think we built a custom UI on top 151 00:13:42,740 --> 00:13:46,900 of it. So they wanted to see what people were doing in terms of like custom UI, but the same, 152 00:13:46,900 --> 00:13:54,820 you know, the same things, uh, apply, right? Meaning if you're writing code, then you're, uh, 153 00:13:54,820 --> 00:14:00,020 not ensuring that the pipeline is green when you're writing code and the team is, is entirely green. 154 00:14:00,580 --> 00:14:05,380 Everything stops and you have to fix the build. And that's something that because it's visual 155 00:14:05,940 --> 00:14:10,100 forces the culture of the company to be like, Hey, you're, you can't like, 156 00:14:10,100 --> 00:14:14,020 everybody should stop what they're working on and fix the build. And so we do have these monitors 157 00:14:14,020 --> 00:14:20,180 up that keep us appraised of the CICD condition because we all also build things, microservices. 158 00:14:20,900 --> 00:14:29,300 It allowed us to abstract away, um, team A from teams B from team B's work in a way that allowed 159 00:14:29,300 --> 00:14:35,540 them to deploy independently, right? Um, you don't want deploying to be some momentous event. 160 00:14:35,540 --> 00:14:40,260 It should be just a normal course of, uh, the course of the day. And so we deploy all the time, 161 00:14:40,260 --> 00:14:45,460 microservices are deployed, different components are deployed. We use the CICD pipeline in that way 162 00:14:45,460 --> 00:14:53,700 to see that the health of the product is, um, green. And that allows us to, um, not worry again 163 00:14:53,700 --> 00:14:58,100 about the overall health. And we can just focus on like, Hey, what am I, what is my, what is in front 164 00:14:58,100 --> 00:15:00,900 of me right now? What do I have to focus on? And I don't have to look around and be like, 165 00:15:00,900 --> 00:15:05,540 like, okay, the thing that I did broke somebody else's thing. Um, so the, that CICD, you know, 166 00:15:05,540 --> 00:15:09,300 again, I, I, we, it wasn't like we were trying to remain on the pivotal stack for whatever reason. 167 00:15:09,300 --> 00:15:14,580 Like, um, I didn't sit down and be like, well, let me just see what pivotal library has. It was 168 00:15:14,580 --> 00:15:20,020 more about each thing kind of came from the team and, um, the team came back and said, well, you know, 169 00:15:20,020 --> 00:15:24,500 this is what we need for the infrastructure. And our team did, you know, our team, um, at its, uh, 170 00:15:24,500 --> 00:15:28,740 largest was about 25 people. So that is a big team. Um, and that did a lot that did allow us to move 171 00:15:28,740 --> 00:15:34,580 faster. Yeah. And, and talk a little bit more about the importance of being able to develop teams 172 00:15:34,580 --> 00:15:39,620 to develop independently of one another. Uh, obviously that's a key component of the whole microservices 173 00:15:39,620 --> 00:15:44,740 architecture, one of the things that supports and why is that so important, uh, for helpful? 174 00:15:46,500 --> 00:15:50,980 Yeah. So I think there's a kind of a couple of ways to think about it. One is, uh, and, and this, 175 00:15:50,980 --> 00:15:55,140 you know, I've had this debate with startups all the time is should you, or should you not, uh, 176 00:15:55,140 --> 00:15:59,300 not adopt microservices when you're building something from scratch? Uh, and then the second 177 00:15:59,300 --> 00:16:04,740 part is, is how do you have teams grow in a way that at a minimum can grow linearly? Like, 178 00:16:04,740 --> 00:16:10,820 so you add two engineers, you get two more engineers worth of output versus what we see in other, um, 179 00:16:11,780 --> 00:16:16,020 team structures is you add people and you get more, but you don't get a full 180 00:16:16,020 --> 00:16:20,580 engineer's output for every person you add, right? Because it asymptotes at some point or approaches 181 00:16:20,580 --> 00:16:26,100 some, um, you know, some, some ratio where you get like 0.2 of somebody or something. And so on the 182 00:16:26,100 --> 00:16:33,940 first question, what we realized that helpful was that if you can create a, an automation, 183 00:16:34,660 --> 00:16:39,460 which allows you to create microservices fairly easily, right? So you can literally like, you know, 184 00:16:39,460 --> 00:16:43,540 type in one command and you can create the stub of microservice. It's automatically, you know, 185 00:16:44,340 --> 00:16:47,380 it's easy to find it connects to the registrar, like whatever you need to do, 186 00:16:48,020 --> 00:16:53,860 then creating microservices is super easy and it encourages people to do that. And the reason we 187 00:16:53,860 --> 00:16:59,140 did it that way versus trying to build something in a monolith first and then try to break it out was 188 00:16:59,140 --> 00:17:03,780 we realized that aggregating microservices is quite easy. So if you, you know, if you were going to have 189 00:17:03,780 --> 00:17:08,820 something be one giant monolith, uh, and then break it out when it, when it, uh, made sense to break it 190 00:17:08,820 --> 00:17:13,860 out versus it's already a microservices and it's 10, let's say microservices and you could just 191 00:17:13,860 --> 00:17:17,460 aggregate eight of them together because they all belong together. The aggregation step is quite easy 192 00:17:17,460 --> 00:17:22,180 and you know, it's not a lot of work to do that, but the breaking apart the monolith typically requires 193 00:17:22,180 --> 00:17:26,580 large swaths of like refactoring time. And so we went the other way. We went like microservices first 194 00:17:27,300 --> 00:17:33,460 in for the second question around how do you maintain, uh, growth and, and, uh, efficiency and effectiveness 195 00:17:34,020 --> 00:17:40,020 was by breaking up components of the T and maybe get the team into components that could go as fast 196 00:17:40,020 --> 00:17:45,140 as they wanted to on their own. That allowed us to move more quickly. So for example, in the microservices 197 00:17:45,140 --> 00:17:52,260 terminology, if you had a pair that owned, you know, five or six microservices and those microservices were built in a way that 198 00:17:52,260 --> 00:17:56,180 to not be dependent on other microservices, like how you're supposed to architect it, then they can move 199 00:17:56,180 --> 00:18:01,380 really, really quickly because they're not worried about, um, running into someone else's, uh, 200 00:18:02,500 --> 00:18:06,260 code infrastructure or having to, having to deploy and saying, oh, I'm deploying this, but 201 00:18:06,260 --> 00:18:10,980 that test from that other service deploys is a requirement for my service to run correctly. 202 00:18:10,980 --> 00:18:16,740 Like that infrastructure problem doesn't happen. And so now you're always thinking about how do I test 203 00:18:16,740 --> 00:18:21,460 my microservice? How do I deploy my microservice quickly? How do I deploy functionality just for my microservice? 204 00:18:21,460 --> 00:18:25,300 And then you're independent. You're like on a completely different team and your API, 205 00:18:25,300 --> 00:18:31,140 your API to everybody else's rest. So you're not really focused on, um, having to worry about what 206 00:18:31,140 --> 00:18:34,740 other people are doing from that perspective in terms of running, uh, running into roadblocks. 207 00:18:34,740 --> 00:18:38,820 Hmm. So it sounds like, sure. It lets, it lets each team develop at their own pace, 208 00:18:38,820 --> 00:18:44,180 but it also lets them focus. Yes. Yeah. And they, I mean, we did, we did a few things, 209 00:18:44,180 --> 00:18:50,340 right? One was we rotate pairs often. So yes, at the time you're focused on your infrastructure, 210 00:18:50,340 --> 00:18:54,180 your services, but because we rotated, you're getting a view of the entire system. 211 00:18:54,180 --> 00:18:58,660 Um, but yes, any, any, any individual week, like we didn't, we don't rotate as much as 212 00:18:58,660 --> 00:19:02,900 Pivotal likes to rotate, which is daily. We would rotate, you know, more weekly, sometimes even 213 00:19:02,900 --> 00:19:07,620 monthly, but that gave people enough of a view of the system to understand what's going on and 214 00:19:08,500 --> 00:19:13,140 a deep, a focused enough view of the system over weeks at a time to then be able to move really, 215 00:19:13,140 --> 00:19:14,180 really quickly on those services. 216 00:19:15,060 --> 00:19:19,380 Yeah. So what else did you, did you look at or, um, or, or not? 217 00:19:19,380 --> 00:19:23,460 So when we were, yeah, basically in 2016, when we realized, you know, you realize you need to back 218 00:19:23,460 --> 00:19:28,900 in the first, my first thought was actually Amazon EC2, just because of, you know, probably brand, 219 00:19:28,900 --> 00:19:33,060 brand recognition. And it was my engineer who said like, why you want me to, you know, 220 00:19:33,060 --> 00:19:39,540 you want me to manage servers now. But, um, outside, outside of EC2, um, you know, 221 00:19:39,540 --> 00:19:44,020 I think in my mind, anyway, I taught about, I thought about digital ocean thought about, 222 00:19:44,020 --> 00:19:49,380 obviously there's GCP didn't spend any time on Azure. I think we did look at Heroku, um, 223 00:19:49,380 --> 00:19:53,220 just at a high level, mostly because my co-founder's company got bought by Salesforce. 224 00:19:53,220 --> 00:19:58,580 And so there's a Heroku connection there. Um, but outside of that, um, we, you know, 225 00:19:58,580 --> 00:20:04,740 I think our idea was, well, let's see if PWS can work for us as a startup. And so by jumping 226 00:20:04,740 --> 00:20:09,460 into PWS early and having all the connections there from a support perspective, 227 00:20:09,460 --> 00:20:15,300 um, we never had to leave for, you know, we're coming on three years now on, on PWS. And we were 228 00:20:15,300 --> 00:20:19,860 able to, you know, pretty much stretch it into different areas that we require. Now, there are 229 00:20:19,860 --> 00:20:23,700 times when there isn't a service in the marketplace or something like that, that we have to go direct. 230 00:20:23,700 --> 00:20:29,620 But for us, the, um, the best way for us to use PWS is to use the services that are then in the 231 00:20:29,620 --> 00:20:37,140 marketplace, because that allows us to have this, um, very low friction way to adopt things that 232 00:20:37,140 --> 00:20:42,660 potentially would be harder to do when not in PWS. So we always check the marketplace first. And then, 233 00:20:42,660 --> 00:20:48,660 um, if it, if it's not there, we'll then do an external connection, but we've been able to move as 234 00:20:48,660 --> 00:20:56,180 fast as we have scale as quickly as we have scaled and, um, build new products, um, that are quite different 235 00:20:56,180 --> 00:21:04,100 from each other all on PWS, uh, very successfully. And I think it's one of our competitive advantages, 236 00:21:04,100 --> 00:21:08,980 uh, versus other companies, because there isn't, uh, any infrastructure we have to manage 237 00:21:08,980 --> 00:21:13,140 that is, uh, you know, the undifferentiated heavy lifting that we don't have to do. 238 00:21:13,140 --> 00:21:17,140 And so we're very easily like, we like literally, I see, sometimes I go to my engineer and I'll say, 239 00:21:17,140 --> 00:21:20,180 Hey, so we have this big, you know, I was telling you weren't their domain, but we have 240 00:21:20,180 --> 00:21:25,780 some pretty big celebrities on our dialogue platform. Um, Jake Tapper, Ben Stiller, Greta Van 241 00:21:25,780 --> 00:21:29,220 Susteren. Yeah. And so sometimes we have these big shows and we're like, Hey, we need to, you know, 242 00:21:29,220 --> 00:21:32,260 ramp up the infrastructure. And my engineer looks at me and he goes, yeah, that just means pressing 243 00:21:32,260 --> 00:21:36,580 buttons. Like I just have to press some buttons. Like he like, right. Like I don't have to do any code. 244 00:21:36,580 --> 00:21:39,300 I'm like, Oh, don't you have to do it? He's like, no, no, I just got to press some buttons. 245 00:21:39,300 --> 00:21:43,940 And so, um, and now with even auto scaling and things like that, we don't even have to do that. 246 00:21:43,940 --> 00:21:49,220 Yeah. Okay. Um, and, and so in terms of how that becomes a competitive advantage is, 247 00:21:49,940 --> 00:21:55,780 you know, every person that you're able to bring on in engineering is able to work on new code, 248 00:21:55,780 --> 00:22:00,420 updates, product oriented work, as opposed to the undifferentiated heavy lifting. 249 00:22:00,420 --> 00:22:04,100 Yeah. And I think, I mean, there's a couple of things going on. One is we bring on lots of smart 250 00:22:04,100 --> 00:22:08,820 folks who may, may not necessarily know about cloud native computing. And so at the, at the beginning, 251 00:22:08,820 --> 00:22:12,740 it's a little bit of, you know, repair, so it's not so hard, but it's, it's an introduction to like 252 00:22:12,740 --> 00:22:17,380 the infrastructure. But once they see the power and they realize that they don't have to do things 253 00:22:17,380 --> 00:22:20,980 that they potentially had to do in other jobs, they're like, wow, this is actually quite fast. 254 00:22:20,980 --> 00:22:24,660 Like, I don't have to worry about all these things. Um, I think during the last three years, 255 00:22:24,660 --> 00:22:28,820 right, you had, you know, a heart bleed and other big security vulnerabilities come across. 256 00:22:29,380 --> 00:22:33,300 And all that happened was I got an email from PW saying like, you know, uh, you know, 257 00:22:33,300 --> 00:22:38,260 redeploy your applications. Like there wasn't anything else that we had to do. And, you know, 258 00:22:38,260 --> 00:22:42,020 given that we work with a lot of big enterprises and we go through rigorous security reviews with them, 259 00:22:42,020 --> 00:22:47,700 um, to go back and they would specifically ask us about those, um, vulnerabilities. I'm like, 260 00:22:47,700 --> 00:22:51,620 oh yeah, it's, it was, it was taken care of by the platform. And I could even other forward 261 00:22:51,620 --> 00:22:57,380 them the email I got from P dubs or, um, or just show them that it was certified fixed from an 262 00:22:57,380 --> 00:23:03,140 infrastructure perspective. Um, again, as a startup with not, you know, not thousands of developers, 263 00:23:03,140 --> 00:23:06,980 right. Um, we're not focused on that. It's like, yeah, I just don't have to worry about it. It's 264 00:23:06,980 --> 00:23:09,860 like, yep, that's just taken care of. Right. We were actually talking about it last week. I can't 265 00:23:09,860 --> 00:23:11,940 remember what it was, but it was, there was something else. And I said, oh, is that just 266 00:23:11,940 --> 00:23:13,620 a CF push? Like I don't have to do anything else. Right. 267 00:23:13,620 --> 00:23:19,300 Yeah. Okay. I mean, and so what, what do you do when, when there's something that's not on the 268 00:23:19,300 --> 00:23:23,300 marketplace where, where kind of the edges or areas? And then I guess kind of on a related 269 00:23:23,300 --> 00:23:31,860 question would be, what advice would you have for another startup out there in looking at P dubs, 270 00:23:31,860 --> 00:23:36,100 getting started? What do you, what would you recommend to them to, to just make that smoother? 271 00:23:36,100 --> 00:23:40,980 Yeah. So, I mean, I think the first thing, I mean, Leila asked the second question first is that, 272 00:23:40,980 --> 00:23:46,100 um, like you want things to be as boring as possible, right? On the infrastructure side. 273 00:23:46,100 --> 00:23:49,300 And I think that's where people get caught up. Cause you're like, I want to try the newest thing. 274 00:23:49,300 --> 00:23:54,420 And for me, it wasn't about the newest thing. Like we weren't trying to like use like the latest in 275 00:23:54,420 --> 00:23:57,380 containerization technology. It was like, it's like, what's the most, like, what's the thing that allows 276 00:23:57,380 --> 00:24:03,220 us to focus on the work, right? Not the undifferentiated work. And so, um, it's pretty boring 277 00:24:03,220 --> 00:24:06,900 around here, infrastructure wise. And maybe that's not as fun as elsewhere. We're like, oh, 278 00:24:06,900 --> 00:24:10,100 the thing went down and I got to bring it back up and like, I can put a security patch. And like, 279 00:24:10,100 --> 00:24:14,740 it's always like nerve wracking when you do an upgrade. So from that perspective, we weren't 280 00:24:14,740 --> 00:24:19,540 necessarily trying to be heroes on like the latest and greatest thing. We were just trying to find 281 00:24:19,540 --> 00:24:26,020 the thing that allows us to focus on features, talking to customers, getting feedback, iteration, 282 00:24:26,020 --> 00:24:31,780 like the, the, the actual value, right. Versus like the, the thing that had no value. So that was, 283 00:24:31,780 --> 00:24:35,540 that was the, you know, your answer to your second question. On the first one, there were times when 284 00:24:35,540 --> 00:24:41,300 we're doing something specific that may or may not fit in directly with what we need from, um, PWS. 285 00:24:41,300 --> 00:24:45,540 And I don't think it's a, like a miss. It's just something very specific. So an example would be, 286 00:24:45,540 --> 00:24:51,460 um, when we were doing the video infrastructure, we were sending, you know, thousands and I think, 287 00:24:51,460 --> 00:24:57,860 you know, tens of thousands of videos a week. And that necessarily required us to have some fast 288 00:24:57,860 --> 00:25:02,900 storage access and the fast storage access isn't necessarily something that's going to go through 289 00:25:02,900 --> 00:25:08,020 a P dubs, just given that, um, it didn't make sense to go through P dubs. And so we, you know, 290 00:25:08,900 --> 00:25:14,980 spent time with Amazon S3, spent time with GCP file storage, uh, to figure out and benchmark the 291 00:25:14,980 --> 00:25:21,540 things that we needed to do to make those as quick as possible for our customers. And, um, 292 00:25:21,540 --> 00:25:25,460 so those are just direct connections to those things. Um, and that's typically what you end up 293 00:25:25,460 --> 00:25:30,020 doing when you have something that in that case was like a high bandwidth, low latency request. 294 00:25:30,020 --> 00:25:33,860 You didn't want you, you didn't want to have any more hops, um, that were required. 295 00:25:34,580 --> 00:25:38,420 Yeah. And are you still able to make those direct connections that, that you've had to build 296 00:25:38,420 --> 00:25:45,220 yourself something that's easy to maintain? Well, not as I mean, everything, our first choice is always 297 00:25:45,780 --> 00:25:50,020 go through the marketplace because it's easy. It's, it's much, much easier to maintain there. 298 00:25:50,020 --> 00:25:54,500 And so that is our, our preference always actually, there have been times when I think we worked on 299 00:25:54,500 --> 00:25:58,100 something that was not in the marketplace and then it came into the marketplace that we would do that 300 00:25:58,100 --> 00:26:04,180 refactoring, um, because it made our lives easier again, like the more you can do in the infrastructure, 301 00:26:04,180 --> 00:26:10,980 the better. Um, so it's never as good as having it be in the marketplace. So we try to keep up to 302 00:26:10,980 --> 00:26:15,460 date with what's there, but, uh, it doesn't mean, I think the other way to think about it is you don't 303 00:26:15,460 --> 00:26:19,140 have to just use everything in the marketplace. You can use anything and, and you don't have to be 304 00:26:19,940 --> 00:26:23,380 stuck with only what's in there, but it is more to maintain when it's outside. 305 00:26:23,380 --> 00:26:28,340 When you're talking about how things are really boring from an infrastructure perspective, 306 00:26:28,820 --> 00:26:32,500 um, but that's a good thing. And maybe this is something you already talked about. What are 307 00:26:32,500 --> 00:26:37,460 the kinds of things that you, you do celebrate? What are the metrics, the kinds of things that 308 00:26:37,460 --> 00:26:42,500 you do want to see? Like this is, this is where it shouldn't be boring. Like infrastructure should 309 00:26:42,500 --> 00:26:47,460 be boring. Great. We're really focused on product. We're really focused on our customers. 310 00:26:47,460 --> 00:26:51,620 Therefore, this is where the exciting stuff happens. How do you measure and look at that? 311 00:26:51,620 --> 00:26:56,660 Yeah, sure. So, I mean, I think the way I mean, think about our newest product dialogue, 312 00:26:56,660 --> 00:27:03,700 right? Um, the things that we don't worry about are, um, scale. So lots and lots of, 313 00:27:03,700 --> 00:27:07,300 you know, it's an audio product, so lots and lots of listeners, we don't worry about there being too 314 00:27:07,300 --> 00:27:11,380 many listeners. Um, lots and lots of shows. We don't worry about lots of people creating shows. 315 00:27:11,380 --> 00:27:17,460 Um, we do, we actually do load tests on these things also to see that the infrastructure can scale to, 316 00:27:18,020 --> 00:27:22,420 you know, five to 10 X what the current load is. So we don't have to worry about the re-architecture 317 00:27:22,420 --> 00:27:27,140 until the next phase. Um, but that's what makes it boring. Like it's pretty, it's, it's, we're not 318 00:27:27,140 --> 00:27:30,900 even looking at those things. It makes it, uh, very easy to focus on things that do matter. So in our, 319 00:27:30,900 --> 00:27:35,940 in our case, the things that do matter are, you know, when, um, we have a show with Jake Tapper, 320 00:27:35,940 --> 00:27:40,340 how many live listeners are there? How many are supported? How many people are trying to call in? 321 00:27:40,340 --> 00:27:44,260 How many shows do we have per day and per week? Like all this, all the business metrics that we 322 00:27:44,260 --> 00:27:48,980 care about are things we can actually concentrate on versus having to worry that will the infrastructure 323 00:27:48,980 --> 00:27:55,060 support it? Is it going to be up? Is the latency too high? Um, is the user experience, 324 00:27:55,060 --> 00:27:59,140 are they experienced extreme slowdowns? Like we don't worry about the things that, um, you know, 325 00:27:59,140 --> 00:28:02,420 I would call the non-functional requirements. The things you assume are already there, 326 00:28:02,420 --> 00:28:04,980 but only when you run into them, you're like, oh crap, we have to do work to get that. 327 00:28:05,700 --> 00:28:09,860 Uh, we don't, we just focus on the end user. And those are the metrics we report to our customers. 328 00:28:09,860 --> 00:28:12,420 Like, hey, here's how many listeners you had. Here's how many people listen to it afterwards. 329 00:28:12,420 --> 00:28:15,060 Here's how many shows were created per week. Here are the top shows of the week. Like all the 330 00:28:15,060 --> 00:28:20,500 things you would imagine as a business user. Um, we never send things like six nines in reliability, 331 00:28:20,500 --> 00:28:25,940 latency, like the things that people, people should not care about. We never report them. 332 00:28:25,940 --> 00:28:29,380 Like I actually don't even know what they are because we don't care. It's a non-issue. 333 00:28:29,380 --> 00:28:37,620 Okay. Um, so, and you know, what's, what's next looking ahead, uh, where are you looking to sort 334 00:28:37,620 --> 00:28:42,820 of expand or, or build from either a technology perspective or product perspective, whatever you 335 00:28:42,820 --> 00:28:49,460 want to kind of get into? Yeah. So what's amazing about dialogue and what an audio in general is 336 00:28:49,460 --> 00:28:55,860 it's a, it is a growing, um, it's a growing marketplace of content, right? So, you know, 337 00:28:55,860 --> 00:29:00,340 I would say in on the internet, right, you, of course you have text, text is everywhere, text is 338 00:29:00,340 --> 00:29:06,180 growing. Um, but you have like video and audio, which are now because of, you know, better hardware, 339 00:29:06,180 --> 00:29:12,580 better networks, social norms and behavior. People want to share things that are more compelling. 340 00:29:13,300 --> 00:29:18,660 And so with audio in specific, it's a very, very fast growing medium, right? It probably 341 00:29:19,860 --> 00:29:26,660 a combination of AirPods, podcasting, audio books, all growing at the same time, 342 00:29:26,660 --> 00:29:30,420 and then social norms changing so that it doesn't look weird when you see people walking around with 343 00:29:30,420 --> 00:29:32,820 talking, looking like they're talking to themselves, but they have AirPods in or 344 00:29:32,820 --> 00:29:39,620 Bluetooth headset or earphones. Um, and that makes it a place where, you know, audio can be consumed in 345 00:29:39,620 --> 00:29:44,580 so many more places than video can, for example, or even text. Um, I take the streetcar or subway or I 346 00:29:44,580 --> 00:29:50,420 walk and I'm always got a podcast on. So from that perspective, it's a growing medium. And what we, 347 00:29:50,420 --> 00:29:56,740 like I mentioned, what we're focused on is making it interactive. And we think that the things that we 348 00:29:56,740 --> 00:30:03,060 have to build in order to make that more compelling for folks are deeper, enriching experiences. So for 349 00:30:03,060 --> 00:30:08,980 example, right now our entire platform's on the web, it's mobile web, it's desktop web. And with lots of 350 00:30:08,980 --> 00:30:15,700 people say, I really want, I really want this to be a mobile app because then I can do, um, things that 351 00:30:15,700 --> 00:30:19,780 I couldn't do in the mobile web, get notifications, potentially subscribe to different people. Like 352 00:30:19,780 --> 00:30:24,100 there's lots of different UX models that can be built. And so then for next for us, it's probably 353 00:30:24,100 --> 00:30:29,700 interacting with our backend in a slightly different way, but via a mobile app. There's probably a few 354 00:30:29,700 --> 00:30:33,460 things that like, you know, we're recording this on Zencaster. I want to be able to have high quality 355 00:30:33,460 --> 00:30:39,300 audio broken down by audio stream. We assemble those, um, so that it's a higher quality experience 356 00:30:39,300 --> 00:30:44,260 for the, for the recording. So there's all sorts of things that we can do to make the experience even 357 00:30:44,260 --> 00:30:51,540 more compelling and more interactive, um, for folks so that we are able to, um, really, really bring 358 00:30:51,540 --> 00:30:55,380 audio everywhere for those, for those people and make it live and interactive, which you can't do right 359 00:30:55,380 --> 00:31:00,020 now on podcasting and audio books. Right. I'm a little terrified what that would look like though. 360 00:31:00,020 --> 00:31:06,740 If what terrified, terrified how, uh, we'll just, you know, here we get to make up all of our own, 361 00:31:06,740 --> 00:31:16,340 um, uh, left field analogies or, um, examples and yeah, you know, it would just be, it would 362 00:31:16,340 --> 00:31:20,580 just be a little bonger balls to have like anyone in the world kind of being like, why are you talking 363 00:31:20,580 --> 00:31:24,820 about margaritas? It's a tech, you know, podcast. Yes. Yeah. Well, I mean, I think, but I think 364 00:31:24,820 --> 00:31:27,300 that's, that's what's, that's what's interesting about the content. He literally hit the nail on the 365 00:31:27,300 --> 00:31:32,580 head. Even though if it's, even if you listen to it afterwards as a recording, live content is 366 00:31:32,580 --> 00:31:39,060 different because like you said, anything can happen. Right. And what we're seeing is that 367 00:31:39,060 --> 00:31:43,940 because of that, anything can happen attitude. And for these people with large followings to 368 00:31:43,940 --> 00:31:47,620 want to interact with their audience, not just talk at them. Right. If you're talking on TV 369 00:31:47,620 --> 00:31:51,780 or even on a podcast, you're just, you're talking to people and you can't get feedback. They can't 370 00:31:51,780 --> 00:31:56,580 talk to you back. And that's where literally the idea came from us was we were listening to all these 371 00:31:56,580 --> 00:32:02,260 great people over podcasts and like, Oh, I wanted to, I want to respond to that. And I can't. And it 372 00:32:02,260 --> 00:32:10,260 does make, yeah, right. It's you yelling at the TV and instead, um, and you do see this happening all 373 00:32:10,260 --> 00:32:14,020 the time, like product on called it Twitter for voice because you do see this, it's just on Twitter. 374 00:32:14,900 --> 00:32:19,380 You don't see it in any other way. Right. Um, and I think that's, uh, that's what's going to change. 375 00:32:19,380 --> 00:32:24,180 Okay. Yeah. No, that's me kind of shouting at the crowds as I walked down the street. 376 00:32:26,020 --> 00:32:30,420 Okay. And then we'll have to like, um, actually, instead of just all capsing when we're trying to 377 00:32:30,420 --> 00:32:32,420 yell on Twitter, I'm just like, actually yell. 378 00:32:32,420 --> 00:32:35,780 Actually. Yeah. Well, and actually it turns out because we're human, right? Like that 379 00:32:36,500 --> 00:32:41,620 you don't usually in react the same way when you're, when you've connected with someone over 380 00:32:41,620 --> 00:32:48,260 voice that you would over text, right? Like there's a, there's a podcast where this person calls 381 00:32:48,260 --> 00:32:52,340 everybody that was yelling at them on Twitter. And the first thing they always say, they're like, 382 00:32:52,340 --> 00:32:56,020 they say, Oh, I'm sorry. Like I didn't, I didn't know. Like they, that's the first thing they say, 383 00:32:56,020 --> 00:33:01,140 because you're talking to another human versus in text, you can just like all caps and like WTF. 384 00:33:01,140 --> 00:33:06,100 And I don't understand. Right. You can do that. And with, with voice and connecting audio, 385 00:33:06,420 --> 00:33:09,940 it's much more civilized, but even without video, even just voice, it changes things. Like when 386 00:33:09,940 --> 00:33:14,500 somebody calls you, it's a big difference. Well, then, um, clearly we're just going to be 387 00:33:14,500 --> 00:33:20,740 cleaning up the world and making it a better place. Uh, so thanks so much for, for joining us 388 00:33:20,740 --> 00:33:26,580 and taking the time. Um, where can folks kind of learn more and follow you and, um, just continue 389 00:33:26,580 --> 00:33:30,820 their, their conversation with you? Sure. So get dialogue is at 390 00:33:30,820 --> 00:33:36,260 uh, I just said it get dialogue.am. That's the name of that's the URL where dialogue is hosted 391 00:33:36,260 --> 00:33:41,780 and people can follow me on Twitter. It's just, uh, F N as in Nancy and my last name flower, 392 00:33:41,780 --> 00:33:56,100 T H A W A R at on Twitter. Okay. All right, folks. Thank you so much.