We are in Austin, and we're lucky enough to have someone to talk with. Would you like to introduce yourself? I am Luke Knies. I'm the founder of Reducted Labs, and I'm the principal author of Puppet, which is a server automation tool. And before that I was doing consulting and other work in system administration. So would you say you're sort of in the IT management provisioning area, open source sort of thing? I don't like the word provisioning because that implies I provisioned the machine and then I went home. And really Puppet is about managing the entire life cycle. More like change management maybe then. What word or phrase do you like? I don't think there is one that I like. I spent the last three years trying to come up with a good term. The best term I've come up with so far that I kind of like is meta infrastructure. I produce the infrastructure that produces your infrastructure, that manages your infrastructure. So Puppet is the thing behind everything else. That's the best I can come up with. So then what does Puppet actually do? You explain to Puppet what services you want, how you want those services to look. I've got these packages, these processes, they have this relationship. When this happens I want these to react. When this configuration file is updated I want the service to react in this way. So you explain all these things to Puppet and then Puppet makes sure all your machines look that way. Well that's a mail server, it's a Solaris server, it's in this data center, which means it acquires these traits. So Puppet will configure those traits with all those resources. And as you change those definitions over time, as you update the definition of a mail server, or as you update what it means to be a Solaris server, Puppet will update all those machines that match those criteria. So you don't have to go to those machines and say, oh by the way, here's a patch up, here's an upgrade, or, you know, update this configuration file and restart the service. Right. Puppet just does that. Yeah, it sounds like it's sort of, I don't really know what you call this, but there's this sort of part of IT management where it's kind of model driven. Right. Where you establish a model and then you hand it over to a tool to implement that model or to sort of make sure that model is executed properly. Right. Rather than necessarily directly modifying scripts and things directly.
2:01And executed is the right word, like I always talk about having executable content. This is your, you're basically speaking to Puppet, you're writing, you're writing code in Puppet. Right. And that, it is executable code, it's an executable configuration. You can write security policies in Puppet and they're executable. So everyone, you know, most companies have this Word document or this Excel spreadsheet that says, here are the things that must be true for us to be secure. And then they have some dude whose job it is to SSH to each of the machines and run a bunch of commands to see if it's true. Right, right. And that's, you know, it's not true. Whereas if you have an executable security policy or an executable configuration policy, then it's always true, you know, within reason. What is that process like with Puppet as far as, I mean, I'm assuming you sort of start with that document or that model. I mean, what does it look like to the full life cycle of the model? So what I do with customers, as an example, so one of the services I provide is I come in the door and I provide some combination of training and implementation. And the goal upon leaving is they'll have at least one person who really knows the product well and they'll have a basic implementation. And what we do is we start with a kernel and you say, okay, well, the first thing we're going to do is we're going to build our Puppet server. So we define a Puppet server class and we flesh out what that means. Well, it means that I've got these packages, I've got these configurations, and I've got these services. And so what you do is you define these resources, you specify them all, and you say, you wrap those resources in a class, and then you define this node. Okay, this box is my Puppet server, so its name is blah, its name is Johnny, and include the Puppet server class in it. And then you run Puppet on that machine and it goes, ah, and it goes, compiles its configuration, which results in this list of resources. And the resources are the things that we already think about but we can't usually talk about directly. Users, packages, groups, services, hosts, cron jobs. The things that we think about but are obscured by file formats or weird commands or things like that. So you just talk about these entities to Puppet, compile that into configuration, and then it handles that for you. In terms of the actual user process, you're usually doing something like you've got an SVN repository or my last client, we used a Git repository.
4:04You make your changes in, you know, you check out that repository, make your changes, you commit that change, and then it works its way. The server automatically pulls the changes from the repository, and then the clients, by default every half an hour, recompile their configuration, reapply the configuration. The data flow looks, you have your idea, you commit that idea, you turn that idea into code, commit that code to the repository, the server pulls it down, and the clients pull it from the server. And there's a client on each machine that's going to be configured, right? Exactly. So there's sort of a little agent-based network with a central brain in the middle. Indeed. I imagine there's a bunch of stuff to sort of normalize the different ways that these machines and servers get configured, right? Exactly. In the sense that a person expresses a mail server should look like this, and that could mean different things depending on which OS or Linux distribution or whatever you have on there. Right. So Puppet has two ways to talk about portability. At the lowest level of portability, the resources themselves are built to be portable. So when you talk about a package or a user or a service, you can talk about it in the same way on FreeBSD or Solaris or Gen2 or Red Hat. You can just say a package, and Puppet will say, "Ah, well, I'm on Gen2, so you're talking about Portage," or, "I'm on Solaris, so you're talking about PackageAd." So that's the lowest level of portability, and so that isolates you from needing to know system detail. What commands you need to know and things like that. But then the higher level is the language itself provides a lot of portability features. So, and this is one of the really big frustrations I had in earlier generations of tools. What do you think the SSH service is called? You know, everyone, you can go to almost anyone in the sysadmin world and say SSH, and they go, "Yes, I know what you mean." But then they go to their platform and they go, "Well, what's the package called?" Well, on Debian it's called OpenSSH and OpenSSHServer, and on FreeBSD it's called SSHD. And so you have all these package names that are different, and then the services themselves have different names. And then the configuration file is stored in a different place. But for you, what you have is, "I've got a package, it's got a configuration file, and there's a service." Right. And the service depends on the package and the configuration file, and if those change, I want the service to get restarted.
6:07So, you think about it in one way, and you need to be able to explain to your system, "When I say SSH, I always mean this." Right, right, right. So the puppet language provides the facility to say, "Here's a thing that I think of as SSH, but FreeBSD thinks of it as this, and Solaris thinks of it as that, and, you know, Debian thinks of it as another thing." You've got to have, like, the distribution translator for-- Exactly. The language itself has portability features to provide this different naming across all the different platforms you have to deal with. Right. So-- And is there, like, a puppet language or something? There is a puppet language. And what's that like? It's probably--the best way to describe it is it's some combination of--I mean, it's heavily influenced by--the functionality is heavily influenced by CFEngine. It looks a little bit like Ruby and a little bit like--almost like the Nagios language, in fact. The resources are as syntax-free as I could get. You have a resource type, a resource name, and a bunch of attributes. Yeah. And that's the thing. And then those are collected into classes. And if all your--if you don't have any selections, if you don't have any portability problems, it looks just like that. Right, right. As you get more complicated, as you have to have sub-classes and things like that, you get a little more complication, but it's a very simple and not very powerful-- So would you call it, like, a DSL, a domain-specific language? It is exactly it. And what are you--did you, like, implement your own parsers and everything for it? I have a real, like, Yak-style parser, and I have my own Lexer, and I've got a parser generator that I--you know, so the whole thing, yes, I do. Right, right, right. And this is actually--this is interesting because this is about my fourth parser. It's the first parser I've written that I actually created an abstract syntax tree. Oh, you did it, like, correctly. Yeah, exactly. So when the compiler--when you run the compiler and you parse the thing, you build this parse tree, you parse it once, and every client that connects uses the same parse tree. So you can go through and parse once and compile many times. So it's very interesting. It was a very big learning experience for me, definitely. Well, that's exciting. Puppet is open source. Puppet is. And I wonder, it's not so much a question of why is it open source, but what sort of benefits and un-benefits have you gotten from it?
8:09How has it been working out that it's an open source project? It's been really great marketing-wise. I mean, this is--I think a lot of people realize that open source is great for marketing, but they don't know what that really means exactly. Right. But I've been able to give talks in five countries in the last year because it's open source. Almost all the talks, you know, I've been working on OSCON, Linuxcon Australia, FOS.IN in Bangalore, FOSDAM in Belgium, these kind of things. You don't get invited to these talks if you're a commercial product. Right. And you don't get people--you know, there's apparently people wandering around Europe talking about how great Puppet is. They aren't going to do that if they don't feel like they're part of a community, right? Yeah. So they're not going to--there's a guy who's got his entire Puppet configuration. He works very hard on it and he publishes the entirety of it. He wouldn't do that if it weren't an open community. Right. And so having it be that open and having people know that they can trust that the product's going to be there. And that even if I go away, yes, I'm one small dude. I'm one guy and it's a small project. But if I go away, somebody else can take my code and move on with it. And so there's that kind of general benefit you get. Because I am a commercial company behind this product, I'm not--you know, commercial is the wrong word. I'm trying to make money on producing open source. Being open source has its own limitations. The business model is harder. But it also--it's really easy to market open source people, but they're open source people. They're used to doing it themselves. Right. They need your help. So I've gotten a very good distribution across the open source world. And they're great. They're great people. They're really helpful. They invite me to conferences. And they work really hard on the product and on the community. It seems like one of the benefits, and this isn't the only thing, but essentially your marketing costs are zero, if you will. Or the initial boost traffic. Well, the flight here wasn't exactly true, but yeah. It's a lot more than just marketing, too, because you get all these people who are working on documentation. Right. I've been able to pass most of the documentation efforts and a lot of ticket maintenance efforts off to the community. I've got people who--I don't have to have, you know, tier one support people in the forums answering questions, because I have people who use the product every day and are in the forums answering questions. And so that really makes a difference. I mean, since we're kind of on this topic is--I see a lot of when the community attacks, so to speak, to borrow a name from an old South by Southwest panel.
10:13How--whether it's specific to your community or you want to generalize, like, how rich do you think you can get off of the project before--like, what's the limit where the community starts to think that you're kind of ripping them off? I have to say that I haven't tested the limits of that very much, to put it realistically. That's a real question that I struggle with all the time, because you--I do want to monetize this, but I want to do a lot more than monetize it. I want to change the way people think about computers. I mean, I was just on this panel that you were--I was on with you, and I want every operating system in the world to either be using Puppet or using something that was inspired by Puppet. Right. I'm embarrassed with the way things work. The only way to get that kind of change, the only way to get that kind of reach, is to get big and to get everyone talking about me and talking about the software. And that kind of inherently means acquiring power and acquiring wealth and acquiring more people working for me in a larger organization. And as that changes, I think there is a lot of concern. The ways that I'm planning on allaying that are trying to avoid--there are a lot of ways you can do that incorrectly. You can start being concerned about whether--so I was talking to another open source company, and they were saying, "Well, we're open source, but we require patches to have their copyright assigned over to you." So I'm not going to pick a business model that requires that I own the copyright for all the code that gets committed to Puppet. I don't think that makes sense. I don't think that the community will respond well to that. I also don't want to pick a business model that encourages me not to release some stuff. So I want to find a business model that more openness makes more money for everyone. Right. And I think one of the other ways, too, is that I want to make other people in the community make money, too. So I think there's money to be made. I think that there's a space that isn't visible. That if you're outside right now, if you're in the systems world, it's not obvious that there's money to be made. You can't--if you are the best sysadmin in the world right now, you literally--there's somebody out there who is the best. Do we know who he is? Is he making more money because of that? Right. Does he get speaking engagements because of that? He doesn't. It's not the same as in programming, where there's a very easy-to-spot rock star system.
12:13Right. Brian Bellendorf is sitting right over there, and everyone knows him because he's done these great things. There's a Brian Bellendorf in the systems world who's drudging away someplace because he can't be treated like a rock star. So how can I build a community that, yes, it's going to make me money? I certainly hope so anyway. But how can I make it to that sysadmin, that rock star, can also make a lot of extra money? He can get $10,000 speaking engagements. So I think that's one of the ways that you navigate that is that you make it clear that you're not the only one who's making money. Right. You're putting the whole thing up with you. Right. It seems like, and it's hard to articulate this without seeming like a non-cynical idiot, right? But basically that you sort of treat success like a tool in order to get a grander sort of goal. And it seems like the successful open source people and companies that maintain the goodwill of the community, that seems to be what they do. Instead of saying, "Oh, now we've cashed out." They realize that they can't really just be the one person that's going to change everything. They need to build up capital, if you will, in order to do things. And the thing that's great about open source is the movement, is the churn. Right. And if you draw a line and say, "These people are getting rich and these people aren't," then you lose that churn. Because there's this step up. Right. And you can step into the people who are making money or step back out. And you don't want that. You want this nice gradient of, "I can step in, start doing some things, start helping a little bit. I'm learning, I'm producing." And then, over time, I get a reputation. It's easier to get a job because people know me. You know, it's just starting to the point where your reputation in the puppet community can help you get a job. Right. In a year, I think your reputation in the puppet community is going to be likely to get you a job. In two years, I hope people will be building jobs based on the puppet community. There's a consulting company right now that's all their business is based on. They don't tell their customers, "Hey, we're a puppet consulting company." But all their consulting is on puppet. Well, all of your business is on puppet, right? Right. But they're a derivative company. Yeah, yeah. They aren't puppet committers. They're just people who use puppet to do their job better.
14:14So, there's going to be this, "I want to build an ecosystem for people who aren't just me, people who aren't just my company." Sure. And the more that this company can say, "We're a puppet company," then if I can get the marketing right so that they can advertise they're a puppet company and they make more money as a result, there's not going to be any kind of jealousy. Right, right. Nobody's going to say, "Well, Luke's making all the money and Adam Jacobs working his butt off, but he's not making any money." So, I think that's the main way is there's this, I'm kind of a pedant about some things and one of them is the difference in definition between jealousy and envy. Right. And envy is, "I also want what he has." And jealousy is, "I want what he has instead of him having it." Right, right, right. So, jealousy is there's one thing and he has it, but I want it. And envy is, "There's one thing, but we can have two." Sort of zero sum versus additive, right? Right. Yeah. So, in this case, I think that in open source, your goal is build envy. You want people to go, "Man, that thing he's got, I want that. And if I work hard enough, I can get that." And the fear is, you end up with jealousy where people say, "Only one person can have the place that he has. And I want that place instead of him." Right. And so, every time you see anything that looks like jealousy, you have to work to make it so that that gets just kind of, you know, you can't push it out, but you want to assuage that fear and say, "I'm not going to do that." And so, you find business models or you look for business models that are going to say, "I'm going to enable you to make money, I'm going to enable you to get..." And I talk about money, right? Because this is what I do for a living. If I don't succeed, then the way that I can tell I'm not succeeding is that I'm really hungry all the time. Right, right. But other people's definition of success isn't that. You know, maybe you've got a job that you love, you're not going anywhere. Maybe you've got, you know, enough medical issues, you don't want to change jobs, you can't work for yourself, you don't want to consult. But, you'd like people to know your name, you'd like people to read your blog, you'd like people to understand who you are, and you'd like people to use the code that you write. And I think this is, a lot of people's definition of success is, I do stuff that's important to other people, right? Absolutely.
16:14So, if you've got great sysadmin work that you've done, and you produce it, and other people are using it, then that's reward and that's success. And if I can have part of the way of succeeding in a public community is, the work you do is used by thousands or hundreds of thousands of people, I don't think a lot of people are going to go, Luke's getting rich off the, you know. Right, right, right. You know.