Hey, everyone. This week, we've got another in the series of, I don't know, platform engineering talks. This one is with a couple of people from the front side who have been doing a lot of work around Backstage, helping large organizations customize it and integrate it with what they're doing. Now, before we get to it, I didn't fiddle around with my stuff enough, and I recorded off the microphone on my laptop. So that's why it sounds like I'm in some sort of different environment. But whatever, it's fine. I don't think I do too much talking in this as much as I usually do. So maybe it's not that big of a deal. Now, I've been collecting together a lot of stuff relevant to platform engineering and other things on my website. If you go to cote.io slash platform, you can find talks that I've done there, papers that I'm always referencing, some other things like that. If you find anything relevant, you should send it to me, and I'll try to put it up there. But that's a good place to go to check out more. And of course, you can always go to tanzutalk.com to find the show notes for this episode, videos that we do, all sorts of things, and be sure to subscribe to it. Now with that, let's get right into the interview. All right. Well, today, I have one of my oldest friends here, and one of his co-workers who recently talked at BackstageCon. Now, do you camel case that? Do you uppercase the B, the S, obviously the C, or do you lowercase the S? Lowercase the S. Yeah, lowercase the S. It's all one word. Like, isn't that impossible to resist, uppercasing the S? I think it's a matter of intonation. Like, if you're talking about, you know, you're at, I don't know, Pick Your Poison, you're at a Pink Floyd concert, and you're like, we're going to go backstage! Yes. With that one, you would capitalize the S. But if you're, you know, you're saying there's a lot going on backstage, then you would do that. So it's, you know, just depends on how you are excited you are in the moment. I suppose backstage, the software can get pretty exciting at times. And in that case, you'd want to camel case it.
2:01Our bigger issue is that we are called Frontside, our company is called Frontside, and when people talk about Frontside and backstage, they end up saying backside and frontstage. So there's that whole thing that happens. I have often reflected that it's sort of a delightfully ironic name for the company, given that you work on the backstage stuff now. But anyways, so why don't y'all introduce yourself? Let's go in alphabetical order. I guess that would be me. That's right. Good job. So, well, I think so. It depends on if we're going first name or last name. But, okay, my name is Charles Lowell. I'm the CTO of Frontside. Frontside actually refers to the experience of any particular given software. So, you experience everything from what you can see right in front of you. And so whether we're doing user experience or whether we're working on back-end systems, the way that we're going to approach it is from the front side. So there's the etymology of the company. And so even from that sense, it does accord with backstage and the things that we're trying to do in backstage, which is focus on the user experience, not only working with it, but what it's like to use it. Hi, I'm Taras Minkowski. I'm the CEO of Frontside. And Frontside, so Charles is the founder, sets the spiritual and the inspirational lighthouse for us. And so, but Frontside as a company, what we do is provide backstage support and we're a professional, backstage professional services partner. So we help companies adapt backstage. And a lot of our approaches are very much inspired by Charles's view and it's very developer experience centric and we bring a lot of that developer experience focus to the conversations around supporting backstage and growing developer portals.
4:16And then, and then, so, so how did, how did y'all, I mean, what, so, so walk us through, and by us, I guess, I mean, me like through like, how did, how did you end up doing this? Cause you know, uh, I've known Charles a long time and the front side has done all sorts of things over the years. So at some point y'all must've been like, we're going to do this backstage stuff. Like how, how did, how did that come about? Yeah. It's, we had a client who, we still have a client, we actually, we work with them that, uh, that needed to develop a developer portal and, uh, we've always been, uh, developer experience centric. Um, and then when we started talking to them about creating develop, uh, develop, uh, create, creating a developer portal, we were talking about a lot of things that backstage had, but I didn't even know that backstage existed. So someone actually within the, within the organization mentioned backstage and we looked at backstage. We're like, yes, we should do this because, uh, one of the biggest problems with a lot of portals is that they never get enough attention. Every portal is, is just enough, um, was invested, invested into it to make it work, but not enough to make it good. And that's the underlying problem with a lot of develop portals. So having essentially a framework that is available and is iterated independently, so you can then just use it to create a developer experience you want was very appealing and it's paid off. And, and then, so, so like, okay, I mean, you, you, you, you, you've, uh, you've hit on it there a little bit, but what, what is like with the, with the, that initial, you know, client and the ones that you have now, like what, what are they doing with backstage? Like what are the needs they have for, I don't know, an internal development portal beyond like a wiki page or, or whatever. Like, and, and, and because that, like, it's, there's a lot more going on than just like a static set of pages that one might call in the old days, knowledge management and, and like, you know, project status, things like that.
6:18And so like, there's something, there's something that what hasn't been satisfied that backstage is kind of filling in there. That's always hard for me to, as I'm doing here, talk about concisely or describe. Yeah, I think there is some, um, um, so if we look at the history of, of, of backstage to Spotify, like it originally emerged as a tool to help streamline certain processes and it's grown from there. And I think a lot of, uh, because, because, because of Spotify kind of engineering organization in their organization, it was very natural for developers to own their, like developer experience. So, so backstage internally, um, is a combination, I think they have something like 40 different teams contributing to backstage where each team, um, is using a kind of like putty, you know, they all add their little bits to make their, to, to add the pieces that they need. They need to improve their, you know, so they, they, people within Spotify, they all go to backstage to, to, uh, basically do anything. Like you essentially can't do, you can't get work done without going to Spotify, without going to backstage at Spotify, because everything that you need to, to be productive is there. Um, and, and over time from that, like there's emerged kind of patterns. So like the software catalog is, is a pattern that is, is now represented by plugin. So it's a, it's a tool that organizations can use to, um, allow developers to control metadata for, uh, different, uh, assets within their organization. So, and for those assets, you know, typically are like, you know, you might have a service or you might have, um, libraries or, uh, or components that are, that are, that you share across the organization. So develop, so the catalog gives developers control over metadata about those things. And from, from that metadata, you then create the actual user experience that you see in the, in the backstage portal.
8:18Yeah, I think it's, it's an, it's an important way to strike the proper trade-off between distribution and centralization. You know, I think, you know, the, the pendulum always swings in between those two poles where you have a very centralized system where, uh, of, of software development, um, and deployment, where you kind of hand things off to, to, uh, you know, your deployment team or whatever. That was kind of the old school way of doing it. And then, you know, the, the, the revolution that's been happening for a while now is giving agency to the individual developers to, to manage their own deployments and manage their own services and break things apart. And, and, but the problem is it can be difficult to share knowledge, uh, between those, those different things. So if I, if I'm an organization and I've got a hundred different services and I've got each one of those services is deployed in five different environments, you've got this fractal complexity that could emerge. Um, and so it's a way of kind of, as I see it, reining that in and almost inverting control. Uh, so what's interesting is that rather than being a portal that you maintain, the individual process, uh, the individual projects can give information about themselves and then backstage can, can pull that in. And so if you're familiar with the CI CD model, it would be kind of the difference between old school Jenkins and a system like circle CI or Travis, where, um, you know, in, in, uh, the, the old school CI, you kind of had a central management, uh, of, of the CI processes. Whereas kind of the key innovation of those distributed CI models was you provided the general infrastructure, but each product could just kind of plug in and say, this is how you build me. This is how you distribute me. And then it, it, uh, it carries away. So I think the inversion of control is an important aspect of it.
10:20So, so I, I mean, I mean, to, to, to, to crudify it, so to speak, basically like we need one UI for all the development we're doing in this organization. And, and, and you could call it a console, hence, hence one of my favorite words ever portal, uh, you know, the, the, the place that we go to do that. And the opposite of having one UI would be, uh, worst case scenario, like every single team, however, whatever a team is defined, whether it's like component or product or geography, every team basically just has like their own way of like writing up and adding the metadata about their projects. And, and you just have to figure that out team by team. Right. And then, and then, but then the part that, that, that, that y'all, y'all are talking about that's in there as well is, and then in addition to that, the way that, uh, each, the developers on each team, like are interacting with the whole, I don't, for lack of a better phrase, like just the software delivery process. There's basically a standardization of that as well, right? Right. Like in, in this kind of the same way that if you do like an inverted, like CICD pipeline, there, there is like a, there's a specification of how you're going to be doing builds. Now, like everyone can't just come up with their own way of doing it. So you really like, you're getting, you're standardizing how software is done, not, not at the, the coding level, but how you package it up and move it along towards production essentially. And, and everything else that's involved that yeah, it's a combination of giving developers, um, what they need to be productive at the stage that they're at, because for some, for some developers in the organization, being productive means give me a really good starting point. Because I don't know what are all the pieces that I need to use, right? So, so having a place to start, so like, um, having all of your resources aggregated in a single place.
12:21So when I go and look, I could find everything, but then also being able to use a, an existing template to create like a new react app or create a new service that automatically deploys, uh, using best practices on our platform. So for, for someone who is not familiar with the ecosystem within the organization, having a starting point is really helpful. But for someone who is, has a lot of experience, the flip side of that is being able to be able to adjust that, uh, uh, that experience to match their understanding. Like if they need something new and they want to be able to like, they already know everything that's available, but they realize that it could be better having the ability to change it will be able to contribute upstream. So that's the, that's the other, that's the other aspect of the developer experience using a portal that's now happening as a result of, of backstage being adopted. There's this inner source practice that's being cultivated where it's being built, uh, starting with a developer portal. Yeah. I mean, I mean, you're that, that, that, that, that part's interesting in, in the, the way of, uh, I don't even know what the phrases for this are, but you know, usually, uh, in, in like a large organization, you get all your enterprise architects and governance people together and they define what the, the software delivery factory or process is like, you know, we've got all these stages and these documents you have to fill out. So, and to some extent, like if you were to take the maximum idea of whatever it is we're talking about here, you could kind of like start to program that more than it have a, be a bunch of like word documents or Google documents or whatever. And, and like, so the role of whoever's specifying how software is done in an organization is more to say, uh, it's going back to your thing Charles, like here, here's like the, the execution pipeline that's going to happen. And here's like the little plugins that you have to add in that describe your product every year, your software. And then you can do whatever else. And then also like, because it's all actually code instead of like, you know, turning on suggest changes in some documents somewhere or going to some terrible meeting to discuss like some policy change.
14:30You could actually like, just like suggest some way we're going to change this and like submit a PR or whatever. And, and we'll get it, we'll get these people together and see if we want to just like accept that, uh, which seems better, hopefully. Yeah, I think it applies for every aspect of the development. So, you know, we were talking, um, a little bit in the green room phase about why not just use a wiki, uh, for example. And, you know, the, the problem is wikis have this, this issue where they end up going dead, uh, at some point because nobody's maintaining them or someone has migrated to some other technology. And so, you know, kind of the, the inverted, say backstage ways, let a project have control over its own documentation and then communicate whatever, whatever your organization's documentation platform is. Let, uh, let the project say, Hey, I'm using, you know, I'm using this markdown or I'm using TS doc, or I'm using this, this documentation tool. And if that changes, well, the, the project can, you know, uh, essentially update the pointer, so to speak. Um, and, and your portal doesn't go stale, um, because you know, you've now evolved. So you're not locked in to one particular technology. All right. All right. So now, now we had a, or at least I do have a little bit of the dream, so to speak, like, like what the point is and what happens. But like, so like, what is the actual, uh, if, if, if I go down to comp USA and I buy the, uh, I buy the backstage, uh, kit, like what, what am I going to get? Like what, what are the actual components and like the, the parts of it? Like, I'm pretty sure Kubernetes is involved somewhere, whether I want it to be or not. And then I, you know, like, what, what do I, what do I like, what do I do? Like, what, what's the architecture of it basically? What are all the parts? Um, so there, there are five core components that are four plugins.
16:31Let's call them plugins. There are five core plugins that are included with backstage, um, which are built on a framework. That is a fairly standard, like client server architecture. So there's a, um, there's a react application on the front end. And then there is, uh, node processes running in the backend, which provide the server that the client connects to. Um, that's the kind of the general architecture. Um, but with the, the five plugins that come out of the box are the catalogs as a software catalog, the scaffolder. So it's a, uh, a way to create templates that generate, um, generate code that is pushed into repositories. And then those, so that code is registered as a component in the catalog, the research, which makes it possible to find things within your ecosystem. Um, there is the Kubernetes plugin, which allows you to hook up your, um, if your components in the catalog represent services, those services are probably deployed to Kubernetes. So the Kubernetes plugin allows you to show information about the services in the catalog. Um, and there is, uh, I think that's four and there is, uh, what else is there? And then there's like, uh, Oh yeah. And tech docs. Yeah. So the tech docs is documentation as code where the markdown files are stored in the repository and then they're auto-generated and made visible in backstage. Right. Right. Right. Right. Okay. And, and then, so you're, you're saying like there's components and a component, does that represent like the software that a team's working on? Like, like, like what is a component? A component is, yes, there's different types of components. So, uh, a service, a website, a library, and then you can create your own types of components. Okay. All right. Yeah. The object model is kind of deliberately loose. Uh, so I think component is more, one of the, uh, better understood.
18:34But the other thing that's neat about it is essentially the catalog, which really underpins so much of, uh, backstage is it's, it's just a graph of things that your organization, uh, and you can kind of draw arbitrary connections between them. So you've got these components, but you also can have references to where these repositories house these components. You have, you can draw connections of which people are the contributors to these components. So if you have a question, uh, about a particular service, you can actually look up and you can say, ah, this person is, is, um, is associated with this. Now, I don't know if you actually get that out of the box, but those are the kinds of inferences that you can, you can draw. Um, right, right. Yeah. And, and so that, I mean, that's the other thing I was interested since, since I'll work with, with several people on this is hearing is like, it seems like. With my crude understanding, the, the, the, one of the original problems that the backstage was looking to solve is like, especially in the Spotify since it's like, I forget the number of some of, you know, we got like 10,000 microservices running around here. And what the hell is going on? Yeah. And, and, and, and yeah. And, and not, not only, not only what are they, how do they work, but like, who can I talk to about them? Like who owns this? Right. And, and then I remember like, like there was an intriguing. Which ones are on Kubernetes? Which ones are on AWS? Right. How much are they costing me? Yeah. Right. And so, so you have that catalog, like one, and it's more, it's, it's still kind of like just this more or less static catalog of like, here's all these different services. And now you at least know what they are. There's documentation for them. As you were saying, you can check in and here is a, here's, here's at least some person or an email address that owns it. Or like that you could talk to about it. I'm being a little facetious here. Who's here's who owns it. Right. And then I don't, I don't know what the, what the object model, as you say, is for this stuff, but then you can also add in, like, I remember one case they had that was interesting.
20:40That was like, well, we want to, we want to convert everyone over to Python. So we stuck this thing in here that basically said you need to convert to Python or you've already converted to Python or like you need to upgrade your Python version. And so it's almost like, I don't know which direction this injection goes, but it sounds like once you set up that, that catalog or whatever it is, you could actually start going in there and adding in like all these other little embellishments and layers of governance. That's probably not a word people like to use, but just to say that like, you know, this project needs to upgrade its version of Python or this one doesn't have the right kind of observability stuff built into it. Or, or it does. I, I mean, it's like, like what's that notion called where you're like, you're moving way past just like a static wiki. It seems like. Yeah, I don't, I don't know what, what it would be called. Um, I know there is, um, the way, the way I look at it is it's giving developers control over the shortcuts that they use to access information. So like when you go to the backstage catalog, um, when you find a specific component, like when you find your component on that page, you can, you can add metadata that points to external services. So you could say, so for example, my logs are at this URL in the Kubernetes, in, in, in Kavana. So when you click on the link, it will take you directly to the logs for, for that component. So, so, so, and then you can use that link to essentially, um, extract information for external services and then show that information on the catalog page. So you could say, um, now, instead of showing the, now, instead of going to Kavana, I can actually like programmatically pull information from Kavana and show those latest logs in the UI. So it gives you the kind of control over, um, what you're seeing on a component per component basis, because it's not unusual for two, for multiple pieces of software to, or technologies to play the same role.
22:52You know, so you could have like, um, you could, there might be different ways of getting, uh, different systems for viewing logs, but they could also be different, like infrastructure providers being used within the same company for hosting the same thing. So, like if you needed to go access, uh, for example, secrets for one component versus another, uh, it is not uncommon to see like, you know, AWS secret manager and HashiCorp bulk being used within the same company. So from different, depending on which component, depending on your particular component, where you access those secrets might be different. Um, so being able to use the catalog pages to, as a jumping point to where, what, where it is specifically for that component is really helpful for developers. They don't have to, so they can find what they're looking for faster. Yeah. And then, and then I suppose you could also like standardize on what's on each of those pages, right? Like to your point, you could say like, every time you have a, uh, a project or a component or whatever you need, you need to, it would be fantastic if you pointed to where the logs are for the current version you're running. And it would be great if you pointed towards wherever your issue tracker is or whatever. And that way you, you more standardized where all that stuff is instead of emailing around things. Yeah. And even, I think maybe the, the, the, the, the word that you're looking for, and I don't know if you mentioned it is just annotation. Like annotation is both a mechanic within backstage. Like they have this concept that you can annotate any entity in the catalog. Uh, but it's also kind of a abstract process of, uh, deriving metadata that's available to the rest of the system downstream from the actual components. So just the fact that a component is registered in the catalog and you know where it lives, you know, now your portal can actually follow that location and say, okay, you registered this component. I can go visit it and I can see, oh, this is, this is a Python project because it's got a PI project file.
24:56And I can look at it and I can see its dependencies and I can see which version of Python I think was the, the thing, the example you used. I could see, Hey, it's got these security vulnerabilities and these are kind of enhancements and enrichments that you can do downstream of the actual project. But all they had to do was kind of, uh, register themselves, uh, or be discovered. Um, so, so maybe I, I, it's both a mechanic and the process, this process of annotation, um, or, or the derivation of knowledge based on just the mere fact of a component existing. That's a mouthful that you should write that down. That'd be a good, uh, a good blog. Uh, yeah. So, so like, like, you know, like, like we mentioned several times, like y'all have been doing a lot of work with, with clients kind of like building up their backstage stuff and customizing it. And so like, you know, amalgamating that, like, if you were like, like when you do, or when you think about like sitting down with someone like, all right, we have nothing. Like, like, like what, uh, like what's the, like, what's like the six, 12 months sort of like thing that they do? Like, what's the first thing that they build out? What's like now it's, you know, usually it's like after three months, we proven that we should keep doing this and actually pay the front side. And then, and then like after that, you want to add, you want to expand to like some more teams and then maybe they want to like customize it and add some stuff. I don't know. I'm just making stuff up based on software, but like, what's kind of like, not generic in a bad way, but what's like the generic path that people go down as far as building up their, their, their backstage stuff. I think that the, the best starting point and hopefully companies do this, but this is not always that this is not always, um, the case, but hopefully the first step is asking their developers what it's like to be a developer at their company. Ask them like, what is your actual workflow? Like, how do you go about doing your work? And then what happens if you do ask what, what often happens is, um, the kind of feedback that you will get is, oh, um, it's really difficult to find stuff.
27:05Like, I don't know where, um, like when, when we, when we've done this for our clients, what you will hear developers say something like, I have a list of 170 links that I've maintained that I share with my friends, my colleagues at the company, because if you don't have the links, you might actually not be able to get your work done. Cause you don't know how, where, where the, where the, the, uh, really complicated URL is for a specific service. So, um, when they get feedback like that, then they can map that to something, you know, to a feature of, of, of backstage. And then they have, now they're starting to form their roadmap and they develop, like developers will tell you exactly what their problems are. It's just, I think what we're not used to is thinking about develop, like thinking about the platform as a product, thinking about it, like thinking about your internal develop platform. The product and thinking about like the actual, um, what, what is it like to experience the platform. But when you ask the question, it's actually pretty easy to map those things to features because these things that Spotify open source, the plugins of Spotify open source, those plugins, um, are solutions to very common developer problems. Yeah. Yeah. Yeah. And then, and then also like, as, as I don't know, I almost feel like as it's frequently overlooked about stuff like this, because it comes from a third party, no one gets the chance to argue it into doing nothing. Right. Like, like, like if you were coming up with your own way to build something like this in any large organization, you'd probably spend a lot of time talking about how you should do things. And one person would have this standard of doing it. Another person would have this way of doing it. And then maybe you would just like, forget about it. Cause now it's time to have vacation. Whereas like, if you've got a third party thing, like there's no debate about what you do. You just, you just start using the tool, which seems useful. I think the challenge with that is that I, I, what, what I'm observing in working with, with, with our clients is that there's definitely a path. Um, but I think what's generally missing is this concept that, that we need to be actively solving developer experience problems.
29:12Like there is friction and there is, um, latency in the, in the developer, in the engineering organizations, but we don't have either the tools or the, uh, we don't have the tools to observe it or to optimize it. So the best case in the best, um, example of this is, uh, you know, in the past we didn't have a good way to like releasing software was very, um, you know, we, we all knew that it would take months to do, but now, you know, as a result of DevOps, we have, you know, different stages for releasing software. So we could say, for example, our CI time is now, you know, it was taking an hour to run. Now we, you know, brought that down to, to seven minutes. So we had both the mean, by having tools that give us the way to observe, um, how long it takes, we were able to reduce that latency. Similar, um, doesn't get exists for developer experience, um, in other aspects. So, um, I think, and that's one of the challenges for a lot of companies is that it's, it's a figuring out what to improve on their platform, given where they are right now is something that people learning to do now. Now, now that backstage exists, it's kind of like laying out this foundation where people start thinking about how they can improve their developer experience. Yeah. Yeah. No, like, like again, it's something I think of in this area a lot, which is, uh, well, to put it one way, it's, it's always shocking when you discover the need for something that you thought existed already in, in, in the sense of, I mean, to use the, the old tired analogy, right? Like the, the cobbler's kids wear enough shoes. Right. And in this case, it's sort of like you have software developers who are by definition, writing a bunch of software and in general, collectively, they don't spend a lot of time writing software that helps the process of what they're doing.
31:15There's all sorts of people who write frameworks and like, I mean, that's, you almost have to stop developers from doing that. So they actually work on the software that runs your business, but like, but, but as far as like, what, what if we like wrote software that like makes the way that we collaborate and run better. And, and like we have, we have plenty of stuff that's like, you know, from the old application lifecycle management and software delivery lifecycle, like we have all sorts of tools for that, but none of it, like, I obviously can't describe it very well either, but none of it sort of is like, well, here's, well, let me, as, as you were saying in your talk at BackstageCon, it's like, yeah, but when can I have Heroku? Right? Like, like, like when can I have, when can I have that set of web pages that I go to that has all the same interface for finding stuff explains to me how to use it, and allows me to just work on the process, for lack of a better phrase, just collaborating with people. And, uh, I mean, I guess it's sort of like astonishing to think this, but like, I guess no one's done that before. Right? Like that, that hasn't existed. Yeah. I mean, I think people have tried a lot, uh, to do it individually and for themselves, but they're just certain, it's like you said, how much of your effort? Is it just from a business sense feasible to extend on that? Is it 10% is it 20% is it 70% and there's definitely a threshold there where you can't have a business, uh, if you're spending all of your time working on your internal developer platform. Uh, and so where, you know, shared solutions have to emerge in order to make it feasible for, uh, everybody to do it. Um, because you know, one thing is for sure, you can't there, you know, like your developer experience, it exists, whether you're observing it and where you're visible of it, not like you can't not have a developer experience.
33:17Um, and so if it's, you know, if you, if you can't invest in it, it's just, it's going to be bad. Um, and so I just, I think that, um, I think it just takes enough people acknowledging that, that it is something that needs to be solved and trying to sit down to do the work to solve it collectively. I just, it's one of those things where it's very difficult for it to be emergent in a single place. I think it also couldn't have been emergent because, um, it was almost impractical to have it emergent previously because there's nothing you can do about it. Like before, before cloud native, before companies start to build internal developed platforms, when you're relying on, on third-party software that you do not control, when you're relying on monoliths that you cannot easily change, you can't really do much about like, even if you wanted to change, you don't have any means to do it. And then I think what's happened with cloud native is that by giving you the, the granular pieces that you can use to assemble your internal developed platform, you now have control over the individual pieces. So now you can, you actually have something that you can manipulate. And then what's happened as a result is now people realize, okay, well, I have an internal developed platform that consists of a whole bunch of tools. But what's, what we're finding is that it's actually really difficult for developers to be productive. And so now there is a, um, you can attach a monetary value to not having solved this problem, like not having, not, not thinking about a developer experience actually has a cost. And this is what we're seeing now, I think what's changed in the last like five years is that companies are actually realizing that it's really expensive for them to not think about the developer experience on the internal developed platforms. And so now they're thinking, now they're standing up teams that are dedicated to, um, to use, to improve that. And I think what's happening with backstage is it's emerged as a, as a, at the right time, you know, we went from, you know, a few years ago, it had 50 adopters a year ago, it had 125 now has 400 adopters.
35:18Um, and it's being adopted really quickly because it is a well, like well understood, the, the need is well understood, but the solution might not be, you know, but that's something that we're collectively learning. But the problem is now known. So, so then how about like, like two, two more things to go before we close out one is, uh, so, so I, I, I got all excited and drove us off the track of like the, uh, the kind of like, you know, the build out that people have, right. And we stopped around, uh, uh, just discoverability, right? Like, like having, having that your, your, your, your, your magic cheat sheet of like all the funny URLs you need to go to, to find the work to do. Like what, so once you establish like the, the catalog, if you will, you can discover things and find them. Like what are the next things that people start working on? Like what are the more kind of advanced stuff that people build in that I imagine, you know, y'all get hired to help out with a little bit. Um, I mean, you do, uh, I would say the, you know, you, it really depends on where, what, what aspect of that network of information is the most valuable to you. Um, so, you know, at the, at, at its core, what you're constructing is a graph of the internal software ecosystem and all constituents. And so, you know, what aspect of that graph is you want to surface, I think is where people focus on next. Now that's the abstract answer. I would say, uh, surfacing the people associated with software. Um, we've seen be something that's an extremely high value target. Um, I don't know if you wanted to add something to that. Yeah, it's, it's very much, it very much differs for each company because each company is going to have a different, like their, their platform is going to be more complete in other areas.
37:20And we can, uh, in, uh, in some, so the most standard thing that we see happening is, uh, creating templates that you can use to scaffold new components. So starting to standardize and make it easier for developers to create things. Um, but depending, so that, that, that, that, that's a pretty common one as a kind of a next step. Um, but what specific, what, what information you surface in the catalog very much depends on your, your biggest needs. Like what are the biggest problems? Cause at some companies, they might have, um, a service built out. Like it's not uncommon for us to see like some clients have an entity service that provides information about all the different things. So like they might already have a catalog. So for them to pull the information from the existing catalog and to surface it in backstage, it's just a matter of like integrating with their, with their service, uh, with the existing service catalog. Um, or another company might have a system for, uh, querying relationships between services. So for them, it might be, okay, make that information available in backstage. So you can represent the structure. So you could, so there's in a, in backstage, there's a concept of systems and domains. It's kind of organization of, of different information. So you might use, um, you might use backstage to represent the relationships between different systems within, within your organization. Right. And then going one level deeper, it'd be like, okay, showing how those things depend on each other. So like, you know, if a service uses another service and depending on what you have, depending on what you already have, like your approach to that problem or what your needs are going to be different. And so you kind of have to prioritize based on where your biggest pain points are. Right. So you could generate the, uh, that old infamous, like Netflix death star Mondrian thing, you know, they used to have, that was like, here's, here's, uh, how all of our microservices connect together. And, uh, isn't that great. But I mean, you know, that, that would be useful to see the, uh, the graph of all your stuff.
39:22And, you know, if you have like thousands and thousands of applications and services that that's hard to track down. So, so then like kind of connected to that, like, so where, at, at, at, at, at what point does like the, the intentions of what backstage want to do, like kind of crossover scope creep. Like if I were to say, so let's take like your, your, your monitoring stuff and we're just going to move that all into backstage rather than being in your monitoring tool. Like that doesn't seem. Right. Like against the philosophy of. Right. No, no. Exactly. And so like, where, where's that edge of like, here's something we could do that, that we need done. And like, it doesn't belong over here in backstage land. That's not like in, I mean, it's just programming. So we could put whatever you want on a screen. And like, when you click a button, it could do whatever you want. But like, that doesn't make sense to have it here. It's not part of it. Like, uh, well, here's another one is like, so would you use backstage to like, to all of the human part of lease management? Or would you not like, like, like, is that the UI you would use to release your software all the way to production? Or would that be scope creep and you wouldn't do that? It could be if you're, if you're, if that process touches multiple systems. Some place you need to touch multiple systems, like from developer experience perspective, like you need to, I think this is where backstage shines is showing information from multiple systems. If you want to go, if you need to get deeper into that, into like, you don't want to hide the system that you're integrating with because it might have really useful information that you need at a specific time. So developers need to go to jump to that system and find what they're looking for. So, uh, so you don't want to, I don't think it's appropriate to bring everything into backstage.
41:25Um, but in the way that the systems overlap backstage might be the only place where this is actually possible. Right, right. That's, that's a good guideline that if, if you're trying to integrate together multiple systems to do something integrates the wrong word. But if you have to use multiple systems to accomplish some task, it's at least good to consider doing something. Right. I mean, I think that backstage is definitely not designed to unwind the progress of distributing stuff in microservices. It's to make it more powerful. So it doesn't do, I mean, it does a lot, but it's, it's not, uh, it's, it's not designed to do, uh, particular tasks so much as aggregate information. About what is being done. Right. To be a portal. Which, which I guess. Did we have avoided that word the entire, uh, the entire time? No, I mean, you know, uh, I, I was, I was, uh, I was raised in the late nineties and early two thousands when portal first came about. So I have, I have weird feelings about the word portal. You even had a site called portal potty. Yes. Like, I don't know if you still hold onto that domain. Like, I, I don't know. I mean, I, I don't, do people use the word portlet? That, that would be great. If that, if that reemerged. I never knew they emerged in the first place. Yeah. Portlet is scope. It's just like the same way that you always have a sitcom that's about 20 years in the past. So, you know, at any, uh, at any given moment, there's going to be a sitcom that takes place 20 years ago. Same thing with the vocabulary and tech. Yeah. There's some law. Yeah. Well, great. Well, that's been good. See, I always just use these to educate myself and ask the questions I'm confused about. And I think I'm a little more cleared up and smarter. I don't know. We'll see. But is there, is there anything, uh, is there anything you're thinking of that maybe we've left out to kind of defining, like, like, like, you know, what I want to do here is like define, like what backstage is beyond like, you know, a portal.
43:31And then like how people like, like the immediate gain that you would get. Right. Which, which, which y'all went over. Well, it's like, well, it's really hard to just find stuff, which sounds like a trivial problem, but it's very important. And, and then, and then, you know, there, there's basically the, the kind of like what the stack looks like from a very high level. But, uh, I don't know what, what, what, what do you think is going to be happening over the next year or so in this area? Like what, uh, what is the community talking about? As far as like, here's what we need to do now. Like, this is the next thing we need to work on. I think the biggest, the, the, the most important thing that's going to happen now is going to be, um, making it easier for, to engage, for people to get engaged with backstage or for the users of backstage to be engaged with backstage in a way that is, that is easy for them. So, because right now it's the, the process of building a port, the portal is, is a little bit, um, like it follows a typical, like, you know, you, you, you have a source code, you build it, you build a Docker image and you push it. So, so it has that entire, it has that, um, process built into the, into develop development, which means that if, if someone, you know, if someone is using backstage and they want to add, let's say like, oh, I want for our project. We use, um, uh, like a, it's a SAS for doing, for doing logging or forgetting, uh, getting error logging from like a client application. Like I would like to integrate that. So what they would need to do is they need to clone the repo at the, at that integration as like a, as a react component or, you know, as an, or node process. And then create a pull request, it's going to get merged and it's going to get redeployed and it's going to be running. So that whole process takes a lot of, a lot of work. So I think what's going to happen over the next, over this year is try to find, increase the number of things that people, that teams can do without having to go through that whole cycle.
45:34Right. Right. Yeah. That makes, I mean, basically just making it more usable for developers to add things. Exactly. Yeah. Of their experiences. So they don't have to use react or TypeScript. They could just use, they could just use the UI to add the things that they need and be able to amend the, like modify their experience based, modify their like backstage experience based on. Uh, their needs without having to, without having to go through a build cycle. Yeah. Well, hopefully it won't reduce on the typical, I mean, it seems like the cycle, whatever the format is, is you do the, uh, the current configuration format squared where the first phase is to write another layer. days in our contemporary era let's simplify the yaml on top of the yaml and uh just just make that easier to configure stuff which always seems i don't know maybe maybe i'm i'm too snarky but that seems a little silly that you're going to simplify something by adding more of the thing on it instead of just simplifying the original thing but i don't know i just make slides i don't know anything anymore yeah and then also i mean you entered like you brought it up several times it i forget what you call this kind of thing but it seems like like one way of figuring out where to get started in addition to asking developers as you're saying is like it's sort of like a little illustrative uh thing to say like all right tell me how you find the logs it's always logs like to kind of make a point about how to organize stuff it's just like like how long does it take you just to find the most recent logs and then like you know that's that's a good use case to go out and solve really quickly and then how do you find the logs for all the applications like how long would it take you to actually get those logs so you can start looking at them never mind the thrill of right through a bunch of logs just finding right always seems like a uh something to test out as far as how your internal portaling for developers is working out well great well thanks but so if people are interested in more of y'all what would you point in that give us a call to action call to
47:36action we can go to frontside.com slash backstage and we also have uh we we offer enterprise backstage support i don't know if you want to talk a little bit about that for us but i know we have uh plans that are built to accommodate people wherever they are in their backstage journey uh whether yeah what what does that business look like like i mean y'all are also like some of the early people figuring out what the the market in this area looks like like what what do uh what have you discovered you can get paid for so far yeah i think one of the so right now what that support looks like is um helping teams that are adopting backstage uh be productive which is essentially giving them the information they need when they need it um and um and we have because there are teams that are being stood up specifically for building this but learning all the tools that are necessary to be to to be able to create a backstage portal is not easy so we make it easier for them and then uh another part of that is uh when backstage doesn't do exactly what they need we as part of the service we actually go and work upstream to make those changes for them on behalf of the team so they can then just upgrade um and get the latest version without having to fork their instance of backstage so they can keep staying like current and be able to upgrade at any time um and uh i think the third thing is uh also navigating like how do you actually accomplish something from product management perspective because if you have uh you you know you have a problem there are a variety of different ways of solving it and so knowing which solution to apply i mean this is one of the benefits that we have in our particular positioning because we work with with adopters and we and we look like we see their roadmaps we see how they solve problems we help them design solutions we have right now um we support development of portals with over a hundred thousand users combined um so we have a quite a varied perspective on who on how these portals are implemented so uh we we make that information like we make them
49:42the information the product managers need to design their roadmaps roadmaps for developer experience kind of available to them what they need it so that's kind of like without what would that kind of how that work you you can you can i mean when you have when you have well not not even that scale when you have like just even a small scale of different organizations doing something like this you can uh you can speed up people's product management thinking right because they you know because that way they're not the first they don't what it's rarely the case that someone is the first person who's encountered this problem and there's usually other people who solve it and they don't have to like reinvent the solution over and over again it's the old like you know on on the uh especially in the enterprise software space uh us us on the sell side are always joking like yeah everyone thinks they have special problems and i just talked to three people this morning who had that same exact special problem and like you have to you can help people out understand that health is available and their problems are not uh detrimentally unique and special for them which is one thing yeah one thing that's really common to a lot of our company a lot of our customers is that um they look to spotify for for inspiration but spotify is a very advanced engineer organization like they created backstage because they're very advanced engineering organization so the practices that spot spotify has they're kind of like the end goal where everybody wants to end up but it's really difficult to implement the end goal when you're like if you're like here on the left side trying to move up to the right side where you where you want to be acting like you are already arrived there it doesn't really work you know so because if you're you you don't you're not not in the same place like spotify has 40 teams already contributing to backstage if you have no teams contribute to backstage the way you're going to solve problems is going to be different and this is what this is what a lot of our work is helping them uh like think about how to solve problems at their particular
51:43place as opposed to how spotify would solve it because it's just a different way of like you you act differently depending on where you are in your journey yeah yeah for sure it's uh it'd be a bad idea to learn how to drive stick in like a formula one race that's exactly what it is yeah all right well great well thanks for being on and uh you know see everyone next time bye all right thank you