Inter-Service Communication, Consumer-Driven Contract Testing, and Service Versioning (Ep. 39)

Episode 26 · 54:16 · SRT · archive.org · all Tanzu Talk

Well, we've got a super special episode this week, Richard, and I have to say, I have not prepared for it at all. So what I'm going to do is take the reins sort of like patinaed and withered and warm and comfortable as they are and just give them over to you. And then at the end of this, I know that with the August panel that we have, I'm going to be fully educated and ready to go back to coding. So I'm looking forward to it because I'm getting tired of slides. It's not working out for me. So with that, why don't you take the reins over and have everyone introduce themselves? Yeah, thank you. I'm going to be like the engineering whisperer. It's going to be great. They're going to do all their thing. And yeah, this is a very special episode. You're going to gather your kids around afterwards and explain it to them. So I'm glad we get to do one of these. We have some smart people on the line, so I'll quick call you out and ask you to do just a couple second intro. So Mr. Schaefer, can you tell everyone who you are? I'm Andrew, often known as Andrew Clay Schaefer on the internet. I work for Pivotal. I think everyone here works for Pivotal today. We're just going to talk about some of the things we see happening, things people are struggling with, with respect to managing deployments of or continuously deploying microservices or die trying. Awesome. Kenny, you are here. Say hello. Hey, guys. I'm Kenny Pistani. I'm a Spring Developer Advocate at Pivotal. I'm writing a book with Josh Long called Cloud Native Java, which has many of the topics we're going to be discussing today in the book. So it's a topic near and dear to my heart. Awesome. And Ali, you're on. Say hi. Yeah. Hi. I'm Oliver Gierke. I'm the project leader of the Spring Data Project at Pivotal. And outside my engineering role, oftentimes involved into customer projects, software architecture, REST, and distributed systems. Yeah. Fantastic. So we are going to talk a little bit, as Schaefer teed up, this idea of how do I deal with these, obviously, dynamic distributed systems, updating them, contracts, things like that. But before we jump into it, I do want to have us define a few terms so that those listening,

you know, we're all working on the same set of definitions. I think we should actually back up a little bit. We should back up one step, which is the reason this podcast is happening is the three of us had a little bit of an open discussion on Twitter about some of these things. And I wouldn't say we were arguing, but we didn't necessarily agree on how some of this should work. So I think it'll be fun to talk through this with the whole group. No, that's great. Kenny was on all caps, which when we knew it was done, we weren't going to, we had to take it into the podcast because he was yelling. So no, this was great. We wanted to, you know, it was great that we had this sort of forum because 140 characters never let you get the nuance out. And Ali wrote a fantastic blog post, which also was about 1,400 characters or more to get some things out there. So yeah, thanks for teeing that up. So let's, let's talk about some of this first. Who wants to define contract for me? We should, we should go right to the dictionary. There you go. There you go. Break out your Websters. Ali, do you want to take that one since you wrote that post? Yeah, of course I can take, I can take it. So we were basically like getting into a discussion about like how systems are, interact or have to interact with each other, how they're basically coming up with an agreement of how they actually interact with each other. That means like besides defining the protocols they want to use to talk to each other, decide what kind of semantics they share, what kind of shared understanding about like the data they share and the business processes that are actually implemented through that communication. And contracts and especially consumer driven contacts, the topic that we're going to get into are well or getting kind of a lot of intention these days. And we basically got into some argument about the nuance in terms of the effects of applying that kind of contract definition on the overall architecture and on the teams, how they communicate

with each other. And that's what basically struck a chord to let me write that blog post, which is basically about evolving distributed systems in general, and the forces that are like applied to that to that topic in general, why you actually distribute systems or separate or divide systems into parts and what approaches to take to evolve them then independently. And contracts, of course, play a role in that, as I said. I mean, Kenny, how would you take a contract for someone to say, OK, when we talk contracts, we're not talking legal terms necessarily. But what's the contract to you when you have a service definition? So the contract is something that a service is going to publish that tells all of its potential consumers what exactly is the interface for communicating with that service. Now, you can take it further and say that in the case of consumer driven testing, that a certain method, an API method, has an expected response. Now, the producer can publish that as a stub to let its consumers know exactly how to interact with that API and for what conditions it can expect certain responses. And so that's the main. So if somebody says, who cares about contracts? They would say that because they don't want to deal with that or they don't like, you know, they have scars from soap and wisdom times or, you know, for someone who doesn't look to those things, why look at contracts? Yeah, I think I think the core of the nuance that we were just having our Twitter discussion about is what's the implications with respect to coupling and what can be changed independently and what can't. And just to just to pull because I did pull up dictionary dot com. The number one definition under contract is an agreement between two or more parties for doing or not doing something specified. And I think that kind of fits under, you know, an umbrella for a lot of the stuff we're talking

about. But, you know, Kenny went right to this thing that would get exposed by an API that would be able to, in some cases, even negotiate what the contract will be for that particular request, that particular session. But in general, there's lots of things that are kind of contracts that go that go unspecified and unspoken about, you know, just the fact that it's even at HTTP requests or some of these other like protocol decisions have implications about the contract that you could keep. Now, I mean, this comes up more and Ali, your post hit on this, because when we talk about microservices, whether we want to use that term or system of systems, as you framed it, that as you start teasing apart these monoliths, you end up introducing a lot of new communication if you don't refactor or make these sort of self-inclusive systems. And if you want to reduce some of that communication, so maybe these contracts matter more as you have more services, more communication. Do you want to summarize just quickly kind of the position you put out there around teasing apart systems and considering what that communication is? And then to Andrew's point, what happens now when you have these systems with contracts between them? How do you start to deal with deployment concerns? Yeah. So let me start with the first thing. So what we started discussing was, OK, what if you if you like if you're dividing a monolith into into separate systems, right, no matter if it's like two or 10 or something, the one thing that that fundamentally changes is the amount of effort that you have to communicate with the other system. So if you have a monolithic Java application, like talking to a different part of the system is basically a method call. There's no network in between. There is nothing. If you have like that split up into systems, you now have a network in between which can fail. The other system can fail. You have to marshal your data into some JSON or some some serialization format. So if you keep the same communication patterns around for like this in systems, inter systems

communication, then you'll run into issues because you you accumulate a lot of latency and you generally want to communicate less, right. So it's like in a distributed systems world, you want to have more coarse grained requests and you want to communicate less often, really. So that might already change the the approaches to architecture quite a lot because you might even want to design your system in a way that it doesn't actively reach out to other systems, but rather publish events or something. But even if you go with events and the same messaging, right, you still the systems have to agree on on on something because as Kenny already mentioned, even if you don't formally go ahead and define like the shared semantics, you will end up with something that's just an implicitly defined. Right. So the fields that that adjacent document has, for example, and then you, of course, do yourself a favor to to negotiate that between the involved parties and and define those define those things and then be able to to make some guarantees around that. The thing that I was kind of slightly disagreeing or not even disagreeing, I was kind of questioning or just wanted to to put an attention on the on the topic is that it's quite important. So the level of coupling you create with that is totally is very much depending on what exactly you put into that contract and what what aspects you focus on when defining the contract and different approaches to that basically having different consequences in the in the amount of coupling you create. So I want to point out one quick thing, and then I heard you had a question, Kenny, but the the things that all of us have to kind of reconcile across all this is there's tension. And you mentioned this in the blog post that what you would do to optimize solely for performance and latency might be different than what you would do to optimize for, you know, this this flexibility and decoupling.

And just kind of like upfront recognized that there's definitely tension and compromises have to be made when when you're when you're engineering these things. So there's not necessarily a right answer or a wrong answer. It's a question of what are you optimizing for in this specific case? Definitely. Yeah. So that was it was definitely one of the core drivers that we also got into discussion on Twitter about like what's the or actually we didn't get into discussion about that on Twitter. Not explicitly, but it's sort of flew around in between the tweets. So like the question of like, why do you actually separate the systems in the first place? What's your actual goal? And how does your your your your architecture actually support that? And I think with with microservices, it's usually that you want to to enable the different teams to ship features independently. And you the teams get more independently, the less they have to talk to. And that's probably then something that Andrew is going to disagree with me. The less you have to you have to talk to others. That's probably the nuance I want to emphasize here. I'm not saying you don't want to communicate. But if if you're forced to talk to to other parties in in in that scenario, that's that's not it. That's not a good thing in the first place. That's rather something you want to avoid. Right. But of course, if you if you if you have to agree or have to talk at some at some points, because you you have to define like the shared semantics of APIs. But yeah, I think that's it. So I think I want to add one thing there, Andrew. Just to make sure that we kind of follow that thread of what you're talking about, the expectations between certain microservices that expose REST APIs. Now, Ali, you having worked on and led the Spring Data REST project, you found yourself quite familiar with hypermedia, specifically Hot OS. Can you speak to how Hot OS can help alleviate the problem of having these contracts between services?

Yeah, it's also I mentioned that in the in the blog post or it's more in the end where I basically get to the point where what I propose like in a maybe not as an alternative, but what I propose to to focus more on when it comes to the definition of these kind of contracts in the broader sense of like, what do we agree on like between systems? Right. So the the aspect of hypermedia in REST systems is often a very overlooked one. But it's actually a very, very effective means to reduce the amount of coupling between systems. And the way it does that is, I mean, there's like it's a two level thing, basically. The first thing is adding hypermedia means to the to your representation, which means you would add links to your JSON, just like a web page has links. You would go ahead and use one of the of the of the of the formats that that that are around. There's a couple of them in JSON hell and collection JSON and Siren and whatnot. But it basically enables you to put links into the responses and the client could actually follow them that in the first place basically only links data to data. So that's there's nothing really, really fancy, but already helpful. And the tricky part and the one that's that's actually one of the most important ones in REST in terms of the the evolvability aspect of APIs is the so-called hypermedia as the engine of application state, which basically means that depending on the resources state, you would add a link to to the representation. And when that state changes, you might even remove it. And that can be used to basically signal the client that it can do something under certain conditions. And you might ask, ask yourself, OK, why? How does that help in terms of evolvability that helps because the client doesn't have to look at the JSON content, maybe the individual fields in too much detail and then basically replicate logic.

That is already on the server itself, which the replication actually would lead to the coupling. Right. So let me give you an example. There's like let's say you have this this kind of order in your in your system and there's a business rule that says the order can only be canceled if it's not paid yet. Right. And there's a human readable status field in the representation. It says payment expected or something. Right. And while that's the case, it can be canceled. So if you now bake the logic into the client and literally program it like if the status field is payment expected, then present the payment button, let's say for a mobile client or something. Then you've added you've added you've added additional semantics to that field. Right. Stuff that's not even in a in a contract somewhere, not not something it's probably documented somewhere, but it's not not technically in a in a in a in a JSON schema or something because you just can't define it that way. If you then have to, for some reason, change the the the the field or the the condition. Right. Let's say we can now cancel orders if they are paid already. Right. Then that that client replicating that logic becomes a problem because now we have to communicate the that change to the client and then potentially negotiate a new contract or whatnot. And if you're instead introduce hypermedia means and only tell the client if there is a cancel link in the representation, you can issue a delete request to issue the cancellation. Right. Then that's canceling can appear in different in different other scenarios. And the change in when that cancellation is actually out is basically condensed into a Boolean yes, no question on the client. Right. It just needs to look for is the link there or not. And if so, I could do something. If not something else, I can't do it. Right.

So and that helps helps the to to basically to keep a lot of the the logic on the on the server or the details of that logic, basically. Um, and actually allows it to or makes the the this the scenario of a breaking change in a in in some in some in some in some logic less likely or that it yeah, makes it less likely that that you run into that breaking change scenario. Does that make sense? Yeah, I agree with everything you said and it definitely makes sense. Because the the thing that is kind of interesting, or what I was reacting to with respect to communication is when when you introduce something and you gave this example of the cancel, and that's going to be, you know, in there's more than one way to solve this problem. But, um, I think it's actually pronounced hatios, but whatever, um, you know, going forward, you still have to tell like someone else on the client to implement any logic to be able to use that. So, so certainly introducing those things and not breaking, I think generally, there's a bunch of tricks you can use with respect to how fields appear and what the logic is. And, you know, that, that could be, um, some implicit, um, but, but the kind of general Postel's law robustness principle still applies that you want to be conservative in what you do, and you want to be liberal in what you accept from others. And that solves like a bunch of problems people run into with really strong coupling. And I think what you were reacting to is when, when you do things in such a way that it creates a lot of unnecessary communication, it creates a lot of overhead that is, it's not inherently adding any value. It's just there because it's part of the, the quote unquote process of the contract that, that I also have a visceral reaction against.

Yeah. I mean, the, the, what, what, um, um, um, one aspect coming up here is that if you, if you, if you just from the example I mentioned, you, uh, you realize it's, it's an, it's an, it's an, it's been some functionality that's like actually implementing more, more business value. Let's say like the, the, the, the, the cancellation of an order, right. That that's not just like an, an arbitrary, oh, we get some Jason, we change some values and we put it back. Right. So we poked at, at the data basically. It's more looking at the API from a, okay, the API allows me to actually implement business functionality. So it's not a, not a, not really a sort of a mimicking a data access API over HTTP really. Um, and that's probably, um, that's a, that's a, that's a crucial point because if you're not writing these kinds of, um, these kinds of APIs, then that benefit might not even be as, um, as significant to you. Right. And I just read up the, the, the, the original, uh, article from Ian about consumer driven context. He has this notion of a conversation. So when between the client and the server, there's more of a high level conversation. Um, then that's kind of approach is kind of, um, is, is, is, is very useful. So, whereas in other scenarios, which are, which might be perfectly fine, right. Where you just want to have an HTTP interface to your, to your data, right. Uh, just because you want to add security to it or whatnot. Um, and that, that, that those high level conversations are not the main driver, um, then the hypermedia aspect might not be as important to you, but then you create more coupling between the services. Um, but if you just, if you're aware of that and, um, um, deal, deal with it accordingly, then, then that's just fine. It's just, um, I just don't want to like create the impression that there's like a black and white and here's the right thing to do always.

Uh, it's, it's more of a, you have to find out what kind of, of API you're actually creating. And I'd argue the more coarse grained your systems are and the more coarse grained the systems that interact are, uh, the more you actually want to expose real business functionality via the API and the more important those, those aspects actually get. The more, the more, the more you get into the smaller and basically let's say have a, like your, your shopping system. And then you realize you have to like scale your, your reads and writes differently. You might even want to split up them internally, like, like have the, the, the business split on the, on the macro level and then internally even, um, create different, uh, different services, uh, just to be able to scale them differently. And then it's a totally different scenario because you're, you're, you're controlling the scope, like on a, on a different level and, um, um, can then also live with a, with a bit more stronger company. You might not even want to use rest in that case, but like a more, uh, more, um, efficient protocol here because like on the, on the, on the smaller scale, the, the performance implications might be different ones as well. So, um, yeah, trade-offs as always, right? Exactly. Good. So, yeah, I mean, Kenny, I want to make sure Kenny Schaefer, you have other thoughts on that before we dig in. Yeah, I think I'm fine with that. I, I, I think we can, if Andrew doesn't have anything to add to that, I think we can move on to kind of the next topic. Well, actually, I want to, I don't know if this is moving on to the next topic, but I kind of want to introduce a scenario and, and, and we'll throw it back and forth about what people think you should do. Uh, so, so you have some service, has APIs, it's, it's, I don't know if we need to get more specific about, you know, functionality or canceling or whatever, um, that functionality is, but things talk to things over network. There's, there's, there's clients. And at some point you're successful and you have a bunch of clients and maybe some of those clients are third party.

And, and you, you need to make changes or you want to introduce functionality. What, what is your general strategy to, uh, either support backwards compatibility or, you know, kind of broadcast, negotiate. And people have tried a bunch of different things and you brought up in your blog post, which is also one of my pet peeves. Uh, people start putting, you know, slash V1, slash V2 in the endpoints. They do like a bunch of, I don't know, there's a bunch of like weird version juggling things. People try to do this, solve this problem. But I'm, I'm curious to see or hear what you, what you think the options are or what you would advocate, uh, to solve this problem. Um, to be completely honest, the, the, the, the point I'm, I'm, that's not the point that I made really. I've just referred to it. Uh, it's, it's getting back to one of the interviews with the Roy, actually Roy guy behind the rest dissertation. Um, is that it, the, the problem is not really like which means to choose, but that you're actually better off if you're not thinking of versions, but rather, uh, consider the thing that you build. If you, if you have a breaking change or, and you need to, to ship it, um, think of it as a completely new service that by actually doesn't actually have any relationship to the, to the, to the, to others. Because then all of a sudden you, you're not creating these, these implications about like, okay, uh, what, what, what parts of, or which, which resources of that API do actually change? What about, but the, the caching state, uh, can I actually negotiate between, between the two quote unquote versions? Um, or, um, is it, is it just better to, to think of it as a, as a completely different, different service? So, I mean, there, you, you already can just like go ahead and use different like host names or something. Right. So that's, again, this is, again, uh, rather considering it a different service than a version of the same thing, because, um, then the clients just like get, get all confused about, okay, which version do I request?

And what kind of answer do I get? And in the end, it doesn't really matter if you're using them like an HTTP header or a, a, a path segment or a subdomain for, for, um, for, uh, your services. Uh, because yeah, the client will have to just detect it either way and then, uh, talk to it and potentially be, is potentially able to, to reuse part of the, of the logic, but it will also have to just like rewrite other parts of it. Basically the ones that, that you broke. But just to play devil's advocate, that's kind of at odds with, uh, the recommendation from the Restafarians that you should just have that one be a canonical, you know, like the, the argument people often make is there's, there's only one website, right? And that, that website's going to change over time. Right. So, so like you don't, you wouldn't all of a sudden have, you know, another, another website for your company just because you wanted to update the, the thing and like keep that one. So like old clients would go to the old one. Yeah. I mean, of course the, the, the, the further you can get away with, let's say a content negotiation or something with, with version media types and something that's, that's something you can do. But at the same time, um, if you're, if you're, if you're, if you're deploying a new website of your company, you're actually not doing a breaking or you're not, you're not introducing a breaking change because it's still just the website. You just go to the root site and you get something and then you, you find your ways through, through the website by following the links and stuff. So, I mean, having, having been, uh, the victim of a new website, uh, more than once in my life, there's definitely breaking changes from a human usability perspective. When you've got a routine that you're, you're doing on a, on a website that has a certain user interface and that changes it from under you. Right. I'm not sure that that principle is that much different than, than the, the, the, the adding a cancel button to the, to the, the, the payload.

The point I'm trying to make is that what do you do as a human being? If you find yourself like with going to eBay, like after a decade or something and not finding the buttons at the places that, that you used to find them, they're either gone, right? I mean, they, they, they basically just dropped functionality or they moved it to some other place. And then you have to find some, discover it to some, to some degree. Uh, but it's not that you, you, you, you, you, you exchange the, the eBay website with a eBay website. And then you all of a sudden need to be reprogrammed completely as a, as a user. You just like go ahead and still use it. And it, it, it, it basically, uh, because from, from the web point of view, from the website point of view, nothing has changed. It's still HTML and HTTP and get and post requests. It's really, um, the point is actually that that's a, that that's, it's, it's not a good, a website change is not a good example for breaking the API because a breaking change in the API is usually, um, semantics of the fields changing, right? Or, uh, some, some, some, the structure may be in the JSON change. Um, but that's not going to happen in the, uh, in HTML. And also the HTML is still HTML. So the media type is still the same. Um, do you see the supportability change there though? I mean, is it realistic to have teams maintain three or four separate services instead of a, I mean, maybe it's the same as maintaining versions, but it seems like an operational burden as well goes on there. Exactly. I mean, that's exactly the point that they, um, there is a great blog post. I think we can, we can link to that in the show note, uh, the, the, the hidden costs of versioning APIs where someone, uh, took a really like, um, he did some, some scientific analysis about what or in how far the effort of maintaining API, API versions in parallel actually increases. It, it, it, it's, it's not increasing linearly as you can probably imagine.

And, uh, he's basically making the point that it's, it's, it's worth investing, um, the time to, uh, to try to avoid that because it's, because, because of the fact that it's so much effort. And it's, it's, it's actually replicating into the clients, right? Because, um, they might, might have to, um, support it, have to be upgraded as well to the, uh, to the, uh, to the new version, because it's not only the effort to create a new version of the API, but also, uh, the, all the clients. And sometimes you're not even aware of, of, of clients. And, um, um, if you, let's say you have a public API, right, you don't even know how many clients are out there that are actually using the API and, and, and all these things complicate the matter. So that's, that's, that's, that's all arguments. Why I'd argue you, you want to avoid that as much as possible, uh, those breaking changes. And, uh, oftentimes the APIs I see, or there's, there's a lot of potential or would have been a lot of potential to actually avoid that in the, in the, in the first place. Um, but yeah, what, Kenny, what do you think about versioning, uh, sort of system of systems or microservices world? Yeah, I think in theory, you can say a lot of things about versioning and how backwards compatibility is going to affect the rate at which your consumers are going to be able to upgrade. But in reality, things are messy in enterprise software. So you can't necessarily, um, kind of put a standard forth that says we're not going to version our APIs because you'll always run into this problem where you have this brand being changed. Maybe it's a, uh, thinking incorrectly about the design of the API and you want to be able to support future changes from, from more consumers to be able to utilize that functionality, uh, in the way that was intended. And you're just always going to have those clients that, that need, just need that V1 compatibility and they don't need to upgrade.

So in, in practice, versioning your APIs is fine. I, I think that the way I would frame this right now, and this is the way I think about all these problems is there's no such thing as a free lunch. And, and, and all these are points of, of what you're going to, so, so this notion of the hidden cost, if you, if you think you're going to save something in one regard, you often end up paying for it somewhere else. And, and, and so just be thoughtful about what you're trying to optimize and whose life you're trying to optimize, right? So I definitely have this pet peeve for, uh, the versioning in the API or the versioning in the endpoints or, you know, the thing people do if they put a .m in front and redirect you for your mobile device. Uh, I, I don't personally, like, it's annoying to me as a user, um, but someone did that because they're optimizing for their local, you know, lifestyle for their, for their experience as a developer or whatever, and had some, some operational or developer benefit that they were optimizing for, um, that might make someone else pay that price in another way. Right. So, Kenny, so let's, you know, a lot of our customers are starting to do some pretty big re-platforming efforts. I mean, we're, I tweeted out something today, one of our customers doing a, something with millions of lines of code that it's worth a billion dollars a year to them as an application. They're still refactoring it into microservices, doing some other things. How, how do you advise people to, you know, what are those tangible things up front? Should they use Spring Cloud Contract if they're using Java? Should they be, you know, what are they supposed to be doing? You know, where are those pitfalls that you say, gosh, especially up front, don't do this right away. And it might feel like premature optimization or things like that. Help people who are starting to tackle some pretty big efforts to get into more microservices or getting more into cross-service communication, not make mistakes that they're going to regret 12 months from now. So, I think this is a really interesting topic. It's a good question. I think right now there's just a lot of confusion around how do you effectively test a microservice architecture.

I don't think we've seen the patterns emerge yet for how that's done, especially cross-platform. If you're using different languages, if you're running in a polyglot environment. So, what contracts give you, in a way, and I think directly to how we implement them in Spring Cloud contracts, which is a little bit different than what Ian Robinson had posted with Martin Fowler on his website, which really takes this perspective of, I'm going to publish the functionality of my service as a contract stub. I'm going to, as a producer, assert against that functionality so you know that in production, this is how I intend to use that API. Now, the consumers don't have to implement the same exact test. They can implement just certain parts of that test or they can assert against different conditions. But essentially, you're going to be able to publish that contract as a producer, as a test, that your consumers can just download as they go through their build and test environment and then run integration tests against that functionality. Now, this doesn't couple you to that service's expectations. You're just coupled to how you expect to use it. And not exactly in production, but your advancement into the production environment. So, you have to go through that promotion process to get there. Schaefer, do you have war wounds from this? How are you going to keep somebody from shaking their head 12, 18 months from now going, that was a poor choice up front? I think, honestly, if you're going to be involved in computers and software, you're going to have that problem, no matter what you do. Because today's best practices are always tomorrow's legacy code. But where do you see premature optimization, that people sometimes start too quick? Do you see any here? I mean, I think that the, kind of going back to some of all these original responses, I think that there's things that are strategically decomposed to enable teams to deliver, continuously deliver.

And then there's, oh, buzzword compliance. Let's, like, make everything small and separated by high latency network communication. Those things are not necessarily the same. And I think that, you know, outside of the contracts and whether you're going to go to a tool to enforce them and test them, I do, I love some of the stuff that's going on with the contract testing as a mindset. At the same time, I feel like every generation re-implements CORBA. Or then, you know, then they call it SOAP. Or, and then, like, let's put XML in the middle or whatever. But in general, what you really want to be able to do is have the ability to have a high confidence that what you're going to do is going to work. And the tooling can certainly help enforce the properness of that communication. I also, going back to some of the things I was reacting to, Folly, don't think you ever get out of the fact that smart people solve problems. And that if you have smart people on both sides of an equation talking to each other, just going back to the kind of truism that is Conway's law, if there's problems between two groups of people working on the subsystem of the larger system, then that's going to be reflected in the API. And whether you're going to enforce that with code that has, you know, hard enforcement of the policy as a contract, or you're going to enforce that by convention as human beings cooperating with each other, is kind of left to, as an exercise for the reader. But to me, it all goes back to people. This is all, software's about people optimizing the experience and the performance of other people. And you can never get away from that high value of communication. That's well said. I wanted to quick click on that one thing because you tweeted something. Not that it's not always sarcastic, but you tweeted something today about that.

Once you add message queues, you have two problems. You know, when you think about... I said some, because it's actually, there's a very famous quote about regular expressions. Right, right. Now you have... Now you have two problems. And so I said, sometimes when people have a problem, they think, I know I'll solve this with messaging. Now they have queue problems. I appreciate the pun. So when you think of inter-service communication, do you think that that's... Queuing, again, it's probably based on scenario, at least from my experience. But other cases where you see a crawl, walk, run of, hey, first, maybe you even have synchronous web service communication. Then maybe you track latency and things go to async or eventually go to messaging. Or, you know, do you have any sort of decision criteria you'd like to make with those? I mean, I think that there is this missing language. This is one of my favorite topics, actually, around architecting systems. And often when people talk about software architecture, they start talking about this kind of ivory tower of abstractions and how they communicate together. But when you sit and think about the system and start especially mapping out what people will euphemistically refer to as nonfunctional requirements, the way that you queue, the way that you cache, the way that you do all these things can have a huge impact on the qualities of that system. And I think that there's kind of like a missing pattern language that the industry doesn't have to discuss this properly. So kind of as a rule of thumb, like I look at a problem and it's like I can think about the throughput, I can think about the numbers, and that starts to kind of have an intuitive sense of like where I would queue or where I would cache or where you could get away with synchronous or whatever. But I don't feel – and I'd love to hear what Kenny and Oliver – I don't feel like we got that proper language to discuss those patterns across the industry.

There's like people that have an intuitive sense of it. And, you know, there's little patterns and people, you know, pull out specific things and, you know, whatever, say things like CQRS or whatever, and that's great. But there's not really like, hey, when you see this thing, you should solve it exactly this way. I don't feel like we really have well-defined patterns. Kenny, what do you think? Yeah, I mean, I just want to reiterate what Andrew just said about us not having a pattern. So if you look at the last 15 years of software development, it's really about companies who have innovated, have been pioneers, who have led the way with a certain approach that was seen to be successful in the long run. And then you'll have these thought leaders or experts come along and write books and establish the vocabulary. We've seen that a lot, especially with Agile. But I just wanted to reiterate that point. I think that's a great point. I feel like we're at the phase where it's kind of a pop culture and there's pockets of people who've got the experience and the intuition, but the language is yet to emerge where we're really able to have kind of semantically dense conversations about these patterns yet. Yeah. Yeah. Ollie, thoughts on are we missing patterns? I mean, I've got enterprise integration patterns up here on my bookshelf and read that for the last 10 years. But I know what you're saying. I mean, the integration space has been talking about some of these just straight up messaging patterns for a while. But I understand we're talking differently as you factor in all the different communication aspects that involve connecting systems. Well, I think there's certainly patterns that are referenceable in the book with respect to queuing, caching, whatever. I think those are – I wouldn't say they're well-established, certainly not to the degree that some quote-unquote patterns got established in other aspects of software. But the thing that's absolutely missing is being able to kind of qualify and quantify the needs of a system and then map that into those patterns.

That's well said. I think that's what's missing is some of that when is this the right fit versus like I'm just going to use this sort of wire tap pattern here. I'm going to use this pub sub here. Some of the worst things I've ever seen in my technical career were people misapplying patterns to solutions that they were never intended to be. Actually, it's one of the aspects that I like about the REST dissertation so much. Who of you was it who wanted me to define REST, actually? I threw that as throwing it down the gauntlet to really rile people up. I don't have to define it because it is defined. I especially like the way it's defined because that sort of plays into what Andrew just said is people usually think of it like URIs and there's HTTP methods and whatnot. And that's actually already like the implementation step to some degree because fundamentally it's about like there's certain characteristics or here's a set of constraints that were applied to the thing called the internet or the web and HTTP as its protocol implementation. There were some constraints about, let's say, the uniform interface, right? Or there's a well-defined set of methods that really have to get proposed to the internet. It could be something else. But you constrain yourself. You put constraints onto yourself and by that you get some effects, right, of the scalability, evolvability, and whatnot. And certain constraints imply or have a much bigger impact on certain effects on the system. Just like I mentioned, the hypermedia or HATOS aspect being the most important one of all of these in terms of evolvability of APIs. It also plays into scalability, but maybe not as much as statelessness or what have you. So fundamentally, the idea of defining the constraints and the effects that those apply or adhering to those constraints then causes might be an approach.

To actually define the same thing for the things we just discussed, like, okay, do I want to avoid the inter-systems communication, right? Like, while I'm serving a user request, I don't actually call out to other systems. What effects does that have on the way I need to deal with data, the way I have to approach consistency within the overall system and whatnot? Maybe one example that we can give our listeners is that the self-contained systems approach. There's even a website about that, scs-architecture.org. It's basically coming up with a set of these constraints or some, what is it, heuristics or – I'm lacking a good term for that – but a set of rules, basically, that you force yourself into when designing a system of systems. And then giving you, like, or describing the consequences that that has on the system that you will be creating. It's a special kind of flavor or granularity of microservices to some degree. But I found it to be a good example for that kind of, okay, let's agree on a set of rules and see what kind of consequences do we get from them. I agree with Oliver, and I'm just trying to tie it back to what we were saying. I feel like the people who've had experience over the last 10 years are starting to have those heuristics. But what we haven't done is taken that next step where those are codified and that we can have rich conversations about them. And who do you – I'll be interested in your take. I mean, this isn't just an architecture discussion, especially Ivory Tower, but not even – so is this engineering – every developer should start to be internalizing some of these things?

Does this stuff emerge from an architecture over time, or do you not really believe in that and think that these are some overarching things that should factor into a system design? Kind of who plays the role on projects to make sure that some of these things are done in a good fashion? Kenny, what do you think? Yeah, so I would say the architect still has a place. Maybe not the ivory tower architect, but an architect who can maybe embed themselves on a team and kind of give a set of rules for how you're able to test, especially, against microservices that you consume. Because at the end of the day, if we have 500 microservices in an architecture, and sure, there may be modularity in a certain set of microservices, but how do you really think about going into a testing environment and testing against maybe 20 or 30 microservices without having to strap an entire environment to test one-to-one between each service? So I think that's what we're seeing with things like Spring Cloud Contract, where you can simulate a service based off of a contract. But yeah, I think that you still just need knowledgeable people on your team to be able to set forth these, I guess, experiences that they've had in the past developing in this architecture. Yeah, Oliver, thoughts on who plays the role in this to make sure that you are thinking of the right principles? Everybody or certain folks? I'm totally with Kenny here. So I guess it's for some of these aspects, we just have to learn from the people that built these kind of systems, that tried different approaches, and found out what was working for them. But at this point, it's kind of important to abstract from the concrete situation they were in to the fundamental context. So not every company is a Netflix, right? So what might work for Netflix will only work for you if you're in the same situation, if you have the same driving forces.

And what else did I want to get to? That thing, the... I'm sorry, I totally lost it now. Anyway. No problem. Schaefer, your thoughts on that? Who takes responsibility here? I think that the proper way for an organization to answer that question is it's everyone's responsibility. At the same time, everyone probably has a little bit of a specialization role. And the role of the PowerPoint jockey architect should probably be an endangered species at this point, although I know they're still thriving in certain pockets of the enterprise. The things that have been successful, if you just want to look at examples, you can see the things that happen at Amazon. You can see the things that happen at Google, Netflix, whatever. As an example, there's often a role of the architects or SREs or whatever who they don't have a project that they're trying to deliver the functionality for the business. So much as the mandate to roam around and look at how things are working and focus on making them better. And then another thing I would add, and this is something I think people learn the hard way one way or another, sometimes by accident, is that there's a high value to experimentation. And one of my mentors early as an engineer always said you should never implement the first thing you think of, that you should force yourself to think of a few solutions and articulate, at least in your own kind of mental process, what the tradeoffs of those solutions are before you make a decision. But a lot of things you can never really know how things will work until you make them work. And that is sort of undervalued in a lot of pockets. But going back to – and I also feel very strongly that a lot of what's happening with microservices and trying to build reliable systems can learn a ton of lessons by studying Erlang and the thought processes of Joe Armstrong in particular around reliability.

And one of the things that the Erlang said that I think just resonates, and I say this all the time in meetings with customers, is that if you don't experiment before you put something into production, then production is always an experiment. Mm-hmm. Very good. Oli, did you get your idea back? Yeah, I got it. The thing I wanted to mention is that the reason we're actually talking about stuff like that, like about microservices and systems of systems, is that we've all made experience with the other approach to that, to bigger systems, which were monoliths, right? And we tried, and some succeeded, and some drew the necessary conclusions for new projects from that, and we drew conclusions from the failure. And that's basically what we just have to go through for the applied distributed systems area that we're – or era that we're actually in right now. Absolutely. So that's just – I guess we're just going to have to learn. That's probably it, and that's a good thing, I guess. So just to add the way that I say the same thing is that the way that we got to these microservice architectures was not because someone set out and said, oh, I need to build microservices. It was a Darwinian evolution that was solving the problem, right? Like Netflix didn't set out to build microservices. It was the way that they could work that solved all the constraints of the problem that they were trying to solve. To piggyback off that, so having done some research on the topic for the book, kind of looking at this evolution of where did microservice architectures originate from, the earliest reference that I could find went back to 2005 or 2006 when Werner Vogel's CTO of Amazon explained the processes for how internally their engineers were thinking about delivering Amazon.com without having to have this one monolithic application.

And it came down for him when he explained in this article. He explained that it really was about control where different teams were unable to control their own destiny because they were sharing resources, and that gave them a constraint which prevented them from being able to control how they got features into production. So I think that's interesting, too. Very good. Well, hopefully for our listeners, they've got lots of chances to try to keep tapping into this stuff. Kenny, I think you're part of this Refresh Cloud Native Roadshow that's kicking off. Is that right? Yes, yes. With Casey West and Andrew. Yeah, so we'll be able to find you on the road. And obviously, Pivotal folks love helping teams do some of these decomposition exercises. Hey, we're always learning, too. It's not like we come in always with the exact answers. I think we learn with our customers of what fits their environment. But hopefully they can work with us or continue to try things out on their own. And the most important thing is get out there and start doing this stuff. All right, fellas. I appreciate you coming on for this very special edition of Pivotal Conversations. Well, and thanks for listening. As always, this has been Pivotal Conversations, which you can find if you go to pivotal.io slash podcast. Or if you're feeling really sprightly with your fingers, you can go to pivotal.io slash pivotal hyphen conversations. You might be feeling lonely that you haven't talked to the hyphen for a while. You can find our not-so-secret back end also at SoundCloud, which is at soundcloud.com slash Pivotal Conversations. And Richard, what's the thing that we like to ask everyone to do if they've enjoyed this episode? I mean, other than send us raw bags of cash. Right, right. Small unmarked bills. Loose change, Dakota. We do love if you add some reviews, you know, rate the podcast if you find this interesting. It always helps us and helps our boss justify our time spent doing podcasts. That's right. Well, we'll see everyone next time. And thanks again to everyone for being on. This is fun.

Tanzu Talk · 🍕 Get more pizza.