Concourse, with James Ma and Lyle Murphy (Ep. 83)

Episode 68 · 48:24 · SRT · archive.org · all Tanzu Talk

well uh this week we've got two people on as a guest the four person four person uh episode here why don't you introduce yourself to guests you can however you want to do if you want to battle it out do rock paper scissors whatever figure out which order you want to go in let's see what happens all right so i guess i'll go first uh this is james mod the product manager for concourse here in toronto i work with wow murphy who like can introduce himself um the product director for total r&d um one of my focuses is concourse here in toronto as well and and is that uh like like what's what's going on up in toronto at the moment what what season is it you know like you gotta i mean it's obviously you got halloween and things like that but like what uh what do people do up there now it's like a it's like fluctuating between spring and winter yeah honestly my favorite time of year it's uh 10 degree 10 to 15 degrees celsius it's jacket weather um but you know it's still nice it's sunny the grime of the november rains haven't come down yet so you know trying to enjoy the outdoors while you can before the slush sets in oh yeah it's it's like it's like the ideal of how life is supposed to be that i got from all those john hughes movies in the 80s like like that's that sounds nice there weren't a lot of movies filmed down here in texas it wasn't that that gritty so i forget do y'all have like do y'all have like those sky bridges and tunnels and stuff up there like we have an extensive underground network yeah i don't know if this is true but i heard the second largest um indoor network of like walkway paths i think it's second to some american city so i'm not too sure someone can call me out on that um but yeah in the winters you can basically made it for make it from the main trade station to our office with only a few minutes of uh braving the outdoors so now now

what's like what's like the social culture of those tunnels is it is it sort of like you're not supposed to acknowledge other people exists or do you smile at people like how does how does it work out like lemmings and people say sorry yeah it's kind of like our own highway because like a lot there's like a huge economy underground economy of coffee shops and convenience stores so if you want to make an exit you gotta you gotta like do lane lane changes do a little shoulder check make sure uh you're not you're not stepping in front of someone if you try to make it to timmy's you use use the uh pair of pair of pathetic blinkers as it were that's that's good so now now last question before we get to whatever our ostensible topics are like is uh are the is the coffee down there good do you have the thing where it's like good coffee down there or is it just like you know coffee coffee oh no we we have we're i i as an office we we are very proud of our coffee selection chosen probably at least two of the top five coffee snobs in uh in toronto i see it's arguably better than the offerings you have in san francisco oh that's impressive i know shots fired uh blue bottle watch it yeah well it's not new york blue bottle has like i feel i feel like blue bottle has podcast ads now so they're definitely mainstream so maybe maybe maybe it all all success and coffee it it leaves room for new entrants to come and try new things so i said that was the last question but now now actually since you've professed your coffee snobbery or intelligence so what what exactly is a piccolo what is that the cortado that you get to leave with a cortado that okay that's fair a lot of uh most i would say a lot of coffee places worth their dime there's certain drinks that you're not able to put in a paper cup um they reduce the flavor nodes and everything else cortado is one of those um so a lot of places will not let you give a cortado to go but a piccolo is a clever way to ask for a cortado to go

yeah that's an unexpected answer i like that that's good i was just reviewing uh someone's someone's notes for a SpringOne platform talk that was basically like um at the management layer here's all sorts of like um uh hopefully devious ways to get around management stuff and i and i feel like that's the answer you just gave there is like i would like to take this drink out of the coffee shop but i have to trick the whole ecosystem into this being an okay thing to do so i'll rename it which which i think there's probably a lot to learn there right for doing uh the old digital transformation well we have uh we have some news items to go through then we'll get back to our uh our guests of course if y'all have any uh input beyond uh you know ways to get a cortado out of a coffee shop about what we talk about feel free to chime in but i think the first item i i guess uh i guess i saw a lot of write-ups about this all all of the uh the public cloud stuff according to the numbers seems to be going uh rip-roaring good i was thinking i was reading through the earning stuff of last week and thinking like oh i should i should have gotten a time machine and invested in those companies that that seems like a good idea but uh yeah the results seem pretty good i think if i remember wasn't there some big uh benchmark that that google was approaching there was some article that they're uh they're about to reach some new run rate on on their their public cloud spend i think it was microsoft was hitting the 20 billion dollar yeah run rate and then uh google still doesn't break out their cloud revenue so you're still guessing as to like other buckets so they growth still seems good microsoft had really good azure growth it's funny they they still bunch it in with some other stuff as well in some areas so you wonder is azure kind of lifting up some other kind of flatter server revenues and things like that but microsoft did great amazon did great what bezos is the richest person on the planet now again so good for him but uh you know the growth rates are a little different i think microsoft's growing a little faster but i think amazon is still five times bigger than the next four or five cloud providers combined so all of this stuff is

scale related but most importantly everybody's making uh or the big three are making lots of money in cloud yeah lots of usage that that seems to be uh you know when i go gallivanting about i i think i think pretty solidly those are the three that come up nowadays like i was i was in uh i was in dc last week and there there was there was a fair amount of consideration i mean it was nice hearing about various agencies who actually uh want to use public cloud and are intent on it which i think is a pretty big change since i don't know two or three years ago where you know uh no one in in government seemed to want to move to public cloud and in in mass if you will but uh yep those are the three that people tend to cycle through so then over in microsoft land they have uh i i guess we didn't cover this last time but they they uh they have a uh a kubernetes service as well so everyone's got it that's right hot off the press yeah they did this last week uh i mean and they did the k too so clearly you know it's it's the azure container service with a k so that's uh that's a thing now so yeah everybody's doing some sort of cluster somewhere which is interesting and hopefully the conversation continues to go down to though what is still the right way to use these things what is the smart things to put inside let's just not i don't know move on to the next shiny thing and necessarily create ourselves a bunch of technical debt we're all going to deal with in two more years but hey you know clearly uh popular stuff after you and i talked last i mean that that scenario you described i mean surely i mean that never happens right that's that that seems like a pattern that us in the tech industry could easily break out of absolutely we'll be fine this time it's different as the t-shirt yeah exactly and then i didn't look at the uh i i saw this uh github contributor thing floating around but i didn't look at it so now here we are what's what's what's in it yeah it was some analysis of kind of which companies are contributing the most to open source and you know it's just kind of surprising i guess to some who may not be as aware of pivotal but you know and then james governor red monk also did some analysis but you know like pivotal

contributes as much to open source as ibm and significantly more than companies like amazon and others so great to see you know microsoft google others leading the pack there but you know and the methodology is not perfect you know people could be submitting using their their personal accounts and it's not representing the company or this or that so it's not going to be perfect but there's still a clear indicator that you know companies like pivotal are investing a lot in open source and other companies are doing some but maybe not as much given they're they're much larger size so let me ask all three of you this i'm i'm a little bit uh you know as i'm fond of saying i'm good at slides so i'm a little bit out of the at least i like to think i am uh i'm a little bit out of the coding thing but like what is what how would you describe the currency of github stars like like what is that like how does that what what how is it a good thing or a bad thing or like what do you do with that i mean is it let me is it is it like as bad as a linkedin endorsement thing or is it somehow uh no no it's not quite that bad i mean uh stars i think are useful as like a validation of like hey people want to keep track of my project so i mean is it worthwhile to track your rising stars over time probably not uh have we done that maybe right you know at the end of the day it's a number and like it's up to you whether or not you want to track that number and uh short of some drastic like unstarring event uh we don't really pay attention to uh stars on a track an unstarring event i like that that's that sounds like something from a uh some part of like a bruce sterling or william gibson book that they decided to delete there's a short story about an unstarring event or a really bad black mirror episode yeah um it is a good hint though if you see something with uh not a lot of stars to unpack it to see like wait a second like how old or how big is this project and like it's a good cue to actually see if you need to dive deeper um into the project if you want to if you actually want to truly commit

to something yeah like ryan morgan uh who had the spring for us um from measuring perspective has like a theory around like it's an indirect correlation but maybe not a good one in that it matters to a point when you're really small to make sure that's growing but what you want to make sure over time is like your ratio of people who are actually contributing and whether it's writing code or just participating and like getting issue comments and everything else in theory as your stars go up hopefully that grows at least the or maintains the same percentage um but i think it's really easy to get i think to Coté's point it's really easy to get kind of um tied to like reddit upvotes or it indicates something happened a lot like there may have been a good blog post or something that generated people to go and start that repo but if you're not getting that actual follow-up um and commitment back to the community then it becomes almost like a smell oh smell you know when i first started at red monk a while ago i wrote this paper called the smells of agile but you know i know what a smell is that's uh sorry i got distracted there was a deer in the street which is really odd uh so uh yeah that's that's good input maybe i should go figure out how github works do do a pull request github stars is just as relevant to the success of your community as velocity is to predicting what's going to happen in your agile working team oh no that's good that's good it it can be a pot it's likely that it's a positive indicator but you shouldn't assume that it is you should check in on it yeah yeah huh all right well uh you know uh there's also some neo4j news what's that richard i was i was seeing that and i was thinking uh you know kenny on my team he used to work there i should i should see if he's excited about it did he yeah you know they've been around a while i just you know i look for kind of interesting product news that

isn't just the top three cloud providers and you know neo4j has been around a while as a graph database graph databases are cool they've kind of gotten rolled into other things sometimes you'll see these multimodal databases that represent graph and key value and other stuff like that but neo4j is alive and kicking they kind of kicked up this newer kind of graphing platform and again i just like seeing some of these smaller database providers whether them or mongo or others not just uh kind of rolling over just because oracle or microsoft or others add these capabilities to their products they still think they can offer best of breed and i think that's a cool example so if we have customers interested in things like this they should let us know to see if we should get them into our marketplace that's a good point and nothing sounds cooler than a graph database except maybe like right machine learning that yeah you're gonna be a you're gonna be hot this year at thanksgiving aren't you just dropping like graph database and yeah github stars you're gonna be big yeah you know i've been thinking i've been in in well i don't know we'll see and in such a uh i feel positive steady state people keep i run into them they keep asking me what's new and i have to say like well it's pretty much the same as last time which uh i need to work on that for my thanksgiving game come up with some new uh exciting stuff i can tell them i ate grub worms in china that'll be a good that'll blow a good 20 minutes of conversation while we're waiting for the turkey yeah no you'll be saw a deer today that that's a highlight there's a lot of things happening for you right now it's another thrilling life well speaking of a lot of things happening two guests so uh we we had you on i've i've been i've been excited for this topic for quite some time to talk about uh concourse and and how it's going and what it does and how it sort of fits into the overall scheme of uh uh making my life more exciting so first of all thanks for being on what why don't you why don't uh what one of you give us like an introduction of what uh what concourse is like what it gets used for yeah um thanks for having us so uh concourse is we jokingly call it the continuous thing doer um it's very much a cicd that was born out of pivotal's cloud native journey and

in building out a path that was worthy of like sitting on uh enterprises and in multi in a multi cloud environment and first and foremost um you know it it comes with a lot of unique innovations that make continuous integration delivery in an enterprise and in a cloud native environment that much better and that much easier and that much more powerful and it comes down to three kind of core tenants the declarative configuration so no faffing around with a ui um everything you need is defined in your pipeline um and that's codified in your github repository or whatever your code management system of choice and that's a lifesaver right so if your system goes down there's no oh i got to remember to install that plug-in where's the license file for this everything's in a single place where you put your code you set your pipeline and off you go the other big innovation with concourse has been um it's reliance on containerization for consistent execution so uh a lot of engineers can relate to you know messing around with configurations and flaky builds and uh trying to get their pipelines cleaned up after an execution you don't have to deal with a lot of that with concourse we built it from the ground up uh with that kind of uh stability and consistency in mind and we we take a lot of advantage of containerization to do that and finally uh because we don't rely so much on the ui to configure pipelines we say we make a lot more use of the ui to uh be an information radiator so every time you walk around a pivotal labs office every tv you see has that concourse pipeline emitting key information about the status of a project how it's performing and you know is there any fires that you need to stop drop and roll for so so i think in in that context and i don't really i'm kind of fishing for something here but like it feels like uh builds and and build pipelines are historically

fraught with terribleness right like like you spend a lot of effort on them and and the results are always weird and things and they're i mean part of why because it's a extremely critical part of the whole uh process i mean it's it's it's good to have your software run by building it and testing it and so and you you kind of hit upon this but i mean are like what about containerizing things kind of like addresses some of that fraughtness like how does that how does that help out beyond like uh using vms or using really like you know strict puppet and chef things or whatever like how does a container improve the uh the difficulties yeah so containers are a pretty neat innovation because they're just so lightweight and easy to generate create and destroy so with containers what we do is say like hey this thing that is traditionally flaky it's flaky a lot of times because um you have to make sure the things that you need to run your tests are there and also you have to make sure that cleans up properly after itself by specifying containers we can say hey we've pre-baked a certain amount like the things that you require for a test so you can just grab a docker image that has go and the dependencies that you need pull that down from the cloud know for a fact that your test could run with the dependencies there and then um once you're done we just nuke it we just destroy it and then every time we want it we can create a new one so because of that um life cycle it becomes a lot cleaner a lot more consistent and a lot more reliable makes sense you get this benefit of a lot of these really powerful awesome tools that are in like the entire like devops um and app ops space is this idea of like things like you mentioned like ansible chef bosch um terraform which is a really cool one um you get all these really powerful tools that end up needing their own kind of like dead man switch type of thing which is like go up and

kind of clean up after themselves and like they're independently they're all very very powerful tools to do one thing very well um and and unfortunately or i guess fortunately for us in concourse these things don't need to be used one after another in certain some sort of predictable order and we get that kind of that layer of obfuscation in which this can all happen now obviously you still have the problem of a dead man switch for concourse but uh one area of failure is considerably better than say six or seven right right so you have sort of like the lightweightness which is also so so here you have the lightweightness of containers to hear it seems like tell me if i'm wrong but a fair amount of it translates to if you can do everything really fast then um it's better in in the sense of bringing something up and bringing it down and versus like if it would take several minutes even to like bring up a part of like i don't know a 50 part system as as that as a failure happens there something screwy it just adds in a time whereas if you have a much faster cycle you might still have errors but you discover them really quickly and you can fix them and and all that stuff plus maybe it's just more reliable to build build up container things and then you add to that to like all the various things for like a cloud native e which is to say a distributed application uh running all the stuff that that you need to bring a cluster like that up and coordinate everything there's lots of good little uh best of breed applications in their own silos but having a way to orchestrate all that stuff not to mention having them clean up after themselves i mean it sounds like they're like my four-year-old daughter and son who can clean up after themselves but you gotta have to have something that's orchestrating uh all of those various steps in building out your your dev and your staging and then eventually your production cloud exactly and i think it's important to highlight the fact that at pivotal we believe in in tdd like wholeheartedly right we don't have a qa department we codify all of our experiences our

testing and our acceptance into our suite of tests and now our pipelines and uh the containerization aspect allows us to do that because we run into a lot of issues where well given a piece of code how do i test that it runs on as well in azure gcp um aws vsphere and you know all these other combinations in a consistent way using a single vm it's just it's just very frustrating to do so tapping into containerization with uh cheap setups and teardowns makes it so much easier and so much more reliable i guess it can be like simply summarized like um adding consistently consistency to to things that inherently are inconsistent yeah that's a nice way of thinking about it so lyle you told me a while back a little bit of the uh origin story of some of this i mean you mentioned like james i mean there's a great ecosystem out there of devops the app ops tooling so why another one like why why did we build something that's now open source and gotten fairly popular so revisionist history what a great move but at the time like where did this come from why did we go down this path so james actually spends most of his days it's not all of his days with alex rachi who was one of the initial um it's like brainchild of uh of concourse so i think he can better answer that one yeah so um it didn't start off as this grand idea of hey we need to build a new ci cd system right it came out of us genuinely want to wanting to apply our td principles into this very massive and interesting problem of building this platform as a service known now known as uh pivotal cloud foundry and over time we ran into those issues that we talked about around how do we make um this kind of pipeline of tests and build instructions to run consistently automatically and against all these uh permutations and combinations and

i think we tried for a very very long time to work with the tools that we had and we basically ran into a wall where you know it's like we we got to try something else we need we're seeing all these technologies come around containerization in fact we're using it and incorporating it into cloud foundry it's just silly that to not take advantage of those learnings and that knowledge to create a better system so alex and chris brown kind of worked on it on the side uh as a side project um over time and it wasn't again it's not like it's easy to think now like hey it's awesome it's it's super obvious to use and pivots love it um they they had the task of creating a system to win over uh pivots and that's not an easy task uh we really like our tools and we really like our workflows and so for them to build this tool and evolve it and to get um to convince more and more folks to adopt it and use it and understand it and um over time it just became such a powerful tool that it has become an official project within pivotal that um is basically used in every corner of our offices and i i don't know what we would do without it um we rely on it for automated testing pulling in security patches stem cell updates i mean you name it it's probably codified into a concourse pipeline somewhere that's pretty awesome i know i love even uh in seattle we've got concourse pipelines up on the walls from our our pivotal labs teams so you mentioned you know some of the different scenarios that concourse gets used stem cell updates this and that so i mean i think that again when i would chat with with you guys as we were getting down this path of productizing concourse a bit for our customers and it sounds like we learned some things that maybe they weren't using it for apps as much as we thought they were actually using it differently themselves so can you tell us a little bit about concourse for pcf and kind of this journey that made us think about ci cd not for just apps but for

platforms like where in the world did that come from yeah so um it came out of um i guess the the big moment was in december and january of i guess 2016 2017 when the concourse top beta went out and we started getting this influx of issues from operators which really surprised us because at that point we had the context of like hey our app developers love doing this um we are sure that enterprise or our pcf customers would love using it too and that's still the case but it turns out that um there's a set of people who love it even more than our pcf uh app developers and that was the operators and they found it as an incredible tool for uh consistently and reliably setting up uh pcf foundations installing tiles and applying updates automatically from pivnet um because they too want to be uh up to date with security patches and all the good things that we consistently push out through our pivnet and other distribution channels and they uh found that concourse was one of the best ways to do that and uh shout out to customer zero who you know experimented and kind of distributed those first pipelines that let operators do that um it's because of them that it really surfaced this uh incredible use case from the devops side that we basically saw as this is something that's really going to take off and we really need to cater towards these folks because at the end of the day if the devops person signs off on concourse you can be sure that the application developer is going to have no problem getting authorization and encouragement to use that tool once they see the power from on high of that artifact promotion process yeah it's it's an interesting continuation of um i don't know whatever to to retcon it whatever the sre devopsy uh soup has been in recent years like the idea that your your developers and your operations people should probably have a huge

amount of overlap in the tools that they use which uh certainly not been the case in the past but in in that as as as as i think you know you're saying your your application developers would would use this tool and then your operations people would use the tool as well for their own platform so to speak the production that they run it's it's uh it seems like it seems like it's useful to consolidate down to uh the same tool and the method across across that divide well it's this theme of consistency right that keeps coming up which is like in an area that inherently forever for many many decades has been inconsistent which is like managing kind of infrastructure and everything else like we're now in this world where infrastructure has code and everything else is now a a thing what everyone wants to do um like infrastructure as code isn't a new idea it was now it's an idea that everyone can actually do um and like the great thing about that is like the biggest areas that either operators have um in terms of managing what gets to happen on their internal platform and what app developers need to go through in terms like the governance of mainly like pushing your app to production um and and so much happened before someone or something gets to type cf push into a command line and the nice thing that we're noticing with concourse being used to kind of continuously consistently do anything for you is that it gives this awesome layer to like the cso individual and auditors and governance people within these enterprise companies to say if you're using these pipelines or these set of pipelines we know what's happening we know it's consistent and if it's not consistent it fails and that causes to fix something and it helps these customers get past this these kind of internal check boxes and approvals they've made for themselves because they had to over the past many many years um and we can start optimizing and automating that

during production which faster feedback loops more else being pushed and it means our customers get to make better applications faster for their customers yeah but that ops change has to be fascinating right of actually using tooling like this to update the base platform so like i haven't seen too many places where you'd be updating you know the database platform or you know even some some of those components in this sort of rolling fashion so did you get much pushback when you first started talking about this or were ops folks eager for a way to say look you're updating os's you know maybe a dozen times a month i'd love to not patch this stuff manually especially without taking a maintenance window what was the kind of reaction when you first started approaching this i think that james's point and he can correct me if i'm wrong but so we have a team internal team customer zero which is a marriage between our platform services team so the ones who do the really hard work helping our customers get their platform stood up um from absolutely nothing um and to be honest i actually don't think r and d saw this uh to what jane was saying we actually initially were making a tile for concourse which is a thing like an app that's installed within your deployed foundation um the reason we chose not to do a tile was um the management plane that you provide yourself by installing a concourse on a specific ops manager you can no longer control the management plan your many many foundations and it was actually that team that just by working really close with our customers and seeing the problems they're trying to solve ended up doing this and we kind of just learned from them and adapted what that meant over time um which ultimately meant the first i guess quote unquote service thing from pivotal that was not a tile but supported as a bosh release yeah and i think it's a it was a combination of uh the operators screaming hey this is great we really love this possibility of automation and me being able to have weekends and not worry about the state of my platform but on the other hand

it's looking at the realities of what we were making and saying like hey this doesn't really help me if it's a tile and a foundation how does that help me operate against my other foundations so um it was this kind of conversation that we started with customers and our field team and customer zero around okay what's the right thing to do here and uh the right thing that we saw was okay let's go back to basics let's focus on um meeting some of the operator needs for uh concourse as a product that controls foundations uh from a security and stability perspective really kind of hardening that up and then putting a version of concourse out there that we could kind of put our put our full support behind and say hey use this to um manage your foundations and we're really going to support you as you go on that journey towards being a fully automated self-updating platform thing lots of thingies involved as always continuously so so speaking of thingies could can you give us an example of like for an application like uh so if i had a thingy that i wanted to uh to to build and deploy and run through the whole pipeline like what is i mean obviously it's all verbal so you can't have any visuals but like uh like what do i do what does that look like i mean i'm assuming uh that there's some yaml involved right like i mean yeah so so what what are the artifacts and what do i do and and what's the process like and how's yaml just yaml all the way down um so it starts off with uh what we call like a three box pipeline right so it starts off with a single with a yaml file that you describe your your build your test and your deploy actions and and so even that is kind of very powerful in the concourse world where you know you have to hook up your pipeline in your yaml file to uh get repositories so hey instead of explicitly executing modes you say just watch this repository every time

we do a change just test it for me because you know we might not remember to to execute those changes every time we commit so just watch this repository and feed it through these three stages so that that's like the most basic concourse 101 get up and running um you are now continuously testing and delivering things and then you can start adding in your complexity and automation to capture more use cases maybe your product um needs to go against a security checker that your team um your devops team forces you to go through just to be compliant so for example um we have osl tooling right internally and and that's to check for open source licenses that to make sure we're using them correctly so you can add another box that says hey um maybe you need to put your code in a certain location or signal another uh team to say pull in my code um we we need to do a check and once that passes then proceed to deploy and and so the the basic elements of concourse go way beyond just hey take this piece of code and test it it's actually um extended to what we call resources so everything's a resource we have um things like hey watch this um bit bucket or watch this s3 repository every time a file changes um do something with it so um there's a there's a very uh basic but powerful language behind concourse that allows you to really add complexity to to relatively basic uh pipelines and and then and then so so let me see if if i can rephrase it so you have uh you have some triggering events right like something changes and that means you need to like uh recheck everything essentially and and uh you have those resources of like it's you know whether it's just like a uh like a git repo or or maybe even just some storage somewhere and something's on there and you sort of pull that in and then i assume there's

some sort of um like from my understanding like sort of so then you'll have like a container and i assume it gives it some configuration kind of like 12 factory style and and then it just like executes whatever that box is and then i guess i assume it has a good or a bad result right and then maybe it goes to the next part of the workflow to do something else uh is that is that right so far exactly it can be as simple as that just um and that's the that's the convenience factor of concourse where you just need to specify the right docker file right and docker has such a large community around it where if you need something it probably already exists and if you don't it's relatively easy to just roll your own even if you don't need something it probably yeah yeah i think a really good way to simplify it code you're going towards there which is like um if you have like a state machine like really what ends up happening is whether you're a developer um platform engineer or anything like normally those people know the inputs that they want to have um and what's going to trigger like that black box that state machine to happen um and all they really know is the ultimate output they want so more simply put if i'm an app developer i know i need to put my bits um onto something like a like a github or a bit bucket or whatever else and a bunch of things are going to happen that i may or may not care about depending on what it is but the eventual output is my app is in the hands of users um and like the strength of concourse and all its resources is really being able to do all that stuff in the middle um for however enterprise like grow and adopt and change uh one of the things you brought up for before was the question around um not just the easy solutions but where customers are going with the more difficult um and harder challenges and we're seeing as more customers adopt microservices um which is great with that becomes a complexity of maintaining the architecture of your arch container deployment

and you have these apps of apps so you have the the final application that say we all use for say on amazon.com or else it's actually made up of hundreds of separate different applications um and we need to manage like which one of them are deployed in which order and how does it get there and if this one goes down what's the mean for this one um and the strength that we're starting to uncover the concourse is what other partners and people can we find in the space to help us all these problems together like as your as you build these very complex applications you get the need of release management which is like oh like maybe your entire pipeline takes 18 hours to finish so you have to figure out what kind of deployment topology do i want to use like rolling or canary or even like rollback roll forward type of thing um when you're trying to stage these apps of apps and you get to the challenge that james bring up in terms of like policy management so if my three vps need to approve this thing before see a push can finally happen how do we take that off the hands of the of the app developer and all they know is i don't know it goes in the black box and eventually get cf pushed um and find a way to actually make that happen consistently right right and that that brings back the aspect of like uh it's it's it's not it's not at all just like a development stage thing like it reaches all the intent is for it to reach all the way to production as as i mean i guess as you would want the most highfalutin definition of a pipeline to be right it's not right uh it's not the wagil fall stuff of like uh we can make our golden images as fast as you want but it takes three months to deploy them right and and so so i'm assuming uh you know what i should do is just go read the documentation but as i like to say here we are uh i'm assuming there's like a set of common services that that are available in the black box right like here is the uh request the configuration that you want and here's like here's how you pass some kind of state between each of the boxes in there and things like that i mean is that is that accurate that there's some like common

support services in there so we we do bake in a lot of uh resources with concourse and that's mostly to make internally out of the box because a lot of folks don't have access they run their concourse in a firewall environment so they can't really happen to occur as freely as possible um so one of the things we had to do before we could become concourse for pcf was to give folks the ability to pre-bake uh docker images and have them referenced uh offline with access to docker hub um so so that's one aspect of it and another aspect that we're really trying to improve on is kind of the day one experience of hey like we hear a lot about these three box pipelines or we hear a lot about these integration points with um these typical services and stories like can you give me some uh reference points and some pointers and examples and now short of paying us on slack we don't really have a scalable consistent way of uh socializing uh that information so one of the first steps we've really taken is to commit to a concourse engineering blog that really tries to dive into some of the core mechanics of concourse and the typical patterns that we've ended up recommending and seeing uh both around the office and within the community all right now that makes sense it it reminds me like i always figured there's uh probably thousands of applications that are the uh the the the sort of descendants of the spring music and the the old j2e pet shop app right like you need some template to start with i don't know what they have in dot net world wasn't it what was the database name richard like wind river or something a good that's close north wind north wind there you go yeah yeah i mean would you uh so i mean with that what do you think of the spring cloud pipeline stuff i was reading the new docs again this morning and that seems like some nice canonical examples opinionated but of how you would take a concourse pipeline and take an app through the various stages and smoke tests and you know blue

green sort of stuff has that helped you think humanize some of the more complex aspects of a deployment pipeline yeah absolutely and we've worked closely with the author and i feel bad i can't remember his name or not i think it's marston yeah marston but we can't say the last name good luck with that one yeah there's like no there's like four k's and a couple of b's but marston in the spring team yeah so we we've had conversations with him and we we uh well it first started as like a collaboration of hey uh congress team we've made these pipelines what do you think how can we improve um so it started off that way and then it ended up being something that i've relied on consistently to as a recommendation to a lot of our spring users to say hey if you're interested in concourse and really changing how you want to do continuous integration delivery that's one of the pipelines i sent to them and i think one of the great things that uh they did with the pipeline was to actually do a jenkins version in a concourse version so a lot of folks get to see side by side what the what the differences are and and kind of like understand at least the general concepts of continuous integration and delivery um and then let them kind of decide what they need to do um on their journey right to see you know are they ready for that level of automation and and really kind of understand what we think a full-blown pipeline might look like and then they can choose uh what to incorporate and that probably ties into the power of conquest right because it's just a bunch of code it's readable it's something that they can like add and remove from it freely modify yeah like we benefit by having um what might be one of the most complicated app teams that would be at like an enterprise uh customer site which is the spring team seeing all the problems they're tackling and having to face by trying to constantly package up um a very quickly growing constantly growing um like spring framework and spring applications um

we've been working that really closely i know skipper which is a tool that was recently just uh like baited or launched by mark pollock in the spring team um it's helping spring solve their release management problems so in this in this challenge where i have this app it's made up of 12 other apps i need app one two three and four to be up running and healthy before you can do four five and six and then if like six rolls back how do i like recover and everything else so um i know something we're looking at there is a resource for concourse to make sure that you can use skipper um and as we continue to find these very complicated um application architectures making sure we're exploring uh different partnerships um there's a few we're exploring right now policy management with some like the larger business process management tools that our customers are using and these will just kind of strengthen the ecosystem of concourse and and everything else um and making sure that we're doing it in a way that we can also partner with others who and leverage like they're very powerful tools yeah now that's great so i've got one last question for you so as you look at this uh both of you kind of keep looking at this landscape and looking at how people and what our customers are trying to do continuously do you see any things emerging i mean specifically i'm thinking about those sort of things you mentioned skipper and we think about even spring cloud data flow and these more non-traditional assets it's not just an application code and a binary it's kind of a system of sorts so as we think about databases as you think about these other components do you see those really getting roped more into a cicd pipeline or tools going to have to adjust what are you thinking about the uh the next couple years of cicd on the first thing yeah so i see two aspects one is kind of the payload what you mentioned is what what are people really interested in continuously deploying and a lot of it is this idea of you know um folks are adopting this microservice strategy uh there are

apps of apps how how do it make it easier for folks to track and redeploy and really create tools and techniques to help with that automation and the second thing that we're really excited about is how to incorporate um more um kind of information radiation around uh workflow efficiency so more analytics and data around your performance of the pipeline how well you've been doing how consistent is your execution um not not in flakes but more uh how are you trending over time are you doing better are you doing worse and i think those are kind of the tools that will really help people understand the value and reap the rewards of um of cicd to really understand now that i've all this automation in place maybe i can start uh figure out where to improve my processes and where to find bottlenecks remove them and uh improve the overall kind of experience yeah that would be awesome if you had some sort of uh i don't know hybrid like software meatware value stream map that was always uh updated i guess it'd be hard to bring in like the meat wary stuff at some point it would be nice to have have have that end-to-end view of everything and i mean going all the way back to our uh github stars thing right like have have a good uh a good heuristic if you will that was telling you if things were uh good or bad or somewhere where you might need to look and do investigate some more yeah that that'd be fun stuff all right well thanks for being on this episode this is good um you'll have to send us a um uh a link to the uh the the sort of like side by side of of a jenkins versus a concourse thing because i think that would be very interesting to see um but uh if people want to uh check out more concourse stuff where should they uh go look uh concourse.ci and our slack channel oh yeah and how about y'all self where do y'all scurry around on the internet um i'm at pie over pie on twitter i'm at what is it freak style on twitter i should

probably redo that one f-r-e-k-s-t-i-l-e oh you know it's it's pretty good this so what what is pie over pie zero or is that some magic math thing uh it's a bit of a it's a bit of a joke uh from my university days where i was trying to answer some smart computer theory question and i answered with pie over pie but we all know that's just zero yeah or a delicious dessert that uh you know you can double up on yeah yeah you know what you maybe you put a pumpkin pie over a cobbler and see what you get with that that seems like a good good solution all right well well as always this has been pivotal conversations if you want to get the latest episodes and browse our back catalog if you're looking for a way to spend uh the winter time in the tunnels enjoying your takeaway cortada uh you can go to soundcloud.com slash Pivotal Conversations and you know obviously you should just subscribe to this whether you listen to it or not we don't really care it's just nice that you download it it would be good to listen to it i guess uh and if if uh you know if if you're interested in more of this stuff you can check out the show notes we post them more or less about every thursday if you go to pivotal.io slash podcast and with that we'll see everyone next time bye bye bye

Tanzu Talk · 🍕 Get more pizza.