1
00:00:00,000 --> 00:00:13,420
domain t-mobile i think we've heard from t-mobile before they uh i've done a lot of good stuff uh

2
00:00:13,420 --> 00:00:19,900
with pivotal so yeah yeah i think um no i don't think we've had t-mobile on the podcast but we

3
00:00:19,900 --> 00:00:26,340
right only had t-mobile speak at cf summit and i've heard them speak at SpringOne platform

4
00:00:26,340 --> 00:00:33,960
um and it's a great story uh so it's it's always exciting to hear and i know that the team's

5
00:00:33,960 --> 00:00:39,000
really been kind of pushing the boundaries and the limits of a lot of stuff and so i think a lot

6
00:00:39,000 --> 00:00:46,620
of folks stand to gain by just hearing what the t-mobile team is up to and we've also got uh ryan

7
00:00:46,620 --> 00:00:51,800
on from altoros and i think that's also another thing that's really valuable for folks is to

8
00:00:51,800 --> 00:00:58,420
understand how you know other folks in the community consultants like altoros are are how

9
00:00:58,420 --> 00:01:04,380
they're helping because i think some people might assume that sure those are great for getting started

10
00:01:04,380 --> 00:01:11,000
getting off the ground but you know here's a team like t-mobile that's three years into the journey

11
00:01:11,000 --> 00:01:19,300
four years into the journey um and you know there's still a lot of help that uh folks at altoros could

12
00:01:19,300 --> 00:01:25,680
do so i can't wait to hear more about this i'm i'm eager to learn yeah so why don't we start uh

13
00:01:25,680 --> 00:01:31,800
talking a little bit uh to to t-mobile matt and uh muhammad are online joining us uh and i think you

14
00:01:31,800 --> 00:01:36,580
know we're going to drill into some of the things you've done around platform automation and some of

15
00:01:36,580 --> 00:01:41,960
the benefits there uh especially working with pks maybe we could start uh either matt or muhammad

16
00:01:41,960 --> 00:01:45,980
and just a quick intro uh introduce yourselves and then maybe just tell us a little bit about

17
00:01:45,980 --> 00:01:54,420
like set the foundation uh kind of your role um and and and how you guys uh kind of got started with

18
00:01:54,420 --> 00:02:02,160
pks so my name is mama salman been with t-mobile for last uh five to six years uh i joined uh pks

19
00:02:02,160 --> 00:02:09,980
pks team uh last year and we started using kubernetes on pks so i think there was a huge ask of automation

20
00:02:09,980 --> 00:02:17,400
automation around that time and uh how we are going to manage um uh our pipelines and our uh upgrades and

21
00:02:17,400 --> 00:02:23,840
our third-party software upgrades uh uh because we knew we are going to grow really fast so uh we

22
00:02:23,840 --> 00:02:29,360
started working on concourse with altoros that's that's when we brought in uh brian from altoros and

23
00:02:29,360 --> 00:02:35,320
started pairing up with him to make sure uh we don't run into any kind of issues going forward

24
00:02:35,320 --> 00:02:40,920
uh so we we started with uh one foundation we built it manually and after that we started

25
00:02:40,920 --> 00:02:47,380
i started um uh working on automation pipelines uh like platform automation and stuff like that

26
00:02:47,380 --> 00:02:57,240
so uh i think we uh we started with uh we are at uh 10 to 12 upgrades now on pks level and we did it

27
00:02:57,240 --> 00:03:04,160
all by uh platform automation and uh there are some other kubernetes uh kubernetes level pipelines

28
00:03:04,160 --> 00:03:08,700
that we worked on i think we can talk about it too later on that actually helped us build a cluster

29
00:03:08,700 --> 00:03:17,560
within one hour 24 node cluster uh a prox a prox one hour uh with all third party uh software installs

30
00:03:17,560 --> 00:03:25,580
on top of pks and um customer ready cluster i'll let mad at some some some of his thoughts to it

31
00:03:25,580 --> 00:03:30,200
yeah my name is matt burphy uh i've been with the pks team for about a year and a half

32
00:03:30,200 --> 00:03:38,380
uh here at t-mobile um like a lot of things that someone said uh one of the nice things that uh

33
00:03:38,380 --> 00:03:44,260
pivotal's brought forward for us is the platform automation uh in the past uh we've had to kind of

34
00:03:44,260 --> 00:03:51,160
write or um create our own pipelines and um really start from ground zero and pivotals

35
00:03:51,160 --> 00:03:56,740
brought forward the pivotal automation uh support which has really helped us a lot in

36
00:03:56,740 --> 00:04:04,040
um um just building our foundation and then once a foundation is built we move on with the pks

37
00:04:04,040 --> 00:04:11,620
cluster installed um i believe we're at uh 3.0 right now right we just went to

38
00:04:11,620 --> 00:04:20,440
platform automation version we are at t-01 and um we just tested that with ryan um and updated one of

39
00:04:20,440 --> 00:04:26,480
our foundation with that other than that uh yeah as it's working for us uh really smooth we did some

40
00:04:26,480 --> 00:04:32,140
we did uh entire foundations i believe there are 10 foundations how many foundations i think we're up to

41
00:04:32,140 --> 00:04:37,520
12. 12. good foundation within one night using uh platform automation patching uh applying a security

42
00:04:37,520 --> 00:04:44,220
patch on all the all the foundation so it's working pretty pretty good for us um uh and uh ryan has

43
00:04:44,220 --> 00:04:50,980
helped us a lot in that in that project and and um uh we continue uh work with him um on all the

44
00:04:50,980 --> 00:04:58,260
pipelines and and uh and making things better yeah it brings a lot of uh a lot of a lot of good

45
00:04:58,260 --> 00:05:05,540
things for us like uh uh even when you get down to ssl certificate management and bbr backups all of

46
00:05:05,540 --> 00:05:11,900
that's being provided with the platform automation which has really helped us out tremendously so and

47
00:05:11,900 --> 00:05:19,600
then you know maybe just to help folks listening who may not be as familiar with um pcf or or pks

48
00:05:19,600 --> 00:05:27,420
specifically you know you mentioned up to 12 foundations for for pks which is pivotals

49
00:05:27,420 --> 00:05:35,940
kubernetes distribution what does the foundation represent uh that's um it's a lot like uh in

50
00:05:35,940 --> 00:05:44,160
pcf cloud foundry um uh we a foundation we call it's like an ops manager installation um so once that

51
00:05:44,160 --> 00:05:51,260
is once that's done you can build several kubernetes clusters within that that ops man installation so

52
00:05:51,260 --> 00:06:00,040
um that's we we just call that a a foundation here but uh okay so that that's sort of the the bosch

53
00:06:00,040 --> 00:06:10,800
layer underneath exactly okay exactly exactly um and then on top of that you're running you actually

54
00:06:10,800 --> 00:06:17,520
then build the the kubernetes clusters themselves and you mentioned that your platform automation

55
00:06:17,520 --> 00:06:23,480
is not just building out the foundation but it's it's building the clusters on top of it

56
00:06:23,480 --> 00:06:30,040
um is do i have that right that that is correct so we have uh seven hardware regions if i recall

57
00:06:30,040 --> 00:06:36,620
correctly uh and then we have 12 uh pks foundations on underneath running on top of that and then

58
00:06:36,620 --> 00:06:42,100
and then we have uh i don't remember the number exactly on kubernetes cluster because we are every

59
00:06:42,100 --> 00:06:48,680
day building clusters every day uh but we are over we are over approximately 40 clusters running uh

60
00:06:48,680 --> 00:06:54,900
running on the all those 12 pks foundations so uh that's how that's how the infrastructure is laid

61
00:06:54,900 --> 00:07:02,360
out here at t mobile and it's all on on-prem on-prem solution uh del hardware running in our data centers

62
00:07:02,360 --> 00:07:08,700
okay so you've got you've got about 40 clusters running across the various foundations and you mentioned

63
00:07:08,700 --> 00:07:15,120
you're building clusters every day so you're not building them and then letting them run for a while

64
00:07:15,120 --> 00:07:22,900
uh like how often are they getting rebuilt and and what what's the reason behind rebuilding these clusters

65
00:07:22,900 --> 00:07:30,500
well but maybe every day was a stretch but uh we're um we we do have sandbox environments where

66
00:07:30,500 --> 00:07:35,380
we'll tear them down and build them up and and they're for our use um as a matter of fact we have

67
00:07:35,380 --> 00:07:46,140
one cluster that uh is rebuilt once a week via a pipeline uh cicd stuff so um to you once we roll

68
00:07:46,140 --> 00:07:52,480
out a production cluster or a um a non-production environment for our developers uh that's pretty

69
00:07:52,480 --> 00:07:59,080
stable and we we don't just go rebuilding those so okay but however we do use the pipelines to patch

70
00:07:59,080 --> 00:08:04,500
those and upgrade and various things but when once they're built and delivered to the customer they're

71
00:08:04,500 --> 00:08:09,980
they're pretty stable uh we have a sandbox cluster that actually rebuilds every every week just to

72
00:08:09,980 --> 00:08:14,900
make sure we are up to date everywhere software wise and everything that uh that we want to apply

73
00:08:14,900 --> 00:08:20,340
we apply on that cluster first before going to the customer cluster uh and that's our that's our

74
00:08:20,340 --> 00:08:25,440
patching standards and applying any patches to our current kubernetes cluster that's that's the path

75
00:08:25,440 --> 00:08:32,220
we follow so that's how the rebuilds of the cluster uh works here but once the customer are on the

76
00:08:32,220 --> 00:08:38,020
cluster customer facing cluster doesn't get rebuilt or any it's just we apply patches on that and that's

77
00:08:38,020 --> 00:08:44,940
uh that's actually pipeline and um and um a hundred percent uptime on on those most of the time

78
00:08:44,940 --> 00:08:52,060
um uh there's no customer impact and um when we apply patches and and uh or up or update the

79
00:08:52,060 --> 00:09:00,320
foundations um on those on those clusters most of the time however we we still are in uh we are still

80
00:09:00,320 --> 00:09:08,980
ramping up we are building clusters and and uh new environments um um quite often so we're we're not

81
00:09:08,980 --> 00:09:15,840
gonna we're not stable here at 12 installs of ops fan and bosch you know that that's going to be

82
00:09:15,840 --> 00:09:22,840
increasing quite a bit so okay great i think you had a question yeah well i just wanted to bring ryan

83
00:09:22,840 --> 00:09:27,160
into the conversation uh and you know we definitely want to get into some of the before and after we'd

84
00:09:27,160 --> 00:09:33,280
love to hear uh you know what what the what the process was like before you implemented a lot of

85
00:09:33,280 --> 00:09:39,100
this platform automation uh and and i i suspect part of the reason uh you're working with ryan and

86
00:09:39,100 --> 00:09:43,500
altoros was to was to help with that automation uh but ryan just bringing you into the conversation

87
00:09:43,500 --> 00:09:50,080
could you just tell us a little bit about um altoros for those who don't uh know the company uh and and

88
00:09:50,080 --> 00:09:54,880
your i guess your perspective you know from a 30 000 foot level on on platform automation

89
00:09:54,880 --> 00:10:03,000
yeah of course jeff um so altoros has been going as a consultant company for the last 18 years it was

90
00:10:03,000 --> 00:10:08,440
founded in 2001 um we've got our headquarters out in the san francisco bay area um but we also

91
00:10:08,440 --> 00:10:14,220
operate at a global scale scale we have offices in the us and norway denmark the uk and bs start in

92
00:10:14,220 --> 00:10:21,300
ireland at the moment and i work remotely with t-mobile um we have about uh 350 staff now um of which we

93
00:10:21,300 --> 00:10:29,900
have about um just over 30 uh cloud foundry consultants and we have a whole um team dedicated to uh pcf and

94
00:10:29,900 --> 00:10:36,480
cloud foundry um so i've been working basically as a consultant with several different companies that are

95
00:10:36,480 --> 00:10:44,340
implementing pivotal cloud foundry and pivotal kubernetes uh services um and i just get a large

96
00:10:44,340 --> 00:10:51,260
uh view of uh different different ways that it can be deployed and some of the challenges that are in

97
00:10:51,260 --> 00:10:56,240
enterprises today and just with that experience i can take that from many different companies and

98
00:10:56,240 --> 00:11:02,800
i've used it to apply to t-mobile um in terms of platform automation um it's really been an iterative

99
00:11:02,800 --> 00:11:12,060
process um we originally started with pcf and there was a a suite of uh templates or pipeline examples

100
00:11:12,060 --> 00:11:18,140
that uh were called pcf pipelines uh that we used to use we started implementing that automation just

101
00:11:18,140 --> 00:11:23,940
because when you're doing things manually you can make a lot of human error and mistakes and when

102
00:11:23,940 --> 00:11:30,720
t-mobile is growing so rapidly um it's easy for technical debt to spiral out of control so we just want to

103
00:11:30,720 --> 00:11:37,100
keep things fast and uh deploy things as quickly as possible so um there's been massive improvements

104
00:11:37,100 --> 00:11:44,860
and uh pcf pipelines has been built upon to kind of create this uh product platform automation um and

105
00:11:44,860 --> 00:11:49,260
it's seen several improvements throughout the years and it's become stable and it was now like generally

106
00:11:49,260 --> 00:11:56,640
available as of may there so we're running on 3.0 like uh selman mentioned and uh yeah it's it's been a

107
00:11:56,640 --> 00:12:02,120
great experience so far so it's it's really helped reduce the amount of time to keep uh code and the

108
00:12:02,120 --> 00:12:07,060
platform up to date especially when there's such a fast release cycle from pivotal with security patches

109
00:12:07,060 --> 00:12:13,020
on the operating system with stem cells um coming out every other week um it's really important that we

110
00:12:13,020 --> 00:12:17,780
keep the the kubernetes environment secure up to date and that we can rebuild it often if there's any

111
00:12:17,780 --> 00:12:23,460
issues or errors yeah it sort of reminds me of that adage of uh you know if it if it hurts do it more

112
00:12:23,460 --> 00:12:32,160
often uh so you know things like uh building out foundations or patching which as you mentioned

113
00:12:32,160 --> 00:12:39,640
is happening all the time with the the volume of and velocity i should say of uh stem cells and patches

114
00:12:39,640 --> 00:12:47,900
that pivotal releases so um rather than make that something that is a painful one-off exercise that

115
00:12:47,900 --> 00:12:53,360
just has to be repeated automated uh so and it's interesting to hear that this has been kind of a

116
00:12:53,360 --> 00:13:00,520
journey i remember when pcf pipelines uh and back in like 2017 or so was starting to get used by a lot

117
00:13:00,520 --> 00:13:07,020
of customers so it's great to hear how it's evolved yeah we've definitely seen the potential

118
00:13:07,020 --> 00:13:12,800
and um it's it's it's kind of one of those things where cloud security is quite different these

119
00:13:12,800 --> 00:13:17,820
days and it's definitely a big question for large enterprises such as t-mobile it's like how do we

120
00:13:17,820 --> 00:13:22,660
ensure that the platform uh and these foundations and these environments are secure so we try to

121
00:13:22,660 --> 00:13:28,900
implement the the three hours of security which is rotate uh repave and repair um and that that is

122
00:13:28,900 --> 00:13:36,500
that is um uh one of the main strategies with such a large distributed platform such as pcf and

123
00:13:36,500 --> 00:13:42,220
kubernetes um the the quicker that you can rebuild something uh the the less likely it is that

124
00:13:42,220 --> 00:13:48,920
security breaches will actually cause a significant impact and uh i think matt said yesterday a great

125
00:13:48,920 --> 00:13:58,240
thing was like uh we want to fail forward and fail fast um matt and uh and muhammad i love the way

126
00:13:58,240 --> 00:14:06,640
that you guys were talking about your customers but it might be worth clarifying here your customers that

127
00:14:06,640 --> 00:14:13,660
you're talking about you're talking about t-mobile's internal developers correct correct uh but you've got a

128
00:14:13,660 --> 00:14:20,560
very sort of product orientation towards how you think of those developers they're not just these other

129
00:14:20,560 --> 00:14:26,340
people who work at my company who keep asking me for stuff right like they're your customers so you're

130
00:14:26,340 --> 00:14:31,060
you're thinking a lot about i can just tell from the language you're using you're clearly thinking a lot

131
00:14:31,060 --> 00:14:36,600
about the experience that those developers have what can you tell tell me about that experience

132
00:14:36,600 --> 00:14:43,100
now that um t-mobile's been kind of ramping up with providing kubernetes environments to those

133
00:14:43,100 --> 00:14:49,560
developers to those customers um and and rolling out pks what's the the developer experience that

134
00:14:49,560 --> 00:14:56,640
that you strive for and and what have you kind of learned along the way uh so we our customers are

135
00:14:56,640 --> 00:15:03,160
mostly internal um uh t-mobile uh developers and um uh since we are a new platform one year old so

136
00:15:03,160 --> 00:15:09,660
we are making sure everybody's aware uh what we are doing and um so we are hosting this uh pks 101

137
00:15:09,660 --> 00:15:19,640
kind of pks slash tke 101 um uh classes uh at t-mobile making sure customers are aware of it how how things

138
00:15:19,640 --> 00:15:27,720
works at in t-mobile world we just had a um uh a tk 101 class in in bellevue in our headquarters where

139
00:15:27,720 --> 00:15:35,300
uh 40 developers showed up and then kind of uh bent over environment um and and showed them how um and

140
00:15:35,300 --> 00:15:41,400
help them deploy a simple app on our uh kubernetes cluster which are running on pks so uh the experience

141
00:15:41,400 --> 00:15:46,320
has been really good since uh last one year and uh we have seen a lot of improvements 100 percent

142
00:15:46,320 --> 00:15:52,680
uh some of that went 100 100 on pks environment i think four of our apps are running

143
00:15:52,680 --> 00:15:59,060
taking 100 production traffic at this moment and we are uh we continue continue to make a lot of poc

144
00:15:59,060 --> 00:16:05,020
work with our uh app teams and making sure that they they have whatever they need to be successful

145
00:16:05,020 --> 00:16:11,160
on this platform yeah it's been a learning experience for both the developers and us

146
00:16:11,160 --> 00:16:19,600
um with pks um we're able to um patch you know in the middle of the day provided that the developers

147
00:16:19,600 --> 00:16:26,620
have used 12 factor rules and um have followed certain standards uh for microservices or container

148
00:16:26,620 --> 00:16:34,740
type orchestration platforms so um uh it's definitely important to us that

149
00:16:34,740 --> 00:16:41,520
even when you know our non-prod environment um to us everything's prod because if if our non-prod

150
00:16:41,520 --> 00:16:47,920
environment goes down that's where all the development is happening um so um it's important to both us and

151
00:16:47,920 --> 00:16:52,520
the developers and it's it's it's been a learning experience and we're we're starting to get good

152
00:16:52,520 --> 00:16:58,760
at it so it's pretty exciting some uh some of the patches you applied in the past um on the pks layer

153
00:16:58,760 --> 00:17:04,960
uh with the 100 percent um production traffic running on them and uh a few times we didn't hear anything

154
00:17:04,960 --> 00:17:11,040
from the customer because the patching went so smooth uh we got we got into some issues uh later on but we

155
00:17:11,040 --> 00:17:15,480
are uh can we continue to make improvements on our environments and making infrastructure layer better

156
00:17:15,480 --> 00:17:19,260
uh to uh best fit for our customers

157
00:17:19,260 --> 00:17:26,320
so you mentioned something that's that's kind of interesting and um i love the idea of the classes

158
00:17:26,320 --> 00:17:30,960
where you're you're kind of engaging with those developers and teaching them what they need to know

159
00:17:30,960 --> 00:17:37,260
so they could be successful in the platform and that sounds a lot like what i remember um hearing from

160
00:17:37,260 --> 00:17:43,380
the t-mobile team that they were doing a couple years ago with pcf back then just using what's now

161
00:17:43,380 --> 00:17:50,920
called the the pivotal application service um and matt you mentioned that uh folks are or should be able

162
00:17:50,920 --> 00:17:57,580
to have you know no downtime for their apps running in pks provided that they're following 12 factors so

163
00:17:57,580 --> 00:18:03,460
what i wanted to ask even before you mentioned that but now i'm really curious is really what's sort

164
00:18:03,460 --> 00:18:09,240
of different that you've seen especially in terms of how you need to enable those developers to be

165
00:18:09,240 --> 00:18:17,060
successful what's different between um pks and the the application service that that you've been

166
00:18:17,060 --> 00:18:24,400
offering to those developers for some time i wouldn't say there's much difference um it's kind of the

167
00:18:24,400 --> 00:18:31,360
same the same concept um some some things that we're we're you know we there was a lot of lessons

168
00:18:31,360 --> 00:18:37,200
learned with pcf we were on a long journey there and i think we've we've uh done a good job of

169
00:18:37,200 --> 00:18:43,240
learning those lessons and applying them here in pks land um take for instance in our prod environment

170
00:18:43,240 --> 00:18:50,780
we have two prod environments and we encourage our our customers to um spread them out across both

171
00:18:50,780 --> 00:18:56,940
environments so if there is something critical uh where they can't have pod disruption at all

172
00:18:56,940 --> 00:19:04,940
um uh hopefully they have um they can switch their a10 services or their level loading services to

173
00:19:04,940 --> 00:19:09,800
say the prod 2 environment and then we can do whatever we need in the prod 1 environment

174
00:19:09,800 --> 00:19:17,060
so things like that there's some lessons learned we've used yeah so it doesn't sound like there's much of a

175
00:19:17,060 --> 00:19:25,800
kind of shortcut or anything it's not like by using pks it's somehow easier like i think there's sometimes a

176
00:19:25,800 --> 00:19:31,400
perception that trying to get my application to be ready to run in the application service is

177
00:19:31,400 --> 00:19:36,180
oh it's hard it's too much work right because you have to make it you know minimum of four factors

178
00:19:36,180 --> 00:19:40,920
but it sounds like really best practices even in pks you should be doing the same thing

179
00:19:40,920 --> 00:19:51,280
yeah yeah definitely um pks seems uh pretty popular here because uh the pcf world um there are some

180
00:19:51,280 --> 00:19:59,160
stricter rules for microservices um okay so it is perceived it is perceived oh sorry to cut you off

181
00:19:59,160 --> 00:20:05,820
go ahead go ahead just just to add to what matt's saying there is uh the pivotal application

182
00:20:05,820 --> 00:20:11,840
service pcf it can be a quite opinionated platform um for web applications and you have to follow

183
00:20:11,840 --> 00:20:18,440
12 factor applications i believe that kubernetes gives you more control um at a lower level at the

184
00:20:18,440 --> 00:20:23,900
container level there is like a large data workloads and anything that needs persistent stateful storage

185
00:20:23,900 --> 00:20:29,700
um is a good fit for pks so it's really when the application and customers like developers

186
00:20:29,700 --> 00:20:35,740
which are our customers come to us we try to go for an assessment with them to kind of see where

187
00:20:35,740 --> 00:20:41,120
their application is the best fit a lot of the time for legacy systems um it's easier to just do a

188
00:20:41,120 --> 00:20:47,020
lift and shift instead of uh rebuilding the application from scratch um there's no real

189
00:20:47,020 --> 00:20:51,460
shortcut uh if you want your application to run in the cloud these days you should really be following

190
00:20:51,460 --> 00:20:57,460
the 12 factors um for creating applications um but sometimes that's not possible and you need to get

191
00:20:57,460 --> 00:21:03,920
something um migrated quicker um so pks uh and kubernetes people believe it's easier but there's so much

192
00:21:03,920 --> 00:21:10,980
of things that you have to uh integrate into your um microservice container application system such as

193
00:21:10,980 --> 00:21:15,160
like logging and monitoring and there's a whole different range of things that go into it so

194
00:21:15,160 --> 00:21:18,940
there's no real shortcuts you should still be applying best practices and a lot of them from

195
00:21:18,940 --> 00:21:26,960
uh pivotal uh pivotal's application service we try to apply those and uh teach customers about that

196
00:21:26,960 --> 00:21:33,940
even if they're going to pks and i wonder you you ryan have seen uh you know as a as a consultant you

197
00:21:33,940 --> 00:21:39,020
work across a number of different companies what is your experience been uh not just t-mobile but but

198
00:21:39,020 --> 00:21:46,920
broader in terms of uh your customers thinking about when to use pas versus pks a more opinionated

199
00:21:46,920 --> 00:21:52,660
platform like the application service versus kubernetes you mentioned you have kind of a criteria you you

200
00:21:52,660 --> 00:21:57,460
look to uh when deciding which which is the best fit for different applications how has that played

201
00:21:57,460 --> 00:22:03,660
out in the real world uh in the different customers you've worked with yeah um so that's a really good

202
00:22:03,660 --> 00:22:10,740
question jeff um so basically um we we do a different a load of different uh capabilities one of them

203
00:22:10,740 --> 00:22:18,440
is uh very popular uh is the the pivotal dojos the platform dojos um and some application

204
00:22:18,440 --> 00:22:25,040
transformation can go into that as well um but but when when a customer is having we try to identify

205
00:22:25,040 --> 00:22:31,960
pain points um it could be something from not being able to deploy or maybe there's some downtime or

206
00:22:31,960 --> 00:22:37,860
there's latency or errors um on the vm level or maybe they're just looking to cut costs and move from vms

207
00:22:37,860 --> 00:22:43,620
to containers um but i do see a lot of similar patterns across a lot of customers um so we've kind of

208
00:22:43,620 --> 00:22:50,260
created some um projects and uh deliverables that are fairly common to to a lot of people

209
00:22:50,260 --> 00:22:56,080
uh and a lot of large enterprises uh but i do see them the same questions being asked time and time

210
00:22:56,080 --> 00:23:03,660
again um so there's no real easy solution um but we try to take those lessons learned from from our

211
00:23:03,660 --> 00:23:09,620
companies and customers uh and put it into like best practices and recommendations uh and we often make a lot

212
00:23:09,620 --> 00:23:18,100
of blog posts about it um but yeah it's really it's really about teaching the developers um how to do

213
00:23:18,100 --> 00:23:23,520
cloud native development and what those best practices are uh and if that falls into place then it's really

214
00:23:23,520 --> 00:23:29,800
really easy and simple to use and they see much much faster deployments um much more reliability with

215
00:23:29,800 --> 00:23:35,480
the production workloads and we actually have the developers owning the applications so that if there is an

216
00:23:35,480 --> 00:23:44,200
issue um it's fixed much much faster so if we come back to um the platform automation kind of journey that

217
00:23:44,200 --> 00:23:53,680
the pks team at t-mobile has been on you mentioned uh muhammad that uh you that built the first

218
00:23:53,680 --> 00:23:59,640
foundation manually and then started to introduce platform automation working with ryan how did that

219
00:23:59,640 --> 00:24:06,760
sort of uh how did it come to be that you started to work with ryan and alturos on that platform

220
00:24:06,760 --> 00:24:16,200
automation project for pks so uh ryan has worked um with us previously on a pcf site so we knew that ryan

221
00:24:16,200 --> 00:24:24,600
can provide us um a lot of help there and um so that's why we kind of um brought in ryan and and um to use the

222
00:24:24,600 --> 00:24:30,200
expertise uh over there and then um kind of build the testing environment and started using um

223
00:24:30,840 --> 00:24:35,560
platform automation over there making sure the upgrades will go smooth and um with a minimum

224
00:24:35,560 --> 00:24:41,560
customer impact and um and um kind of uh kind of started uh testing the whole whole platform

225
00:24:41,560 --> 00:24:47,080
automation uh layer over there making sure ops manager upgrades are a smooth pks style um

226
00:24:47,720 --> 00:24:54,520
um uh creates a smooth with uh with minimum customer impact uh so uh kind of kind of making the start

227
00:24:54,520 --> 00:25:02,680
making improvements over there making sure uh uh uh making sure our customers are um uh at at minimum

228
00:25:02,680 --> 00:25:08,840
impact when we do these upgrades and i can kind of start to expand uh from there and and the upgrades

229
00:25:08,840 --> 00:25:17,240
in np environment and then uh then prod and uh uh right now we are uh 100 percent automated on 99 percent

230
00:25:17,240 --> 00:25:26,360
automated on um on uh pks uh pks layer okay yeah we we have a really we have a really good uh relationship

231
00:25:26,360 --> 00:25:34,440
with ryan and alturas um especially with uh in the pas world and um using concourse and the nice thing

232
00:25:34,440 --> 00:25:41,320
about uh platform automation is it's the same tool used in pas as well as pks so it was a real good

233
00:25:41,320 --> 00:25:47,240
fit to have alturas and ryan help us out to get the pks stuff off the ground in the cicd world

234
00:25:49,080 --> 00:25:54,840
yeah just to add to what matt's saying there um just to put it into a bigger high-level perspective

235
00:25:54,840 --> 00:25:59,640
um we have really small operations team i've mostly been working with matt and salman

236
00:25:59,640 --> 00:26:08,840
um and you're talking about like a team of two or three operators or devops uh engineers um running

237
00:26:09,640 --> 00:26:15,880
all these foundations and thousands of containers and thousands of pods for for hundreds of developers

238
00:26:15,880 --> 00:26:23,640
and like hundreds of applications um and really that's that's why the automation is essential um there's

239
00:26:23,640 --> 00:26:28,600
no way that you can manage something with such a small operations team and i see this in so many

240
00:26:28,600 --> 00:26:33,400
customers is like the ops team can be really really small and if you have the correct automation in

241
00:26:33,400 --> 00:26:39,720
place not just with uh platform automation um but uh expanding out into the infrastructure automation

242
00:26:39,720 --> 00:26:46,920
with terraform as well um it's it's really the platform automation is very modular and it gives you

243
00:26:46,920 --> 00:26:52,760
the building blocks to kind of piece together what bits you want and then expand upon that so we've

244
00:26:52,760 --> 00:27:00,360
expanded the platform automation to include building out the the kubernetes clusters with the the pks cli

245
00:27:00,840 --> 00:27:07,560
and we've expanded that into building out the low balancers um uh that connect up to pks and uh

246
00:27:07,560 --> 00:27:13,400
configuring the ingress and configuring splunk collectors and just configuring the monitoring and

247
00:27:13,400 --> 00:27:19,480
we've hooked all that into the platform automation that pivotal has provided with but uh it's really um

248
00:27:20,680 --> 00:27:25,960
building blocks that we can use and then expand upon so the the idea is to automate everything so that

249
00:27:25,960 --> 00:27:31,400
we can have this small team of two or three people running the entire platform and it basically runs itself

250
00:27:32,680 --> 00:27:41,080
okay so what i'm hearing is you know these aren't just raw kubernetes clusters that are getting spun up

251
00:27:41,080 --> 00:27:47,960
for those internal customers they're they're pretty well outfitted with a lot of um

252
00:27:49,000 --> 00:27:55,720
you know important infrastructure around it like the load balancing and uh splunk logging aggregation and

253
00:27:55,720 --> 00:28:05,480
um so you're really building out a more uh whole platform for those developers that is just built on

254
00:28:05,480 --> 00:28:12,680
kubernetes uh just yeah and that you know rings familiar with folks like kelsey hightower out there

255
00:28:12,680 --> 00:28:18,040
tweeting it's like hey kubernetes is a platform for building platforms it's not really the developer

256
00:28:18,040 --> 00:28:25,880
experience itself do i have that right yeah absolutely there's so much more to it um and we've just kind

257
00:28:25,880 --> 00:28:31,560
of expanded it and expanded it and everything that we need so it's at a click of a button we run the

258
00:28:31,560 --> 00:28:35,720
pipeline and it builds out a full cluster that's ready for the developers to go on instantly

259
00:28:36,360 --> 00:28:41,800
um and if we rebuild it we have everything there um again from the login to the dashboards to

260
00:28:42,920 --> 00:28:49,080
the cluster role bindings to the persisted storage at us attached uh and then we also run like sonoboy

261
00:28:49,080 --> 00:28:55,320
which is like a conformance tool um to make sure that uh the best practices and standards are met with

262
00:28:55,320 --> 00:29:00,840
the kubernetes cluster and that it's configured correctly um so we try to package everything and

263
00:29:00,840 --> 00:29:07,400
we do make it a platform so that um it's kind of similar to the pivotal application service that

264
00:29:07,400 --> 00:29:12,120
everything that that's provided we also provide in kubernetes um all in one

265
00:29:13,880 --> 00:29:18,440
yeah as soon as we run the pipeline on on a kubernetes cluster we make sure that

266
00:29:19,320 --> 00:29:27,560
the cluster is customer ready for them to deploy their workload on it and making sure um uh that we

267
00:29:27,560 --> 00:29:33,400
don't have any manual work after that uh that's the whole point of uh kubernetes cluster pipeline was

268
00:29:34,440 --> 00:29:39,000
delivering a customer ready cluster with all third-party components on on top of pks layer

269
00:29:39,880 --> 00:29:45,640
and what by by working with ryan um so thank you by the way for kind of explaining the background and

270
00:29:45,640 --> 00:29:50,600
and the the good experience and relationship that you've had with ryan and alturos that made it kind

271
00:29:50,600 --> 00:29:59,960
of a natural fit for building out the kubernetes space platform um but muhammad and matt i'm curious what did

272
00:29:59,960 --> 00:30:08,120
having ryan uh helping with this platform automation build out what did that free you guys up to go do

273
00:30:09,480 --> 00:30:16,520
uh well it it did definitely help us uh because we're also involved uh uh salmon and i in kubernetes

274
00:30:16,520 --> 00:30:23,240
support as well and um certainly before ryan came we were really buried in the concourse stuff trying

275
00:30:23,240 --> 00:30:29,960
to home cook this ourselves um and just you know kind of going through public get repos and looking for

276
00:30:29,960 --> 00:30:36,280
things and just really trying to get a grasp on concourse and had no time for um actual kubernetes

277
00:30:36,280 --> 00:30:42,040
support so it's really free to have to be able to do um things like that we're now now we're able to

278
00:30:42,040 --> 00:30:48,120
take on different tasks and you know more r d stuff or gangway or um any new stuff coming up in the

279
00:30:48,120 --> 00:30:56,840
kubernetes down the line so it uh it truly uh uh once the hard work is over it it does kind of run itself

280
00:30:56,840 --> 00:31:04,280
until you know an upgrade or something like that or um it's just made rolling out a new environment

281
00:31:05,080 --> 00:31:10,520
so much easier uh and i think you know we've been me and matt and someone we've been parried

282
00:31:10,520 --> 00:31:15,080
like pretty much every day for the past six months which is how long this engagement's been running

283
00:31:15,560 --> 00:31:21,720
and we picked up even small things hopefully they've learned um anything from restful apis to bash

284
00:31:21,720 --> 00:31:26,120
scripting just doing the whole pair programming really is good for knowledge transfer and skills

285
00:31:26,120 --> 00:31:31,480
transfer so hopefully that they if if they need to add even more automation in the future they can

286
00:31:31,480 --> 00:31:38,040
build it themselves uh would you agree with that matt definitely definitely it wasn't like you came in

287
00:31:38,040 --> 00:31:43,160
here and did all the work for us it's uh uh definitely a lot of knowledge transfer taught us how to fish i

288
00:31:43,160 --> 00:31:49,640
guess for the old analogy that's how it's used yeah that never gets old no it's so many different

289
00:31:49,640 --> 00:31:53,720
scenarios but but it's a good it's a good point and i wonder maybe if you could expand a little bit on

290
00:31:53,720 --> 00:31:58,920
and this is i guess kind of tangential to the main topic but you know what what is the relationship

291
00:31:58,920 --> 00:32:04,520
expand a little bit on the relationship uh working with uh a service provider like alturos how do you

292
00:32:05,320 --> 00:32:11,240
how do you build that trust you know what how does it actually work on the ground how how do you

293
00:32:11,240 --> 00:32:15,000
develop that that working relationship when you're coming from you know somebody coming from the

294
00:32:15,000 --> 00:32:21,720
outside how do you make the most of that engagement well again for us uh we we had already had ryan for

295
00:32:21,720 --> 00:32:29,320
uh pas support but i i guess what it started with is uh just um you know uh someone and i told our

296
00:32:29,320 --> 00:32:34,920
management hey look you know we're kind of swimming here we need some help um we need someone like ryan

297
00:32:34,920 --> 00:32:40,760
and alturos to come in and kind of show us the ropes and bringing ryan in and uh having a meeting and

298
00:32:40,760 --> 00:32:47,160
kind of set in the bar and this is what we expect uh we we had weekly meetings on uh how we're doing

299
00:32:47,160 --> 00:32:54,440
take things off the list that are done and adding things and um so the trust was built uh quite early

300
00:32:54,440 --> 00:32:59,800
in the process so it's uh i don't know do you have anything more to put i mean we were doing a lot of

301
00:32:59,800 --> 00:33:05,800
stuff uh a lot of stuff manually uh uh before ryan came in and we were installing um a lot of components

302
00:33:05,800 --> 00:33:12,600
on kubernetes cluster uh uh kind of manually and and uh we had some pipelines separated out pipelines

303
00:33:12,600 --> 00:33:17,800
that would do work but we kind of needed something that uh ties everything together and then kind of

304
00:33:17,800 --> 00:33:23,800
needed somebody who knows um who knows how these pipelines and architecture work on the from the from the

305
00:33:23,800 --> 00:33:29,640
on the ground level so that's when that's when uh ryan uh comes into play and kind of uh coming back to

306
00:33:29,640 --> 00:33:35,640
the pcf player ryan already worked with pcf team uh i think last last year a couple of years ago so we

307
00:33:35,640 --> 00:33:41,400
kind of had that trust already so as soon as we uh we started talking about this ryan was the first person

308
00:33:41,400 --> 00:33:48,520
who who came to our mind and and kind of um started um talking to alturos how to bring him in again

309
00:33:48,520 --> 00:33:56,120
and uh work with us on pks uh pks side so uh yeah i mean we man we achieved a lot and uh on platform

310
00:33:56,120 --> 00:34:03,160
layer um third-party installation uh building kubernetes cluster customer ready um that's that's how it went

311
00:34:04,760 --> 00:34:12,040
yeah and uh just just to add to what salmon and matt said there what i see from customers is uh there is

312
00:34:12,040 --> 00:34:18,040
a lot of stuff done manually like large enterprises are under so much time pressure to get production

313
00:34:18,040 --> 00:34:25,560
workloads in on the really strict deadlines and what ends up happening happening is uh all of this stuff

314
00:34:25,560 --> 00:34:30,920
is done manually to kind of do proof of concept and kind of prove the value and value is really important

315
00:34:30,920 --> 00:34:37,560
but once you've got that initial value you then have to go back and say can we automate this um because

316
00:34:38,120 --> 00:34:43,640
uh from now on i like if you're if we're doing anything new like as the projects went on the

317
00:34:43,640 --> 00:34:50,280
scope's gotten bigger and bigger and uh new things have been added so rather than building something

318
00:34:50,280 --> 00:34:57,000
else we we initially try to build it out automatically first time um rather than building it out and

319
00:34:57,000 --> 00:35:01,960
revisiting it um later and because it never really gets done it gets pushed back and pushed back and

320
00:35:01,960 --> 00:35:07,800
higher priorities come so what i try to get matt and salmon to do is anytime we've got a new thing to build

321
00:35:07,800 --> 00:35:15,000
out it has to be done automatically out of the box and we try to automate it first thing um and that that

322
00:35:15,000 --> 00:35:21,080
it takes longer to automate something but the benefits long-term pay off um because you're not having

323
00:35:21,080 --> 00:35:28,120
the repeat tedious processes over and over again um which if you're a linux system admin uh you

324
00:35:28,120 --> 00:35:34,600
probably that's the worst thing you can be doing in the job right so you got to put in that investment

325
00:35:34,600 --> 00:35:42,280
early uh and it pays off over time yeah yeah and and it also helps us extend extend things across across

326
00:35:42,280 --> 00:35:49,240
the environment and you know um i mean uh we couldn't imagine building 4d clusters um uh different set

327
00:35:49,240 --> 00:35:55,160
of uh different set of eyes and hands and and um every cluster would look a little bit different in that

328
00:35:55,160 --> 00:36:01,160
scenario but when when automation comes into play you know uh you know everything is going to look exactly the

329
00:36:01,160 --> 00:36:07,240
same the way you want it to be yeah even just the little things like naming standards can make a huge

330
00:36:07,240 --> 00:36:12,200
difference in trying to find something when you've got tons of data centers and tons of resource pools

331
00:36:12,840 --> 00:36:18,440
on premise and just trying to find like troubleshoot and bug if everything's named consistently the same way

332
00:36:18,440 --> 00:36:24,360
it just makes it much much easier to operate troubleshooting so just one one thing that uh

333
00:36:25,400 --> 00:36:31,880
matt you had mentioned uh the like the weekly meetings just in terms of how you're kind of

334
00:36:33,000 --> 00:36:37,480
i guess running the engagement with el toros you mentioned these weekly meetings where sort of

335
00:36:37,480 --> 00:36:43,640
things come off of that backlog and then and new things come on um which sounds a lot like you know

336
00:36:43,640 --> 00:36:49,400
know iteration planning meetings and this is not a a consulting engagement where

337
00:36:50,600 --> 00:36:54,520
things were all defined up front as like okay here's a statement of work and this is what's

338
00:36:54,520 --> 00:36:58,520
going to happen in the next six months it's like you know every week what's going to get done actually

339
00:36:58,520 --> 00:37:05,480
changes a little bit and that that also ties then um to me to at least the way i heard it to

340
00:37:05,480 --> 00:37:10,600
what you were saying ryan in terms of every new thing that shows up like the scope keeps changing and

341
00:37:10,600 --> 00:37:15,480
growing but every new thing that shows up we make sure to automate it the first time

342
00:37:16,120 --> 00:37:22,680
um and i just i mean if am i thinking about that right in terms of part of the way that you're

343
00:37:22,680 --> 00:37:28,680
working with el toros is with things like weekly iteration meetings where you know what we thought

344
00:37:28,680 --> 00:37:32,120
we needed that and we actually decided we don't need that this other thing is more important and

345
00:37:32,120 --> 00:37:35,320
that's what we're going to build and by the way we're going to build it automated the first time

346
00:37:36,600 --> 00:37:43,720
definitely definitely we um it was a definitely a living breathing document almost like a real loose

347
00:37:43,720 --> 00:37:49,080
sprint type meeting it's like okay what did we get done this these two weeks um yeah we're done with

348
00:37:49,080 --> 00:37:57,240
that and oh by the way um well you know pks 1.3 came out and uh we haven't really built something

349
00:37:57,240 --> 00:38:03,160
around an upgrade or um let's make sure we have this down so that comes on the list and uh we just all

350
00:38:03,160 --> 00:38:10,040
kind of check our things and ryan gives us a yep that looks good or hey let's do this or that um it

351
00:38:10,040 --> 00:38:15,720
was a very uh fluent engagement with him it wasn't like you said uh we're going to do these 10 things

352
00:38:15,720 --> 00:38:21,720
and it's going to cost this much and it's got to be done in six months um it was it was quite flexible

353
00:38:21,720 --> 00:38:28,600
which was awesome it's something that we really needed so yeah i can i mean just you know that

354
00:38:28,600 --> 00:38:35,480
obviously uh aligns really closely with the the pivotal way of working on building software right so

355
00:38:36,200 --> 00:38:43,640
not necessarily setting up kind of a hard uh it's you know waterfall style statement of work or something

356
00:38:43,640 --> 00:38:52,200
but um having a a general kind of guiding light right um north star if you will but then having

357
00:38:52,200 --> 00:38:59,080
that fluidity that agility if you if you will if you must so it sounds like that would be kind of a

358
00:38:59,720 --> 00:39:05,320
secret success that could be passed on to others in terms of how they think of working with consultants

359
00:39:05,320 --> 00:39:10,040
um whether it's alturos or someone else is kind of looking for the ability to engage

360
00:39:10,680 --> 00:39:16,200
in that iterative way because it aligns to the iterative journey that you've been on with

361
00:39:16,920 --> 00:39:24,760
pks platform automation and has allowed you to kind of map to what those internal customers your

362
00:39:24,760 --> 00:39:29,400
developers really need and build out a platform that that meets those needs

363
00:39:29,400 --> 00:39:35,000
and and so and not to put too fine a point on it but it would be great i think we could reiterate

364
00:39:35,000 --> 00:39:41,560
the value that platform automation delivers to your customers i.e the developers um you know you

365
00:39:41,560 --> 00:39:46,840
talked about obviously there's some security benefits um but i think you know what about things like

366
00:39:46,840 --> 00:39:52,360
you know getting new capabilities new features and pcf out to them quickly i mean if you could if

367
00:39:52,360 --> 00:39:58,680
you had to you know tick off the top two or three benefits to developers for really automating

368
00:39:58,680 --> 00:40:06,920
platform operation what would they be i i i i think for one big thing is uh it makes us very um uh

369
00:40:06,920 --> 00:40:12,440
efficient when when a developer team comes and says you know we have a new product coming out a new

370
00:40:12,440 --> 00:40:18,040
phone's coming out a new model uh we're going to need you know three clusters and we need them tomorrow

371
00:40:18,600 --> 00:40:24,040
that's something that's actually realistic and we could spin up and deliver to them um with with with

372
00:40:24,040 --> 00:40:30,360
our solid configuration proven and um with their namespaces and give them give them the keys and

373
00:40:30,360 --> 00:40:37,880
say go ahead and go and and to to add to the uh uh to that at the pks layer we almost did like all

374
00:40:37,880 --> 00:40:45,560
almost nine upgrades coming from 1.1.5 to 135 now and we have used platform automation and that and

375
00:40:45,560 --> 00:40:51,320
those upgrades include new features that we would need in in uh pks style and that eventually the customer

376
00:40:51,320 --> 00:40:57,480
would uh need it or those feature uh would help us out in in some Kubernetes cluster configuration so

377
00:40:58,360 --> 00:41:03,720
i can't imagine doing those upgrades manually from 1.1.5 to one three one three five that we are

378
00:41:03,720 --> 00:41:13,160
currently right now on on uh on almost uh uh 24 uh pks foundations right and and ultimately again i'm a

379
00:41:13,160 --> 00:41:17,560
i'm a very concrete person so let's let's take that one step further benefits for the developers so how

380
00:41:17,560 --> 00:41:24,600
does that benefit um t-mobile's actual customers who are you know uh have phone plans and you know how

381
00:41:24,600 --> 00:41:30,280
does how does this all translate to value for them in the business well well for one instance i'd say

382
00:41:30,280 --> 00:41:35,160
about a month and a half ago two months ago a cve came out that was that was quite important that

383
00:41:35,160 --> 00:41:41,480
could expose a lot of data and customer data for that matter on some of our sensitive environments

384
00:41:41,480 --> 00:41:48,680
and we were able to upgrade all of our environments very quick and uh without much interruption to

385
00:41:48,680 --> 00:41:55,800
anyone and that cve was behind us where in the old days we would have had thousands of boxes to patch

386
00:41:56,440 --> 00:41:59,880
would have had to put a group together to make sure everything was done and all of that

387
00:42:00,280 --> 00:42:07,000
uh this was literally running five to six pipelines and we were done so right that's one nice thing

388
00:42:07,000 --> 00:42:11,400
when when a security bulletin comes out or something needs to be patched immediately

389
00:42:12,040 --> 00:42:17,560
um this allows us to do that quite well yeah that's a great example obviously security is you know top

390
00:42:17,560 --> 00:42:22,760
of mind for a lot of people a lot of consumers uh and so this you're able to keep them safer more

391
00:42:22,760 --> 00:42:27,400
quickly uh and that's that's obviously valuable to the customer and that's hopefully gonna you know that

392
00:42:27,400 --> 00:42:33,960
provides a somewhat of a differentiation for t-mobile uh to to you know retain customers and win new

393
00:42:33,960 --> 00:42:40,040
customers so uh that's a great example fantastic and i think we should also link in the show notes here

394
00:42:40,680 --> 00:42:49,320
um your colleagues uh james and brendan presented at kubecon maybe six months ago on you know another

395
00:42:49,320 --> 00:42:55,240
sort of dimension to the journey towards building the the on-premise kubernetes platform

396
00:42:55,240 --> 00:42:59,960
um and they touched on kind of another not so much on the platform automation side but

397
00:42:59,960 --> 00:43:04,920
on some other dimensions to that so i i think that would be a good compliment for folks to check out

398
00:43:05,720 --> 00:43:09,320
but that was six months ago so i know there's been probably some updates and changes and it does

399
00:43:09,320 --> 00:43:14,680
sound like the platform automation story has come from a long way uh so that is very cool to hear thank

400
00:43:14,680 --> 00:43:19,480
you so much yeah absolutely and uh yeah as we wrap up i mean what's what's next on your agenda uh

401
00:43:20,680 --> 00:43:24,520
in terms of you know maybe next six months to a year what's what do you hope to accomplish let's say we

402
00:43:24,520 --> 00:43:27,880
we have you back on the show uh this time next year what what do you think we'll be talking about

403
00:43:29,480 --> 00:43:35,640
small more more more and more automation minimum uh minimum manual work uh for us for us for our

404
00:43:35,640 --> 00:43:43,960
customers and making sure uh uh making sure we are uh we are up to date and um on our on our customer

405
00:43:43,960 --> 00:43:51,800
providing all the features um in automated fashion uh that would be uh our goal and uh that's our um

406
00:43:51,800 --> 00:43:59,960
company goal as well i'm making sure um we are 99 automated across the board on on pksdk land

407
00:44:01,960 --> 00:44:06,920
yeah but we'll be much larger we'll be definitely much larger there'll be uh there's more more

408
00:44:06,920 --> 00:44:11,720
infrastructure coming in more foundations being built more clusters being built more containers

409
00:44:11,720 --> 00:44:19,880
everything and uh just just to add to that as well um we've been working on a pks reporter tool um that

410
00:44:19,880 --> 00:44:26,600
runs and collects information from all of the different pks foundations all the different uh kubernetes

411
00:44:26,600 --> 00:44:31,160
clusters that are running there the different plans the different sizes what kind of workloads are on it

412
00:44:31,960 --> 00:44:38,440
and from that reporter we're trying to output that data in a meaningful way and generate an automatic

413
00:44:38,440 --> 00:44:44,760
architecture diagram which basically gives us a high level overview of all of our environments what

414
00:44:44,760 --> 00:44:50,600
versions they're at what kind of state they're in um similar to monitoring but more of the architecture

415
00:44:50,600 --> 00:44:57,400
diagram um we've been working with uh fabric js to kind of draw that out um patch together from some json

416
00:44:58,120 --> 00:45:04,760
and then we can export that as a png image or uh send it to as a report to stakeholders and uh

417
00:45:04,760 --> 00:45:09,320
basically share it out to developers to kind of show hey we're at the latest and greatest features

418
00:45:09,320 --> 00:45:15,800
um you've got all the benefits of kubernetes with your scalability and uh you're you can basically run

419
00:45:15,800 --> 00:45:23,160
anything um on on these clusters because they're underutilized and so on and so forth so that that that's

420
00:45:23,160 --> 00:45:28,280
something that's in the works at the moment uh and hopefully it'll be of uh benefit once it's

421
00:45:28,280 --> 00:45:35,080
complete that sounds cool can i can i sneak in maybe one more question which is all this automation

422
00:45:35,080 --> 00:45:41,080
that you've built around kubernetes you're building around pks you're not building around open source

423
00:45:41,080 --> 00:45:50,120
kubernetes um why what's the difference that from your perspective why is it why have you chosen to do this

424
00:45:50,120 --> 00:46:00,280
around pks and not just open source kubernetes i i think that um going with pks allows us to um have you

425
00:46:00,280 --> 00:46:05,320
pivotal do all the research when new things are coming out right you have a whole team that um

426
00:46:06,040 --> 00:46:15,080
is building your your technology as new technology comes out um you're able to soak that in and maybe

427
00:46:15,080 --> 00:46:21,480
a month or two down the line once pivotal gets their pks package with the new kubernetes versions out and

428
00:46:21,480 --> 00:46:28,040
the new new bells and whistles um is delivered to us rather than have us have a whole team that

429
00:46:28,680 --> 00:46:35,080
is dedicated to research and development or research and you know of all the new things that are coming

430
00:46:35,080 --> 00:46:40,840
out with kubernetes we kind of put that on you and then it's delivered to us and and also give us gives

431
00:46:40,840 --> 00:46:46,200
us flexibility to focus more on automation and our customers customers definitely

432
00:46:47,960 --> 00:46:53,720
yeah okay that makes sense you know you want to be if you have a team of doing r d instead of having

433
00:46:53,720 --> 00:47:00,440
them do the same r d that it's probably happening in other places have them focus on what what matters at

434
00:47:00,440 --> 00:47:08,440
t-mobile okay and and our our uh our mts uh mts uh mts i james webb and and um um our principal on team

435
00:47:08,440 --> 00:47:14,360
just here see uh these guys work really closely with pivotal to bring in new features and we have a um

436
00:47:15,000 --> 00:47:20,200
Tuesday call every day every week with pivotal to kind of go over what we need and uh what are

437
00:47:20,200 --> 00:47:28,040
features coming in in new versions so uh that's kind of nice and i think as well is that kubernetes is

438
00:47:28,040 --> 00:47:34,360
is a very complex system like it's rapidly uh being developed like there's new features every other

439
00:47:34,360 --> 00:47:38,920
day and there's like a massive community behind it uh putting new features in and we really need

440
00:47:38,920 --> 00:47:44,760
somebody to check those features are production ready um i believe and maybe you can edit this out or

441
00:47:44,760 --> 00:47:51,560
whatever but t-mobile um tried to do open source kubernetes before um and it was just too much to

442
00:47:51,560 --> 00:47:59,960
manage it's too complex and that that's why at altaurus we recommend pks is because uh managed kubernetes

443
00:47:59,960 --> 00:48:04,920
is is much easier to run especially when you don't have a full team of hundreds of engineers to kind

444
00:48:04,920 --> 00:48:11,160
of actively develop and work on it and troubleshoot it um it's just too complicated um so for for

445
00:48:11,160 --> 00:48:17,400
enterprises that are not purely technology focused um and want to stay on the cutting edge technology

446
00:48:17,880 --> 00:48:23,800
uh a managed kubernetes solution is always the best approach in my opinion okay thank you well said

447
00:48:23,800 --> 00:48:30,360
thank you so much for for coming on the show it was really great to hear about um you know the the

448
00:48:30,360 --> 00:48:37,080
the journey that t-mobile has been on around building out the platform automation for the the pks

449
00:48:37,080 --> 00:48:45,480
environment um and how how far along you guys have come in a relatively short amount of time with really

450
00:48:45,480 --> 00:48:54,440
building out a very developer centric platform uh at t-mobile so congratulations can't wait to hear more

451
00:48:54,440 --> 00:49:04,520
absolutely thank you thank you