1
00:00:00,000 --> 00:00:06,180
Hello, everybody. Here we are at Velocity 2009 in lovely San Jose.

2
00:00:06,360 --> 00:00:08,680
And we've got a guest with ourselves. Would you like to introduce yourself?

3
00:00:09,040 --> 00:00:12,380
Sure. My name is Jeff Hammerbacher, and I'm Vice President of Products at Cloudera.

4
00:00:12,380 --> 00:00:18,240
You actually just did a talk about Hadoop and obviously a little bit of how Cloudera hooks up into Hadoop.

5
00:00:18,460 --> 00:00:24,660
But it's a good startup for you. Actually, when you did a survey of people who you asked people if they'd heard about Hadoop,

6
00:00:24,660 --> 00:00:29,560
what was the response that you got this afternoon kind of representative of what you usually get?

7
00:00:30,000 --> 00:00:34,500
Yeah, so this conference is for operations people. So I expect it to be a little higher.

8
00:00:34,640 --> 00:00:37,100
I think there was a little less than half said that they played with Hadoop,

9
00:00:37,180 --> 00:00:40,920
and I'd say maybe a little under a quarter who said that they were operating Hadoop Cluster right now.

10
00:00:40,920 --> 00:00:44,280
Now, all of that said, can you give us a quick overview of what Hadoop is?

11
00:00:44,540 --> 00:00:47,980
Sure. It's an open source project. It's part of the Apache Software Foundation.

12
00:00:48,960 --> 00:00:53,640
And it's many sub-projects together, but the main ones are distributed file system,

13
00:00:53,640 --> 00:00:57,720
modeled after Google's file system, and implementation of MapReduce,

14
00:00:57,800 --> 00:01:00,140
which is a distributed data processing framework,

15
00:01:00,420 --> 00:01:03,680
which is very fault-tolerant and capable of scaling to thousands of nodes.

16
00:01:03,860 --> 00:01:07,720
So both the distributed file system and the MapReduce execution framework

17
00:01:07,720 --> 00:01:10,840
are designed to be run on thousands of nodes over petabytes.

18
00:01:10,920 --> 00:01:13,800
So what do people do with these thousands of nodes and petabytes of data?

19
00:01:14,480 --> 00:01:15,680
You know, a lot of interesting things.

20
00:01:15,820 --> 00:01:19,340
So Yahoo was really the first company to take Hadoop to massive scale,

21
00:01:19,440 --> 00:01:22,600
and their first application was, well, they had two.

22
00:01:22,700 --> 00:01:25,540
One was building the web map, which is basically their internal representation

23
00:01:25,540 --> 00:01:28,000
of all the pages and links on the Internet,

24
00:01:28,200 --> 00:01:31,860
and then generating an index, an inverted index, out of that data.

25
00:01:32,400 --> 00:01:35,240
And then they also built a shared cluster for research,

26
00:01:35,620 --> 00:01:37,580
where they could just take data from across their properties,

27
00:01:38,060 --> 00:01:39,060
stick it into a single cluster,

28
00:01:39,280 --> 00:01:42,060
and let sort of research scientists ask questions

29
00:01:42,060 --> 00:01:44,200
and do things like statistics and machine learning over their data.

30
00:01:44,320 --> 00:01:46,800
Back with my team at Facebook, it was used to a similar purpose

31
00:01:46,800 --> 00:01:48,160
for the kind of shared research.

32
00:01:48,600 --> 00:01:51,260
And it evolved into a more traditional business intelligence platform there.

33
00:01:51,680 --> 00:01:53,920
But a lot of other places we see it being used for interesting things

34
00:01:53,920 --> 00:01:56,520
like image processing or genome sequence analysis.

35
00:01:57,140 --> 00:02:00,400
So it's kind of got a wide spectrum of applications so far.

36
00:02:00,540 --> 00:02:04,960
I remember in your slide you said that it's actually kind of a funny background

37
00:02:04,960 --> 00:02:08,220
that used to be a quant, which I'm always mispronouncing.

38
00:02:08,220 --> 00:02:11,600
But I'm kind of personally curious how you,

39
00:02:11,980 --> 00:02:14,380
because you kind of alluded to I was doing this 10 months,

40
00:02:14,440 --> 00:02:15,200
and then I went to Facebook.

41
00:02:16,800 --> 00:02:18,740
You know, I actually, so I had a lot of fun.

42
00:02:18,840 --> 00:02:21,160
I worked at a company called Bear Stearns.

43
00:02:21,760 --> 00:02:23,920
The late Bear Stearns, now a part of J.P. Morgan.

44
00:02:24,720 --> 00:02:29,080
And, you know, the mathematical modeling was very sophisticated

45
00:02:29,080 --> 00:02:31,280
in that it was complex mathematics.

46
00:02:31,940 --> 00:02:33,760
You know, some people might say it wasn't very sophisticated

47
00:02:33,760 --> 00:02:38,000
in terms of the sort of probability that it took into account.

48
00:02:38,000 --> 00:02:43,540
But in finance, you're doing very complex modeling of sort of stochastic processes,

49
00:02:44,000 --> 00:02:46,460
whereas in the web domain, they've kind of hit on this other way

50
00:02:46,460 --> 00:02:49,340
of doing data analysis, which is let's keep our models very simple

51
00:02:49,340 --> 00:02:52,140
and let's be very empirical and just collect as much data as possible

52
00:02:52,140 --> 00:02:52,700
and do that analysis.

53
00:02:52,700 --> 00:02:55,600
I actually just returned last night from New York,

54
00:02:55,680 --> 00:02:57,920
where I spent the week kind of talking to different financial institutions

55
00:02:57,920 --> 00:03:00,660
and learning more about how they're putting Hadoop to use.

56
00:03:00,780 --> 00:03:03,640
You know, there's actually a couple of large banks who are using Hadoop.

57
00:03:03,960 --> 00:03:05,580
And it's a different paradigm, you know.

58
00:03:05,580 --> 00:03:08,120
They're used to sort of hiring sophisticated mathematicians and physicists

59
00:03:08,120 --> 00:03:10,680
to try and model complex phenomenon

60
00:03:10,680 --> 00:03:13,300
and then fit small amounts of data to that model.

61
00:03:13,700 --> 00:03:16,900
And instead, you know, I think they have a lot that they could sort of incorporate

62
00:03:16,900 --> 00:03:19,180
in their infrastructures of just collect as much data as possible

63
00:03:19,180 --> 00:03:21,460
and then try and find some signal inside that data.

64
00:03:21,560 --> 00:03:23,780
Yeah, it's interesting that you were launching in that

65
00:03:23,780 --> 00:03:26,240
because that was one thing I was curious about whenever,

66
00:03:26,820 --> 00:03:28,940
or at least there seems to be a reoccurring pattern

67
00:03:28,940 --> 00:03:32,120
whenever an open source framework for something comes along,

68
00:03:32,120 --> 00:03:35,060
an enterprise framework that, you know, it's sort of,

69
00:03:35,060 --> 00:03:37,500
it's not like exactly replicating.

70
00:03:37,680 --> 00:03:42,280
It's more like you could do these things in the past usually,

71
00:03:42,500 --> 00:03:44,680
and this is a different, hopefully better way

72
00:03:44,680 --> 00:03:47,560
and usually cheaper way of doing, you know, things in the past

73
00:03:47,560 --> 00:03:51,480
that you could do with large data sets or analysis or things like that.

74
00:03:51,540 --> 00:03:54,220
And I'm curious, with sort of, with your Wall Street background,

75
00:03:54,380 --> 00:03:57,420
like how, I mean, is Hadoop sort of different

76
00:03:57,420 --> 00:03:59,960
in the way it goes about dealing with large data sets

77
00:03:59,960 --> 00:04:01,840
than traditional ways of doing things?

78
00:04:02,340 --> 00:04:04,880
Yeah, there's nothing like it in the commercial realm.

79
00:04:05,100 --> 00:04:06,800
I think that was what was most fascinating to me

80
00:04:06,800 --> 00:04:10,000
and led me to want to start the company around the project

81
00:04:10,000 --> 00:04:11,860
because it's one of those rare occasions

82
00:04:11,860 --> 00:04:15,560
where the open source world is just, it's inventing something new

83
00:04:15,560 --> 00:04:18,100
and it's out in front, and it's enterprise class.

84
00:04:18,220 --> 00:04:22,400
I mean, there's, you know, tens of thousands of nodes running Hadoop within Yahoo

85
00:04:22,400 --> 00:04:24,160
and solving all kinds of production problems.

86
00:04:24,180 --> 00:04:26,880
The approach that they took was essentially, you know,

87
00:04:26,880 --> 00:04:30,060
worry about scalability of simple operations first

88
00:04:30,060 --> 00:04:32,660
and then worry about persisting all kinds of data

89
00:04:32,660 --> 00:04:35,460
and not just structured data, which is, you know, there was,

90
00:04:35,680 --> 00:04:38,780
so it's somewhere in between, you know, very large-scale file systems

91
00:04:38,780 --> 00:04:41,160
and, you know, traditional relational databases

92
00:04:41,160 --> 00:04:43,920
where you can do a lot of the complex analytics

93
00:04:43,920 --> 00:04:45,480
that you normally associate with databases

94
00:04:45,480 --> 00:04:48,040
but you have the simplicity and scalability

95
00:04:48,040 --> 00:04:51,860
and the wide range of, you know, data types which can be stored

96
00:04:51,860 --> 00:04:54,080
which you would think of when you get a distributed file system.

97
00:04:54,300 --> 00:04:56,320
So it really is, it's something new, it's something different.

98
00:04:56,500 --> 00:04:56,600
Right.

99
00:04:56,600 --> 00:04:59,100
And we're all kind of, we're all kind of just watching how people use it

100
00:04:59,100 --> 00:05:01,180
and, you know, we haven't quite given it a name yet.

101
00:05:01,260 --> 00:05:02,600
We're just kind of watching how people pick it up.

102
00:05:02,600 --> 00:05:05,240
We haven't quite named, you know, our market segment or anything

103
00:05:05,240 --> 00:05:07,580
but we do, we're just watching people put it in

104
00:05:07,580 --> 00:05:10,460
in front of databases, behind databases, replacing databases,

105
00:05:11,080 --> 00:05:12,200
you know, replacing ETL.

106
00:05:12,440 --> 00:05:14,860
It's just, it's going into all different places in our organization

107
00:05:14,860 --> 00:05:18,640
and we're not really ready to say, you know, exactly what it is yet.

108
00:05:18,660 --> 00:05:22,300
Would you say like a database is the main sort of front end,

109
00:05:22,520 --> 00:05:24,720
if you will, that gets encountered or like,

110
00:05:25,060 --> 00:05:27,260
I mean, and the broader question is sort of

111
00:05:27,260 --> 00:05:29,780
what gets layered on top of Hadoop?

112
00:05:29,880 --> 00:05:30,060
Right.

113
00:05:30,060 --> 00:05:32,780
Like, I mean, you can't, Hadoop isn't like, you know, Excel,

114
00:05:32,920 --> 00:05:34,780
you install it and you're set to go, right?

115
00:05:35,360 --> 00:05:36,040
Yeah, yeah.

116
00:05:36,320 --> 00:05:38,020
That's some, that's a different sort of world.

117
00:05:39,180 --> 00:05:42,000
You know, so the main data sources for Hadoop,

118
00:05:42,080 --> 00:05:44,520
so sort of in front of Hadoop, we see a lot of people stick,

119
00:05:44,620 --> 00:05:46,620
like log files or message logs.

120
00:05:46,660 --> 00:05:47,900
So if you have an enterprise message bus

121
00:05:47,900 --> 00:05:49,160
or you're generating a lot of log files

122
00:05:49,160 --> 00:05:50,800
from your application servers or your web servers,

123
00:05:51,260 --> 00:05:52,720
those generally get ingested into Hadoop

124
00:05:52,720 --> 00:05:55,520
and then people also ingest data from relational databases.

125
00:05:55,980 --> 00:05:57,200
And those could be relational databases

126
00:05:57,200 --> 00:05:58,400
which are serving data to end users

127
00:05:58,400 --> 00:06:00,240
or they could be kind of internal data marts

128
00:06:00,240 --> 00:06:02,000
and you kind of want to consolidate a bunch of data marts

129
00:06:02,000 --> 00:06:03,980
into a larger data repository.

130
00:06:04,600 --> 00:06:06,280
And then the way we see people interact with Hadoop

131
00:06:06,280 --> 00:06:08,960
is both programmatically via MapReduce

132
00:06:08,960 --> 00:06:11,320
and also one thing that was very successful

133
00:06:11,320 --> 00:06:12,400
at my previous employer, Facebook,

134
00:06:12,560 --> 00:06:14,040
was a web interface that we built

135
00:06:14,040 --> 00:06:17,780
to the SQL interface to Hadoop.

136
00:06:17,900 --> 00:06:19,760
So we built a system called Hive

137
00:06:19,760 --> 00:06:21,680
in which it enabled you to write SQL queries

138
00:06:21,680 --> 00:06:23,380
against your Hadoop cluster.

139
00:06:23,860 --> 00:06:25,820
And then we put a web interface on top of that

140
00:06:25,820 --> 00:06:27,720
which allowed the operators

141
00:06:27,720 --> 00:06:29,060
to construct parameterized queries

142
00:06:29,060 --> 00:06:30,600
and then the end users to interact

143
00:06:30,600 --> 00:06:33,300
with just a combo box on a web UI.

144
00:06:34,180 --> 00:06:35,940
So that was one way to deliver solutions with Hadoop.

145
00:06:36,100 --> 00:06:38,080
And so there are people there in sales and marketing

146
00:06:38,080 --> 00:06:40,740
who are running multi-terabyte queries

147
00:06:40,740 --> 00:06:41,860
and they don't even know it.

148
00:06:41,960 --> 00:06:44,540
They're just kind of interacting with the web UI down to it.

149
00:06:44,540 --> 00:06:45,880
I'd say what's most common for developers

150
00:06:45,880 --> 00:06:47,980
is to pull up just a terminal

151
00:06:47,980 --> 00:06:49,860
or your favorite text editor

152
00:06:49,860 --> 00:06:51,280
and write some Python code

153
00:06:51,280 --> 00:06:52,240
and run that over the cluster.

154
00:06:52,480 --> 00:06:52,660
Right, right.

155
00:06:52,720 --> 00:06:54,000
So just sort of write a script

156
00:06:54,000 --> 00:06:55,720
that outputs something for you.

157
00:06:55,800 --> 00:06:56,660
Yeah, so the same way

158
00:06:56,660 --> 00:06:57,740
that you used to write scripts,

159
00:06:58,600 --> 00:06:59,900
you know, which may be like awk scripts

160
00:06:59,900 --> 00:07:01,820
to munch data on a local file system.

161
00:07:02,040 --> 00:07:03,740
That's kind of how developers interact with you.

162
00:07:03,740 --> 00:07:03,880
Right, right.

163
00:07:04,220 --> 00:07:05,320
But just on a much larger scale.

164
00:07:05,320 --> 00:07:06,620
What's your sense for how many people

165
00:07:06,620 --> 00:07:08,620
are out there with Hadoop installs

166
00:07:08,620 --> 00:07:09,460
doing something with it?

167
00:07:09,620 --> 00:07:11,800
Man, I was blown away, actually,

168
00:07:11,880 --> 00:07:12,880
just when I started looking.

169
00:07:13,020 --> 00:07:14,460
And so, you know, every rock you turn over,

170
00:07:14,920 --> 00:07:15,860
there's another one.

171
00:07:15,920 --> 00:07:17,080
So I'd say, you know,

172
00:07:17,080 --> 00:07:18,060
on the order of hundreds

173
00:07:18,060 --> 00:07:19,500
in terms of production applications,

174
00:07:20,100 --> 00:07:21,140
and some, you know,

175
00:07:21,180 --> 00:07:22,260
one order of magnitude larger

176
00:07:22,260 --> 00:07:23,060
in terms of people

177
00:07:23,060 --> 00:07:24,420
who are kind of poking with it,

178
00:07:24,480 --> 00:07:24,980
playing with it.

179
00:07:25,180 --> 00:07:25,380
Right.

180
00:07:25,620 --> 00:07:26,480
The interesting thing, I think,

181
00:07:26,540 --> 00:07:27,360
for this coming year

182
00:07:27,360 --> 00:07:29,320
is as Hadoop marches towards 1.0

183
00:07:29,320 --> 00:07:31,440
and you start to see more stability

184
00:07:31,440 --> 00:07:32,200
around the APIs

185
00:07:32,200 --> 00:07:33,880
and consolidation

186
00:07:33,880 --> 00:07:34,860
of the development processes,

187
00:07:35,380 --> 00:07:37,100
the ecosystem above and below Hadoop,

188
00:07:37,280 --> 00:07:38,460
I think there's a tremendous opportunity

189
00:07:38,460 --> 00:07:39,840
for, you know,

190
00:07:40,220 --> 00:07:42,200
once Hadoop kind of simplifies

191
00:07:42,200 --> 00:07:43,080
the software,

192
00:07:43,180 --> 00:07:43,800
which you're going to be running

193
00:07:43,800 --> 00:07:44,640
over thousands of nodes,

194
00:07:45,120 --> 00:07:46,340
hardware vendors can come in

195
00:07:46,340 --> 00:07:47,500
and optimize hardware to run

196
00:07:47,500 --> 00:07:49,020
for a large Hadoop cluster.

197
00:07:49,400 --> 00:07:50,180
So I think there's going to be

198
00:07:50,180 --> 00:07:52,020
some interesting innovations there.

199
00:07:52,120 --> 00:07:52,840
And then above Hadoop,

200
00:07:53,120 --> 00:07:54,000
you can think about frameworks

201
00:07:54,000 --> 00:07:55,120
for doing traditional business

202
00:07:55,120 --> 00:07:55,900
intelligence work

203
00:07:55,900 --> 00:07:57,000
or even, you know,

204
00:07:57,060 --> 00:07:57,640
text mining

205
00:07:57,640 --> 00:07:59,700
or image processing.

206
00:08:00,020 --> 00:08:01,260
So people building frameworks

207
00:08:01,260 --> 00:08:03,580
to do higher order activities

208
00:08:03,580 --> 00:08:04,240
above Hadoop.

209
00:08:04,500 --> 00:08:04,940
Right, right.

210
00:08:04,940 --> 00:08:05,900
And sort of use Hadoop

211
00:08:05,900 --> 00:08:07,060
as a back end, if you will.

212
00:08:07,180 --> 00:08:07,360
Exactly.

213
00:08:07,360 --> 00:08:07,840
And, I mean,

214
00:08:07,840 --> 00:08:08,800
speaking of the ecosystem,

215
00:08:09,140 --> 00:08:10,020
you, you know,

216
00:08:10,100 --> 00:08:10,800
a little while ago,

217
00:08:10,840 --> 00:08:12,140
I guess it was the fall of last year,

218
00:08:12,480 --> 00:08:14,160
you guys started up Cloudera, right?

219
00:08:14,360 --> 00:08:15,320
And so what,

220
00:08:15,860 --> 00:08:16,240
I mean,

221
00:08:16,300 --> 00:08:16,980
what do you,

222
00:08:17,460 --> 00:08:19,020
what's the position of Cloudera,

223
00:08:19,180 --> 00:08:19,400
you know,

224
00:08:19,460 --> 00:08:20,220
kind of in relation

225
00:08:20,220 --> 00:08:21,440
to the Hadoop community

226
00:08:21,440 --> 00:08:22,220
and Hadoop itself?

227
00:08:22,300 --> 00:08:22,760
We see ourselves

228
00:08:22,760 --> 00:08:23,800
as very active contributors

229
00:08:23,800 --> 00:08:24,500
to that community.

230
00:08:24,980 --> 00:08:26,760
We put our training online for free.

231
00:08:27,400 --> 00:08:27,580
You know,

232
00:08:27,620 --> 00:08:28,520
we hired Tom White

233
00:08:28,520 --> 00:08:30,600
and employed him

234
00:08:30,600 --> 00:08:31,340
while he was producing

235
00:08:31,340 --> 00:08:32,680
the O'Reilly definitive guy

236
00:08:32,680 --> 00:08:33,140
for Hadoop

237
00:08:33,140 --> 00:08:34,460
and continue to work with him.

238
00:08:34,460 --> 00:08:35,380
And we've got a lot of guys

239
00:08:35,380 --> 00:08:36,360
contributing patches

240
00:08:36,360 --> 00:08:37,540
and working towards committer status

241
00:08:37,540 --> 00:08:39,300
to the open source Apache project.

242
00:08:39,780 --> 00:08:41,000
I think the role that we play

243
00:08:41,000 --> 00:08:42,360
in relation to the Apache project

244
00:08:42,360 --> 00:08:44,220
is as kind of the usability

245
00:08:44,220 --> 00:08:46,140
and consolidation team.

246
00:08:46,260 --> 00:08:46,360
You know,

247
00:08:46,380 --> 00:08:47,600
we're able to basically say,

248
00:08:47,980 --> 00:08:48,680
you know,

249
00:08:48,720 --> 00:08:49,600
what would it take

250
00:08:49,600 --> 00:08:49,940
to get,

251
00:08:50,060 --> 00:08:50,800
to lower the bar

252
00:08:50,800 --> 00:08:51,620
to Hadoop adoption?

253
00:08:51,960 --> 00:08:53,080
And how can we bring

254
00:08:53,080 --> 00:08:53,560
the viewpoint

255
00:08:53,560 --> 00:08:54,640
of as many enterprises

256
00:08:54,640 --> 00:08:55,160
as possible

257
00:08:55,160 --> 00:08:56,000
to Hadoop development?

258
00:08:56,460 --> 00:08:57,200
Because right now,

259
00:08:57,280 --> 00:08:57,520
you know,

260
00:08:57,740 --> 00:08:58,780
the majority of Hadoop development

261
00:08:58,780 --> 00:08:59,340
takes place

262
00:08:59,340 --> 00:09:00,940
in web-oriented organizations

263
00:09:00,940 --> 00:09:01,820
like Apple and Facebook.

264
00:09:02,520 --> 00:09:04,460
And to really break Hadoop out

265
00:09:04,460 --> 00:09:05,700
and to get it picked up

266
00:09:05,700 --> 00:09:06,660
in places like retail

267
00:09:06,660 --> 00:09:07,680
and finance and telecom,

268
00:09:08,120 --> 00:09:08,600
you need someone

269
00:09:08,600 --> 00:09:09,660
who's really the active,

270
00:09:09,780 --> 00:09:10,660
voice for those customers.

271
00:09:11,120 --> 00:09:12,220
And they're not going to have

272
00:09:12,220 --> 00:09:13,260
large engineering teams

273
00:09:13,260 --> 00:09:14,260
who are serving as that voice.

274
00:09:14,700 --> 00:09:15,780
So we try and consolidate

275
00:09:15,780 --> 00:09:17,240
the feedback we get

276
00:09:17,240 --> 00:09:18,600
from the different enterprises

277
00:09:18,600 --> 00:09:19,380
who we work with

278
00:09:19,380 --> 00:09:20,540
and bring that

279
00:09:20,540 --> 00:09:22,180
to how we target Hadoop development.

280
00:09:22,300 --> 00:09:23,120
Sort of an advocate

281
00:09:23,120 --> 00:09:24,420
for the enterprise user.

282
00:09:24,480 --> 00:09:24,780
Exactly.

283
00:09:25,260 --> 00:09:26,220
And then we ourselves

284
00:09:26,220 --> 00:09:27,400
work on a lot of sort of

285
00:09:27,400 --> 00:09:28,640
usability and packaging things.

286
00:09:28,940 --> 00:09:29,980
So, you know,

287
00:09:30,120 --> 00:09:30,920
Hadoop has become,

288
00:09:31,340 --> 00:09:31,980
I think it's eight

289
00:09:31,980 --> 00:09:33,060
different sub-projects now.

290
00:09:33,380 --> 00:09:34,180
So when you say Hadoop,

291
00:09:34,240 --> 00:09:35,140
you mean a lot of things.

292
00:09:35,600 --> 00:09:36,080
And, you know,

293
00:09:36,080 --> 00:09:37,100
you have things like Yahoo

294
00:09:37,100 --> 00:09:39,080
making their source code available

295
00:09:39,080 --> 00:09:39,640
that they run

296
00:09:39,640 --> 00:09:40,280
on their own clusters,

297
00:09:40,420 --> 00:09:41,440
which is a slight different

298
00:09:41,440 --> 00:09:42,280
from what's available

299
00:09:42,280 --> 00:09:43,200
in the open source project,

300
00:09:43,360 --> 00:09:44,020
which is slightly different

301
00:09:44,020 --> 00:09:45,560
from what our distribution is.

302
00:09:45,700 --> 00:09:47,060
So what we see our role as

303
00:09:47,060 --> 00:09:47,720
is basically saying,

304
00:09:47,800 --> 00:09:47,880
you know,

305
00:09:47,900 --> 00:09:48,520
there's a lot of good things

306
00:09:48,520 --> 00:09:49,020
Yahoo does.

307
00:09:49,140 --> 00:09:49,740
There's a lot of good things

308
00:09:49,740 --> 00:09:50,220
which come out of the

309
00:09:50,220 --> 00:09:50,880
open source community.

310
00:09:51,080 --> 00:09:52,220
And there's a lot of good things

311
00:09:52,220 --> 00:09:52,720
that we build

312
00:09:52,720 --> 00:09:53,380
when we're working

313
00:09:53,380 --> 00:09:54,220
with our enterprise customers.

314
00:09:54,420 --> 00:09:55,540
So we see our distribution

315
00:09:55,540 --> 00:09:56,340
as kind of a way

316
00:09:56,340 --> 00:09:58,220
to take all that good feedback

317
00:09:58,220 --> 00:09:59,680
and bundle it up

318
00:09:59,680 --> 00:10:00,720
in a really easy to install

319
00:10:00,720 --> 00:10:02,440
and easy to manage distribution

320
00:10:02,440 --> 00:10:03,340
and just get more people

321
00:10:03,340 --> 00:10:04,540
using both Hadoop

322
00:10:04,540 --> 00:10:05,980
and all the related sub-projects.

323
00:10:05,980 --> 00:10:07,400
I guess it's analogous to

324
00:10:07,400 --> 00:10:08,460
you have Linux

325
00:10:08,460 --> 00:10:09,140
and then you have a bunch

326
00:10:09,140 --> 00:10:10,060
of Linux distributions

327
00:10:10,060 --> 00:10:10,980
and you guys have

328
00:10:10,980 --> 00:10:11,880
a Hadoop distribution

329
00:10:11,880 --> 00:10:14,280
essentially that packages things up

330
00:10:14,280 --> 00:10:15,020
and maybe adds

331
00:10:15,020 --> 00:10:15,720
some other things

332
00:10:15,720 --> 00:10:16,680
and of course you have support.

333
00:10:16,680 --> 00:10:17,000
So yeah,

334
00:10:17,160 --> 00:10:18,380
and so one of the important things

335
00:10:18,380 --> 00:10:19,840
is you know that Red Hat

336
00:10:19,840 --> 00:10:20,820
as a vendor

337
00:10:20,820 --> 00:10:21,740
has the distribution

338
00:10:21,740 --> 00:10:23,400
around which people can build on.

339
00:10:23,840 --> 00:10:24,520
You know so Red Hat

340
00:10:24,520 --> 00:10:25,260
is making a commitment

341
00:10:25,260 --> 00:10:26,400
to the APIs they publish

342
00:10:26,400 --> 00:10:27,740
and they're making a commitment

343
00:10:27,740 --> 00:10:28,440
to say you know

344
00:10:28,440 --> 00:10:29,480
we're on this release cycle

345
00:10:29,480 --> 00:10:30,300
and it's going to be coming out.

346
00:10:30,600 --> 00:10:31,220
And we think that

347
00:10:31,220 --> 00:10:32,680
given kind of the breadth

348
00:10:32,680 --> 00:10:33,540
of the Hadoop community

349
00:10:33,540 --> 00:10:34,000
these days

350
00:10:34,000 --> 00:10:35,220
it's important to kind of

351
00:10:35,220 --> 00:10:36,080
have a commercial vendor

352
00:10:36,080 --> 00:10:36,960
to pull some of that

353
00:10:36,960 --> 00:10:37,440
back together

354
00:10:37,440 --> 00:10:38,080
and give people

355
00:10:38,080 --> 00:10:39,300
sort of a solid set of APIs

356
00:10:39,300 --> 00:10:41,300
across you know

357
00:10:41,300 --> 00:10:42,440
whether it's a

358
00:10:42,440 --> 00:10:43,620
you know a web property

359
00:10:43,620 --> 00:10:44,760
or a financial services

360
00:10:44,760 --> 00:10:46,060
companies using it.

361
00:10:46,060 --> 00:10:46,600
We're at a

362
00:10:46,600 --> 00:10:47,940
more or less

363
00:10:47,940 --> 00:10:49,400
an operation centric conference

364
00:10:49,400 --> 00:10:49,940
kind of with

365
00:10:49,940 --> 00:10:51,680
slight tinges of cloud computing

366
00:10:51,680 --> 00:10:52,520
and things like that.

367
00:10:52,580 --> 00:10:52,760
Right.

368
00:10:52,940 --> 00:10:53,760
And I wonder

369
00:10:53,760 --> 00:10:54,460
with the

370
00:10:54,460 --> 00:10:55,140
you went over

371
00:10:55,140 --> 00:10:55,700
some of these things

372
00:10:55,700 --> 00:10:56,500
but with the

373
00:10:56,500 --> 00:10:58,140
the sort of experience

374
00:10:58,140 --> 00:10:59,960
that you and Cloudera

375
00:10:59,960 --> 00:11:00,760
has had supporting

376
00:11:00,760 --> 00:11:01,700
Hadoop installations

377
00:11:01,700 --> 00:11:02,220
what the

378
00:11:02,220 --> 00:11:04,120
for the operations people

379
00:11:04,120 --> 00:11:05,040
and for data center people

380
00:11:05,040 --> 00:11:05,660
what's sort of like

381
00:11:05,660 --> 00:11:06,320
the path

382
00:11:06,320 --> 00:11:07,580
for starting to use Hadoop?

383
00:11:07,580 --> 00:11:08,380
It's usually adapted

384
00:11:08,380 --> 00:11:09,340
by developers right?

385
00:11:09,580 --> 00:11:10,460
So we see developers

386
00:11:10,460 --> 00:11:11,860
get very enthusiastic

387
00:11:11,860 --> 00:11:13,380
about the capabilities

388
00:11:13,380 --> 00:11:14,240
that Hadoop brings.

389
00:11:15,220 --> 00:11:16,020
And so you know

390
00:11:16,020 --> 00:11:16,980
some developer

391
00:11:16,980 --> 00:11:17,900
will scrounge together

392
00:11:17,900 --> 00:11:19,420
some internal resources

393
00:11:19,420 --> 00:11:20,880
or maybe they'll go out

394
00:11:20,880 --> 00:11:21,740
to Amazon Web Services

395
00:11:21,740 --> 00:11:22,780
and spin up a cluster there.

396
00:11:23,880 --> 00:11:24,260
So they'll take

397
00:11:24,260 --> 00:11:25,040
a small data set

398
00:11:25,040 --> 00:11:26,080
say it's maybe like

399
00:11:26,080 --> 00:11:26,900
the search query logs

400
00:11:26,900 --> 00:11:28,580
or something for a website

401
00:11:28,580 --> 00:11:30,580
and then they'll work

402
00:11:30,580 --> 00:11:31,180
on some features

403
00:11:31,180 --> 00:11:31,840
they wanted to build.

404
00:11:32,280 --> 00:11:32,760
So for instance

405
00:11:32,760 --> 00:11:33,600
like you know

406
00:11:33,600 --> 00:11:34,720
you search for X

407
00:11:34,720 --> 00:11:35,540
did you mean Y?

408
00:11:35,680 --> 00:11:36,220
That's a common

409
00:11:36,220 --> 00:11:37,340
search application.

410
00:11:37,900 --> 00:11:38,200
And really

411
00:11:38,200 --> 00:11:39,060
the way that's built

412
00:11:39,060 --> 00:11:40,360
and Peter Norvig

413
00:11:40,360 --> 00:11:41,620
has a great blog post on it

414
00:11:41,620 --> 00:11:42,660
is through just a bunch

415
00:11:42,660 --> 00:11:44,100
of conditional probability calculations.

416
00:11:44,560 --> 00:11:44,980
And that's something

417
00:11:44,980 --> 00:11:45,480
that MapReduce

418
00:11:45,480 --> 00:11:46,040
is very good at.

419
00:11:46,140 --> 00:11:47,040
So we see developers

420
00:11:47,040 --> 00:11:48,420
pick up a small cluster

421
00:11:48,420 --> 00:11:49,700
whether it's development boxes

422
00:11:49,700 --> 00:11:50,680
you know

423
00:11:50,680 --> 00:11:52,040
maybe some nodes

424
00:11:52,040 --> 00:11:52,780
that were getting retired

425
00:11:52,780 --> 00:11:54,420
or it's up on Amazon Web Services

426
00:11:54,420 --> 00:11:55,780
they take the small data set

427
00:11:55,780 --> 00:11:56,760
you know

428
00:11:56,760 --> 00:11:57,700
they get used to

429
00:11:57,700 --> 00:11:58,580
the MapReduce paradigm

430
00:11:58,580 --> 00:11:59,320
they write a few

431
00:11:59,320 --> 00:12:00,460
lines of code

432
00:12:00,460 --> 00:12:00,960
and all of a sudden

433
00:12:00,960 --> 00:12:01,920
they have a feature that works

434
00:12:01,920 --> 00:12:02,740
and they go show it to someone

435
00:12:02,740 --> 00:12:03,040
and it's like

436
00:12:03,040 --> 00:12:03,360
oh hey

437
00:12:03,360 --> 00:12:04,300
let's put that in production.

438
00:12:04,840 --> 00:12:05,260
And all of a sudden

439
00:12:05,260 --> 00:12:05,660
they go like

440
00:12:05,660 --> 00:12:06,280
oh geez

441
00:12:06,280 --> 00:12:06,580
you know

442
00:12:06,580 --> 00:12:07,440
the seven nodes

443
00:12:07,440 --> 00:12:07,920
that I stole

444
00:12:07,920 --> 00:12:08,540
from underneath

445
00:12:08,540 --> 00:12:09,880
the build team's desk

446
00:12:09,880 --> 00:12:11,260
probably shouldn't be

447
00:12:11,260 --> 00:12:11,800
going into production.

448
00:12:11,960 --> 00:12:12,320
I actually

449
00:12:12,320 --> 00:12:13,640
I saw one of our clients

450
00:12:13,640 --> 00:12:14,680
it was literally running

451
00:12:14,680 --> 00:12:16,360
on one of the

452
00:12:16,360 --> 00:12:17,500
data analysts' laptop.

453
00:12:17,800 --> 00:12:18,360
They had a feature

454
00:12:18,360 --> 00:12:18,680
that was

455
00:12:18,680 --> 00:12:19,760
the data flow

456
00:12:19,760 --> 00:12:20,920
was coming off the live site

457
00:12:20,920 --> 00:12:21,760
onto their laptop.

458
00:12:22,040 --> 00:12:23,420
Sort of like desktop computing.

459
00:12:23,740 --> 00:12:23,880
Yeah

460
00:12:23,880 --> 00:12:24,960
and so you'd see people

461
00:12:24,960 --> 00:12:25,440
doing this

462
00:12:25,440 --> 00:12:27,280
and so

463
00:12:27,280 --> 00:12:28,720
the next thing they say

464
00:12:28,720 --> 00:12:29,060
is okay

465
00:12:29,060 --> 00:12:29,660
well how can

466
00:12:29,660 --> 00:12:30,700
how can we buy

467
00:12:30,700 --> 00:12:31,340
a Hadoop cluster

468
00:12:31,340 --> 00:12:32,440
that's you know

469
00:12:32,440 --> 00:12:33,120
going to serve us

470
00:12:33,120 --> 00:12:34,220
as like an enterprise class

471
00:12:34,220 --> 00:12:34,620
and so

472
00:12:34,620 --> 00:12:36,380
then we see a lot of people

473
00:12:36,380 --> 00:12:37,180
go to server vendors

474
00:12:37,180 --> 00:12:37,740
and say like

475
00:12:37,740 --> 00:12:37,980
you know

476
00:12:37,980 --> 00:12:38,660
do you have a

477
00:12:38,660 --> 00:12:39,440
do you have a server

478
00:12:39,440 --> 00:12:39,960
which is meant

479
00:12:39,960 --> 00:12:40,820
to be right for Hadoop

480
00:12:40,820 --> 00:12:41,820
so that's one of the first

481
00:12:41,820 --> 00:12:42,400
questions a lot of

482
00:12:42,400 --> 00:12:42,880
our support customers

483
00:12:42,880 --> 00:12:43,200
Oh right

484
00:12:43,200 --> 00:12:43,800
so they're looking

485
00:12:43,800 --> 00:12:44,920
for like hardware specs

486
00:12:44,920 --> 00:12:45,340
exactly

487
00:12:45,340 --> 00:12:46,120
for Google

488
00:12:46,120 --> 00:12:46,340
Yeah

489
00:12:46,340 --> 00:12:46,840
so that's one of the

490
00:12:46,840 --> 00:12:47,320
first questions

491
00:12:47,320 --> 00:12:47,900
people ask

492
00:12:47,900 --> 00:12:48,760
and so that allows

493
00:12:48,760 --> 00:12:49,280
them to grow

494
00:12:49,280 --> 00:12:50,200
out a larger cluster

495
00:12:50,200 --> 00:12:51,020
and then the next

496
00:12:51,020 --> 00:12:51,620
thing they want to know

497
00:12:51,620 --> 00:12:52,300
is a lot of things

498
00:12:52,300 --> 00:12:52,880
that I talked about

499
00:12:52,880 --> 00:12:53,480
today around

500
00:12:53,480 --> 00:12:54,000
you know

501
00:12:54,000 --> 00:12:54,580
how do I administer

502
00:12:54,580 --> 00:12:55,340
this file system

503
00:12:55,340 --> 00:12:55,880
in such a way

504
00:12:55,880 --> 00:12:56,360
that the data

505
00:12:56,360 --> 00:12:56,960
I put in there

506
00:12:56,960 --> 00:12:57,700
isn't going to disappear

507
00:12:57,700 --> 00:12:59,240
so they start asking

508
00:12:59,240 --> 00:12:59,820
a lot of questions

509
00:12:59,820 --> 00:13:00,760
about you know

510
00:13:00,760 --> 00:13:01,660
how do we do backups

511
00:13:01,660 --> 00:13:02,240
you know

512
00:13:02,240 --> 00:13:03,080
how do we do security

513
00:13:03,080 --> 00:13:03,460
right

514
00:13:03,460 --> 00:13:04,500
all those things

515
00:13:04,500 --> 00:13:05,620
and then we see

516
00:13:05,620 --> 00:13:05,940
people

517
00:13:05,940 --> 00:13:06,800
the developers

518
00:13:06,800 --> 00:13:07,300
so that's how

519
00:13:07,300 --> 00:13:08,020
the operations team

520
00:13:08,020 --> 00:13:08,600
kind of follows

521
00:13:08,600 --> 00:13:09,340
and we see the developers

522
00:13:09,340 --> 00:13:10,040
start experimenting

523
00:13:10,040 --> 00:13:10,460
with some of the

524
00:13:10,460 --> 00:13:11,220
higher level frameworks

525
00:13:11,220 --> 00:13:12,180
so things like

526
00:13:12,180 --> 00:13:13,800
Pig, Hive, Cascading

527
00:13:13,800 --> 00:13:14,440
we see a lot of people

528
00:13:14,440 --> 00:13:15,340
start playing around with

529
00:13:15,340 --> 00:13:16,680
and getting comfortable

530
00:13:16,680 --> 00:13:17,820
not writing just

531
00:13:17,820 --> 00:13:18,460
plain MapReduce

532
00:13:18,460 --> 00:13:19,460
but stepping up a layer

533
00:13:19,460 --> 00:13:20,500
so digging down

534
00:13:20,500 --> 00:13:21,000
on that a little bit

535
00:13:21,000 --> 00:13:21,400
like what

536
00:13:21,400 --> 00:13:22,600
what are the typical

537
00:13:22,600 --> 00:13:22,920
sort of

538
00:13:22,920 --> 00:13:23,320
and you went over

539
00:13:23,320 --> 00:13:23,660
some of this

540
00:13:23,660 --> 00:13:24,060
in your talk

541
00:13:24,060 --> 00:13:24,840
what is the typical

542
00:13:24,840 --> 00:13:25,740
kind of hardware

543
00:13:25,740 --> 00:13:26,700
like setups

544
00:13:26,700 --> 00:13:28,200
that get installed

545
00:13:28,200 --> 00:13:29,140
best case

546
00:13:29,140 --> 00:13:29,800
is when people

547
00:13:29,800 --> 00:13:30,640
take a 1U

548
00:13:30,640 --> 00:13:31,340
or 2U

549
00:13:31,340 --> 00:13:32,060
you know

550
00:13:32,060 --> 00:13:33,620
not bargain basement

551
00:13:33,620 --> 00:13:35,000
but commodity node

552
00:13:35,000 --> 00:13:37,000
and they stick in

553
00:13:37,000 --> 00:13:38,500
some 7200 RPM

554
00:13:38,500 --> 00:13:39,220
SATA drives

555
00:13:39,220 --> 00:13:40,540
fairly high capacity

556
00:13:40,540 --> 00:13:41,320
like 1 terabytes

557
00:13:41,320 --> 00:13:42,200
1.5 terabytes

558
00:13:42,200 --> 00:13:44,080
so the kind of

559
00:13:44,080 --> 00:13:44,520
standard node

560
00:13:44,520 --> 00:13:44,840
that a lot of

561
00:13:44,840 --> 00:13:45,420
the benchmarking

562
00:13:45,420 --> 00:13:45,980
is done on

563
00:13:45,980 --> 00:13:46,480
at Yahoo

564
00:13:46,480 --> 00:13:48,000
is a 1U box

565
00:13:48,000 --> 00:13:50,320
with two sockets

566
00:13:50,320 --> 00:13:50,800
four quarters

567
00:13:50,800 --> 00:13:51,440
in each socket

568
00:13:51,440 --> 00:13:52,820
8 gig of RAM

569
00:13:52,820 --> 00:13:54,180
and 4 by 1 terabyte

570
00:13:54,180 --> 00:13:54,760
SATA drives

571
00:13:54,760 --> 00:13:55,560
and with a gig

572
00:13:55,560 --> 00:13:56,040
E interconnect

573
00:13:56,040 --> 00:13:57,000
and then commodity

574
00:13:57,000 --> 00:13:57,780
switches at the top

575
00:13:57,780 --> 00:13:58,180
of the rack

576
00:13:58,180 --> 00:13:59,040
and a core switch

577
00:13:59,040 --> 00:14:01,100
connecting many racks

578
00:14:01,100 --> 00:14:02,820
so that's kind of

579
00:14:02,820 --> 00:14:03,300
the standard hardware

580
00:14:03,300 --> 00:14:03,580
setup

581
00:14:03,580 --> 00:14:04,040
we're seeing

582
00:14:04,040 --> 00:14:04,420
some people

583
00:14:04,420 --> 00:14:05,080
start to experiment

584
00:14:05,080 --> 00:14:05,960
with 2U boxes

585
00:14:05,960 --> 00:14:07,200
with 12 drives

586
00:14:07,200 --> 00:14:07,560
per node

587
00:14:07,560 --> 00:14:08,980
which is interesting

588
00:14:08,980 --> 00:14:09,500
because you think

589
00:14:09,500 --> 00:14:10,080
about what happens

590
00:14:10,080 --> 00:14:10,720
to a Hadoop cluster

591
00:14:10,720 --> 00:14:11,300
over time

592
00:14:11,300 --> 00:14:12,500
capacity becomes

593
00:14:12,500 --> 00:14:13,140
one of the

594
00:14:13,140 --> 00:14:14,540
sort of limiting factors

595
00:14:14,540 --> 00:14:15,880
so those 2U

596
00:14:15,880 --> 00:14:17,260
12 drive boxes

597
00:14:17,260 --> 00:14:19,520
are more storage dense

598
00:14:19,520 --> 00:14:20,760
and they tend

599
00:14:20,760 --> 00:14:21,360
to be a better fit

600
00:14:21,360 --> 00:14:21,700
for people

601
00:14:21,700 --> 00:14:22,160
who have been running

602
00:14:22,160 --> 00:14:22,760
Hadoop in production

603
00:14:22,760 --> 00:14:23,440
for a few years

604
00:14:23,440 --> 00:14:23,820
it's a pretty

605
00:14:23,820 --> 00:14:24,440
interesting time

606
00:14:24,440 --> 00:14:25,040
we're able to

607
00:14:25,040 --> 00:14:25,880
collect a lot of things

608
00:14:25,880 --> 00:14:27,020
so one good example

609
00:14:27,020 --> 00:14:28,020
is if you're building

610
00:14:28,020 --> 00:14:29,320
an ad serving algorithm

611
00:14:29,320 --> 00:14:30,500
I've never worked

612
00:14:30,500 --> 00:14:30,780
at Google

613
00:14:30,780 --> 00:14:31,680
but one story I hear

614
00:14:31,680 --> 00:14:32,440
is that there was

615
00:14:32,440 --> 00:14:33,980
a significant performance

616
00:14:33,980 --> 00:14:35,000
increase in their ad serving

617
00:14:35,000 --> 00:14:36,180
when they started logging

618
00:14:36,180 --> 00:14:37,640
not just the ad

619
00:14:37,640 --> 00:14:38,360
that actually got served

620
00:14:38,360 --> 00:14:38,780
to the user

621
00:14:38,780 --> 00:14:40,140
but sort of the 10 ads

622
00:14:40,140 --> 00:14:41,420
that were closest

623
00:14:41,420 --> 00:14:41,900
to being served

624
00:14:41,900 --> 00:14:42,180
to the user

625
00:14:42,180 --> 00:14:43,480
so they started being able

626
00:14:43,480 --> 00:14:44,640
to collect a lot more context

627
00:14:44,640 --> 00:14:45,500
or for instance

628
00:14:45,500 --> 00:14:46,100
one thing that you see

629
00:14:46,100 --> 00:14:46,860
in a lot of search companies

630
00:14:46,860 --> 00:14:47,760
is instead of just logging

631
00:14:47,760 --> 00:14:48,660
which link was clicked on

632
00:14:48,660 --> 00:14:49,760
you log every link

633
00:14:49,760 --> 00:14:50,260
which was shown

634
00:14:50,260 --> 00:14:52,040
so you start seeing people

635
00:14:52,040 --> 00:14:52,740
just kind of open up

636
00:14:52,740 --> 00:14:53,320
their minds

637
00:14:53,320 --> 00:14:55,360
to oh what if we logged

638
00:14:55,360 --> 00:14:56,180
all this information

639
00:14:56,180 --> 00:14:57,560
and then had the ability

640
00:14:57,560 --> 00:14:58,780
to go process it later

641
00:14:58,780 --> 00:14:59,740
what can we learn

642
00:14:59,740 --> 00:15:00,340
about our system

643
00:15:00,340 --> 00:15:01,640
what can we do better

644
00:15:01,640 --> 00:15:02,600
or even like

645
00:15:02,600 --> 00:15:03,940
a huge collection of images

646
00:15:03,940 --> 00:15:05,100
is there value in that

647
00:15:05,100 --> 00:15:06,040
can we start doing things

648
00:15:06,040 --> 00:15:08,260
like automatically tagging

649
00:15:08,260 --> 00:15:08,940
you've seen a lot of

650
00:15:08,940 --> 00:15:09,540
companies start up

651
00:15:09,540 --> 00:15:09,800
around that

652
00:15:09,800 --> 00:15:10,460
I think people are finding

653
00:15:10,460 --> 00:15:12,120
both volumes of data

654
00:15:12,120 --> 00:15:13,100
that are now addressable

655
00:15:13,100 --> 00:15:13,940
with systems like Hadoop

656
00:15:13,940 --> 00:15:15,660
and new kinds of data

657
00:15:15,660 --> 00:15:16,380
that are addressable

658
00:15:16,380 --> 00:15:17,260
with systems like Hadoop

659
00:15:17,260 --> 00:15:17,780
yeah that they could

660
00:15:17,780 --> 00:15:18,480
previously choose to

661
00:15:18,480 --> 00:15:19,240
throw away

662
00:15:19,240 --> 00:15:20,500
or not have the resources

663
00:15:20,500 --> 00:15:21,460
to do anything with

664
00:15:21,460 --> 00:15:22,200
do you have like

665
00:15:22,200 --> 00:15:23,020
a sense of a road map

666
00:15:23,020 --> 00:15:23,720
for what Cloudera

667
00:15:23,720 --> 00:15:24,540
is going to be up to

668
00:15:24,540 --> 00:15:26,180
and like what's going on

669
00:15:26,180 --> 00:15:26,680
in the Hadoop world

670
00:15:26,680 --> 00:15:27,140
like you said

671
00:15:27,140 --> 00:15:27,680
1.0 is

672
00:15:27,680 --> 00:15:29,660
for as popular as it is

673
00:15:29,660 --> 00:15:30,500
1.0 is coming out

674
00:15:30,500 --> 00:15:31,200
sometime soon

675
00:15:31,200 --> 00:15:33,200
but across those two

676
00:15:33,200 --> 00:15:33,840
domains there

677
00:15:33,840 --> 00:15:35,000
I mean what are you guys

678
00:15:35,000 --> 00:15:35,680
looking at in the future

679
00:15:35,680 --> 00:15:36,160
so we're putting a lot

680
00:15:36,160 --> 00:15:37,240
of our open source

681
00:15:37,240 --> 00:15:38,080
development energy

682
00:15:38,080 --> 00:15:40,000
into usability

683
00:15:40,000 --> 00:15:41,400
just making it

684
00:15:41,400 --> 00:15:42,020
as easy as possible

685
00:15:42,020 --> 00:15:42,520
to install

686
00:15:42,520 --> 00:15:42,980
configure

687
00:15:42,980 --> 00:15:44,180
and get up and running

688
00:15:44,180 --> 00:15:45,040
and then

689
00:15:45,040 --> 00:15:46,700
next to usability

690
00:15:46,700 --> 00:15:47,800
is getting data

691
00:15:47,800 --> 00:15:48,240
in and out

692
00:15:48,240 --> 00:15:49,640
so we built a tool

693
00:15:49,640 --> 00:15:50,640
called Scoop

694
00:15:50,640 --> 00:15:51,440
SQL to Hadoop

695
00:15:51,440 --> 00:15:52,400
which allows people

696
00:15:52,400 --> 00:15:52,940
to basically just

697
00:15:52,940 --> 00:15:53,760
point at a database

698
00:15:53,760 --> 00:15:55,100
dump it to a flat file

699
00:15:55,100 --> 00:15:55,620
and get it into

700
00:15:55,620 --> 00:15:56,680
or not even dump

701
00:15:56,680 --> 00:15:57,140
to a flat file

702
00:15:57,140 --> 00:15:57,760
just get it directly

703
00:15:57,760 --> 00:15:58,380
into HDFS

704
00:15:58,380 --> 00:15:59,880
so once you can

705
00:15:59,880 --> 00:16:01,080
get a cluster up and running

706
00:16:01,080 --> 00:16:01,960
you can configure it easily

707
00:16:01,960 --> 00:16:03,060
and you can get data

708
00:16:03,060 --> 00:16:03,700
in and out easily

709
00:16:03,700 --> 00:16:04,520
you know some things

710
00:16:04,520 --> 00:16:04,820
that we're starting

711
00:16:04,820 --> 00:16:05,580
to think about also

712
00:16:05,580 --> 00:16:07,100
include monitoring

713
00:16:07,100 --> 00:16:08,820
you know

714
00:16:08,820 --> 00:16:09,460
out of the box

715
00:16:09,460 --> 00:16:10,240
having your distribution

716
00:16:10,240 --> 00:16:12,020
be really easy

717
00:16:12,020 --> 00:16:12,940
to monitor and administer

718
00:16:12,940 --> 00:16:13,940
and then

719
00:16:13,940 --> 00:16:14,940
sort of some

720
00:16:14,940 --> 00:16:15,740
longer term things

721
00:16:15,740 --> 00:16:16,120
that we're putting

722
00:16:16,120 --> 00:16:16,540
a lot of energy

723
00:16:16,540 --> 00:16:16,780
into

724
00:16:16,780 --> 00:16:18,000
are some of the

725
00:16:18,000 --> 00:16:18,660
work that's going

726
00:16:18,660 --> 00:16:19,280
into 1.0

727
00:16:19,280 --> 00:16:19,900
so a system

728
00:16:19,900 --> 00:16:20,380
like Avro

729
00:16:20,380 --> 00:16:21,080
which is the

730
00:16:21,080 --> 00:16:21,560
cross language

731
00:16:21,560 --> 00:16:22,460
serialization framework

732
00:16:22,460 --> 00:16:23,540
that's going to enable

733
00:16:23,540 --> 00:16:24,300
sort of versioning

734
00:16:24,300 --> 00:16:24,960
and rolling upgrades

735
00:16:24,960 --> 00:16:25,800
and also security

736
00:16:25,800 --> 00:16:27,680
via a unified

737
00:16:27,680 --> 00:16:28,400
RPC interface

738
00:16:28,400 --> 00:16:28,940
we've got a guy

739
00:16:28,940 --> 00:16:29,340
who's working

740
00:16:29,340 --> 00:16:30,560
very hard on Avro

741
00:16:30,560 --> 00:16:32,000
we're thinking a lot

742
00:16:32,000 --> 00:16:32,620
about availability

743
00:16:32,620 --> 00:16:33,440
to the file system

744
00:16:33,440 --> 00:16:33,940
that's something

745
00:16:33,940 --> 00:16:34,560
a lot of enterprise

746
00:16:34,560 --> 00:16:35,320
customers need

747
00:16:35,320 --> 00:16:36,420
so we're putting

748
00:16:36,420 --> 00:16:37,400
a lot of work

749
00:16:37,400 --> 00:16:37,940
into Zookeeper

750
00:16:37,940 --> 00:16:38,520
you know

751
00:16:38,520 --> 00:16:39,060
one of our engineers

752
00:16:39,060 --> 00:16:39,860
built some Python

753
00:16:39,860 --> 00:16:40,380
bindings

754
00:16:40,380 --> 00:16:40,840
and he's working

755
00:16:40,840 --> 00:16:41,620
on dynamic cluster

756
00:16:41,620 --> 00:16:42,400
membership protocols

757
00:16:42,400 --> 00:16:42,880
right now

758
00:16:42,880 --> 00:16:44,020
for Zookeeper

759
00:16:44,020 --> 00:16:45,240
so we see those

760
00:16:45,240 --> 00:16:45,740
things as sort of

761
00:16:45,740 --> 00:16:46,580
second order projects

762
00:16:46,580 --> 00:16:47,520
which will turn into

763
00:16:47,520 --> 00:16:48,840
which will facilitate

764
00:16:48,840 --> 00:16:49,820
building some of these

765
00:16:49,820 --> 00:16:50,740
like very

766
00:16:50,740 --> 00:16:51,900
very desired features

767
00:16:51,900 --> 00:16:52,620
like high availability

768
00:16:52,620 --> 00:16:53,140
of the file system

769
00:16:53,140 --> 00:16:54,040
it seems like a lot

770
00:16:54,040 --> 00:16:54,480
of your attention

771
00:16:54,480 --> 00:16:56,080
is and rightly so

772
00:16:56,080 --> 00:16:57,400
for being sort of

773
00:16:57,400 --> 00:16:58,440
as odd as it may sound

774
00:16:58,440 --> 00:16:59,080
in this context

775
00:16:59,080 --> 00:16:59,640
for making things

776
00:16:59,640 --> 00:17:00,360
more user friendly

777
00:17:00,360 --> 00:17:01,220
but it's targeted

778
00:17:01,220 --> 00:17:02,300
towards just operations

779
00:17:02,300 --> 00:17:02,920
and system

780
00:17:02,920 --> 00:17:03,340
oh yeah

781
00:17:03,340 --> 00:17:04,680
I mean we don't see

782
00:17:04,680 --> 00:17:05,560
why running

783
00:17:05,560 --> 00:17:06,680
a thousand node

784
00:17:06,680 --> 00:17:07,700
cluster shouldn't be

785
00:17:07,700 --> 00:17:08,460
as easy as

786
00:17:08,460 --> 00:17:08,780
you know

787
00:17:08,780 --> 00:17:09,760
administering your laptop

788
00:17:09,760 --> 00:17:10,440
right right

789
00:17:10,440 --> 00:17:10,780
you know

790
00:17:10,780 --> 00:17:11,560
if you build

791
00:17:11,560 --> 00:17:12,400
the right user interfaces

792
00:17:12,400 --> 00:17:12,940
and you build

793
00:17:12,940 --> 00:17:13,540
the right monitoring

794
00:17:13,540 --> 00:17:14,020
infrastructure

795
00:17:14,020 --> 00:17:14,900
yeah

796
00:17:14,900 --> 00:17:16,480
yeah why not

797
00:17:16,480 --> 00:17:17,060
why not

798
00:17:17,060 --> 00:17:18,120
I'd love

799
00:17:18,120 --> 00:17:19,520
so there's this term

800
00:17:19,520 --> 00:17:19,980
that was kind of

801
00:17:19,980 --> 00:17:20,680
introduced by Google

802
00:17:20,680 --> 00:17:21,000
recently

803
00:17:21,000 --> 00:17:22,300
warehouse scale computers

804
00:17:22,300 --> 00:17:23,000
right

805
00:17:23,000 --> 00:17:24,120
where you know

806
00:17:24,120 --> 00:17:24,680
Google thinks

807
00:17:24,680 --> 00:17:25,600
of the unit of computing

808
00:17:25,600 --> 00:17:26,500
as a data center

809
00:17:26,500 --> 00:17:27,760
and Hadoop

810
00:17:27,760 --> 00:17:28,580
is really the first

811
00:17:28,580 --> 00:17:29,540
software project

812
00:17:29,540 --> 00:17:30,360
outside of Google

813
00:17:30,360 --> 00:17:30,740
I think

814
00:17:30,740 --> 00:17:32,200
that has the same mindset

815
00:17:32,200 --> 00:17:32,760
you know

816
00:17:32,760 --> 00:17:33,860
the whole notion

817
00:17:33,860 --> 00:17:34,220
of Hadoop

818
00:17:34,220 --> 00:17:34,600
and all of its

819
00:17:34,600 --> 00:17:35,500
related sub-projects

820
00:17:35,500 --> 00:17:36,680
are what would it take

821
00:17:36,680 --> 00:17:38,360
to make a single

822
00:17:38,360 --> 00:17:38,880
data center

823
00:17:38,880 --> 00:17:40,140
addressable as one entity

824
00:17:40,140 --> 00:17:41,320
as like a single

825
00:17:41,320 --> 00:17:41,720
computer

826
00:17:41,720 --> 00:17:42,140
right

827
00:17:42,140 --> 00:17:42,800
and so the

828
00:17:42,800 --> 00:17:43,560
distributed file system

829
00:17:43,560 --> 00:17:44,100
ties together

830
00:17:44,100 --> 00:17:44,800
all the storage

831
00:17:44,800 --> 00:17:45,700
and MapReduce

832
00:17:45,700 --> 00:17:46,340
is your sort of

833
00:17:46,340 --> 00:17:47,840
process execution engine

834
00:17:47,840 --> 00:17:48,940
and you get this

835
00:17:48,940 --> 00:17:50,100
great sort of API

836
00:17:50,100 --> 00:17:51,600
that you have no idea

837
00:17:51,600 --> 00:17:52,040
whether you're

838
00:17:52,040 --> 00:17:52,960
executing on ten nodes

839
00:17:52,960 --> 00:17:53,620
or a thousand nodes

840
00:17:53,620 --> 00:17:54,580
you're writing the same code

841
00:17:54,580 --> 00:17:55,000
right

842
00:17:55,000 --> 00:17:56,200
so I don't know

843
00:17:56,200 --> 00:17:57,080
it makes me very excited

844
00:17:57,080 --> 00:17:57,800
because the space

845
00:17:57,800 --> 00:17:58,600
of addressable problems

846
00:17:58,600 --> 00:17:59,220
is just getting much

847
00:17:59,220 --> 00:17:59,980
I mean I'm sure

848
00:17:59,980 --> 00:18:00,560
there are going to be

849
00:18:00,560 --> 00:18:02,060
commercial entities

850
00:18:02,060 --> 00:18:03,600
pouring into the space

851
00:18:03,600 --> 00:18:04,160
given some of the

852
00:18:04,160 --> 00:18:04,860
adoption that Hadoop's

853
00:18:04,860 --> 00:18:05,100
had

854
00:18:05,100 --> 00:18:05,840
and we see a lot

855
00:18:05,840 --> 00:18:07,080
of people sort of

856
00:18:07,080 --> 00:18:08,400
reworking their

857
00:18:08,400 --> 00:18:09,980
their product roadmaps

858
00:18:09,980 --> 00:18:10,840
who are maybe

859
00:18:10,840 --> 00:18:11,560
close to Hadoop

860
00:18:11,560 --> 00:18:12,360
to try and look

861
00:18:12,360 --> 00:18:13,040
more like it

862
00:18:13,040 --> 00:18:14,060
so yeah

863
00:18:14,060 --> 00:18:15,240
so I can

864
00:18:15,240 --> 00:18:15,840
but I can

865
00:18:15,840 --> 00:18:16,220
I can

866
00:18:16,220 --> 00:18:16,700
I can sort of

867
00:18:16,700 --> 00:18:17,340
honestly say it

868
00:18:17,340 --> 00:18:17,640
there's really

869
00:18:17,640 --> 00:18:18,380
nothing like it

870
00:18:18,380 --> 00:18:20,680
and it's an

871
00:18:20,680 --> 00:18:21,340
open source project

872
00:18:21,340 --> 00:18:22,580
which is very exciting

873
00:18:22,580 --> 00:18:23,400
always a bonus

874
00:18:23,400 --> 00:18:24,200
yeah exactly

875
00:18:24,200 --> 00:18:26,340
I can't imagine

876
00:18:26,340 --> 00:18:26,980
not you know

877
00:18:26,980 --> 00:18:28,420
we had a lot

878
00:18:28,420 --> 00:18:29,220
of guys in my group

879
00:18:29,220 --> 00:18:30,260
my last employer

880
00:18:30,260 --> 00:18:30,600
Facebook

881
00:18:30,600 --> 00:18:31,220
who came from

882
00:18:31,220 --> 00:18:32,120
companies like Oracle

883
00:18:32,120 --> 00:18:34,840
and they're just

884
00:18:34,840 --> 00:18:35,640
very very excited

885
00:18:35,640 --> 00:18:36,160
to be working

886
00:18:36,160 --> 00:18:36,660
on a project

887
00:18:36,660 --> 00:18:37,640
where you know

888
00:18:37,640 --> 00:18:38,240
you write code

889
00:18:38,240 --> 00:18:38,600
one day

890
00:18:38,600 --> 00:18:39,160
you ship it

891
00:18:39,160 --> 00:18:39,500
and then you're

892
00:18:39,500 --> 00:18:40,280
helping out someone

893
00:18:40,280 --> 00:18:41,020
you know

894
00:18:41,020 --> 00:18:42,140
halfway across the world

895
00:18:42,140 --> 00:18:42,880
you're not caught

896
00:18:42,880 --> 00:18:43,740
behind a 12 or 18

897
00:18:43,740 --> 00:18:44,520
month release cycle

898
00:18:44,520 --> 00:18:45,140
or longer

899
00:18:45,140 --> 00:18:45,860
yeah exactly

900
00:18:45,860 --> 00:18:46,180
well great

901
00:18:46,180 --> 00:18:46,640
well thanks for

902
00:18:46,640 --> 00:18:47,220
taking all this time

903
00:18:47,220 --> 00:18:47,720
to talk with us

904
00:18:47,720 --> 00:18:48,220
that was fun

905
00:18:48,220 --> 00:18:48,600
oh yeah

906
00:18:48,600 --> 00:18:49,220
no problem man