The document discusses the growth and impact of Hadoop over the past 4 years. It notes that Hadoop is already widely used, with over 500PB of data and thousands of implementations. It also discusses how Hadoop enables new types of analytics and insights through its ability to handle large and diverse datasets. The document outlines the growing ecosystem around Hadoop and predicts continued growth and innovation in both the technology and its applications in industries.
2. Reflections On You
12+ months using on average
114.5TB average size
66 average nodes in Use
500+ certified on Hadoop in 1 year
60+PB Total
Data from pre-conference survey
3.
4. Immutable Law of Data
RDBMS
Hadoop
Volume, Variety, Velocity increase
5. Immutable Law of Data
RDBMS
Hadoop
Volume, Variety, Velocity increase
Geopbytes
Brontobytes
Yottabytes
Zettabytes
Exabytes
Terabytes
19. So Much To See Today!
• Optimizing search
• Advanced analytics in the Army
• Using Flume &Hive for log data
• Analyzing VOIP data with R
20. What’s Next?
Market
• Adoption
• Agility
• Flexibility
Technology
• Accelerated innovation
from community
• More tools e.g., monitoring
• More automation
• More stability
• More interfaces
21. • At the core of the open source platform for
data
• Four years old and going strong!
22.
23. Organizational Impact
• More knobs and dials
• Fine grain control
• Achieve previously impossible /
impractical
• Save money
• Save time
• Greater flexibility with data
Copyright 2010 Cloudera Inc. All rights reserved
24. Hadoop World Keynote (NOTES)
• Themes
– Hadoop is already a big deal
• Keep in mind the why
• Solving real problems now
– It is about the platform with Hadoop at the
core
• Why
• Helps you profit
• More accessible now than ever, real people with
enterprise ops and enterprise skills, no longer the
exclusive demand of the PhDs
– What’s on the Horizon for Hadoop
Copyright 2010 Cloudera Inc. All rights reserved
25. Hadoop is Having a
Transformative Impact (notes)
• Continued growth and excitement
• Transformative to your career, your enterprise, your market
– Star maker
– Get ready for Hadoop being a big deal for your companies
– Your market – hyper personalization
– Use data to interact in a more customized fashion
– “It’s hard not to have a TB of data” – Mike
– Operability and SLAs for a critical enterprise platform
– Education and training
– A new stack for analytics (CEP (flume) CDH (Sqoop) dbms/BI)
• Future is now
– Use cases now and impact it is having and where it will be, look at
Facebook, Yahoo, eBay etc.
Copyright 2010 Cloudera Inc. All rights reserved
26. What is on the Horizon for
Hadoop (notes)
• Continued growth and excitement
• Transformative to your career, your enterprise, your market
– Star maker –
• good for your career, help make critical changes in the way customers are supported, major new business opportunities etc.
• Pull cloudera certification #’s
– Get ready for Hadoop being a big deal for your companies
• Enterprise will be more agile and able capture and analyze more data to better target ads, find fraud, etc.
• Agility – impacts the things that matter to you
• What’s happened before the transaction
– Your market – hyper personalization
• 100s’s of vertical apps to be created (developers are you listening?)
• Trend that crosses? Any other trend we can compare to? DBMS growth? Improvements in operations,
• How detailed sources have changed
• Devices, understanding how people interact with your business – retail, online entertainment, fin serv, government
– Use data to interact in a more customized fashion
– “It’s hard not to have a TB of data” – Mike
– Operability and SLAs for a critical enterprise platform
– Education and training
– A new stack for analytics (CEP (flume) CDH (sqoop) dbms/BI)
• Future is now
– Use cases now and impact it is having and where it will be, look at Facebook, Yahoo, eBay etc.
Copyright 2010 Cloudera Inc. All rights reserved
27. Emerging Importance
of Data Scientist
• Able to impact business at many
levels
• New conference focused data and
data related roles — O’Reilly
Strata Conference
Copyright 2010 Cloudera Inc. All rights reserved
28. Unprecedented Data Volume,
Velocity and Variety
Data Growth
Out Pacing
Processing Power
Organizations
Swamped and
Turning to Hadoop
61% CAGR
42% CAGR
Data
Transistors
Copyright 2010 Cloudera Inc. All rights reserved
29. Transforming Analytic
Requirements
• Insight into this data needs more than simple
tabular analysis
– More is needed for meaningful answers
• You can and will do deeper and more
introspective analysis
– Machine learning, natural language processing, clustering,
sophisticated statistical analysis, modeling and back testing
• Looking for patterns
– You can see patterns in lots of data that are invisible in less
data. You need pattern discovery tools
Copyright 2010 Cloudera Inc. All rights reserved
30. Hadoop: Already a Big Deal!!
Massive Adoption
Vibrant & Growing Community
100’s of PB Under Management
1000’s of Implementations
31. Benefitting From a Dynamic
OS Community
• Community around
Hadoop is proliferating
and expanding
• > ½ Hadoop sub-projects
promoted to TLPs
• Dozens of related projects
• 100’s of developers
& growing
Copyright 2010 Cloudera Inc. All rights reserved
32. Interest in Hadoop Has Exploded
More are looking for it
Leading analysts report
significant growth
in inquiries
Major increase
in coverage
Copyright 2010 Cloudera Inc. All rights reserved
33. A Data Management Platform
Applications
Copyright 2010 Cloudera Inc. All rights reserved
** 180 public, self-declared powered by Apache Hadoop and related projects (apache powered by wiki)** that’s a small fraction of users** There are 900+ people hereAttendees right now: 915Responses: 151Avg usage in months: > 12mo (for non 0 responses), 8.76mo overallAverage storage size: 114.5TB (for non 0 responses) 78TB overallTotal data under management: 11.79PB for responses, 104.7PB extrapolatedAverage # nodes: 66.25 (for non 0 responses) 46 overallTotal notes under management: 6957 for responses, 60625 extrapolatedLongest time spent on Hadoop: 36 monthsMost data: 2PBLargest cluster: 1300 nodesPercentage over 10 nodes: 55%Percentage over 10TB: 38.5%** 40% are here to start learning about Hadoop
Last year, we told you that Hadoop was going to be a big deal. It is.You’re here, so you know that. You’re the reason why.Shout outs: Doug, Tom (new book!); Yahoo! for its early and continued contribution (security). Global community.Three things driving success.
The big story is data. There’s a lot of it. The increase is fastest in varieties of data poorly suited to RDBMS’ tabular storage. 2.5 zettaytes (x1000 exabytes) of new information in 2012 Digital universe grew by 62% last year to 800 zettabytes and will grow to 1.2 yottabytes this year
In case you want to know what comes next.Keynotes will sound increasingly ridiculous.
Corollary:With volume, variety and velocity, data is messy.In the 80s and 90s, RDBMS was the answer.Now, we see row stores and column stores (Vertica), emerging class of distributed hash-style systems (Membase, Hbase, Cassandra), Hadoop. Major shift in the data center!
Hadoop was built for data. Cheap storage lets you deal with volume, but also: variety and velocity.This technology was invented by the biggest names on the Web – Google, Facebook, Yahoo! – to solve their business problems. But those are also your business problems. Those companies weren’t different than you; they were just a little bit in the future. Hadoop is the high-value analytics engine for today’s businesses.
It is famous for getting big – no “up to”.
Variety and velocity sometimes come before volume.RECALL: First reason for Hadoop’s success: data.
Second reason: Open source.Don’t hand Oracle a loaded gun.Community is thrashing proprietary. Bill Joy: All the smartest people work somewhere else.Linux, Postgres, MySQL, JBoss, Xensource: Commoditiized existing markets.Not so, Hadoop! No existing commercial product does large-scale, powerful analytics over a mixture of complex and structured data. First truly innovative OSS platform I know.
Third reason: No longer just HDFS and MapReduce.Talking to Kellan Elliott-McRae, VP Eng Etsy. “Toolchain is mature.” Talks like an engineer, but he’s right.Last year: Hadoop. This year: A mature, flexible, powerful data management platform ready for enterprise-grade use.Cloudera has been in the market for two years, solving real problems. We’ve badly needed this platform to emerge from the pieces.We believe deeply in the open source ethos. Not political: Gives us a huge unfair advantage in the market.We spent two years listening to you.
Today, we are announcing the newest release of CDHv3.Integrated the critical pieces of the platform from the global open source community.Incorported Yahoo!’s excellent work on security – see Todd Lipcon’s talk today.Deeply committed to OSS. This is and will always be a 100% pure Apache-licensed platform. No one can turn off your access to your data, and no one can take away your critical business analytics.
Hadoop: Innovative, powerful, platform, but just one piece of the puzzle in your data center.Needs to play nicely with all the other systems and tools you run.Established vendors, with Cloudera, have announced integration with Hadoop. You’ll see this list grow quickly over the next year. “Right tool for the job.”
This platform is ripe for creation of new vertical apps and analytical tools. Expect to see an explosion in the number and variety of solutions built on it.
So what are people doing?You’re surrounded by more than 900 other people today. They’ll tell you how they’re using Hadoop to solve real business problems. Not science projects; not a toy. The platform is driving costs down and letting companies extract profit from data in ways that were impossible just a few years ago.You’ll hear about real use cases from smart people.
You'll hear General Electric talk about how they're able to capture and monitor data on their mission-critical IT systems. They're able to track load and usage, monitor what users are doing and watch for impending failures. GE runs a large internal cloud; with the Hadoop platform, they're squeezing costs out of that cloud, and driving value straight to the bottom line.
You'll hear Bank of America talk about its very fast-growing Hadoop cluster. BofA uses the platform to better understand its customers, and to more precisely score risk in its individual user accounts and in its portfolio as a whole. These days, the player in the market who is the most right about risk wins, and wins big. Banks can reduce their reserve capital requirements, freeing up money to invest elsewhere, and can avoid loss and drive gains by choosing market positions with the best information available.
You'll hear eBay describe how they're using this platform to get much closer to customers. They're able to understand much better the preferences and behavior of individual users. That means they can deliver precisely-targeted products in auctions, tailored exactly to the individual. That precise targeting turns into more purchases, and more purchases drives eBay's top line.
Forty-two sessions across a very long day.Unlike last year, we’ve scheduled lots of breaks for networking and hallway conversations – always the real reason to attend a conference!Hilary Mason from bit.ly, Flip Kromer of Infochimps, Drew Conway, Terry Jones and Clay Johnson are the young Turks transforming the industry, defining a whole new profession of "data scientist" and creating new frameworks and tools for understanding data and creating insight and profit out of information. Hit the coffee breaks and work the hallways hard.
We are, all together, in the middle of a dramatic change in how the world works. Data is central to government, business, security and leisure. That's more true today than it was when we saw you here last year. It will be still more true next year. We have all, together, created this amazing, powerful, innovative new open source software platform.Over the next twelve months, you’ll see the market realize this even more broadly. You’ll see the platform continue to evolve and improve. You’ll see more companies take advantage of the power of Hadoop to unlock information from data and drive value and profit.
Look how far Hadoop has come!Data, open source and the emergence of the platform have made the software mainstream. It’s enterprise-ready, and works with enterprise tools. Look at the progress since Doug created the project back in 2006. Imagine the transformation we’ll see over the next four years.And I love how, even with all that progress, the little yellow elephant is still pushing!Go out and enjoy the day. Hear some great talks and make new friends. Get a book signed by Tom. We’ll see you back here this afternoon – and don’t forget the reception afterward, one floor down.
Tim is the founder and CEO of O’Reilly Media. Its goal: to be a catalyst for technology change by capturing and transmitting the knowledge of "alpha geeks" and other innovators.Privileged to know him for about fourteen years.Learned a great deal about open source, community and technology from conversations, his speeches and his writings.Created a number of very important industry conferences – OSCON, Etech and others. New in February – Strata, aimed squarely at data scientists.Thank Tim for coming.
Let's also try to drive home the message that we expect hundreds of vertical apps to be built on top of CDH over the next year or two. So it's not just about the users, it's about the developers (, developers, developers!!!).
Opportunities for individuals to find and solve interesting, professionally rewarding problems.help make critical changes in the way customers are supported and interacted withContribute to the bottom linemajor new business opportunities etc.
New kinds of data introduced daily – mashups, sensors, log files, videos etc.Data created at machine speedI think it's worth highlighting that this platform acknowledges the three biggest trends in enterprise data management today: growth in data volumes, growth in unstructured data, and clouds/commodity hardware. Analytics and apps are compelling – see the stories at this conference