Making Sense of Data
•Download as PPTX, PDF•
9 likes•1,047 views
Mike Driscoll discusses lessons for start-ups around making sense of data. Some key points include collecting customer data to get feedback, embracing multiple data formats and stores as there is no single database, separating query and storage layers, making data easy to access and fast to query, selectively instrumenting machines and customers, and architecting around business questions rather than letting data architecture drive needs. The technology stack is like a nervous system processing sensory input from data.
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Making Sense of Data
Similar to Making Sense of Data (20)
Recently uploaded
Recently uploaded (20)
Making Sense of Data
- 1. MIKE DRISCOLL CO-FOUNDER + CTO METAMARKETS @medriscoll making sense of data: Lessons for start-ups
- 2. If it is unmanaged, you will be blind to weaknesses, deaf to new opportunities, and dumb to your customers. data IS SENSORY INPUT
- 3. Data is the sensory input that moves through it. your technology stack is your nervous system
- 5. Collecting customer data is a way to “get out of the building.” create feedback loops
- 6. customers
- 8. Complexity lies at the boundaries between systems make etl a priority
- 9. Real-Time Daily Weekly sync data latencieswith decision loops
- 10. All data models are wrong. Some data models are useful. don’t agonize overdata schemas
- 11. Hadoop is a processing layer You also need a query layer hadoop isn’t enough
- 13. Embrace a polyglot architecture of formats and data stores there is no‘One True database’
- 14. A RESTful query layer will reduce pain of migration. separate query& storage layers
- 15. Reduce the barriers to accessing data across systems. make data easy
- 16. “Human-time” means that queries return in seconds. make data fast
- 17. Human activity is small in size fully instrument your customers
- 18. Human activity is small in size. fully instrument your customers
- 19. Machine-generated data can quickly overwhelm. selectively instrument your machines
- 20. Machine-generated data can quickly overwhelm. selectively instrument your machines
- 21. Work backwards from business questions. Don’t let data architecture drive business needs architect aroundbusiness questions
- 23. Someone who can munge, model, & visualize data hire a data scientist
- 24. Engineers with a thin grasp of statistics beat statisticians with thin grasp of engineering. working code beats theoretical models
- 25. Isolated from production systems. Analytics are a different constituency with different needs create an analytics sandbox
- 27. Both internal & external obsess about dashboard design
- 29. Either by directly monetizing them or enhance customer experience extract value from yourdata assets
- 30. YOUR TECHNOLOGYSTACK IS YOUR NERVOUS SYSTEM.YOUR DATA IS YOUR SENSORY INPUT.
- 31. MIKE DRISCOLL CO-FOUNDER + CTO METAMARKETS @medriscoll making sense of data: lessons for start-ups questions?
Editor's Notes
- Feedback loops.
- Over the next set of slides, I’m going discuss some lessons as data moves through a start-ups organization...
- So this is how we frame our technology stack at my start-up, Metamarkets. It’s a four-tiered stack. I believe that many start-ups have similar stacks when they think about how data moves through them.But there’s something important missing here: your technology stack doesn’t exist in a vacuum.
- Over the next set of slides, I’m going discuss some lessons as data moves through a start-ups organization...
- To be successful, we’ve got to incorporate feedback, both from customers, and the larger world.Feedback is critical. Steve Blank and Eric Ries have talked about not iterating in a vacuum.The feedback you can achieve by managing your data can be incredibly important.
- Which begins at ingestion, and ends at the top with products.
- ETL often gets a bad wrap. Nothing could be more important to your company than moving data between systems.That is what ETL does. It should be a first class piece of your architecture, you should put one of top engineers at this layer of the stack.(At Metamarkets, we have a former VP of BlackRock working on ETL, and he’s been outstanding).When our ETL breaks down, the data stops flowing, and our business stops moving.
- * Don’t invest in real-time data if you’re making weekly decisions.* Moving away from batch systems is hard work.Alternatively, some systems – such as those required for monitoring – may need sub-millisecond response times.But as a general rule, reducing latency in systems creates value in unexpected ways.
- Don’t get bogged down in discussions of the perfect data format for your company. “All models are wrong, some models are useful.”There is no such thing.
- Which begins at ingestion, and ends at the top with products.
- You will likely end up using a variety of data stores in your organization.So don’t agonize over your data store choices.
- As you scale and grow, you will have to change storage layers.We went through three different versions, first Postgres, then Greenplum, then HBase, before developing on our own version.
- embrace standardssimple, flat formats wherever possible (XML is the clamshell packaging of data)We recently onboarded a client who gave us JSON data. It’s a beautiful thing.Everyone knows SQL: Cloudera found that Hadoop cluster use went up 10x when HIVE was installed.
- But HIVE isn’t going to cut it for getting quick insights into their data. No wants to wait 15 minutes for answers.Put in ETL flows that summarize data, and keep a core set of key business metrics in a “hot” database, one that can be queried in real-time.
- Feedback loops.
- Requirements for systems should be driven by their business needs.
- Which begins at ingestion, and ends at the top with products.
- but remember...
- 4sq explorepymkkaggle winnerswritten by individuals who were engineers first, statisticians second.when hiring folks to do your analytics, you want those who can roll up their sleaves and actually code the models themselvees.
- don’t make your analytics team compete for resources, or jeopardize production systemsthey will only get burned and then cut outset up systems where analytics folks can play with data, safelyanalytics often falls into the class of problems that are important, but not urgent. don’t let this happen to your organization.
- Which begins at ingestion, and ends at the top with products.
- Data represents the totality of a start-up’s sensory experiences.Absent a well-developed digital nervous system to respond to these inputs, you are blind to your deficiencies, deaf to your customers, and dumb to your opportunities.
- Either externally, as Klout,Flightcaster, and BillGuard have done.4SQ’s Explore and LinkedIn’s PYMK, has both improved User Experience.Having strong analytical talent in your organization is critical to success here.
- Feedback loops.