Download to read offline
Uploaded byChris Swan
Big data debunking some of the myths
The document discusses the complexities and misconceptions surrounding big data, emphasizing the importance of understanding algorithms and data types. It highlights different categories of big data problems, categorizing them by data volume and algorithm complexity. Additionally, it warns against over-reliance on opaque systems and stresses the significance of knowing the specifics of your data and tools.
More Related Content
Similar to Big data debunking some of the myths
![Software Bill of Materials (SBOMs) for C applications [FOSDEM 2025]](https://cdn.slidesharecdn.com/ss_thumbnails/fosdem2025-csboms-250202120401-89aeff6c-thumbnail.jpg?width=640&height=640&fit=bounds)
Big data debunking some of the myths
- 1. copyright 2015 Big Data: debunkingsome of the myths Chris Swan @cpswan
- 2. copyright 2015 Agenda • Mybackground • What do I mean by big data? • Know your algorithm • Know your data • Performance
- 3. copyright 2015 My background CTO CTOClient Experience Co-head CTO Security Corporate Finance fintech, early stage IT R&D – Networks and security Grid, app server engineering Combat System Engineer
- 4. copyright 2015 Recent adventurewith Big Data
- 5. copyright 2015 Misquoting RogerNeedham Whoever thinks their analytics problem is solved by big data, doesn’t understand their analytics problem and doesn’t understand big data 5
- 6. copyright 2015 What doI mean by ‘big data’?
- 7. copyright 2015 Overview 7 Based ona blog post from April 2012 – http://is.gd/swbdla Problem Types Algorithm Complexity DataVolume Simple Big Data Quant
- 8. copyright 2015 Simple problems 8 Lowdata volume, low algorithm complexity Problem Types Algorithm Complexity DataVolume Simple Big Data Quant
- 9. copyright 2015 Quant Problems 9 Anydata volume, high algorithm complexity Problem Types Algorithm Complexity DataVolume Simple Big Data Quant
- 10. copyright 2015 Big DataProblems 10 High data volume, low algorithm complexity Problem Types Algorithm Complexity DataVolume Simple Big Data Quant Types of Big Data Problem: 1. Inherent 2. More data gives better result than more complex algorithm
- 11. copyright 2015 11 Good -Lots of new tools, mostly open source Bad - Term being abused by marketing departments Ugly - Can easily lead to over reliance on systems that lack transparency and ignore specific data points 'Computer says no', but nobody can explain why The good, the bad and the ugly of Big Data
- 12. copyright 2015 It’s importantto know your algorithms
- 13. copyright 2015 Turning anassumption into a line
- 14. copyright 2015 There arelots of algorithms to understand
- 15.
- 16.
- 17.
- 18. copyright 2015 It’s alsoimportant to know your data
- 19. copyright 2015 Whatever wecall our ‘experts’
- 20. copyright 2015 Who’s heardof Anscombe’s quartet?
- 21. copyright 2015 Same statisticalproperties, but… http://en.wikipedia.org/wiki/Anscombe's_quartet
- 22.
- 23. copyright 2015 Don’t agoniseover distros The performance of Hadoop distros are all the same to within 1 server within a cluster Stefan Groschupf One of the creators of Hadoop
- 24. copyright 2015 Small isstill beautiful
- 25.
- 26. copyright 2015 In termsof distance http://loci.cs.utk.edu/dsi/netstore99/docs/presentations/keynote/sld023.htm
- 27. copyright 2015 Interactive >Real time
- 28.