Big Data as Opposed to Small Data Mark Whitehorn
Upcoming SlideShare
Loading in...5
×
 

Big Data as Opposed to Small Data Mark Whitehorn

on

  • 1,045 views

 

Statistics

Views

Total Views
1,045
Views on SlideShare
931
Embed Views
114

Actions

Likes
0
Downloads
13
Comments
0

2 Embeds 114

http://forum.razved.info 100
http://www.vivastream.com 14

Accessibility

Categories

Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment
  • Each scan in the data file requires a lot of highly intensive processing in order to determine what proteins were present in the cell.Some examples…..Currently a single threaded pc based application is used
  • 4 minutes

Big Data as Opposed to Small Data Mark Whitehorn Big Data as Opposed to Small Data Mark Whitehorn Presentation Transcript

  • BIG DATA - AS OPPOSEDTO SMALL DATAMark Whitehorn
  • What is Big data?Is it really just a marketing campaign?http://www.perceptualedge.com/articles/visual_business_intelligence/big_data_big_ruse.pdf“If you’re like me, the mere mention of Big Datanow turns your stomach….Why all the fuss? Why,indeed. Essentially, Big Data is a marketingcampaign, pure and simple.”Stephen Few 2
  • Big dataClearly I am not like Stephen Few.I don’t believe I have a particular axe to grind, Isimply find this interestingThis talk is designed to try to explain:• what Big Data is• what characteristics we have found useful• why it may be of interest to you 3• a paradox
  • DataAll computer applications manipulate data 4
  • DataSo, in the ’60 and ‘70s we rapidly learnt toseparate the data, and its manipulation, fromthe application 5
  • DataSo, in the ’60 and ‘70s we rapidly learnt toseparate the data, and its manipulation, fromthe applicationWhich led directly to the development ofdatabase engines and, ultimately, relationalones (DB2, Oracle, SQL Server) 6
  • DataData has always existed in two, very broad,flavours….. • Data that is treated as small, discrete packages and is a good fit with the relational way of storing and querying data • Data that is not as above 7
  • Data is stored in tables LicenseNo Make Model Year Colour CER 162 C Triumph Spitfire 1965 Green EF 8972 Bentley Mk. VI 1946 Black YSK 114 Bentley Mk. VI 1949 Red 8 Mark Whitehorn
  • Data is stored in tables Each table has a name Car LicenseNo Make Model Year Colour CER 162 C Triumph Spitfire 1965 Green EF 8972 Bentley Mk. VI 1946 Black YSK 114 Bentley Mk. VI 1949 Red 9 Mark Whitehorn
  • Data is stored in tables Car LicenseNo Make Model Year Colour CER 162 C Triumph Spitfire 1965 Green EF 8972 Bentley Mk. VI 1946 Black YSK 114 Bentley Mk. VI 1949 Red Data is atomic 10 Mark Whitehorn
  • Data is stored in tables Columns Car LicenseNo Make Model Year Colour CER 162 C Triumph Spitfire 1965 Green EF 8972 Bentley Mk. VI 1946 Black YSK 114 Bentley Mk. VI 1949 Red 11 Mark Whitehorn
  • Data is stored in tables Columns Car LicenseNo Make Model Year Colour CER 162 C Triumph Spitfire 1965 GreenRows EF 8972 Bentley Mk. VI 1946 Black YSK 114 Bentley Mk. VI 1949 Red 12 Mark Whitehorn
  • Data is stored in tables Car LicenseNo Make Model Year Color CER 162 C Triumph Spitfire 1965 Green EF 8972 Bentley Mk. VI 1946 Black YSK 114 Bentley Mk. VI 1949 Red Each row represents a unique entity in the ‘real’ world…… 13 Mark Whitehorn
  • 14
  • DataThe manipulation consists typically of sub-setting the data by rows and columns andthen doing some sums 15
  • DataNote that this kind of manipulation is treatingthe data as atomic, which is fine, because therelational model assumes atomicity of dataNote also, that the rows are unordered 16
  • Data• Data has always existed in two, very broad, flavours….. • Data that is inherently atomic and is a good fit with the relational way of storing and querying data • Data that is not as above 17
  • Examples• Examples of ‘other’ data: • Images • Music • Word docs • Sensor data • Web logs • Twitter • Machines • Point of Sale 18 • Mass spectrometers
  • What’s in a name?So, what do we call the ‘rest’? • Un-structured? • Semi-structured? • Multi-structured? • Non-relational? • Non-tabular? 19
  • What’s in a name?• What about: • Big data? 20
  • Other definitions? •VVVvvvv • Volume • Variety • Velocity • Value • Very interesting • Various other words beginning with V….. 21
  • Big Data – not new?• So why have we focused, for the last 30 years, almost exclusively on the first flavour?• Because it: • is easy (relatively easy – Jim Gray*) • represents a significant proportion of the available data *Jim Gray and Andreas Reuter - Transaction Processing: Concepts and Techniques (1993) Turning Award 1998 22
  • Big Data has come of age• Two factors have changed • Rise of the Machines • Increase is computational power• There is a great synergy here • We are acquiring far more big data and we have computational power to extract the information it contains 23
  • Big Data is hard• 3 Vs• It is highly variable• We often want to look inside the data • Frequently non-atomic • Need custom functions for virtually every operation • find the rotating wing aircraft in the image • Identify the best customer • What does the blog sphere think of our company? 24
  • Big Data• Examples • Log file • Mass spec. • Images 25
  • Big Data• Examples • Log file • Mass spectrometer • Image 26
  • Big Data • Examples • Log file • Mass spec. • Images 27
  • What is Big Data?• Examples • Log file • Mass spec. • Images BIG DATA
  • Summary so far……• Just as you can always fit an aircraft engine into a car chassis, you can always put Big Data in a table, but you probably don’t want to• The analysis is not sub-setting the data by rows and columns• So each class of big data usually require a (lovingly hand-crafted) custom analysis 30
  • Case StudyBig Data in the Life Sciences World The massed spectrometers Why would anyone do that? 31
  • Human Genome Project$3 billion – 13 YearsSequencing completed(2003). 32
  • Human Genome ProjectHuman Genome Project$3 billion – 13 YearsOur genes define us.Errr…. how doesthat work exactly? 33
  • What is a protein? DNA Protein blueprint product 34
  • Why study proteins PROTEOME GENOME Genes contain Proteins carry outinstructions for creating functions within a cell 35 proteins
  • Protein: ACTIN Example ProteinsFunction: Contracts Muscles Protein: Insulin Function: Controls Blood Sugar Protein: Keratin Function: Forms Hair and NailsO 2 Protein: Hemoglobin Function: Carries Oxygen Protein: Antibody Function: Fights Viruses 36
  • biSCIENCE20-25,000 genes in the humangenome.Every nucleated cell in the samehuman has the same genome.But not all genes are active atthe same time.Perm any 15-18,00 activeproteins in any one cell at any 37one time.
  • slowly changing millions of years over a day rapidly changing 38
  • Studying ProteinsProteins are chopped up using anenzyme to make them easier to measure.A specialised instrument (Mass Spectrometer) isused to measure (‘weigh’) the small proteinfragments.We can use the mass of the small fragments tocarry out intelligent database searches to identifywhich protein was detected. 39
  • Protein PeptidesMKLNISFPATGCQKLIEVDDERKLRTFYEKRMATEVAADALGEEWKGYVVRISGGNDKQGFPMKQGVLTHGRVRLLLSKGHSCYRPRRTGERKRKSVRGCIVDANLSVLNLVIVKKGEKDIPGLTDTTVPRRLGPKRASRIRKLFNLSKEDDVRQYVVRKPLNKEGKKPRTKAPKIQRLVTPRVLQHKRRRIALKKQRTKKNKEEAAEYAKLLAKRMKEAKEKRQEQIAKRRRLSSLRASTSKSESSQK Amino Acids 40
  • Mass SpectrometryAn analytical technique for the determination ofthe elemental composition of a sample. 41
  • Spectra P1 P2 P3 42
  • Mass SpectraFile Sizes: typically severalgigabytes per MS run.Identifications: range from 500- 438000 protein identifications.
  • pepTRACKER 44TRACK. VISUALISE. DISCOVER.
  • 80% 60% 40% 20% 45
  • Localisation Protein Peptide Alignment Map Normalised Profiles for Synthesis, Degradation and TurnoverComparison BetweenCompartments 46
  • Custom analysis and custom visualisation –vital tools in understanding big data 47
  • Intensive Data Processing Required to deriveInformation from the raw data Base Line Correction Peak Detection BIOConductor PROcess R Package Deisotoping 48 Proteomics Volume 3, Issue 8, Article first published online: 12 AUG 2003
  • “proteomics is much more complicatedthan genomics . . . while an organisms genome is more or less constant, the proteome differs from cell to cell and over time”Computationally, perhaps three ordersof magnitude more complex than HGP 49
  • biSCIENCEWhy bother trying to quantify it?Because this is payback time.Documenting the proteomeopens the door to a whole newworld. 50
  • biSCIENCESo, what is a data scientist?My favourite description comes from Twitter:“Yeah, so Im actually a data scientist. I just do thisbarista thing in between gigs.”More cynically:“A data scientist is just an analyst who lives inCalifornia.” 51
  • biSCIENCEPossibly more accurate is that a data scientist (DS) is“a better software engineer than any statistician anda better statistician than any software engineer”. 52
  • biSCIENCEDSs are also part artist and part engineer. Theyneed a toolbox of techniques, skills, processes andabilities from which to construct novel solutions.And they need the ability to create a UI that turnstheir abstract finding into something that the usersof the system can understand, so DSs also need theskills to create elegant visualisations that turn rawdata into information. 53
  • biSCIENCEAnd (yes, there’s more) they need to be able tocommunicate well with people. There is little use increating a superb analytical process if you can’tcommunicate how and why it works to the boardmembers. 54
  • biSCIENCEAnd then there is the curiosity. Duncan Ross(Director of Data Sciences at Teradata) characteriseddata scientists well:The first and most important trait is curiosity. Insanecuriosity. In many walks of life evolution selectsagainst the kind of person who decides to find outwhat happens “if I push that button”. Data Scienceselects for it. 55
  • biSCIENCESo, what are the general characteristics of a DS?They include:• insatiable curiosity (see above)• interdisciplinary interests• excellent communication skills• excellent analytical capabilities 56
  • biSCIENCEDSs also need a good working knowledge of:• machine learning techniques• data mining• statistics• maths• algorithm development• code development• data visualisation• multi-dimensional database design and implementation 57
  • biSCIENCESpecific skills include the technologies to handle bigdata:• NoSQL databases• Hadoop and related technologies• MapReduce and its implementation on differing software platforms 58
  • biSCIENCEDSs also have an intimate knowledge of languagessuch as:• SQL• MDX• R• Functional and OOP languages such as Erlang and Java 59
  • biSCIENCEMost of all, no matter what they are called, all truedata scientists have started playing with some dataat 8:00PM and suddenly found it is 3:00AM.
  • Case Study Twitter Who loves you? Social/text/sentiment 61
  • Consider the humble tweet… 62
  • Consider the humble tweet…As, indeed, Sally Bercow shouldhave done 63
  • Consider the humble tweet…As, indeed, Sally Bercow shouldhave done *Innocent Face* 64
  • Consider the humble tweet…I’d just like to apologise for that last slide but I would point outthat it “contained no accusation whatsoever … Mischievous but not libellous.” 65
  • Case Study Oil Rig data Gone fishing Sensor data 66
  • Lessons learned• Engagement• Choose you battles – look for an area where you can gain competitive advantage• Choose your platform carefully• Programming – algorithm development• Data scientists • Custom algorithms 67 • Custom visualisations
  • Thank you very much for listeningAny Questions?Mark Whitehorn(MarkWhitehorn@computing.dundee.ac.uk) 68
  • BIG DATA - AS OPPOSEDTO SMALL DATA60 minutesMark Whitehorn