Your SlideShare is downloading. ×
1st Birmingham Big Data Science Group meetup
Upcoming SlideShare
Loading in...5

Thanks for flagging this SlideShare!

Oops! An error has occurred.

Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

1st Birmingham Big Data Science Group meetup


Published on

Published in: Technology

1 Like
  • Be the first to comment

No Downloads
Total Views
On Slideshare
From Embeds
Number of Embeds
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

No notes for slide


  • 1. Welcome to the Birmingham Big Data Science Group (BIDS)
    Faizan Javed
    Intermark Group
    Sponsor: Intermark Group
  • 2. BIDS Stats
    Founded April 10, 2011
    9 members (and counting..)
    Founder: Faizan Javed, Co-Founder: QasimIjaz
    Online presence: for co-ordinatingmeetups:
    Also on (for related articles and announcements):
  • 3. Agenda
    What is Big Data?
    Quick overview of related technologies:
    Large-scale distributed systems and platforms
    NoSQL data stores
    Intelligent algorithms/web-mining/information retrieval techniques
    Highly-scalable systems
  • 4. What is Big Data?
    More people connected to the internet
    Social media explosion (Web 2.0): Facebook, Twitter, etc.
    Huge volumes of data being collected: sensors, mobile devices, machine-to-machine communications, social media and retail sites web logs for browsing patterns
    “Big” in Big Data is relative:  today's "big" is certainly tomorrow's "medium" and next week's "small.“
    “Big Data" is when the size of the data itself becomes part of the problem. Going from Gigabytes to Petabytes!
  • 5.
  • 6. Big Data, Big Numbers McKinsey report, May 2011:
  • 7. Why care about big data?
    Deep analysis of data can be a competitive advantage.
    More data  easier to find consistent patterns
    More data usually beats better algorithms
    Ex 1: Predict customer preferences and target ads on an ecommerce website.
    Ex 2: Improve search quality.
    Ex 3: Bank risk modeling (aggregate customer activity from different lines of businesses)
    Key point: “Many different sources” & “unstructured data”
  • 8. Big Players on the Big Data Scene
    The Government
  • 9. The need for new techniques
    Traditional “relational” techniques breakdown at scale.
    NoSQL databases: Cassandra, Hbase, Riak, etc
    Large-scale “commodity” scale-out distributed computing techniques: MapReduce/Hadoop, Percolator, etc
    Analytics platforms: IBM BigInsight, EMC GreenPlum
  • 10. The NoSQL revolution
  • 11. Prominent NoSQL database users
    Cassandra: Facebook, Twitter, Rackspace, Reddit,
    Riak: Mozilla,, Comcast
    Voldemort: LinkedIn
    MongoDB: Foursquare, Etsy,, Intuit
    Hbase: Stumbleupon, Twitter, Infolinks, Adobe,,
  • 12. Hadoop-based SMAQ stack
    public static class Reduce extends Reducer<Text, IntWritable, Text, IntWritable>
    public void reduce(Text key, Iterable<IntWritable> values, Context context)
    throws IOException, InterruptedException
    int sum = 0;
    for (IntWritableval : values)
    sum += val.get();
    context.write(key, new IntWritable(sum));
  • 13. Hadoop-based SMAQ stack
    Hadoop comes with HDFS – Hadoop Distributed File Sytem.
    Can be used alongside various NoSQL systems (Hbase most common)
  • 14. Hadoop-based SMAQ stack
    Pig (yahoo)
    input = LOAD 'input/sentences.txt' USING TextLoader();
    words = FOREACH input GENERATE FLATTEN(TOKENIZE($0)); grouped = GROUP words BY $0;
    counts = FOREACH grouped GENERATE group, COUNT(words); ordered = ORDER counts BY $0; STORE ordered INTO 'output/wordCount' USING PigStorage();
    Hive (facebook)
    INSERT OVERWRITE TABLE xyz_com_page_views SELECT page_views.* FROM page_views WHERE >= '2008-03-01' AND <= '2008-03-31' AND page_views.referrer_url like '';
  • 15. Next-generation systems: going beyond MapReduce/Hadoop
    Mostly Google and Yahoo innovations.
    Percolator – “real-time” MapReduce. Powers Google Instant.
    Dremel – superfast “Hive” to interact with large-datasets. Inhouse-Google.
    Pregel– highly efficient graph computing for analyzing social graphs. In-house Google. Open-source projects available.
    Megastore- scalable NoSQL like system with ACID semantics but lower consistency across partitions. In-house Google.
    Next-gen Hadoop at Yahoo: enhanced scalability (going beyond 4000 clusters), support for multiple programming paradigms, enhanced cluster utilization.
  • 16. Intelligent Web & machine learning
    Recommendation systems, data/web mining, natural language processing
    Recommendation systems:
    A type of collaborative filtering/information retrieval technique.
    Uses user profiles, ratings, browsing habits to recommend items not yet considered.
    First made famous in the commercial arena by
  • 17. & Netflix recommendation systems
  • 18. Foursquare (3/2011) and Google Places (5/2011)
  • 19. Hot area!Netflix and competitions
  • 20. Search Engines (Google, Bing, Wolfram, Lucene/Nutch, etc)
  • 21. Search innovations @ LinkedIn
    Uses open-source Luceneproject for social graph search and real-time indexing and searching.
    Dynamic filters automatically generated based on your query results!
  • 22. Conclusion
    Big Data is a very challenging and promising area
    Can be used to get a competitive advantage
    Usually bring about advances in computer science
    Vast area of topics: NoSQL systems, large-scale distributed computing systems, highly scalable web system designs
    Machine learning techniques: search engines, recommender systems