Your SlideShare is downloading. ×
1st Birmingham Big Data Science Group meetup
Upcoming SlideShare
Loading in...5

Thanks for flagging this SlideShare!

Oops! An error has occurred.


Introducing the official SlideShare app

Stunning, full-screen experience for iPhone and Android

Text the download link to your phone

Standard text messaging rates apply

1st Birmingham Big Data Science Group meetup


Published on

Published in: Technology

1 Like
  • Be the first to comment

No Downloads
Total Views
On Slideshare
From Embeds
Number of Embeds
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

No notes for slide


  • 1. Welcome to the Birmingham Big Data Science Group (BIDS)
    Faizan Javed
    Intermark Group
    Sponsor: Intermark Group
  • 2. BIDS Stats
    Founded April 10, 2011
    9 members (and counting..)
    Founder: Faizan Javed, Co-Founder: QasimIjaz
    Online presence: for co-ordinatingmeetups:
    Also on (for related articles and announcements):
  • 3. Agenda
    What is Big Data?
    Quick overview of related technologies:
    Large-scale distributed systems and platforms
    NoSQL data stores
    Intelligent algorithms/web-mining/information retrieval techniques
    Highly-scalable systems
  • 4. What is Big Data?
    More people connected to the internet
    Social media explosion (Web 2.0): Facebook, Twitter, etc.
    Huge volumes of data being collected: sensors, mobile devices, machine-to-machine communications, social media and retail sites web logs for browsing patterns
    “Big” in Big Data is relative:  today's "big" is certainly tomorrow's "medium" and next week's "small.“
    “Big Data" is when the size of the data itself becomes part of the problem. Going from Gigabytes to Petabytes!
  • 5.
  • 6. Big Data, Big Numbers McKinsey report, May 2011:
  • 7. Why care about big data?
    Deep analysis of data can be a competitive advantage.
    More data  easier to find consistent patterns
    More data usually beats better algorithms
    Ex 1: Predict customer preferences and target ads on an ecommerce website.
    Ex 2: Improve search quality.
    Ex 3: Bank risk modeling (aggregate customer activity from different lines of businesses)
    Key point: “Many different sources” & “unstructured data”
  • 8. Big Players on the Big Data Scene
    The Government
  • 9. The need for new techniques
    Traditional “relational” techniques breakdown at scale.
    NoSQL databases: Cassandra, Hbase, Riak, etc
    Large-scale “commodity” scale-out distributed computing techniques: MapReduce/Hadoop, Percolator, etc
    Analytics platforms: IBM BigInsight, EMC GreenPlum
  • 10. The NoSQL revolution
  • 11. Prominent NoSQL database users
    Cassandra: Facebook, Twitter, Rackspace, Reddit,
    Riak: Mozilla,, Comcast
    Voldemort: LinkedIn
    MongoDB: Foursquare, Etsy,, Intuit
    Hbase: Stumbleupon, Twitter, Infolinks, Adobe,,
  • 12. Hadoop-based SMAQ stack
    public static class Reduce extends Reducer<Text, IntWritable, Text, IntWritable>
    public void reduce(Text key, Iterable<IntWritable> values, Context context)
    throws IOException, InterruptedException
    int sum = 0;
    for (IntWritableval : values)
    sum += val.get();
    context.write(key, new IntWritable(sum));
  • 13. Hadoop-based SMAQ stack
    Hadoop comes with HDFS – Hadoop Distributed File Sytem.
    Can be used alongside various NoSQL systems (Hbase most common)
  • 14. Hadoop-based SMAQ stack
    Pig (yahoo)
    input = LOAD 'input/sentences.txt' USING TextLoader();
    words = FOREACH input GENERATE FLATTEN(TOKENIZE($0)); grouped = GROUP words BY $0;
    counts = FOREACH grouped GENERATE group, COUNT(words); ordered = ORDER counts BY $0; STORE ordered INTO 'output/wordCount' USING PigStorage();
    Hive (facebook)
    INSERT OVERWRITE TABLE xyz_com_page_views SELECT page_views.* FROM page_views WHERE >= '2008-03-01' AND <= '2008-03-31' AND page_views.referrer_url like '';
  • 15. Next-generation systems: going beyond MapReduce/Hadoop
    Mostly Google and Yahoo innovations.
    Percolator – “real-time” MapReduce. Powers Google Instant.
    Dremel – superfast “Hive” to interact with large-datasets. Inhouse-Google.
    Pregel– highly efficient graph computing for analyzing social graphs. In-house Google. Open-source projects available.
    Megastore- scalable NoSQL like system with ACID semantics but lower consistency across partitions. In-house Google.
    Next-gen Hadoop at Yahoo: enhanced scalability (going beyond 4000 clusters), support for multiple programming paradigms, enhanced cluster utilization.
  • 16. Intelligent Web & machine learning
    Recommendation systems, data/web mining, natural language processing
    Recommendation systems:
    A type of collaborative filtering/information retrieval technique.
    Uses user profiles, ratings, browsing habits to recommend items not yet considered.
    First made famous in the commercial arena by
  • 17. & Netflix recommendation systems
  • 18. Foursquare (3/2011) and Google Places (5/2011)
  • 19. Hot area!Netflix and competitions
  • 20. Search Engines (Google, Bing, Wolfram, Lucene/Nutch, etc)
  • 21. Search innovations @ LinkedIn
    Uses open-source Luceneproject for social graph search and real-time indexing and searching.
    Dynamic filters automatically generated based on your query results!
  • 22. Conclusion
    Big Data is a very challenging and promising area
    Can be used to get a competitive advantage
    Usually bring about advances in computer science
    Vast area of topics: NoSQL systems, large-scale distributed computing systems, highly scalable web system designs
    Machine learning techniques: search engines, recommender systems