SlideShare a Scribd company logo
1 of 13
Download to read offline
DATA MINING THE TWITTER API
                         2012 ALAO Annual Conference




Friday, October 26, 12
Photograph by Max Dannenbaum/Getty Images




                         THE AGE OF BIG DATA

Friday, October 26, 12
“It’s a revolution... We’re just getting under way.
                 But the march of quantification, made possible by
                 enormous new sources of data, will sweep
                 through academia, business and government.
                 There is no area that is going to be untouched.”
                                                                                   Gary King, Director
                                         Institute for Quantitative Social Science, Harvard University




Friday, October 26, 12
Wordpress posts
           Industrial Equipment
                                                    Household Appliances

                            Weather                       Instagrams

          Electric Meters                 Tweets Water Meters

                                              Government Data
                            Traffic
                                         Tumblr posts
                         Search Traffic                  Automobiles

Friday, October 26, 12
Why do we want these Tweets?
                                         Vertical scroll
                         Discover & respond to user needs proactively
                               Create social rapport with users
                                       Build social capital
                                         Add followers
                         Gain unique insight into Zeitgeist of user base
                              Capture as an additional data point


Friday, October 26, 12
Friday, October 26, 12
Open Source: take whatever you want, use however you want




                             Proprietary: pay for each tool in the garage, use them as designed




                    API: utilize the services of the garage, if you have authorization and ask the right way




                                       DATA ACCESS
                                     The Neighbor’s Garage Analogy

Friday, October 26, 12
High-Level Overview
       Build URL request with keywords and other parameters
                                     Submit URL request
                                Convert returned JSON data
                                       <?php ?>
                         Insert non-duplicative data into MySQL table
                                    <?php mysql ?>
                          Display data by pulling out of MySQL table
                             <?php mysql html css ?>
Friday, October 26, 12
<?php

 echo '<head>';

 echo '<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />';

 echo '<link rel="stylesheet" href="style.css" type="text/css" />';

 echo '</head>';

 echo '<body>';

 mysql_connect("localhost", "user", "pass") or die(mysql_error());

 mysql_select_db("database") or die(mysql_error());

 $tweet_archive = mysql_query("SELECT * FROM twindicators where loc NOT LIKE '%Beach%' ORDER BY id DESC");

 echo '<div class="content">';

 while ($row = mysql_fetch_array($tweet_archive)) {

      $decoded_text = urldecode($row["text"]);

      echo '<p><span class="user">' . $row[user] . '</span> | <span class="date">' . $row[date] . '</span>
      <br /><br />
      <span class="text"><a href="https://twitter.com/#!/' . $row[user] . '/status/' . $row[twid] . '">' . $decoded_text .
      echo '<hr />';

 }

 echo '</div>';

 echo '</body>';

 ?>


                    DISPLAY DATA BY PULLING OUT OF MYSQL TABLE



Friday, October 26, 12
70 followers


                  NOTABLE INTERACTIONS
Friday, October 26, 12
Future Developments

                           Develop a deeper set of keywords
                         Develop a richer interface for librarians
     Incorporate SMS notifications to subject specific librarians
                 when relevent tweet is recorded
     Analyze data for user satisfaction / areas for improvement




Friday, October 26, 12
SHAMELESS PLUG
                         Web Service APIs and Libraries. ALA Editions

Friday, October 26, 12
Jason Paul Michel
                              micheljp@muohio.edu
                              @jpmichel



     Code:
     Query and store: https://gist.github.com/3846007
     Display: https://gist.github.com/3846055




                         DETAILS & SUCH
Friday, October 26, 12

More Related Content

Similar to Data Mining the Twitter API for Academic Libraries

Crafting Solutions with the Elastic Stack: pragmatic takes and lessons learned
Crafting Solutions with the Elastic Stack: pragmatic takes and lessons learnedCrafting Solutions with the Elastic Stack: pragmatic takes and lessons learned
Crafting Solutions with the Elastic Stack: pragmatic takes and lessons learnedGuido Lena Cota
 
Internet of Things (IoT) HackDay
Internet of Things (IoT) HackDayInternet of Things (IoT) HackDay
Internet of Things (IoT) HackDayAmazon Web Services
 
IoT Hack Day: AWS Pop-up Loft Hack Series Sponsored by Intel
IoT Hack Day: AWS Pop-up Loft Hack Series Sponsored by IntelIoT Hack Day: AWS Pop-up Loft Hack Series Sponsored by Intel
IoT Hack Day: AWS Pop-up Loft Hack Series Sponsored by IntelAmazon Web Services
 
Datascript: Serverless Architetecture
Datascript: Serverless ArchitetectureDatascript: Serverless Architetecture
Datascript: Serverless ArchitetectureLadislav Prskavec
 
Big Data Analytics from Azure Cloud to Power BI Mobile
Big Data Analytics from Azure Cloud to Power BI MobileBig Data Analytics from Azure Cloud to Power BI Mobile
Big Data Analytics from Azure Cloud to Power BI MobileRoy Kim
 
Introduction to the Hadoop Ecosystem (FrOSCon Edition)
Introduction to the Hadoop Ecosystem (FrOSCon Edition)Introduction to the Hadoop Ecosystem (FrOSCon Edition)
Introduction to the Hadoop Ecosystem (FrOSCon Edition)Uwe Printz
 
Dev talks 2021 Data Science @crowdstrike
Dev talks 2021   Data Science @crowdstrikeDev talks 2021   Data Science @crowdstrike
Dev talks 2021 Data Science @crowdstrikeRuxandra Burtica
 
Sharpen your "Architectural Documentation" Saw
Sharpen your "Architectural Documentation" SawSharpen your "Architectural Documentation" Saw
Sharpen your "Architectural Documentation" SawKevin Hakanson
 
Scaling and Modernizing Data Platform with Databricks
Scaling and Modernizing Data Platform with DatabricksScaling and Modernizing Data Platform with Databricks
Scaling and Modernizing Data Platform with DatabricksDatabricks
 
Stargate, the gateway for some multi-models data API
Stargate, the gateway for some multi-models data APIStargate, the gateway for some multi-models data API
Stargate, the gateway for some multi-models data APIData Con LA
 
Semantic Web Standards and the Variety “V” of Big Data
Semantic Web Standards and  the Variety “V” of Big DataSemantic Web Standards and  the Variety “V” of Big Data
Semantic Web Standards and the Variety “V” of Big Databobdc
 
Easy Analytics on AWS with Amazon Redshift, Amazon QuickSight, and Amazon Mac...
Easy Analytics on AWS with Amazon Redshift, Amazon QuickSight, and Amazon Mac...Easy Analytics on AWS with Amazon Redshift, Amazon QuickSight, and Amazon Mac...
Easy Analytics on AWS with Amazon Redshift, Amazon QuickSight, and Amazon Mac...Amazon Web Services
 
SQL Analytics Powering Telemetry Analysis at Comcast
SQL Analytics Powering Telemetry Analysis at ComcastSQL Analytics Powering Telemetry Analysis at Comcast
SQL Analytics Powering Telemetry Analysis at ComcastDatabricks
 
Self-Service IoT Data Analytics with StreamPipes
Self-Service IoT Data Analytics with StreamPipesSelf-Service IoT Data Analytics with StreamPipes
Self-Service IoT Data Analytics with StreamPipesApache StreamPipes
 
Saving Money by Optimizing Your Cloud Add-On Infrastructure
Saving Money by Optimizing Your Cloud Add-On InfrastructureSaving Money by Optimizing Your Cloud Add-On Infrastructure
Saving Money by Optimizing Your Cloud Add-On InfrastructureAtlassian
 
No SQL, No Problem: Use Azure DocumentDB
No SQL, No Problem: Use Azure DocumentDBNo SQL, No Problem: Use Azure DocumentDB
No SQL, No Problem: Use Azure DocumentDBKen Cenerelli
 
NASA Goddard: Head in the Clouds
NASA Goddard: Head in the CloudsNASA Goddard: Head in the Clouds
NASA Goddard: Head in the CloudsAmazon Web Services
 
Time's Up! Getting Value from Big Data Now
Time's Up! Getting Value from Big Data NowTime's Up! Getting Value from Big Data Now
Time's Up! Getting Value from Big Data NowEric Kavanagh
 
AWS를 통한 데이터 분석 및 처리의 새로운 혁신 기법 - 김윤건, AWS사업개발 담당:: AWS Summit Online Korea 2020
AWS를 통한 데이터 분석 및 처리의 새로운 혁신 기법 - 김윤건, AWS사업개발 담당::  AWS Summit Online Korea 2020AWS를 통한 데이터 분석 및 처리의 새로운 혁신 기법 - 김윤건, AWS사업개발 담당::  AWS Summit Online Korea 2020
AWS를 통한 데이터 분석 및 처리의 새로운 혁신 기법 - 김윤건, AWS사업개발 담당:: AWS Summit Online Korea 2020Amazon Web Services Korea
 

Similar to Data Mining the Twitter API for Academic Libraries (20)

Crafting Solutions with the Elastic Stack: pragmatic takes and lessons learned
Crafting Solutions with the Elastic Stack: pragmatic takes and lessons learnedCrafting Solutions with the Elastic Stack: pragmatic takes and lessons learned
Crafting Solutions with the Elastic Stack: pragmatic takes and lessons learned
 
Internet of Things (IoT) HackDay
Internet of Things (IoT) HackDayInternet of Things (IoT) HackDay
Internet of Things (IoT) HackDay
 
IoT Hack Day: AWS Pop-up Loft Hack Series Sponsored by Intel
IoT Hack Day: AWS Pop-up Loft Hack Series Sponsored by IntelIoT Hack Day: AWS Pop-up Loft Hack Series Sponsored by Intel
IoT Hack Day: AWS Pop-up Loft Hack Series Sponsored by Intel
 
Datascript: Serverless Architetecture
Datascript: Serverless ArchitetectureDatascript: Serverless Architetecture
Datascript: Serverless Architetecture
 
Big Data Analytics from Azure Cloud to Power BI Mobile
Big Data Analytics from Azure Cloud to Power BI MobileBig Data Analytics from Azure Cloud to Power BI Mobile
Big Data Analytics from Azure Cloud to Power BI Mobile
 
Introduction to the Hadoop Ecosystem (FrOSCon Edition)
Introduction to the Hadoop Ecosystem (FrOSCon Edition)Introduction to the Hadoop Ecosystem (FrOSCon Edition)
Introduction to the Hadoop Ecosystem (FrOSCon Edition)
 
Dev talks 2021 Data Science @crowdstrike
Dev talks 2021   Data Science @crowdstrikeDev talks 2021   Data Science @crowdstrike
Dev talks 2021 Data Science @crowdstrike
 
Sharpen your "Architectural Documentation" Saw
Sharpen your "Architectural Documentation" SawSharpen your "Architectural Documentation" Saw
Sharpen your "Architectural Documentation" Saw
 
Scaling and Modernizing Data Platform with Databricks
Scaling and Modernizing Data Platform with DatabricksScaling and Modernizing Data Platform with Databricks
Scaling and Modernizing Data Platform with Databricks
 
Stargate, the gateway for some multi-models data API
Stargate, the gateway for some multi-models data APIStargate, the gateway for some multi-models data API
Stargate, the gateway for some multi-models data API
 
Semantic Web Standards and the Variety “V” of Big Data
Semantic Web Standards and  the Variety “V” of Big DataSemantic Web Standards and  the Variety “V” of Big Data
Semantic Web Standards and the Variety “V” of Big Data
 
Easy Analytics on AWS with Amazon Redshift, Amazon QuickSight, and Amazon Mac...
Easy Analytics on AWS with Amazon Redshift, Amazon QuickSight, and Amazon Mac...Easy Analytics on AWS with Amazon Redshift, Amazon QuickSight, and Amazon Mac...
Easy Analytics on AWS with Amazon Redshift, Amazon QuickSight, and Amazon Mac...
 
SQL Analytics Powering Telemetry Analysis at Comcast
SQL Analytics Powering Telemetry Analysis at ComcastSQL Analytics Powering Telemetry Analysis at Comcast
SQL Analytics Powering Telemetry Analysis at Comcast
 
Self-Service IoT Data Analytics with StreamPipes
Self-Service IoT Data Analytics with StreamPipesSelf-Service IoT Data Analytics with StreamPipes
Self-Service IoT Data Analytics with StreamPipes
 
Saving Money by Optimizing Your Cloud Add-On Infrastructure
Saving Money by Optimizing Your Cloud Add-On InfrastructureSaving Money by Optimizing Your Cloud Add-On Infrastructure
Saving Money by Optimizing Your Cloud Add-On Infrastructure
 
Data Science on Google Cloud Platform
Data Science on Google Cloud PlatformData Science on Google Cloud Platform
Data Science on Google Cloud Platform
 
No SQL, No Problem: Use Azure DocumentDB
No SQL, No Problem: Use Azure DocumentDBNo SQL, No Problem: Use Azure DocumentDB
No SQL, No Problem: Use Azure DocumentDB
 
NASA Goddard: Head in the Clouds
NASA Goddard: Head in the CloudsNASA Goddard: Head in the Clouds
NASA Goddard: Head in the Clouds
 
Time's Up! Getting Value from Big Data Now
Time's Up! Getting Value from Big Data NowTime's Up! Getting Value from Big Data Now
Time's Up! Getting Value from Big Data Now
 
AWS를 통한 데이터 분석 및 처리의 새로운 혁신 기법 - 김윤건, AWS사업개발 담당:: AWS Summit Online Korea 2020
AWS를 통한 데이터 분석 및 처리의 새로운 혁신 기법 - 김윤건, AWS사업개발 담당::  AWS Summit Online Korea 2020AWS를 통한 데이터 분석 및 처리의 새로운 혁신 기법 - 김윤건, AWS사업개발 담당::  AWS Summit Online Korea 2020
AWS를 통한 데이터 분석 및 처리의 새로운 혁신 기법 - 김윤건, AWS사업개발 담당:: AWS Summit Online Korea 2020
 

Data Mining the Twitter API for Academic Libraries

  • 1. DATA MINING THE TWITTER API 2012 ALAO Annual Conference Friday, October 26, 12
  • 2. Photograph by Max Dannenbaum/Getty Images THE AGE OF BIG DATA Friday, October 26, 12
  • 3. “It’s a revolution... We’re just getting under way. But the march of quantification, made possible by enormous new sources of data, will sweep through academia, business and government. There is no area that is going to be untouched.” Gary King, Director Institute for Quantitative Social Science, Harvard University Friday, October 26, 12
  • 4. Wordpress posts Industrial Equipment Household Appliances Weather Instagrams Electric Meters Tweets Water Meters Government Data Traffic Tumblr posts Search Traffic Automobiles Friday, October 26, 12
  • 5. Why do we want these Tweets? Vertical scroll Discover & respond to user needs proactively Create social rapport with users Build social capital Add followers Gain unique insight into Zeitgeist of user base Capture as an additional data point Friday, October 26, 12
  • 7. Open Source: take whatever you want, use however you want Proprietary: pay for each tool in the garage, use them as designed API: utilize the services of the garage, if you have authorization and ask the right way DATA ACCESS The Neighbor’s Garage Analogy Friday, October 26, 12
  • 8. High-Level Overview Build URL request with keywords and other parameters Submit URL request Convert returned JSON data <?php ?> Insert non-duplicative data into MySQL table <?php mysql ?> Display data by pulling out of MySQL table <?php mysql html css ?> Friday, October 26, 12
  • 9. <?php echo '<head>'; echo '<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />'; echo '<link rel="stylesheet" href="style.css" type="text/css" />'; echo '</head>'; echo '<body>'; mysql_connect("localhost", "user", "pass") or die(mysql_error()); mysql_select_db("database") or die(mysql_error()); $tweet_archive = mysql_query("SELECT * FROM twindicators where loc NOT LIKE '%Beach%' ORDER BY id DESC"); echo '<div class="content">'; while ($row = mysql_fetch_array($tweet_archive)) { $decoded_text = urldecode($row["text"]); echo '<p><span class="user">' . $row[user] . '</span> | <span class="date">' . $row[date] . '</span> <br /><br /> <span class="text"><a href="https://twitter.com/#!/' . $row[user] . '/status/' . $row[twid] . '">' . $decoded_text . echo '<hr />'; } echo '</div>'; echo '</body>'; ?> DISPLAY DATA BY PULLING OUT OF MYSQL TABLE Friday, October 26, 12
  • 10. 70 followers NOTABLE INTERACTIONS Friday, October 26, 12
  • 11. Future Developments Develop a deeper set of keywords Develop a richer interface for librarians Incorporate SMS notifications to subject specific librarians when relevent tweet is recorded Analyze data for user satisfaction / areas for improvement Friday, October 26, 12
  • 12. SHAMELESS PLUG Web Service APIs and Libraries. ALA Editions Friday, October 26, 12
  • 13. Jason Paul Michel micheljp@muohio.edu @jpmichel Code: Query and store: https://gist.github.com/3846007 Display: https://gist.github.com/3846055 DETAILS & SUCH Friday, October 26, 12