Download as PDF, PPTX
Uploaded byMargriet Groenendijk
ODSC UK 2016: How To Analyse Weather Data and Twitter Sentiment with Spark and Watson
The document discusses a presentation by Margriet Groenendijk on analyzing weather data and Twitter sentiment using IBM's Spark and Watson technologies at the Open Data Science Conference in London. It covers the sources for obtaining weather data, the use of IBM's Cloud services for storage and analysis, and provides examples of how to implement the analysis using Python scripts. Groenendijk highlights challenges and insights gained from analyzing real-time weather sentiment coupled with historical Twitter data.
More Related Content
Similar to ODSC UK 2016: How To Analyse Weather Data and Twitter Sentiment with Spark and Watson
![[DSC Europe 25] Boris Perkovic - Lost in performance.pptx](https://cdn.slidesharecdn.com/ss_thumbnails/uq5hrp7vsuahqkxzifux-1-251204082258-fd2ee09d-thumbnail.jpg?width=640&height=640&fit=bounds)
![[DSC Europe 25] Andy Cotgreave - Nothing is new in analytics.pptx](https://cdn.slidesharecdn.com/ss_thumbnails/mba4vzcurvoh5lfrd5zw-6-251205194645-341bbbbe-thumbnail.jpg?width=640&height=640&fit=bounds)
![[DSC Europe 25] Petar Zivanov - AI meets documents From chatbots to AI-powere...](https://cdn.slidesharecdn.com/ss_thumbnails/xer2bb6nrdc8pdpev0pc-8-251204082258-7c2fa4a1-thumbnail.jpg?width=640&height=640&fit=bounds)
![[DSC Europe 25] Goran Obradovic - The Rise of Sovereign AI: Building the Regi...](https://cdn.slidesharecdn.com/ss_thumbnails/7nw2xxixrxqdxvrb5wca-6-251205085714-ab09a2ac-thumbnail.jpg?width=640&height=640&fit=bounds)
![[DSC Europe 25] Jim Sterne - Adopting Generative AI Capabilities Into the Ent...](https://cdn.slidesharecdn.com/ss_thumbnails/sxhpofuorcagxsaulkmt-3-251204082258-7e66bc48-thumbnail.jpg?width=640&height=640&fit=bounds)
![[DSC Europe 25] Nikola Rajovic - Hardware Technologies Under the Hood: RISC-V...](https://cdn.slidesharecdn.com/ss_thumbnails/o2gptrmtoyqndgoshwgq-dsc2025-tenstorrent-rajovic-251205090438-814685f5-thumbnail.jpg?width=640&height=640&fit=bounds)
![[DSC Europe 25] Dragan Vucic - Building the Learning Organization - How AI Tr...](https://cdn.slidesharecdn.com/ss_thumbnails/8brigo2sbu6qur6gxrra-7-251205085715-6ae07d24-thumbnail.jpg?width=640&height=640&fit=bounds)
ODSC UK 2016: How To Analyse Weather Data and Twitter Sentiment with Spark and Watson
- 1. @MargrietGr Margriet Groenendijk, PhD DeveloperAdvocate for IBM Cloud Data Services Open Data Science Conference UK 9 October 2016, London How To Analyse Weather Data and Twitter Sentiment with Spark and Watson
- 2.
- 3.
- 4.
- 5. @MargrietGr Analyse Weather Dataand Twitter Sentiment using Spark and Watson People love to talk about the weather on Twitter What insights can you find when combining the data? What is the weather sentiment related to?
- 6. @MargrietGr Bluemix Where to findthe data? Insights for Twitter Weather Company Data API services from IBM Bluemix https://console.ng.bluemix.net/
- 7. @MargrietGr Bluemix Where to storethe data? Available from IBM Bluemix Cloudant NoSQL DB
- 8. @MargrietGr Where to analysethe data? http://datascience.ibm.com
- 9.
- 10.
- 11. @MargrietGr IBM Bluemix ▪ Freetrial etc ▪ lots of services etc Free 30-day trial “Big Blue Box” containing all IBMs services https://console.ng.bluemix.net/
- 12. @MargrietGr Add a servicein Bluemix Add a service Search for weather Account + spaces
- 13. Weather Company Datafor IBM Bluemix
- 14. @MargrietGr Weather Company Datafor IBM Bluemix 2 Credentials 3 Ready to use the REST APIs Add service1
- 15. @MargrietGr Your own weatherforecast in a Python notebook
- 16. @MargrietGr Weather Company DataAPI Show json weather file
- 17. @MargrietGr Your own weather forecastin a Python notebook London
- 18.
- 19.
- 20. @MargrietGr Python script rundaily on a Bluemix VM service https://python-cloudant.readthedocs.io
- 21. @MargrietGr Python script rundaily on a Bluemix VM service Add crontab job to run daily
- 22.
- 23.
- 24. @MargrietGr ▪ geospatial index ▪show map with 100 cities :-)
- 25. @MargrietGr ▪ geospatial index ▪show map with 100 cities :-)
- 26.
- 27.
- 28.
- 29.
- 30.
- 31. @MargrietGr Add the dashDBservice in Bluemix Add a service Search for dashDB
- 32.
- 33. @MargrietGr Use an existingservice 3 1 2 posted:2016-08-01,2016-10-01 followers_count:3000 friends_count: 3000 (weather OR sun OR sunny OR rain OR hail OR storm OR rainy OR drought OR flood OR hurricane OR tornado OR cold OR snow OR drizzle OR cloudy OR thunder OR lightning OR wind OR windy OR heatwave) REST API docs: https://new-console.ng.bluemix.net/docs/ services/Twitter/ twitter_rest_apis.html#rest_apis Search for tweets 4 Select table
- 34.
- 35.
- 36.
- 37. IBM Data ScienceExperience
- 38.
- 39.
- 40. @MargrietGr Load tweets fromdashDB with Spark SQL
- 41. @MargrietGr Clean data, summariseand load into pandas DataFrame
- 42. @MargrietGr Add weather totweets Weather data is nested, pyspark.sql struggles with that There is no location data of tweets Only 10% of all tweets available in the free plan through the Decahose stream Weather API only has 24 hours of data available
- 43.
- 44. @MargrietGr Weather Company Data crontab -e 023 * * * /path/to/file/do_something.sh python do_something.py Tweets Weather Sentiment
- 45.
- 46. @MargrietGr Add sentiment -example
- 47.
- 48.
- 49.
- 50. @MargrietGr Use an existingservice 3 1 2 posted:2016-08-26,2016-10-06 followers_count:1000 friends_count:1000 (matthew OR hurricane matthew OR hurricane) REST API docs: https://new-console.ng.bluemix.net/docs/ services/Twitter/ twitter_rest_apis.html#rest_apis #matthew tweets 4 Select table
- 51.
- 52.
- 53.
- 54. @MargrietGr Some lessons learned APIsare great! Can extend and build on this, as all data is in the Cloud Weather data only available for 24 hrs, great for weather apps, but harder to combine weather with historical tweets, need a daily script Now ready to build a more efficient workflow that will be easily able to handle millions of tweets Start a more in depth analysis in the Data Science Experience
- 55. @MargrietGr ▪ analyse data! ▪pretty plots https://github.com/ibm-cds-labs/pixiedust
- 56. @MargrietGr Margriet Groenendijk, PhD DeveloperAdvocate for IBM Cloud Data Services https://developer.ibm.com/clouddataservices/author/mgroenen/ Thank you! Slides will be available on http://www.slideshare.net/MargrietGroenendijk