Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Using Clojure for Sentiment Analysis of the Twittersphere

2,373 views

Published on

Sentiment analysis of online social networks, such as Twitter, is a rapidly growing research area with an increasing number of applications such as: informing disaster response; tracking spread of contagious disease; early forecast of civil unrest. This talk will present an approach for building scalable social media analysis platforms in Clojure

Published in: Technology
  • Be the first to comment

Using Clojure for Sentiment Analysis of the Twittersphere

  1. 1. Sentiment Analysis of the Twittersphere @APassionForCode
  2. 2. Leiningen Versus the Ants! Clojure Versus Java! FP Versus OOP
  3. 3. Apache Ant Leiningen Versus the Ants! Clojure Versus Java! FP Versus OOP
  4. 4. Apache Ant Leiningen Versus the Ants Clojure Versus Java! FP Versus OOP
  5. 5. Apache Ant Leiningen Versus the Ants Clojure Versus Java Functional Versus Object Oriented
  6. 6. “You wanted a banana but what you got was a gorilla holding the banana and the entire jungle.” Joe Armstrong - inventor of Erlang
  7. 7. “Sometimes, the elegant implementation is just a function. Not a method. Not a class. Not a framework. Just a function.” John Carmack - Lead on Doom, Quake, Rage
  8. 8. Blood aches pain sick shivers spasm vomit dizzy fainting colic and in a coma! 1 FAVORITE
  9. 9. Blood aches pain sick shivers spasm vomit dizzy fainting colic and in a coma! 1 FAVORITE Scores HIGH for flu symptoms
  10. 10. Blood aches pain sick shivers spasm vomit dizzy fainting colic and in a coma! 1 FAVORITE Who is Catalina Rubottom?
  11. 11. Blood aches pain sick shivers spasm vomit dizzy fainting colic and in a coma! 1 FAVORITE 30 million geo-tagged tweets sent from UK HDFS/Hadoop Mongo/Aggregation Mongo/MapReduce Postgres
  12. 12. How can we do fast, real time analytics of social media?
  13. 13. Store t his! Not this!
  14. 14. WARNING! You’re about to see real Clojure code!
  15. 15. increment (1)
  16. 16. (increment 1)
  17. 17. increment (1)
  18. 18. (increment 1) Co ng ratu lat io ns - yo u’re no w fu lly qualifie d!
  19. 19. ptaoussanis/carmine
  20. 20. (:require [taoensso.carmine :as car]) ! ! (defn create-tweet-id [] (first (wcar* (car/incr "global:tweet.id")))) ! ! (defn set-values [sentiment location created-at] (let [tweed-id (create-tweet-id)] (wcar* (car/setbit sentiment tweet-id 1) (car/setbit location tweet-id 1) (car/setbit created-at tweet-id 1))))
  21. 21. (:require [taoensso.carmine :as car]) ! ! (defn create-tweet-id [] (first (wcar* (car/incr "global:tweet.id")))) ! ! (defn set-values [sentiment location created-at] (let [tweed-id (create-tweet-id)] (wcar* (car/setbit sentiment tweet-id 1) (car/setbit location tweet-id 1) (car/setbit created-at tweet-id 1))))
  22. 22. (:require [taoensso.carmine :as car]) ! ! (defn create-tweet-id [] (first (wcar* (car/incr "global:tweet.id")))) ! ! (defn set-values [sentiment location created-at] (let [tweed-id (create-tweet-id)] (wcar* (car/setbit sentiment tweet-id 1) (car/setbit location tweet-id 1) (car/setbit created-at tweet-id 1))))
  23. 23. (:require [taoensso.carmine :as car]) ! ! (defn create-tweet-id [] (first (wcar* (car/incr "global:tweet.id")))) ! ! (defn set-values [sentiment location created-at] (let [tweed-id (create-tweet-id)] (wcar* (car/setbit sentiment tweet-id 1) (car/setbit location tweet-id 1) (car/setbit created-at tweet-id 1))))
  24. 24. (:require [taoensso.carmine :as car]) ! ! (defn create-tweet-id [] (first (wcar* (car/incr "global:tweet.id")))) ! ! (defn set-values [sentiment location created-at] (let [tweed-id (create-tweet-id)] (wcar* (car/setbit sentiment tweet-id 1) (car/setbit location tweet-id 1) (car/setbit created-at tweet-id 1)))) find value for key set bit to 1 find bit of index
  25. 25. (wcar* (car/bitcount "SCOTLAND") (car/bitcount "JOVIALITY") (car/bitcount "26062014114532"))
  26. 26. ENG SCT WAL NIR key value
  27. 27. key tweet-id (car/setbit “ENG” 0 1) ENG 1 SCT 0 WAL 0 NIR 0 key value
  28. 28. key tweet-id (car/setbit “SCT” 1 1) ENG 0 1 SCT 1 0 WAL 0 0 NIR 0 0 key value
  29. 29. key tweet-id (car/setbit “ENG” 2 1) ENG 1 0 1 SCT 0 1 0 WAL 0 0 0 NIR 0 0 0 key value
  30. 30. key tweet-id (car/setbit “SCT” 15 1) ENG 0 0 1 0 1 1 0 1 0 0 1 0 1 1 0 1 SCT 1 0 0 0 0 0 1 0 1 0 0 0 0 0 1 0 WAL 0 1 0 1 0 0 0 0 0 1 0 0 0 0 0 0 NIR 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 key value
  31. 31. (car/bitcount "ENG") ENG 0 0 1 0 1 1 0 1 0 0 1 0 1 1 0 1 SCT 1 0 0 0 0 0 1 0 1 0 0 0 0 0 1 0 WAL 0 1 0 1 0 0 0 0 0 1 0 0 0 0 0 0 NIR 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 8 key value
  32. 32. (car/bitcount "SCT") ENG 0 0 1 0 1 1 0 1 0 0 1 0 1 1 0 1 SCT 1 0 0 0 0 0 1 0 1 0 0 0 0 0 1 0 WAL 0 1 0 1 0 0 0 0 0 1 0 0 0 0 0 0 NIR 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 8 4 key value
  33. 33. (car/bitcount "WAL") ENG 0 0 1 0 1 1 0 1 0 0 1 0 1 1 0 1 SCT 1 0 0 0 0 0 1 0 1 0 0 0 0 0 1 0 WAL 0 1 0 1 0 0 0 0 0 1 0 0 0 0 0 0 NIR 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 8 4 3 key value
  34. 34. (car/bitcount "NIR") ENG 0 0 1 0 1 1 0 1 0 0 1 0 1 1 0 1 SCT 1 0 0 0 0 0 1 0 1 0 0 0 0 0 1 0 WAL 0 1 0 1 0 0 0 0 0 1 0 0 0 0 0 0 NIR 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 8 4 3 1 key value
  35. 35. JOVIAL SHY HOSTILE FATIGUE key value
  36. 36. key tweet-id (car/setbit “JOVIAL” 0 1) JOVIAL 1 SHY 0 HOSTILE 0 FATIGUE 0 key value
  37. 37. key tweet-id (car/setbit “HOSTILE” 1 1) JOVIAL 0 1 SHY 0 0 HOSTILE 1 0 FATIGUE 0 0 key value
  38. 38. key tweet-id (car/setbit “SHY” 2 1) JOVIAL 0 0 1 SHY 1 0 0 HOSTILE 0 1 0 FATIGUE 0 0 0 key value
  39. 39. key tweet-id (car/setbit “JOVIAL” 15 1) JOVIAL 1 0 1 0 1 1 1 0 0 1 0 1 0 0 0 1 SHY 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 HOSTILE 0 0 0 0 0 0 0 1 0 0 1 0 0 0 1 0 FATIGUE 0 1 0 1 0 0 0 0 1 0 0 0 1 0 0 0 key value
  40. 40. (car/bitcount [*]) JOVIAL 1 0 1 0 1 1 1 0 0 1 0 1 0 0 0 1 SHY 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 HOSTILE 0 0 0 0 0 0 0 1 0 0 1 0 0 0 1 0 FATIGUE 0 1 0 1 0 0 0 0 1 0 0 0 1 0 0 0 8 1 3 4 key value
  41. 41. How many people in England are Happy about the referendum result? (wcar* (car/bitop "AND" “ENGLAND&JOVIALITY" "ENGLAND" "JOVIALITY") (car/expire “ENGLAND&JOVIALITY" 10) (car/bitcount “ENGLAND&JOVIALITY"))
  42. 42. How many people in England are Happy about the referendum result? (wcar* (car/bitop "AND" “ENGLAND&JOVIALITY" "ENGLAND" "JOVIALITY") (car/expire “ENGLAND&JOVIALITY" 10) (car/bitcount “ENGLAND&JOVIALITY"))
  43. 43. How many people in England are Happy about the referendum result? (wcar* (car/bitop "AND" “ENGLAND&JOVIALITY" "ENGLAND" "JOVIALITY") (car/expire “ENGLAND&JOVIALITY" 10) (car/bitcount “ENGLAND&JOVIALITY"))
  44. 44. How many people in England are Happy about the referendum result? (wcar* (car/bitop "AND" “ENGLAND&JOVIALITY" "ENGLAND" "JOVIALITY") (car/expire “ENGLAND&JOVIALITY" 10) (car/bitcount “ENGLAND&JOVIALITY"))
  45. 45. How many people in England are Happy about the referendum result?
  46. 46. How many people in England are Happy about the referendum result? ENG 0 0 1 0 1 1 0 1 0 0 1 0 1 1 0 1
  47. 47. How many people in England are Happy about the referendum result? ENG 0 0 1 0 1 1 0 1 0 0 1 0 1 1 0 1 JOVIAL 1 0 1 0 1 1 1 0 0 1 0 1 0 0 0 1
  48. 48. How many people in England are Happy about the referendum result? ENG 0 0 1 0 1 1 0 1 0 0 1 0 1 1 0 1 JOVIAL 1 0 1 0 1 1 1 0 0 1 0 1 0 0 0 1 1 AND (car/bitop "AND" “ENGLAND&JOVIALITY" "ENGLAND" "JOVIALITY")
  49. 49. How many people in England are Happy about the referendum result? ENG 0 0 1 0 1 1 0 1 0 0 1 0 1 1 0 1 JOVIAL 1 0 1 0 1 1 1 0 0 1 0 1 0 0 0 1 0 1 AND (car/bitop "AND" “ENGLAND&JOVIALITY" "ENGLAND" "JOVIALITY")
  50. 50. How many people in England are Happy about the referendum result? ENG 0 0 1 0 1 1 0 1 0 0 1 0 1 1 0 1 JOVIAL 1 0 1 0 1 1 1 0 0 1 0 1 0 0 0 1 0 0 1 AND (car/bitop "AND" “ENGLAND&JOVIALITY" "ENGLAND" "JOVIALITY")
  51. 51. How many people in England are Happy about the referendum result? ENG 0 0 1 0 1 1 0 1 0 0 1 0 1 1 0 1 JOVIAL 1 0 1 0 1 1 1 0 0 1 0 1 0 0 0 1 0 0 1 0 1 1 0 0 0 0 0 0 0 0 0 1 AND (car/bitop "AND" “ENGLAND&JOVIALITY" "ENGLAND" "JOVIALITY")
  52. 52. ENG 0 0 1 0 1 1 0 1 0 0 1 0 1 1 0 1 JOVIAL 1 0 1 0 1 1 1 0 0 1 0 1 0 0 0 1 0 0 1 0 1 1 0 0 0 0 0 0 0 0 0 1 (car/bitcount “ENGLAND&JOVIALITY") 4 How many people in England are Happy about the referendum result?
  53. 53. How many people in Scotland are Tired and Grumpy after the referendum? (wcar* (car/bitop "OR" "FATIGUE|HOSTILITY" "FATIGUE" “HOSTILITY") (car/expire "FATIGUE|HOSTILITY" 10) (car/bitop "AND" "SCOTLAND&(FATIGUE|HOSTILITY)" "SCOTLAND" "FATIGUE|HOSTILITY") (car/expire "SCOTLAND&(FATIGUE|HOSTILITY)" 10) (car/bitcount "SCOTLAND&(FATIGUE|HOSTILITY)"))
  54. 54. How many people in Scotland are Tired and Grumpy after the referendum? (wcar* (car/bitop "OR" "FATIGUE|HOSTILITY" "FATIGUE" “HOSTILITY") (car/expire "FATIGUE|HOSTILITY" 10) (car/bitop "AND" "SCOTLAND&(FATIGUE|HOSTILITY)" "SCOTLAND" "FATIGUE|HOSTILITY") (car/expire "SCOTLAND&(FATIGUE|HOSTILITY)" 10) (car/bitcount "SCOTLAND&(FATIGUE|HOSTILITY)"))
  55. 55. How many people in Scotland are Tired and Grumpy after the referendum? (wcar* (car/bitop "OR" "FATIGUE|HOSTILITY" "FATIGUE" “HOSTILITY") (car/expire "FATIGUE|HOSTILITY" 10) (car/bitop "AND" "SCOTLAND&(FATIGUE|HOSTILITY)" "SCOTLAND" "FATIGUE|HOSTILITY") (car/expire "SCOTLAND&(FATIGUE|HOSTILITY)" 10) (car/bitcount "SCOTLAND&(FATIGUE|HOSTILITY)"))
  56. 56. How many people in Scotland are Tired and Grumpy after the referendum? (wcar* (car/bitop "OR" "FATIGUE|HOSTILITY" "FATIGUE" “HOSTILITY") (car/expire "FATIGUE|HOSTILITY" 10) (car/bitop "AND" "SCOTLAND&(FATIGUE|HOSTILITY)" "SCOTLAND" "FATIGUE|HOSTILITY") (car/expire "SCOTLAND&(FATIGUE|HOSTILITY)" 10) (car/bitcount "SCOTLAND&(FATIGUE|HOSTILITY)"))
  57. 57. How many people in Scotland are Tired and Grumpy after the referendum?
  58. 58. How many people in Scotland are Tired and Grumpy after the referendum? HOSTILE 0 0 0 0 0 0 0 1 0 0 1 0 0 0 1 0
  59. 59. How many people in Scotland are Tired and Grumpy after the referendum? HOSTILE 0 0 0 0 0 0 0 1 0 0 1 0 0 0 1 0 FATIGUE 0 1 0 1 0 0 0 0 1 0 0 0 1 0 0 0
  60. 60. HOSTILE 0 0 0 0 0 0 0 1 0 0 1 0 0 0 1 0 FATIGUE 0 1 0 1 0 0 0 0 1 0 0 0 1 0 0 0 BOTH OR How many people in Scotland are Tired and Grumpy after the referendum?
  61. 61. HOSTILE 0 0 0 0 0 0 0 1 0 0 1 0 0 0 1 0 FATIGUE 0 1 0 1 0 0 0 0 1 0 0 0 1 0 0 0 BOTH 0 1 0 1 0 0 0 1 1 0 1 0 1 0 1 0 OR How many people in Scotland are Tired and Grumpy after the referendum?
  62. 62. HOSTILE 0 0 0 0 0 0 0 1 0 0 1 0 0 0 1 0 FATIGUE 0 1 0 1 0 0 0 0 1 0 0 0 1 0 0 0 BOTH 0 1 0 1 0 0 0 1 1 0 1 0 1 0 1 0 OR How many people in Scotland are Tired and Grumpy after the referendum?
  63. 63. How many people in Scotland are Tired and Grumpy after the referendum? HOSTILE 0 0 0 0 0 0 0 1 0 0 1 0 0 0 1 0 FATIGUE 0 1 0 1 0 0 0 0 1 0 0 0 1 0 0 0 BOTH 0 1 0 1 0 0 0 1 1 0 1 0 1 0 1 0 SCT 1 0 0 0 0 0 1 0 1 0 0 0 0 0 1 0
  64. 64. HOSTILE 0 0 0 0 0 0 0 1 0 0 1 0 0 0 1 0 FATIGUE 0 1 0 1 0 0 0 0 1 0 0 0 1 0 0 0 BOTH 0 1 0 1 0 0 0 1 1 0 1 0 1 0 1 0 SCT 1 0 0 0 0 0 1 0 1 0 0 0 0 0 1 0 BOTH 0 1 0 1 0 0 0 1 1 0 1 0 1 0 1 0 AND How many people in Scotland are Tired and Grumpy after the referendum?
  65. 65. HOSTILE 0 0 0 0 0 0 0 1 0 0 1 0 0 0 1 0 FATIGUE 0 1 0 1 0 0 0 0 1 0 0 0 1 0 0 0 BOTH 0 1 0 1 0 0 0 1 1 0 1 0 1 0 1 0 SCT 1 0 0 0 0 0 1 0 1 0 0 0 0 0 1 0 BOTH 0 1 0 1 0 0 0 1 1 0 1 0 1 0 1 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 1 0 2 How many people in Scotland are Tired and Grumpy after the referendum?
  66. 66. RING Journaling Syncing Business logic What Where When Memory Web API Incoming Data
  67. 67. RING Journaling Syncing Business logic What Where When Memory Web API Incoming Data adamwynne/twitter-api
  68. 68. RING Journaling Syncing Business logic What Where When Memory Web API Incoming Data clojurewerkz/meltdown LMAX-Exchange/disruptor
  69. 69. RING Journaling Syncing Business logic What Where When Memory Web API Incoming Data dakrone/clojure-opennlp
  70. 70. “I’m loving #XConfMCR! ”
  71. 71. Positive Affect: enthusiastic, active, alert ! Negative Affect: subjective distress ! Emerged as distinctive, orthogonal dimensions
  72. 72. PANAS 2 3 153 2 4 5 1 1 3 3 3 5 345 2 2 1
  73. 73. PANAS 2 3 153 2 4 5 1 1 3 3 3 5 345 2 2 1 1 3 5 9 10 12 14 16 17 19 2 4 6 7 8 11 13 15 18 20
  74. 74. PANAS positive affect! negative affect 3 2 4 5 1 1 5 3 3 1 3 2 5 3 4 3 5 2 2 1 1 3 5 9 10 12 14 16 17 19 2 4 6 7 8 11 13 15 18 20 39 19
  75. 75. PANAS-x General Dimension Scales! Negative Affect (10) Positive Affect (10) Basic Positive Emotions Scales! Joviality (8) Self-assurance (6) Attentiveness (4) Basic Negative Emotions Scales! Fear (6) Hostility (6) Guilt (6) Sadness (5) Other Affective States! Shyness (4) Fatigue (4) Serenity (3) Surprise (3)
  76. 76. PANAS-t Accounts for bias on social media! Outlines sanitisation Validate against 10 real events
  77. 77. dakrone/clojure-opennlp Sanitisation • Exclude media/advertising/spam • Account for text speak • Account for smileys & emoj’s • Word stemming (or lemmatisation) • Part of Speech Tagging (POS) LMAO GetRichQuick.com
  78. 78. SHYNESS FATIGUE SERENITY SURPRISE FEAR HOSTILITY GUILT SADNESS JOVIALITY SELF ASSURANCE ATTENTIVENESS We have sentiment keys!
  79. 79. RING Journaling Syncing Business logic What Where When Memory Web API Incoming Data mbostock/d3
  80. 80. [-75.14310264, 40.05701649]
  81. 81. Reverse Geocoding Issues • Don’t want external services • Don’t want heavy IO • Don’t want round trips to the database • Accuracy not too much of a concern
  82. 82. Equirectangular Projection
  83. 83. Courtesy of Eric Gaba (Wikimedia Commons User: Sting)
  84. 84. Tissot's indicatrix of deformation x = λcosφ1 y = φ Courtesy of Eric Gaba (Wikimedia Commons User: Sting)
  85. 85. x (px) y (px)
  86. 86. x (px) y (px) [lon, lat] = [55.86140, -4.26477] [x, y] = [329, 601] [x, y] = = Scotland!
  87. 87. x (px) y (px) [lon, lat] = [55.86140, -4.26477] [x, y] = [329, 601] [x, y] = = Scotland!
  88. 88. x (px) y (px) [lon, lat] = [55.86140, -4.26477] [x, y] = [329, 601] [x, y] = = Scotland!
  89. 89. x (px) y (px) [lon, lat] = [55.86140, -4.26477] [x, y] = [329, 601] [x, y] = = Scotland!
  90. 90. x (px) y (px) [lon, lat] = [55.86140, -4.26477] [x, y] = [329, 601] [x, y] = = Scotland!
  91. 91. x (px) y (px) [lon, lat] = [55.86140, -4.26477] [x, y] = [329, 601] [x, y] = = Scotland!
  92. 92. Ser ver si de! (JavaFX ) Mike Bostock & Nick Rabinowitz
  93. 93. (:require [hiccup.page :as hiccup]) (:import (javafx.scene.web WebView)) ! (def scripts ["uk-data.js" "topojson.v1.min.js" "d3.v3.min.js" "geocoder.js"]) ! (defn create-browser [html] (-> WebView. .getEngine (.loadContent html))) ! (defn build-page [script-paths] (hiccup/html5 [:head (for [path script-paths] Java F X [:script (slurp (io/resource path))])])) ! (defn coordinates->name [longitude latitude] (let [browser (-> scripts build-page create-browser)] (.executeScript browser (str "geo.decode([" longitude "," latitude "])"))))
  94. 94. (:require [hiccup.page :as hiccup]) (:import (javafx.scene.web WebView)) ! (def scripts ["uk-data.js" "topojson.v1.min.js" "d3.v3.min.js" "geocoder.js"]) ! (defn create-browser [html] (-> WebView. .getEngine (.loadContent html))) ! (defn build-page [script-paths] (hiccup/html5 [:head (for [path script-paths] Java F X [:script (slurp (io/resource path))])])) ! (defn coordinates->name [longitude latitude] (let [browser (-> scripts build-page create-browser)] (.executeScript browser (str "geo.decode([" longitude "," latitude "])"))))
  95. 95. (:require [hiccup.page :as hiccup]) (:import (javafx.scene.web WebView)) ! (def scripts ["uk-data.js" "topojson.v1.min.js" "d3.v3.min.js" "geocoder.js"]) ! (defn create-browser [html] (-> WebView. .getEngine (.loadContent html))) ! (defn build-page [script-paths] (hiccup/html5 [:head (for [path script-paths] Java F X [:script (slurp (io/resource path))])])) ! (defn coordinates->name [longitude latitude] (let [browser (-> scripts build-page create-browser)] (.executeScript browser (str "geo.decode([" longitude "," latitude "])"))))
  96. 96. (:require [hiccup.page :as hiccup]) (:import (javafx.scene.web WebView)) ! (def scripts ["uk-data.js" "topojson.v1.min.js" "d3.v3.min.js" "geocoder.js"]) ! (defn create-browser [html] (-> WebView. .getEngine (.loadContent html))) ! (defn build-page [script-paths] (hiccup/html5 [:head (for [path script-paths] Java F X [:script (slurp (io/resource path))])])) ! (defn coordinates->name [longitude latitude] (let [browser (-> scripts build-page create-browser)] (.executeScript browser (str "geo.decode([" longitude "," latitude "])"))))
  97. 97. (:require [hiccup.page :as hiccup]) (:import (javafx.scene.web WebView)) ! (def scripts ["uk-data.js" "topojson.v1.min.js" "d3.v3.min.js" "geocoder.js"]) ! (defn create-browser [html] (-> WebView. .getEngine (.loadContent html))) ! (defn build-page [script-paths] (hiccup/html5 [:head (for [path script-paths] Java F X [:script (slurp (io/resource path))])])) ! (defn coordinates->name [longitude latitude] (let [browser (-> scripts build-page create-browser)] (.executeScript browser Stuart Sier ras Co mpo nent lib!! (str "geo.decode([" longitude "," latitude "])"))))
  98. 98. RING Journaling Syncing Business logic What Where When Memory Web API Incoming Data KirinDave/clj-time
  99. 99. “Thu Jun 26 14:35:57 +0000 2014” over 86 thousand seconds in a day over 31 million seconds in a year
  100. 100. mm:260620141435
  101. 101. HH:260620141400
  102. 102. dd:260620140000 Monday Tuesday Wednesday Thursday Friday Saturday Sunday 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 27 28 29 30 31
  103. 103. MM:010620140000 JAN FEB MAR APR MAY JUL AUG SEP OCT NOV DEC
  104. 104. 2013 2015 yy:010120140000
  105. 105. (:require [clj-time.format :as time]) ! ! (def input-formatter "EE MMM d HH:mm:ss Z yyyy") (def formats { "mm:" "ddMMyyHHmm" "HH:" "ddMMyyHH00" ... }) ! ! (defn create-key [date [prefix formatter]] (let [out-date (time/unparse formatter date)] (str prefix out-date))) ! ! (defn date->keys [created_at] (let [date (time/parse input-formatter created_at)] (map #(create-key date %) formats)))
  106. 106. (:require [clj-time.format :as time]) ! ! (def input-formatter "EE MMM d HH:mm:ss Z yyyy") (def formats { "mm:" "ddMMyyHHmm" "HH:" "ddMMyyHH00" ... }) ! ! (defn create-key [date [prefix formatter]] (let [out-date (time/unparse formatter date)] (str prefix out-date))) ! ! (defn date->keys [created_at] (let [date (time/parse input-formatter created_at)] (map #(create-key date %) formats)))
  107. 107. (:require [clj-time.format :as time]) ! ! (def input-formatter "EE MMM d HH:mm:ss Z yyyy") (def formats { "mm:" "ddMMyyHHmm" "HH:" "ddMMyyHH00" ... }) ! ! (defn create-key [date [prefix formatter]] (let [out-date (time/unparse formatter date)] (str prefix out-date))) ! ! (defn date->keys [created_at] (let [date (time/parse input-formatter created_at)] (map #(create-key date %) formats)))
  108. 108. RING Journaling Syncing Business logic What Where When Memory Web API Incoming Data ptaoussanis/carmine
  109. 109. We have 11 keys in total! SENTIMENT ! LOCATION ! mm:date-time HH:date-time dd:date-time MM:date-time yy:date-time What Where When
  110. 110. (:require [taoensso.carmine :as redis]) ! ! (defn create-tweet-id [] (redis/incr "global:tweet.id")) ! ! (defn set-bits [data-keys tweet-id] (doseq [data-key data-keys] (redis/setbit data-key tweet-id 1))) ! ! (defn data->analytics [data-keys] (let [tweet-id (create-tweet-id)] (set-bits data-keys tweet-id)))
  111. 111. (:require [taoensso.carmine :as redis]) ! ! (defn create-tweet-id [] (redis/incr "global:tweet.id")) ! ! (defn set-bits [data-keys tweet-id] (doseq [data-key data-keys] (redis/setbit data-key tweet-id 1))) ! ! (defn data->analytics [data-keys] (let [tweet-id (create-tweet-id)] (set-bits data-keys tweet-id)))
  112. 112. (:require [taoensso.carmine :as redis]) ! ! (defn create-tweet-id [] (redis/incr "global:tweet.id")) ! ! (defn set-bits [data-keys tweet-id] (doseq [data-key data-keys] (redis/setbit data-key tweet-id 1))) ! ! (defn data->analytics [data-keys] (let [tweet-id (create-tweet-id)] (set-bits data-keys tweet-id)))
  113. 113. { 'query-type': 'count', 'params': { 'sentiments': [ 'FATIGUE', 'HOSTILITY' ], 'locations': [ 'SCT' ] } } weavejester/compojure JSON quer y
  114. 114. { 'query-type': 'count', 'params': { 'sentiments': [ 'FATIGUE', 'HOSTILITY' ], 'locations': [ 'SCT' ] } } weavejester/compojure JSON quer y /ap i?_quer y={jso n}
  115. 115. (defmulti execute :query-type) ! ! (defmethod execute “count" [] … logic here) ! ! (defmethod execute “p-affect“ [] … logic here) ! ! (defmethod execute “n-affect“ [] … logic here) { 'query-type': 'count', 'params': { 'sentiments': [ 'FATIGUE', 'HOSTILITY' ], 'locations': [ 'SCT' ] } } weavejester/compojure JSON quer y /ap i?_quer y={jso n}
  116. 116. (defmulti execute :query-type) ! ! (defmethod execute “count" [] … logic here) ! ! (defmethod execute “p-affect“ [] … logic here) ! ! (defmethod execute “n-affect“ [] … logic here) { 'query-type': 'count', 'params': { 'sentiments': [ 'FATIGUE', 'HOSTILITY' ], 'locations': [ 'SCT' ] } } weavejester/compojure JSON quer y /ap i?_quer y={jso n}
  117. 117. weavejester/compojure (defmulti execute :query-type) ! ! (defmethod execute “count" [] … logic here) ! ! (defmethod execute “p-affect“ [] … logic here) ! ! (defmethod execute “n-affect“ [] … logic here) JSON quer y { 'query-type': 'count', 'params': { 'sentiments': [ 'FATIGUE', 'HOSTILITY' ], 'locations': [ 'SCT' ] } } /ap i?_quer y={jso n}
  118. 118. weavejester/compojure (defmulti execute :query-type) ! ! (defmethod execute “count" [] … logic here) ! ! (defmethod execute “p-affect“ [] … logic here) ! ! (defmethod execute “n-affect“ [] … logic here) JSON quer y { 'query-type': ‘p-affect', 'params': { 'sentiments': [ 'FATIGUE', 'HOSTILITY' ], 'locations': [ 'SCT' ] } } /ap i?_quer y={jso n}
  119. 119. weavejester/compojure (defmulti execute :query-type) ! ! (defmethod execute “count" [] … logic here) ! ! (defmethod execute “p-affect“ [] … logic here) ! ! (defmethod execute “n-affect“ [] … logic here) JSON quer y { 'query-type': ‘p-affect', 'params': { 'sentiments': [ 'FATIGUE', 'HOSTILITY' ], 'locations': [ 'SCT' ] } } /ap i?_quer y={jso n}

×