Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Using Clojure for Sentiment Analysis of the Twittersphere (EuroClojure 2014)

2,851 views

Published on

Slides from "Using Clojure for Sentiment Analysis of the Twittersphere" - EuroClojure 2014.

I had to convert from Keynote to pdf for upload, so apologies for loss of animations or notes.

  • Be the first to comment

Using Clojure for Sentiment Analysis of the Twittersphere (EuroClojure 2014)

  1. 1. Sentiment Analysis of the Twittersphere
  2. 2. Leiningen Versus the Ants! Clojure Versus Java! FP Versus OOP
  3. 3. Leiningen Versus the Ants! Clojure Versus Java! FP Versus OOP Apache Ant
  4. 4. Leiningen Versus the Ants Clojure Versus Java! FP Versus OOP Apache Ant
  5. 5. Leiningen Versus the Ants Clojure Versus Java Functional Versus Object Oriented Apache Ant
  6. 6. Blood aches pain sick shivers spasm vomit dizzy fainting colic and in a coma! 1 FAVORITE
  7. 7. Blood aches pain sick shivers spasm vomit dizzy fainting colic and in a coma! 1 FAVORITE Scores HIGH for flu symptoms
  8. 8. Blood aches pain sick shivers spasm vomit dizzy fainting colic and in a coma! 1 FAVORITE Who is Catalina Rubottom?
  9. 9. Blood aches pain sick shivers spasm vomit dizzy fainting colic and in a coma! 1 FAVORITE HDFS/Hadoop Mongo/Aggregation Mongo/MapReduce Postgres 30 million geo-tagged tweets sent from UK
  10. 10. How can we do fast, real time analytics of social media?
  11. 11. Store this! Not this!
  12. 12. ptaoussanis/carmine
  13. 13. (:require [taoensso.carmine :as car]) ! ! (defn create-tweet-id [] (first (wcar* (car/incr "global:tweet.id")))) ! ! (defn set-values [sentiment location created-at] (let [tweed-id (create-tweet-id)] (wcar* (car/setbit sentiment tweet-id 1) (car/setbit location tweet-id 1) (car/setbit created-at tweet-id 1))))
  14. 14. (:require [taoensso.carmine :as car]) ! ! (defn create-tweet-id [] (first (wcar* (car/incr "global:tweet.id")))) ! ! (defn set-values [sentiment location created-at] (let [tweed-id (create-tweet-id)] (wcar* (car/setbit sentiment tweet-id 1) (car/setbit location tweet-id 1) (car/setbit created-at tweet-id 1))))
  15. 15. (:require [taoensso.carmine :as car]) ! ! (defn create-tweet-id [] (first (wcar* (car/incr "global:tweet.id")))) ! ! (defn set-values [sentiment location created-at] (let [tweed-id (create-tweet-id)] (wcar* (car/setbit sentiment tweet-id 1) (car/setbit location tweet-id 1) (car/setbit created-at tweet-id 1))))
  16. 16. (:require [taoensso.carmine :as car]) ! ! (defn create-tweet-id [] (first (wcar* (car/incr "global:tweet.id")))) ! ! (defn set-values [sentiment location created-at] (let [tweed-id (create-tweet-id)] (wcar* (car/setbit sentiment tweet-id 1) (car/setbit location tweet-id 1) (car/setbit created-at tweet-id 1))))
  17. 17. (:require [taoensso.carmine :as car]) ! ! (defn create-tweet-id [] (first (wcar* (car/incr "global:tweet.id")))) ! ! (defn set-values [sentiment location created-at] (let [tweed-id (create-tweet-id)] (wcar* (car/setbit sentiment tweet-id 1) (car/setbit location tweet-id 1) (car/setbit created-at tweet-id 1)))) find value for key find bit of index set bit to 1
  18. 18. (wcar* (car/bitcount "SCOTLAND") (car/bitcount "JOVIALITY") (car/bitcount "26062014114532"))
  19. 19. ENG SCT WAL NIR valuekey
  20. 20. ENG 1 SCT 0 WAL 0 NIR 0 (car/setbit “ENG” 0 1) tweet-idkey valuekey
  21. 21. ENG 0 1 SCT 1 0 WAL 0 0 NIR 0 0 (car/setbit “SCT” 1 1) tweet-idkey valuekey
  22. 22. ENG 1 0 1 SCT 0 1 0 WAL 0 0 0 NIR 0 0 0 (car/setbit “ENG” 2 1) tweet-idkey valuekey
  23. 23. ENG 0 0 1 0 1 1 0 1 0 0 1 0 1 1 0 1 SCT 1 0 0 0 0 0 1 0 1 0 0 0 0 0 1 0 WAL 0 1 0 1 0 0 0 0 0 1 0 0 0 0 0 0 NIR 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 (car/setbit “SCT” 15 1) tweet-idkey valuekey
  24. 24. ENG 0 0 1 0 1 1 0 1 0 0 1 0 1 1 0 1 SCT 1 0 0 0 0 0 1 0 1 0 0 0 0 0 1 0 WAL 0 1 0 1 0 0 0 0 0 1 0 0 0 0 0 0 NIR 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 (car/bitcount "ENG") 8 valuekey
  25. 25. ENG 0 0 1 0 1 1 0 1 0 0 1 0 1 1 0 1 SCT 1 0 0 0 0 0 1 0 1 0 0 0 0 0 1 0 WAL 0 1 0 1 0 0 0 0 0 1 0 0 0 0 0 0 NIR 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 (car/bitcount "SCT") 8 4 valuekey
  26. 26. ENG 0 0 1 0 1 1 0 1 0 0 1 0 1 1 0 1 SCT 1 0 0 0 0 0 1 0 1 0 0 0 0 0 1 0 WAL 0 1 0 1 0 0 0 0 0 1 0 0 0 0 0 0 NIR 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 (car/bitcount "WAL") 8 4 3 valuekey
  27. 27. ENG 0 0 1 0 1 1 0 1 0 0 1 0 1 1 0 1 SCT 1 0 0 0 0 0 1 0 1 0 0 0 0 0 1 0 WAL 0 1 0 1 0 0 0 0 0 1 0 0 0 0 0 0 NIR 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 (car/bitcount "NIR") 8 4 3 1 valuekey
  28. 28. JOVIAL SHY HOSTILE FATIGUE valuekey
  29. 29. JOVIAL 1 SHY 0 HOSTILE 0 FATIGUE 0 (car/setbit “JOVIAL” 0 1) tweet-idkey valuekey
  30. 30. JOVIAL 0 1 SHY 0 0 HOSTILE 1 0 FATIGUE 0 0 (car/setbit “HOSTILE” 1 1) tweet-idkey valuekey
  31. 31. JOVIAL 0 0 1 SHY 1 0 0 HOSTILE 0 1 0 FATIGUE 0 0 0 (car/setbit “SHY” 2 1) tweet-idkey valuekey
  32. 32. JOVIAL 1 0 1 0 1 1 1 0 0 1 0 1 0 0 0 1 SHY 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 HOSTILE 0 0 0 0 0 0 0 1 0 0 1 0 0 0 1 0 FATIGUE 0 1 0 1 0 0 0 0 1 0 0 0 1 0 0 0 (car/setbit “JOVIAL” 15 1) tweet-idkey valuekey
  33. 33. JOVIAL 1 0 1 0 1 1 1 0 0 1 0 1 0 0 0 1 SHY 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 HOSTILE 0 0 0 0 0 0 0 1 0 0 1 0 0 0 1 0 FATIGUE 0 1 0 1 0 0 0 0 1 0 0 0 1 0 0 0 (car/bitcount [*]) 8 1 3 4 valuekey
  34. 34. (wcar* (car/bitop "AND" “ENGLAND&JOVIALITY" "ENGLAND" "JOVIALITY") (car/expire “ENGLAND&JOVIALITY" 10) (car/bitcount “ENGLAND&JOVIALITY")) How many people in England are Happy?
  35. 35. (wcar* (car/bitop "AND" “ENGLAND&JOVIALITY" "ENGLAND" "JOVIALITY") (car/expire “ENGLAND&JOVIALITY" 10) (car/bitcount “ENGLAND&JOVIALITY")) How many people in England are Happy?
  36. 36. (wcar* (car/bitop "AND" “ENGLAND&JOVIALITY" "ENGLAND" "JOVIALITY") (car/expire “ENGLAND&JOVIALITY" 10) (car/bitcount “ENGLAND&JOVIALITY")) How many people in England are Happy?
  37. 37. (wcar* (car/bitop "AND" “ENGLAND&JOVIALITY" "ENGLAND" "JOVIALITY") (car/expire “ENGLAND&JOVIALITY" 10) (car/bitcount “ENGLAND&JOVIALITY")) How many people in England are Happy?
  38. 38. How many people in England are Happy?
  39. 39. ENG 0 0 1 0 1 1 0 1 0 0 1 0 1 1 0 1 How many people in England are Happy?
  40. 40. ENG 0 0 1 0 1 1 0 1 0 0 1 0 1 1 0 1 JOVIAL 1 0 1 0 1 1 1 0 0 1 0 1 0 0 0 1 How many people in England are Happy?
  41. 41. ENG 0 0 1 0 1 1 0 1 0 0 1 0 1 1 0 1 JOVIAL 1 0 1 0 1 1 1 0 0 1 0 1 0 0 0 1 1 AND How many people in England are Happy? (car/bitop "AND" “ENGLAND&JOVIALITY" "ENGLAND" "JOVIALITY")
  42. 42. ENG 0 0 1 0 1 1 0 1 0 0 1 0 1 1 0 1 JOVIAL 1 0 1 0 1 1 1 0 0 1 0 1 0 0 0 1 0 1 AND How many people in England are Happy? (car/bitop "AND" “ENGLAND&JOVIALITY" "ENGLAND" "JOVIALITY")
  43. 43. ENG 0 0 1 0 1 1 0 1 0 0 1 0 1 1 0 1 JOVIAL 1 0 1 0 1 1 1 0 0 1 0 1 0 0 0 1 0 0 1 AND How many people in England are Happy? (car/bitop "AND" “ENGLAND&JOVIALITY" "ENGLAND" "JOVIALITY")
  44. 44. ENG 0 0 1 0 1 1 0 1 0 0 1 0 1 1 0 1 JOVIAL 1 0 1 0 1 1 1 0 0 1 0 1 0 0 0 1 0 0 1 0 1 1 0 0 0 0 0 0 0 0 0 1 AND How many people in England are Happy? (car/bitop "AND" “ENGLAND&JOVIALITY" "ENGLAND" "JOVIALITY")
  45. 45. ENG 0 0 1 0 1 1 0 1 0 0 1 0 1 1 0 1 JOVIAL 1 0 1 0 1 1 1 0 0 1 0 1 0 0 0 1 0 0 1 0 1 1 0 0 0 0 0 0 0 0 0 1 How many people in England are Happy? (car/bitcount “ENGLAND&JOVIALITY") 4
  46. 46. (wcar* (car/bitop "OR" "FATIGUE|HOSTILITY" "FATIGUE" “HOSTILITY") (car/expire "FATIGUE|HOSTILITY" 10) (car/bitop "AND" "SCOTLAND&(FATIGUE|HOSTILITY)" "SCOTLAND" "FATIGUE|HOSTILITY") (car/expire "SCOTLAND&(FATIGUE|HOSTILITY)" 10) (car/bitcount "SCOTLAND&(FATIGUE|HOSTILITY)")) How many people in Scotland are Tired and Grumpy?
  47. 47. (wcar* (car/bitop "OR" "FATIGUE|HOSTILITY" "FATIGUE" “HOSTILITY") (car/expire "FATIGUE|HOSTILITY" 10) (car/bitop "AND" "SCOTLAND&(FATIGUE|HOSTILITY)" "SCOTLAND" "FATIGUE|HOSTILITY") (car/expire "SCOTLAND&(FATIGUE|HOSTILITY)" 10) (car/bitcount "SCOTLAND&(FATIGUE|HOSTILITY)")) How many people in Scotland are Tired and Grumpy?
  48. 48. (wcar* (car/bitop "OR" "FATIGUE|HOSTILITY" "FATIGUE" “HOSTILITY") (car/expire "FATIGUE|HOSTILITY" 10) (car/bitop "AND" "SCOTLAND&(FATIGUE|HOSTILITY)" "SCOTLAND" "FATIGUE|HOSTILITY") (car/expire "SCOTLAND&(FATIGUE|HOSTILITY)" 10) (car/bitcount "SCOTLAND&(FATIGUE|HOSTILITY)")) How many people in Scotland are Tired and Grumpy?
  49. 49. (wcar* (car/bitop "OR" "FATIGUE|HOSTILITY" "FATIGUE" “HOSTILITY") (car/expire "FATIGUE|HOSTILITY" 10) (car/bitop "AND" "SCOTLAND&(FATIGUE|HOSTILITY)" "SCOTLAND" "FATIGUE|HOSTILITY") (car/expire "SCOTLAND&(FATIGUE|HOSTILITY)" 10) (car/bitcount "SCOTLAND&(FATIGUE|HOSTILITY)")) How many people in Scotland are Tired and Grumpy?
  50. 50. How many people in Scotland are Tired and Grumpy?
  51. 51. HOSTILE 0 0 0 0 0 0 0 1 0 0 1 0 0 0 1 0 How many people in Scotland are Tired and Grumpy?
  52. 52. HOSTILE 0 0 0 0 0 0 0 1 0 0 1 0 0 0 1 0 FATIGUE 0 1 0 1 0 0 0 0 1 0 0 0 1 0 0 0 How many people in Scotland are Tired and Grumpy?
  53. 53. HOSTILE 0 0 0 0 0 0 0 1 0 0 1 0 0 0 1 0 FATIGUE 0 1 0 1 0 0 0 0 1 0 0 0 1 0 0 0 BOTH OR How many people in Scotland are Tired and Grumpy?
  54. 54. HOSTILE 0 0 0 0 0 0 0 1 0 0 1 0 0 0 1 0 FATIGUE 0 1 0 1 0 0 0 0 1 0 0 0 1 0 0 0 BOTH 0 OR How many people in Scotland are Tired and Grumpy?
  55. 55. HOSTILE 0 0 0 0 0 0 0 1 0 0 1 0 0 0 1 0 FATIGUE 0 1 0 1 0 0 0 0 1 0 0 0 1 0 0 0 BOTH 1 0 OR How many people in Scotland are Tired and Grumpy?
  56. 56. HOSTILE 0 0 0 0 0 0 0 1 0 0 1 0 0 0 1 0 FATIGUE 0 1 0 1 0 0 0 0 1 0 0 0 1 0 0 0 BOTH 0 1 0 How many people in Scotland are Tired and Grumpy?
  57. 57. HOSTILE 0 0 0 0 0 0 0 1 0 0 1 0 0 0 1 0 FATIGUE 0 1 0 1 0 0 0 0 1 0 0 0 1 0 0 0 BOTH 0 1 0 1 0 0 0 1 1 0 1 0 1 0 1 0 OR How many people in Scotland are Tired and Grumpy?
  58. 58. HOSTILE 0 0 0 0 0 0 0 1 0 0 1 0 0 0 1 0 FATIGUE 0 1 0 1 0 0 0 0 1 0 0 0 1 0 0 0 BOTH 0 1 0 1 0 0 0 1 1 0 1 0 1 0 1 0 OR How many people in Scotland are Tired and Grumpy?
  59. 59. SCT 1 0 0 0 0 0 1 0 1 0 0 0 0 0 1 0 HOSTILE 0 0 0 0 0 0 0 1 0 0 1 0 0 0 1 0 FATIGUE 0 1 0 1 0 0 0 0 1 0 0 0 1 0 0 0 BOTH 0 1 0 1 0 0 0 1 1 0 1 0 1 0 1 0 How many people in Scotland are Tired and Grumpy?
  60. 60. SCT 1 0 0 0 0 0 1 0 1 0 0 0 0 0 1 0 BOTH 0 1 0 1 0 0 0 1 1 0 1 0 1 0 1 0 HOSTILE 0 0 0 0 0 0 0 1 0 0 1 0 0 0 1 0 FATIGUE 0 1 0 1 0 0 0 0 1 0 0 0 1 0 0 0 BOTH 0 1 0 1 0 0 0 1 1 0 1 0 1 0 1 0 AND How many people in Scotland are Tired and Grumpy?
  61. 61. SCT 1 0 0 0 0 0 1 0 1 0 0 0 0 0 1 0 BOTH 0 1 0 1 0 0 0 1 1 0 1 0 1 0 1 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 1 0 HOSTILE 0 0 0 0 0 0 0 1 0 0 1 0 0 0 1 0 FATIGUE 0 1 0 1 0 0 0 0 1 0 0 0 1 0 0 0 BOTH 0 1 0 1 0 0 0 1 1 0 1 0 1 0 1 0 How many people in Scotland are Tired and Grumpy? 2
  62. 62. RING Syncing Journaling Business logic What Where When Memory WebAPI Incoming Data
  63. 63. RING Syncing Journaling Business logic What Where When Memory WebAPI Incoming Data adamwynne/twitter-api
  64. 64. RING Syncing Journaling Business logic What Where When Memory WebAPI Incoming Data clojurewerkz/meltdown LMAX-Exchange/disruptor
  65. 65. RING Syncing Journaling Business logic What Where When Memory WebAPI Incoming Data dakrone/clojure-opennlp
  66. 66. “I’m loving #EuroClojure! ”
  67. 67. Positive Affect: enthusiastic, active, alert ! Negative Affect: subjective distress ! Emerged as distinctive, orthogonal dimensions
  68. 68. PANAS 1 2 3 4 5 1 3 1 5 3 2 3 3 5 3 4 5 2 2 1
  69. 69. PANAS 1 2 3 4 5 1 3 1 5 3 2 3 3 5 3 4 5 2 2 1 1 3 5 9 10 12 14 16 17 19 2 4 6 7 8 11 13 15 18 20
  70. 70. PANAS 1 23 4 5 1 3 1 5 3 23 35 34 5 2 2 1 1 3 5 9 10 12 14 16 17 19 2 4 6 7 8 11 13 15 18 20 39 19 positive affect! negative affect
  71. 71. PANAS-x General Dimension Scales! Negative Affect (10) Positive Affect (10) Basic Positive Emotions Scales! Joviality (8) Self-assurance (6) Attentiveness (4) Basic Negative Emotions Scales! Fear (6) Hostility (6) Guilt (6) Sadness (5) Other Affective States! Shyness (4) Fatigue (4) Serenity (3) Surprise (3)
  72. 72. PANAS-t Accounts for bias on social media! Outlines sanitisation Validate against 10 real events
  73. 73. Sanitisation • Exclude media/advertising/spam • Account for text speak • Account for smileys & emoj’s • Word stemming (or lemmatisation) • Part of Speech Tagging (POS) LMAO GetRichQuick.com dakrone/clojure-opennlp
  74. 74. SHYNESS FATIGUE SERENITY SURPRISE FEAR HOSTILITY GUILT SADNESS JOVIALITY SELF ASSURANCE ATTENTIVENESS We have sentiment keys!
  75. 75. RING Syncing Journaling Business logic What Where When Memory WebAPI Incoming Data mbostock/d3
  76. 76. [-75.14310264, 40.05701649]
  77. 77. Reverse Geocoding Issues • Don’t want external services • Don’t want heavy IO • Don’t want round trips to the database • Accuracy not too much of a concern
  78. 78. Equirectangular Projection
  79. 79. Courtesy of Eric Gaba (Wikimedia Commons User: Sting)
  80. 80. x = λcosφ1 y = φ Tissot's indicatrix of deformation Courtesy of Eric Gaba (Wikimedia Commons User: Sting)
  81. 81. x (px) y (px)
  82. 82. x (px) y (px) [lon, lat] = [55.86140, -4.26477] [x, y] = [329, 601] [x, y] = = Scotland!
  83. 83. x (px) y (px) [lon, lat] = [55.86140, -4.26477] [x, y] = [329, 601] [x, y] = = Scotland!
  84. 84. x (px) y (px) [lon, lat] = [55.86140, -4.26477] [x, y] = [329, 601] [x, y] = = Scotland!
  85. 85. x (px) y (px) [lon, lat] = [55.86140, -4.26477] [x, y] = [329, 601] [x, y] = = Scotland!
  86. 86. x (px) y (px) [lon, lat] = [55.86140, -4.26477] [x, y] = [329, 601] [x, y] = = Scotland!
  87. 87. x (px) y (px) [lon, lat] = [55.86140, -4.26477] [x, y] = [329, 601] [x, y] = = Scotland!
  88. 88. Server side! (JavaFX) Mike Bostock & Nick Rabinowitz
  89. 89. (:require [hiccup.page :as hiccup]) (:import (javafx.scene.web WebView)) ! (def scripts ["uk-data.js" "topojson.v1.min.js" "d3.v3.min.js" "geocoder.js"]) ! (defn create-browser [html] (-> WebView. .getEngine (.loadContent html))) ! (defn build-page [script-paths] (hiccup/html5 [:head (for [path script-paths] [:script (slurp (io/resource path))])])) ! (defn coordinates->name [longitude latitude] (let [browser (-> scripts build-page create-browser)] (.executeScript browser (str "geo.decode([" longitude "," latitude "])")))) JavaFX
  90. 90. (:require [hiccup.page :as hiccup]) (:import (javafx.scene.web WebView)) ! (def scripts ["uk-data.js" "topojson.v1.min.js" "d3.v3.min.js" "geocoder.js"]) ! (defn create-browser [html] (-> WebView. .getEngine (.loadContent html))) ! (defn build-page [script-paths] (hiccup/html5 [:head (for [path script-paths] [:script (slurp (io/resource path))])])) ! (defn coordinates->name [longitude latitude] (let [browser (-> scripts build-page create-browser)] (.executeScript browser (str "geo.decode([" longitude "," latitude "])")))) JavaFX
  91. 91. (:require [hiccup.page :as hiccup]) (:import (javafx.scene.web WebView)) ! (def scripts ["uk-data.js" "topojson.v1.min.js" "d3.v3.min.js" "geocoder.js"]) ! (defn create-browser [html] (-> WebView. .getEngine (.loadContent html))) ! (defn build-page [script-paths] (hiccup/html5 [:head (for [path script-paths] [:script (slurp (io/resource path))])])) ! (defn coordinates->name [longitude latitude] (let [browser (-> scripts build-page create-browser)] (.executeScript browser (str "geo.decode([" longitude "," latitude "])")))) JavaFX
  92. 92. (:require [hiccup.page :as hiccup]) (:import (javafx.scene.web WebView)) ! (def scripts ["uk-data.js" "topojson.v1.min.js" "d3.v3.min.js" "geocoder.js"]) ! (defn create-browser [html] (-> WebView. .getEngine (.loadContent html))) ! (defn build-page [script-paths] (hiccup/html5 [:head (for [path script-paths] [:script (slurp (io/resource path))])])) ! (defn coordinates->name [longitude latitude] (let [browser (-> scripts build-page create-browser)] (.executeScript browser (str "geo.decode([" longitude "," latitude "])")))) JavaFX
  93. 93. (:require [hiccup.page :as hiccup]) (:import (javafx.scene.web WebView)) ! (def scripts ["uk-data.js" "topojson.v1.min.js" "d3.v3.min.js" "geocoder.js"]) ! (defn create-browser [html] (-> WebView. .getEngine (.loadContent html))) ! (defn build-page [script-paths] (hiccup/html5 [:head (for [path script-paths] [:script (slurp (io/resource path))])])) ! (defn coordinates->name [longitude latitude] (let [browser (-> scripts build-page create-browser)] (.executeScript browser (str "geo.decode([" longitude "," latitude "])")))) Stuart Sierras Component lib!! JavaFX
  94. 94. RING Syncing Journaling Business logic What Where When Memory WebAPI Incoming Data KirinDave/clj-time
  95. 95. “Thu Jun 26 14:35:57 +0000 2014” over 86 thousand seconds in a day over 31 million seconds in a year
  96. 96. mm:260620141435
  97. 97. HH:260620141400
  98. 98. Monday Tuesday Wednesday Thursday Friday Saturday Sunday 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 27 28 29 30 31 dd:260620140000
  99. 99. JAN FEB MAR APR MAY JUL AUG SEP OCT NOV DEC MM:010620140000
  100. 100. 2013 2015 yy:010120140000
  101. 101. (:require [clj-time.format :as time]) ! ! (def input-formatter "EE MMM d HH:mm:ss Z yyyy") (def formats { "mm:" "ddMMyyHHmm" "HH:" "ddMMyyHH00" ... }) ! ! (defn create-key [date [prefix formatter]] (let [out-date (time/unparse formatter date)] (str prefix out-date))) ! ! (defn date->keys [created_at] (let [date (time/parse input-formatter created_at)] (map #(create-key date %) formats)))
  102. 102. (:require [clj-time.format :as time]) ! ! (def input-formatter "EE MMM d HH:mm:ss Z yyyy") (def formats { "mm:" "ddMMyyHHmm" "HH:" "ddMMyyHH00" ... }) ! ! (defn create-key [date [prefix formatter]] (let [out-date (time/unparse formatter date)] (str prefix out-date))) ! ! (defn date->keys [created_at] (let [date (time/parse input-formatter created_at)] (map #(create-key date %) formats)))
  103. 103. (:require [clj-time.format :as time]) ! ! (def input-formatter "EE MMM d HH:mm:ss Z yyyy") (def formats { "mm:" "ddMMyyHHmm" "HH:" "ddMMyyHH00" ... }) ! ! (defn create-key [date [prefix formatter]] (let [out-date (time/unparse formatter date)] (str prefix out-date))) ! ! (defn date->keys [created_at] (let [date (time/parse input-formatter created_at)] (map #(create-key date %) formats)))
  104. 104. RING Syncing Journaling Business logic What Where When Memory WebAPI Incoming Data ptaoussanis/carmine
  105. 105. SENTIMENT ! LOCATION ! mm:date-time HH:date-time dd:date-time MM:date-time yy:date-time We have 11 keys in total! What Where When
  106. 106. (:require [taoensso.carmine :as redis]) ! ! (defn create-tweet-id [] (redis/incr "global:tweet.id")) ! ! (defn set-bits [data-keys tweet-id] (doseq [data-key data-keys] (redis/setbit data-key tweet-id 1))) ! ! (defn data->analytics [data-keys] (let [tweet-id (create-tweet-id)] (set-bits data-keys tweet-id)))
  107. 107. (:require [taoensso.carmine :as redis]) ! ! (defn create-tweet-id [] (redis/incr "global:tweet.id")) ! ! (defn set-bits [data-keys tweet-id] (doseq [data-key data-keys] (redis/setbit data-key tweet-id 1))) ! ! (defn data->analytics [data-keys] (let [tweet-id (create-tweet-id)] (set-bits data-keys tweet-id)))
  108. 108. (:require [taoensso.carmine :as redis]) ! ! (defn create-tweet-id [] (redis/incr "global:tweet.id")) ! ! (defn set-bits [data-keys tweet-id] (doseq [data-key data-keys] (redis/setbit data-key tweet-id 1))) ! ! (defn data->analytics [data-keys] (let [tweet-id (create-tweet-id)] (set-bits data-keys tweet-id)))
  109. 109. { 'query-type': 'count', 'params': { 'sentiments': [ 'FATIGUE', 'HOSTILITY' ], 'locations': [ 'SCT' ] } } weavejester/compojure JSON query
  110. 110. { 'query-type': 'count', 'params': { 'sentiments': [ 'FATIGUE', 'HOSTILITY' ], 'locations': [ 'SCT' ] } } weavejester/compojure JSON query /api?_query={json}
  111. 111. (defmulti execute :query-type) ! ! (defmethod execute “count" [] … logic here) ! ! (defmethod execute “p-affect“ [] … logic here) ! ! (defmethod execute “n-affect“ [] … logic here) { 'query-type': 'count', 'params': { 'sentiments': [ 'FATIGUE', 'HOSTILITY' ], 'locations': [ 'SCT' ] } } weavejester/compojure JSON query /api?_query={json}
  112. 112. (defmulti execute :query-type) ! ! (defmethod execute “count" [] … logic here) ! ! (defmethod execute “p-affect“ [] … logic here) ! ! (defmethod execute “n-affect“ [] … logic here) { 'query-type': 'count', 'params': { 'sentiments': [ 'FATIGUE', 'HOSTILITY' ], 'locations': [ 'SCT' ] } } weavejester/compojure JSON query /api?_query={json}
  113. 113. (defmulti execute :query-type) ! ! (defmethod execute “count" [] … logic here) ! ! (defmethod execute “p-affect“ [] … logic here) ! ! (defmethod execute “n-affect“ [] … logic here) weavejester/compojure JSON query /api?_query={json} { 'query-type': 'count', 'params': { 'sentiments': [ 'FATIGUE', 'HOSTILITY' ], 'locations': [ 'SCT' ] } }
  114. 114. (defmulti execute :query-type) ! ! (defmethod execute “count" [] … logic here) ! ! (defmethod execute “p-affect“ [] … logic here) ! ! (defmethod execute “n-affect“ [] … logic here) weavejester/compojure JSON query /api?_query={json} { 'query-type': ‘p-affect', 'params': { 'sentiments': [ 'FATIGUE', 'HOSTILITY' ], 'locations': [ 'SCT' ] } }
  115. 115. (defmulti execute :query-type) ! ! (defmethod execute “count" [] … logic here) ! ! (defmethod execute “p-affect“ [] … logic here) ! ! (defmethod execute “n-affect“ [] … logic here) weavejester/compojure JSON query /api?_query={json} { 'query-type': ‘p-affect', 'params': { 'sentiments': [ 'FATIGUE', 'HOSTILITY' ], 'locations': [ 'SCT' ] } }
  116. 116. The code for this project is about to be open sourced ! We would love to have more contributors! ! To find out more, follow: #APassionForCode

×