2012 09 GDG San Francisco Hackday at Parisoma

1,583 views

Published on

Presentation on Neo4j, Federal Campaign Data at http://www.gtugsf.com/events/52972282/

Published in: Technology
1 Comment
2 Likes
Statistics
Notes
  • Free Download : http://gg.gg/114bb

    Hey Guyz and girls, Today I am gonna show you perfact tool, Remember This video is old, But the download link with hack is brand new. Its very simple to using this tool and here are some instructions in video. Please REDOWNLOAD. Don't Forget to Comment Subscribe & Rate My Video :)

    Virus Scan :- This file has been scanned with avast! Antivirus. -- Status: FILE IS CLEANN.

    Copyright © 2014. All Rights Reserved
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
No Downloads
Views
Total views
1,583
On SlideShare
0
From Embeds
0
Number of Embeds
10
Actions
Shares
0
Downloads
41
Comments
1
Likes
2
Embeds 0
No embeds

No notes for slide

2012 09 GDG San Francisco Hackday at Parisoma

  1. 1. #neo4j The Neo4j Election Data @GDG SF Andreas Kollegger Peter Neubauer Michael Hunger @akollegger @peterneubauer @mesirii 1Saturday, September 29, 12
  2. 2. #neo4j 1Saturday, September 29, 12
  3. 3. #neo4j Follow the Data FEC Campaign Data Andreas Kollegger Peter Neubauer Michael Hunger @akollegger @peterneubauer @mesirii 2Saturday, September 29, 12
  4. 4. #neo4j 2Saturday, September 29, 12
  5. 5. Saturday, September 29, 12
  6. 6. 4Saturday, September 29, 12
  7. 7. Follow the Plan 4Saturday, September 29, 12
  8. 8. Follow the Plan 1.Graph Database Primer 4Saturday, September 29, 12
  9. 9. Follow the Plan 1.Graph Database Primer 1.Why graphs? 4Saturday, September 29, 12
  10. 10. Follow the Plan 1.Graph Database Primer 1.Why graphs? 2.Whats a graph database? 4Saturday, September 29, 12
  11. 11. Follow the Plan 1.Graph Database Primer 1.Why graphs? 2.Whats a graph database? 2.FEC Campaign Data 4Saturday, September 29, 12
  12. 12. Follow the Plan 1.Graph Database Primer 1.Why graphs? 2.Whats a graph database? 2.FEC Campaign Data 1.Data Model 4Saturday, September 29, 12
  13. 13. Follow the Plan 1.Graph Database Primer 1.Why graphs? 2.Whats a graph database? 2.FEC Campaign Data 1.Data Model 2.Import Strategy 4Saturday, September 29, 12
  14. 14. Follow the Plan 1.Graph Database Primer 1.Why graphs? 2.Whats a graph database? 2.FEC Campaign Data 1.Data Model 2.Import Strategy 3.Queries 4Saturday, September 29, 12
  15. 15. 5Saturday, September 29, 12
  16. 16. Follow the Plan - Part 2 5Saturday, September 29, 12
  17. 17. Follow the Plan - Part 2 1. Intro to Google Apps Script by Alex 5Saturday, September 29, 12
  18. 18. Follow the Plan - Part 2 1. Intro to Google Apps Script by Alex 2. Register at Heroku and install the heroku gem 5Saturday, September 29, 12
  19. 19. Follow the Plan - Part 2 1. Intro to Google Apps Script by Alex 2. Register at Heroku and install the heroku gem 3. Create and install a Heroku app (heroku apps:create) 5Saturday, September 29, 12
  20. 20. Follow the Plan - Part 2 1. Intro to Google Apps Script by Alex 2. Register at Heroku and install the heroku gem 3. Create and install a Heroku app (heroku apps:create) 4. Add a Neo4j addon instance to it (heroku addons:add neo4j) 5Saturday, September 29, 12
  21. 21. Follow the Plan - Part 2 1. Intro to Google Apps Script by Alex 2. Register at Heroku and install the heroku gem 3. Create and install a Heroku app (heroku apps:create) 4. Add a Neo4j addon instance to it (heroku addons:add neo4j) 5. Upload existing data to the graph 5Saturday, September 29, 12
  22. 22. Follow the Plan - Part 2 1. Intro to Google Apps Script by Alex 2. Register at Heroku and install the heroku gem 3. Create and install a Heroku app (heroku apps:create) 4. Add a Neo4j addon instance to it (heroku addons:add neo4j) 5. Upload existing data to the graph 6. Create a custom Ruby proxy app on Heroku 5Saturday, September 29, 12
  23. 23. Follow the Plan - Part 2 1. Intro to Google Apps Script by Alex 2. Register at Heroku and install the heroku gem 3. Create and install a Heroku app (heroku apps:create) 4. Add a Neo4j addon instance to it (heroku addons:add neo4j) 5. Upload existing data to the graph 6. Create a custom Ruby proxy app on Heroku 7. Connect to the app using a Google Spreadsheet  5Saturday, September 29, 12
  24. 24. Follow the Plan - Part 2 1. Intro to Google Apps Script by Alex 2. Register at Heroku and install the heroku gem 3. Create and install a Heroku app (heroku apps:create) 4. Add a Neo4j addon instance to it (heroku addons:add neo4j) 5. Upload existing data to the graph 6. Create a custom Ruby proxy app on Heroku 7. Connect to the app using a Google Spreadsheet  8. Build a small bar chart from a Cypher query 5Saturday, September 29, 12
  25. 25. 6Saturday, September 29, 12
  26. 26. Graph Database Primer 6Saturday, September 29, 12
  27. 27. 7Saturday, September 29, 12
  28. 28. Why graphs, why now? D $ ; * , ☕⛵ ) ⛽. 7 4B N % 8 1 = +O @ 5% < J - ☕ 9: ( ⚾ ! ? E F$ P 2 ✈ 3 * - ), L C ( ⚽ IQ H 6 7 >G KMSaturday, September 29, 12
  29. 29. Why graphs, why now? 1.Big Data is the trend 8 /! ⛽C & :D A ⛵0 , # G 7 <7 J$ O " ⚾ ! T⚽ % + 3 X. E C M S D@ A; B * 9N ✈ P 10 4 # Q K D 5 Y EI @ ⛽ , 4 J4 Z &R "☕> G % 5 3+9 =% 8 @F ? $ +7 ; [W . $ B L⚾. ⛵ * ,< ⛵ U )☕ ⛽ XO F> 5 $=1 H = BN ☕✈ ! 1⚽ P - ☕ 8 2⚾: LE H % < J /9 V ( ? F? ( ] ✈ 3 * - ), ; : C O *6 ⚽ IQ H 7 > G KMSaturday, September 29, 12
  30. 30. Why graphs, why now? 1.Big Data is the trend 2.NOSQL is the answer P L 97 N W Y T 5 K 0 S @! ; Q 8 /B . ⛽C& & ⛽ : D !2$% A ⛵0 , # ? G # O- 3⛵ ! T⚽ C " ⚾ X 7$ 7JR+ O J %< ! 3 8 [ * X. E ⚾] " N , ⚽9 C > S D@ A; B D 1 P9 O > E M 10 4 # M UJ I + D 7 ✈ #: Q K & WA ⛽ ✈ R34 S 5 Y EI @ , 5 $ 4 < R T ⛵☕> G Z 9 K %. " @F ? X $ 4& J ⛽[) % 0 7 XOL⚾ - =☕ ;" WI 5A 3B N = . 8 1 H H V * ⛵ ☕. U L + $ B5 + ⛽ N ⛵ U 9 = , > B$1 ✈ G☕ , < M P Z ☕ - ☕ ⚾8 2⚾:[FE H % < J P <: ! F / ]8 V; % FQ ⚽ ? ✈ 4 H Z GV ), 2L E? *6 ⚽ 1 F ✈ 3 ( ( - ] ; : CO ? * @ ⚽ C D Y M Q I H 7 + > G KSaturday, September 29, 12
  31. 31. Why graphs, why now? 1.Big Data is the trend 2.NOSQL is the answer 3.Large in volume, and in P L 97 N W Y P L 97 N W Y T5K 0 S @ . ; T 5 K 0 S @! ; ⛽ . & Q & B & ⛽ Q 8 /B O-G⛵ ⛽ C # !2R C : D !2$% A " # ⛵X 3 0, ? # O- 3⛵ ! T⚽ ? 8 " J 3%$[ ! > X 7$ 7JR+ 1 ] X⚾ A;⚽9 E J % < M DJO C⚾ , 8 9 [ C >✈S * 7✈ O > . ⚾] " N , ⚽9 ! U M IR D@ + E B WA D 1 P + O >E # : 0 4 S M UJ I 1 3 45 T 7 S D X I@ WA " # & ✈ Q K , 4 J & Y EG T ⛽ $ +7 < Z= ⛵R UG # :R34 5 $ 54HZV ☕" I >0 L < $K% . ⛽ ⛵☕ FN? [ : - X ;" WI 5A 3B 9 K %.8 1⚾P [ % 0 L + N ⚾. H V * ⛵ )☕. U ⛽ G☕ , < M PU& N XO ☕ = L B- ☕ 1@ A ]8 ; MH F , < BQ ⛽ ⛵ [F> 4 Q J ;✈ 5 $= GV 1 F / 98 V, ⚽F ? < ✈ ( : F Z ☕ - ☕ ⚾8 2⚾: $ E H % 1⚽ P @ < %B Z ! ?H 2? E ✈ 4 H Z GV ]] 2L + *6 D Y F ✈ 3( - ), ; : C O ⚽ ⚽ * @Y IQ E ? H C + >C D K M G 7Saturday, September 29, 12
  32. 32. P L0 7 N W T 5 K Q S @ . ; L9 Y W & ⛽ BP 07 N9 Y O- 3⛵ !2 ? K T5 S @B ;. 7 N W P L 97 N " # Y X 8 WJ %$ & R 9 Y Q ⛽C7 L >⛵ N W !2 ? 01 [ ! S @ B ; T 5 KDJ ⚾] @ . , ⚽# O- 3 X . P> 0 9Q !2 ? ⛽ M UQ IS Why graphs, why now? 7 + O 58 Q WA @ $[ !R C BS T K ; E J S %. 9 " B⚽ > & R C - 3⛵ O XJ # :R34 5& ⛽ J ⚾] T , ; E > ✈ !I ?M U <I K 7 ! O ? 2RU LD 1 %$[ ! # 8 H"V ☕" ⚽9 X G %$⛽ 0O C AN ✈$ % . S ⛵ [ ! #& - 3⛵:R34 + 2R C - WA , + O M UJ W ⚾] , F >X #"< I [U L < K T > E >1 A Z ☕ ⚽9 M P8]8 [J %$5 & X: BQ , 1E H⚾V ; , ⚽9 ! > ⚾ . ✈ 4 H > GV 1 AG☕" E 0 E N S D I 5U LL < # :. ⚽ 1.Big Data is the trend F Z DJ W] 2 ⛵ R3 7 S 9 9 Y ✈T 4 + O M U @IR ☕7 + 1⚾P D Y ⛽ M A Z ? S O > ]8 [A % B Q F W< ;$ ,: - P& 0[ K - 0N 7: " A H ,G ⛵ & N P A 0[ < YNT ✈ 4 5 ✈ X $ N W I 50 LL XC # :.34 H Z GV9 Y 2. E + K - ⚽F W I P& L < K T M P K8 ; % ⛽ Q M U FT V @ . ⚾ 5 ] Q Z ☕B ;⚾P < 7: " H $ %⛽ S ☕ B 1 F 5 K8 ; G☕⛵ Q+M A V , @B . U @Y 0 LC0[ :%W N D7 N - ? T < G⛽ @ V E 2 ✈ !2 Z GV ] Q Z F F ? ⚽4 H ? ⛽ 1F 5 ] Q $ S ☕ B ;⚾P K8 ; @B Q T ,< S . 8 3 Y 9 Y & O-D⛵ C PJ L + R C C 3⛵ X %$7 N # " W 2.NOSQL is the answer O-D X 9 ⚾] T , ⚽9 ! > ⚾ NJ %$[ ! # E @Y 2 ✈ !2 Z G⛽ @ ? ⚽4 H ? V & R C C D⛵ O- 3 Y 2 E ; B ? !2 ? 1 0 [> 5 K Q S E B1; 8 @ J WA Y L 8 + . W " 7 , ⚽9 P > X %$ J RCJ I 7+ O M U . ] 7+ O5T D I@ 0 > E 1 A " , ⚽9 ! > ⚾] [# :R34 ⛵ ✈& ⛽ S !# :.34 S K M UJ W 7 + O > E ; ✈T S Q ? BR D I WA G# 8 ⛽ ⛵ O-I 5 & 3 XNJ 2R C <? - 0 L H $!2 ⛽V ☕" ⚾ M" A %$[V % " U X : 3.Large in volume, and in K ! I 5 & ⛽ & ✈T 34 S [ < ☕⛵ Q F , ⚽9R! > $%M J P , CG B 8 ;E [ < K - U L X # :. R 0 ⛵3 -[ # % " I 50 L < K T XA HOV N & N 1⚾ ☕4 1 ] 7 + O Z ☕ E⚽ 1⚾PM UJ I GV ✈D HZ F ] > F " ]8 8; $ G☕⛵ Q YU < ⚾ ZP : ⛽ , 9M P B A : - [< % $ , ✈ YS W 9 > E 4 WAGV ⚽ # :R34 @9 2 ✈ > H Z+ 7 ]9 2D ☕F L0 7 N W8 ; ? ⚽F O , I JU ME 1⚾ 1 Y✈ 4 F W NI 5 & P K TS Y9 AL @Y ✈ ? ⚽ K Z GV @ . 2 L0 7 T 5 H W ] E X + . " C 0N Y < + - P43R: # N 7 D UL .CD S @Y ;? H V ☕⛵ P S 0 K 5N W % 5 I K ⛽ G B@ MQ.T K < :& A [< N BQ @B Q L0 7 T $ L U T 5 "Q & ⛽ +. ; C D B ? SX ;☕ ,.A Z F T 5⚾P :8 @! ;F⛽ ⛽☕#Z -!⛵ J F 1 K- ] ⛽ ; 0 M $[ & &⚾ ? ⚽4 H Z GV% < /? P 1 ⛵0 ☕C ⛵⛽G& H V 3 O !2 ✈2! ⛵Y, 8 S-O 8 Q B Q 3D2B # O- 3⛵ F ✈ 8 ; E 2R C %$[ !R C X "? & ⛽ ED ? 8 ] $%8 Z " ⚽ %$ ! , ⚽9 C R + % CX :2 !# ? G A HX, # ] J@ O- 3⛵ " ⚾ Y⚽ V ⚾G M UJ ⚾9 7 + O > E > A [$ J 7 ! T@ CO " 4 3 1[ ! 9⚽ 2R JD ⚽ # > E >8 , " X ]$ JD D J I X⚾ A;IR3 > 1 7J MC S % <9UC + 1 ] + , O > E SWA I[ M U * . EB 4 ✈ @ +# : W O S 9N R ⚽ ⚾] 7 , ✈ > + !D 7 D 1 5 43 : # E # A H # M UJ I P + O 7 S D X I@ >✈ 10DX S M :R34 ✈ I 50 L < K T 45 G☕" < KU . ⛽ ⛵ & T [ : - N . K < # N R34 K 5 V Y E W" I Z & ? "☕ X L 0 I ⛽ G HZV T ⛽ >0 L ✈U Q & & "☕ V A G A M, 5+ 4 < RG☕⛵ ⛵ M F F [ ⚾ % ]8 ; % B N U☕ MP F A $ , ,X : ⛵J : [ 8 ;"⚾ I 5N 3 = Z= . $ $7 % 0☕ + N ⚾. H $ 4 & Z 9 K %⛽ 8 @P $ V B Q 2< E 1: - ; V * P W . U L XOL B - < F ⛽U ⛽ 1M 4 G VG< P , Z⚾ A ☕4 $1☕ ✈ ] H ; H ] ☕⛵ )☕⛵ F ✈ B $✈ 5 =1 H⚽48 Z< @ [ [F % B < J1⚾ ; > F Q GV F , :⚽ F Z ✈ + 2 C D Y , E H% ⚽ P ! G E ? ? Z @F ☕ H8 ⚽98 V L 2 F 2⚾: ☕4 ✈1 / ]] < E - ? Y D C Z GV ), 2 : + ? *6 D Y ✈ ( H @ ? ( + ⚽ H ⚽3 * - Y ; ? CO C @ I H + >C D K M Q G 7 Saturday, September 29, 12
  33. 33. 7Saturday, September 29, 12
  34. 34. 8Saturday, September 29, 12
  35. 35. A Graph? 8Saturday, September 29, 12
  36. 36. A Graph? Yes, a graph 8Saturday, September 29, 12
  37. 37. A graph database... 9Saturday, September 29, 12
  38. 38. A graph database... ๏ no: not for charts & diagrams, or vector artwork 9Saturday, September 29, 12
  39. 39. A graph database... ๏ no: not for charts & diagrams, or vector artwork ๏ yes: for storing data that is structured as a graph 9Saturday, September 29, 12
  40. 40. A graph database... ๏ no: not for charts & diagrams, or vector artwork ๏ yes: for storing data that is structured as a graph •remember linked lists, trees? 9Saturday, September 29, 12
  41. 41. A graph database... ๏ no: not for charts & diagrams, or vector artwork ๏ yes: for storing data that is structured as a graph •remember linked lists, trees? •graphs are the general-purpose data structure 9Saturday, September 29, 12
  42. 42. A graph database... ๏ no: not for charts & diagrams, or vector artwork ๏ yes: for storing data that is structured as a graph •remember linked lists, trees? •graphs are the general-purpose data structure ๏ “A relational database may tell you the average age of everyone in the USA, but a graph database will tell you who is most likely to buy you a beer.” 9Saturday, September 29, 12
  43. 43. A Graph Database 10Saturday, September 29, 12
  44. 44. 11Saturday, September 29, 12
  45. 45. You know relational 11Saturday, September 29, 12
  46. 46. You know relational 11Saturday, September 29, 12
  47. 47. You know relational foo 11Saturday, September 29, 12
  48. 48. You know relational foo bar 11Saturday, September 29, 12
  49. 49. You know relational foo foo_bar bar 11Saturday, September 29, 12
  50. 50. You know relational foo foo_bar bar 11Saturday, September 29, 12
  51. 51. You know relational foo foo_bar bar 11Saturday, September 29, 12
  52. 52. You know relational foo foo_bar bar 11Saturday, September 29, 12
  53. 53. You know relational now consider relationships... 11Saturday, September 29, 12
  54. 54. You know relational now consider relationships... 11Saturday, September 29, 12
  55. 55. You know relational now consider relationships... 11Saturday, September 29, 12
  56. 56. You know relational now consider relationships... 11Saturday, September 29, 12
  57. 57. You know relational now consider relationships... 11Saturday, September 29, 12
  58. 58. You know relational now consider relationships... 11Saturday, September 29, 12
  59. 59. 11Saturday, September 29, 12
  60. 60. 12Saturday, September 29, 12
  61. 61. Were talking about a Property Graph 12Saturday, September 29, 12
  62. 62. Were talking about a Property Graph Nodes 12Saturday, September 29, 12
  63. 63. Were talking about a Property Graph Nodes Relationships 12Saturday, September 29, 12
  64. 64. Were talking about a Property Graph Em Joh il a n knows knows Alli Tob Lar Nodes son ias knows s knows And And knows knows rea rés s knows knows knows Pet Miic Mc knows Ian er knows a a knows knows De Mic lia h ael Relationships Properties (each a key+value) + Indexes (for easy look-ups) 12Saturday, September 29, 12
  65. 65. 12Saturday, September 29, 12
  66. 66. 13Saturday, September 29, 12
  67. 67. And, but, so how do you query this "graph" database? 13Saturday, September 29, 12
  68. 68. 14Saturday, September 29, 12
  69. 69. Cypher - a graph query language ๏ a pattern-matching query language ๏ declarative grammar with clauses (like SQL) ๏ aggregation, ordering, limits ๏ create, read, update, delete 14Saturday, September 29, 12
  70. 70. Cypher - a graph query language ๏ a pattern-matching query language ๏ declarative grammar with clauses (like SQL) ๏ aggregation, ordering, limits ๏ create, read, update, delete // get node 1, traverse 2 steps away start a=node(1) match (a)--()--(c) return c // create a node with a name property CREATE (me {name: Andreas}) return me ๏ more on this later... 14Saturday, September 29, 12
  71. 71. Cypher - pattern matching 15Saturday, September 29, 12
  72. 72. Cypher - pattern matching 15Saturday, September 29, 12
  73. 73. Cypher - pattern matching 15Saturday, September 29, 12
  74. 74. Cypher - pattern matching 15Saturday, September 29, 12
  75. 75. Cypher - pattern matching 15Saturday, September 29, 12
  76. 76. Cypher - pattern matching 15Saturday, September 29, 12
  77. 77. Cypher - pattern matching 15Saturday, September 29, 12
  78. 78. Cypher - pattern matching syntax 16Saturday, September 29, 12
  79. 79. Cypher - pattern matching syntax 16Saturday, September 29, 12
  80. 80. Cypher - pattern matching syntax () --> () 16Saturday, September 29, 12
  81. 81. Cypher - pattern matching syntax 17Saturday, September 29, 12
  82. 82. Cypher - pattern matching syntax A B 17Saturday, September 29, 12
  83. 83. Cypher - pattern matching syntax A B (A) --> (B) 17Saturday, September 29, 12
  84. 84. Cypher - pattern matching syntax 18Saturday, September 29, 12
  85. 85. Cypher - pattern matching syntax A B 18Saturday, September 29, 12
  86. 86. Cypher - pattern matching syntax A B (A) -- (B) 18Saturday, September 29, 12
  87. 87. Cypher - pattern matching syntax 19Saturday, September 29, 12
  88. 88. Cypher - pattern matching syntax LOVES A B 19Saturday, September 29, 12
  89. 89. Cypher - pattern matching syntax LOVES A B A -[:LOVES]-> B 19Saturday, September 29, 12
  90. 90. Cypher - pattern matching syntax 20Saturday, September 29, 12
  91. 91. Cypher - pattern matching syntax A B C 20Saturday, September 29, 12
  92. 92. Cypher - pattern matching syntax A B C A --> B --> C 20Saturday, September 29, 12
  93. 93. Cypher - pattern matching syntax 21Saturday, September 29, 12
  94. 94. Cypher - pattern matching syntax A B C 21Saturday, September 29, 12
  95. 95. Cypher - pattern matching syntax A B C A --> B --> C, A --> C 21Saturday, September 29, 12
  96. 96. Cypher - pattern matching syntax A B C A --> B --> C, A --> C A --> B --> C <-- A 21Saturday, September 29, 12
  97. 97. 22Saturday, September 29, 12
  98. 98. Cypher - common clauses 22Saturday, September 29, 12
  99. 99. Cypher - common clauses // get node 1, traverse 2 steps away START a=node(1) MATCH (a)--()--(c) RETURN c // get node from an index, return it START a=node:people(name=Andreas) RETURN a // get node from an index, match, filter // with where, then return results START a=node:people(name=Andreas) MATCH (a)-[r]-(b) WHERE b.last=Sparrow RETURN r,b 22Saturday, September 29, 12
  100. 100. FEC Campaign Data 23Saturday, September 29, 12
  101. 101. FEC Campaign Data yeah, this is the good stuff.. 23Saturday, September 29, 12
  102. 102. and now, its time for FEC Campaign Data yeah, this is the good stuff.. 23Saturday, September 29, 12
  103. 103. FEC Campaign Data ๏In 1975, Congress created the Federal Election Commission (FEC) to administer and enforce the Federal Election Campaign Act (FECA) – The statute that governs the financing of federal elections. ๏The duties of the FEC, which is an independent regulatory agency, are to disclose campaign finance information 24Saturday, September 29, 12
  104. 104. FEC Campaign Data ๏Detailed files about... • Candidates Committee Candidate • Committees • Individual Contributions Individual Contributions ๏10 years of data ๏Updated every Sunday 25Saturday, September 29, 12
  105. 105. FEC Campaign Data - Committees ๏Committees • one record for each committee registered with the Federal Election Commission. Committee - cm12.txt CMTE_ID: String CMTE_NM: String TRES_NM: String CMTE_ST1: String CMTE_ST2: String CMTE_CITY: String CMTE_ST: String CMTE_ZIP: String CMTE_DSGN: String CMTE_TP: String CMTE_PTY_AFFILIATION: String CMTE_FILING_FREQ: String ORG_TP: String CONNECTED_ORG_NM: String CAND_ID: String 26Saturday, September 29, 12
  106. 106. FEC Campaign Data ๏Candidates • one record for each candidateappeared on a ballot registered with the FEC or who has either list prepared by a state elections office. Candidate - cn12.txt CAND_ID: String CAND_NAME: String CAND_PTY_AFFILIATION: String CAND_ELECTION_YR: String CAND_OFFICE_ST: String CAND_OFFICE: String CAND_OFFICE_DISTRICT: String CAND_ICI: String CAND_STATUS: String CAND_PCC: String CAND_ST1: String CAND_ST2: String CAND_CITY: String CAND_ST: String CAND_ZIP: String 27Saturday, September 29, 12
  107. 107. FEC Campaign Data ๏Individual Contributions • each contribution from an individual to least $200. committee if the contribution was at a federal Individual Contrib - itcont.txt CMTE_ID: String AMNDT_IND: String RPT_TP: String TRANSACTION_PGI: String IMAGE_NUM: String TRANSACTION_TP: String ENTITY_TP: String NAME: String CITY: String STATE: String ZIP_CODE: String EMPLOYER: String OCCUPATION: String TRANSACTION_DT: String TRANSACTION_AMT: Double OTHER_ID: String TRAN_ID: String FILE_NUM: Integer MEMO_CD: String MEMO_TEXT: String SUB_ID: Integer 28Saturday, September 29, 12
  108. 108. FEC Campaign Data - Extra Records ๏Candidate to Committee Linkage • registered candidate to committee linkage ๏Transactions between Committees • inter-committee contribution or independent expenditure during the two-year election cycle ๏Contribution to Candidate • contribution or independent expenditure from committee to candidate during the two-year election cycle 29Saturday, September 29, 12
  109. 109. Import Strategy 30Saturday, September 29, 12
  110. 110. Raw Data Import Committee Candidate Candidate to Committee Inter Committee Contributions Candidate Contributions Individual Contributions 31Saturday, September 29, 12
  111. 111. Raw Data Import Committee Candidate CMTE_ID CAND_ID Candidate to Committee CMTE_ID CAND_ID Inter Committee Contributions CMTE_ID Candidate Contributions CAND_ID Individual Contributions CMTE_ID 31Saturday, September 29, 12
  112. 112. Connected Data Import 32Saturday, September 29, 12
  113. 113. Connected Data Import Committee CAND_ID Candidate CMTE_ID Candidate to Committee CAND_ID OTHER_ID:CAND_ID (from) CMTE_ID Inter Committee Contributions OTHER_ID:CMTE_ID (from) CAND_ID CMTE_ID Candidate Contributions OTHER_ID:CAND_ID CMTE_ID (from) (to) Individual Contributions OTHER_ID:CMTE_ID (from) 32Saturday, September 29, 12
  114. 114. Connected Data Import Committee CAND_ID Candidate CMTE_ID Candidate to Committee CAND_ID OTHER_ID:CAND_ID (from) CMTE_ID Inter Committee Contributions OTHER_ID:CMTE_ID (from) CAND_ID CMTE_ID Candidate Contributions OTHER_ID:CAND_ID CMTE_ID (from) (to) Individual Contributions OTHER_ID:CMTE_ID (from) 32Saturday, September 29, 12
  115. 115. Related Data Import 33Saturday, September 29, 12
  116. 116. Related Data Import Committee CAMPAIGNS_FOR Candidate SUPPORTS INTER_COMMITTEE_CONTRIBUTION INTER_COMMITTEE_CONTRIBUTION CANDIDATE_CONTRIBUTION INDIVIDUAL_CONTRIBUTION EARMARKED_BY Individual Contributions EARMARKED_BY 33Saturday, September 29, 12
  117. 117. Related Data Import Committee CAMPAIGNS_FOR Candidate SUPPORTS Committee INTER_COMMITTEE_CONTRIBUTION Candidate INTER_COMMITTEE_CONTRIBUTION CANDIDATE_CONTRIBUTION Individual Contributions INDIVIDUAL_CONTRIBUTION EARMARKED_BY Individual Contributions EARMARKED_BY 33Saturday, September 29, 12
  118. 118. Dave Fauths Approach 34Saturday, September 29, 12
  119. 119. Advanced Import - Dave Fauth ๏ includes SuperPAC data ๏ custom transform, then import ๏ model then looks like this... Expenditures Committee SUPPORTS Candidate FUNDS superPac Contributions Contribution GIVES Individual 35Saturday, September 29, 12
  120. 120. Advanced Import - Dave Fauth ๏ Extract and Transform • Stored files on S3 • Used MortarData to run Hadoop jobs to prepare data (@MortarData) ๏ Load • Used Neo4J BatchInserter to load • Thanks to Michael Hunger (@mesirii) • Loaded 2M+ nodes in <5 minutes 36Saturday, September 29, 12
  121. 121. Advanced Import - Dave Fauth Java BatchInsert Download Use S3 data Storage Process with Hadoop/Pig Created Neo4J DB 37Saturday, September 29, 12
  122. 122. Wanna learn more? ๏Come hear Dave Fauth present at... 38Saturday, September 29, 12
  123. 123. Next... Your Turn 39Saturday, September 29, 12
  124. 124. From scratch ๏ git clone https://github.com/akollegger/FEC_GRAPH.git ๏ cd FEC_GRAPH ๏ ant initialize • (need Apache ant? install from http://ant.apache.org) ๏ ant • ant will build the importers and create a script ๏ ./bin/fec2graph --force --importer=RELATED ๏ ant neo4j-start • will download and unpack neo4j, then start it 40Saturday, September 29, 12
  125. 125. Investigate with Neo4js Web UI ๏open http://localhost:7474 ๏Dashboard - overview of data records ๏Data browser - examine data records, with visualization options ๏Console - query the database using Cypher 41Saturday, September 29, 12
  126. 126. Querying FEC with Cypher ๏ For Cypher documentation • http://docs.neo4j.org/ ๏ FEC Data Definitions • http://www.fec.gov/finance/disclosure/ftpdet.shtml ๏ Ready for a challenge? 42Saturday, September 29, 12
  127. 127. http://1.usa.gov/uIGzZ 43Saturday, September 29, 12
  128. 128. Cypher Challenges http://1.usa.gov/uIGzZ 43Saturday, September 29, 12
  129. 129. Cypher Challenges // All presidential candidates for 2012 // Top 10 Presidential candidates according to number of campaign committees // find President Barack Obama // lookup Obama by his candidate ID // find Presidential Candidate Mitt Romney // lookup Romney by his candidate ID // find the shortest path of funding between Obama and Romney // 10 top individual contributions to Obama // 10 top individual contributions to Romney http://1.usa.gov/uIGzZ 43Saturday, September 29, 12
  130. 130. 44Saturday, September 29, 12
  131. 131. Cypher Challenges 44Saturday, September 29, 12
  132. 132. Cypher Challenges // All presidential candidates for 2012 start candidate=node:candidates(CAND_ID:*) where candidate.CAND_OFFICE=P AND candidate.CAND_ELECTION_YR=2012 return candidate.CAND_NAME; // Top 10 Presidential candidates according to // number of campaign committees start candidate=node:candidates(CAND_ID:*) match candidate<-[r:SUPPORTS]-(campaign) where candidate.CAND_OFFICE=P AND candidate.CAND_ELECTION_YR=2012 return candidate.CAND_NAME, COUNT(campaign) as count ORDER BY count desc LIMIT 10; // find President Barack Obama start obama=node:candidates(CAND_ID:*) WHERE obama.CAND_NAME =~ .*OBAMA.* return obama.CAND_NAME, obama.CAND_ID; 44Saturday, September 29, 12
  133. 133. 45Saturday, September 29, 12
  134. 134. Cypher Challenges 45Saturday, September 29, 12
  135. 135. Cypher Challenges // lookup Obama by his candidate ID start obama=node:candidates(CAND_ID=P80003338) return obama; // find Presidential Candidate Mitt Romney start romney=node:candidates(CAND_ID:*) WHERE romney.CAND_NAME =~ .*ROMNEY.* return romney.CAND_NAME, romney.CAND_ID; // lookup Romney by his candidate ID start romney=node:candidates(CAND_ID=P80003353) return romney; // find the shortest path of funding between Obama and Romney start romney=node:candidates(CAND_ID=P80003353), obama=node:candidates(CAND_ID=P80003338) MATCH p=shortestPath(romney-[*..10]-obama) return p; 45Saturday, September 29, 12
  136. 136. 46Saturday, September 29, 12
  137. 137. Cypher Challenges 46Saturday, September 29, 12
  138. 138. Cypher Challenges // 10 top individual contributions to Obama start obama=node:candidates(CAND_ID=P80003338) match obama<- [:SUPPORTS]-(campaign)<-[:INDIVIDUAL_CONTRIBUTION]- (contribution) return contribution.NAME, contribution.TRANSACTION_AMT order by contribution.TRANSACTION_AMT desc limit 10; // 10 top individual contributions to Romney start romney=node:candidates(CAND_ID=P80003353) match romney<-[:SUPPORTS]-(campaign)<-[:INDIVIDUAL_CONTRIBUTION]- (contribution) return contribution.NAME, contribution.TRANSACTION_AMT order by contribution.TRANSACTION_AMT desc limit 10; 46Saturday, September 29, 12
  139. 139. Customize the Data Importer ๏Java-savvy and feeling brave? ๏make a copy of • CODE/fecGraph/src/importer/fec/RelatedFecImporter.java ๏add your class to • CODE/fecGraph/src/importer/Tool.java ๏read docs about batch insertion • http://docs.neo4j.org/chunked/milestone/batchinsert.html ๏Ideas: • extract States and Zip Codes into "location index" • extract individual contributors from contribution list 47Saturday, September 29, 12
  140. 140. #neo4j 48Saturday, September 29, 12
  141. 141. #neo4j Lets have some Fun! :) 48Saturday, September 29, 12
  142. 142. #neo4j Neo4j Google REST Cypher Ruby Heroku 49Saturday, September 29, 12
  143. 143. Follow the Plan - Part 2 #neo4j Neo4j Google REST Cypher Ruby Heroku 49Saturday, September 29, 12
  144. 144. 50Saturday, September 29, 12
  145. 145. Follow the Plan - Part 2 50Saturday, September 29, 12
  146. 146. Follow the Plan - Part 2 1. Register at Heroku and install the heroku gem 50Saturday, September 29, 12
  147. 147. Follow the Plan - Part 2 1. Register at Heroku and install the heroku gem 2. Create and install a Heroku app (heroku apps:create) 50Saturday, September 29, 12
  148. 148. Follow the Plan - Part 2 1. Register at Heroku and install the heroku gem 2. Create and install a Heroku app (heroku apps:create) 3. Add a Neo4j addon (http://addons.heroku.com/neo4j) instance to it (heroku addons:add neo4j) 50Saturday, September 29, 12
  149. 149. Follow the Plan - Part 2 1. Register at Heroku and install the heroku gem 2. Create and install a Heroku app (heroku apps:create) 3. Add a Neo4j addon (http://addons.heroku.com/neo4j) instance to it (heroku addons:add neo4j) 4. Create a custom Ruby app (code below, GitHub) https:// github.com/neo4j-examples/heroku-neo4j-proxy 50Saturday, September 29, 12
  150. 150. Follow the Plan - Part 2 1. Register at Heroku and install the heroku gem 2. Create and install a Heroku app (heroku apps:create) 3. Add a Neo4j addon (http://addons.heroku.com/neo4j) instance to it (heroku addons:add neo4j) 4. Create a custom Ruby app (code below, GitHub) https:// github.com/neo4j-examples/heroku-neo4j-proxy 5. Upload the data from example-data.neo4j.org 50Saturday, September 29, 12
  151. 151. Follow the Plan - Part 2 1. Register at Heroku and install the heroku gem 2. Create and install a Heroku app (heroku apps:create) 3. Add a Neo4j addon (http://addons.heroku.com/neo4j) instance to it (heroku addons:add neo4j) 4. Create a custom Ruby app (code below, GitHub) https:// github.com/neo4j-examples/heroku-neo4j-proxy 5. Upload the data from example-data.neo4j.org 6. Connect to the app using a Google Spreadsheet , http:// bit.ly/GDG-GCALC 50Saturday, September 29, 12
  152. 152. Follow the Plan - Part 2 1. Register at Heroku and install the heroku gem 2. Create and install a Heroku app (heroku apps:create) 3. Add a Neo4j addon (http://addons.heroku.com/neo4j) instance to it (heroku addons:add neo4j) 4. Create a custom Ruby app (code below, GitHub) https:// github.com/neo4j-examples/heroku-neo4j-proxy 5. Upload the data from example-data.neo4j.org 6. Connect to the app using a Google Spreadsheet , http:// bit.ly/GDG-GCALC 7. Build a small bar chart from a Cypher query 50Saturday, September 29, 12
  153. 153. http://1.usa.gov/uIGzZ 51Saturday, September 29, 12
  154. 154. Heroku Challenges http://1.usa.gov/uIGzZ 51Saturday, September 29, 12
  155. 155. Heroku Challenges //Point the Database Instance to FEC http://bit.ly/SmkwUx/db/ data // Build a Google Data table endpoint // https://developers.google.com/chart/interactive/docs/ php_example http://1.usa.gov/uIGzZ 51Saturday, September 29, 12
  156. 156. The Heroku Neo4j proxy App 52Saturday, September 29, 12
  157. 157. The Heroku Neo4j proxy App > gem install heroku > git add github git@github.com:neo4j-examples/heroku-neo4j- proxy.git > heroku apps:create <app-name> > heroku addons:add neo4j > //add the project files > git add *; git commit -m"neo4j demo" > git push heroku master 52Saturday, September 29, 12
  158. 158. The Google Spreadsheet Cypher driver https://docs.google.com/spreadsheet/ccc? key=0AsSBFHSo5OaPdGhzT1RTbDVaR0R3NW5iNUFpejVuSHc#gid=0 53Saturday, September 29, 12
  159. 159. The Google Spreadsheet Cypher driver function cypherUrlREST(payload, url, user, pwd) { var auth = Utilities.base64Encode(user+":"+pwd); var response = UrlFetchApp.fetch( url, {"method":"POST", "payload": payload, "contentType": "application/json", "headers":{ "Authorization":"Basic "+auth, "accept":"application/json", } }); return response.getContentText(); } https://docs.google.com/spreadsheet/ccc? key=0AsSBFHSo5OaPdGhzT1RTbDVaR0R3NW5iNUFpejVuSHc#gid=0 53Saturday, September 29, 12
  160. 160. http://1.usa.gov/uIGzZ 54Saturday, September 29, 12
  161. 161. Google Challenges http://1.usa.gov/uIGzZ 54Saturday, September 29, 12
  162. 162. Google Challenges // Build a cypher parser in GoogleAppsScript // Build a Cypher query Google Widget // Visualize Cypher Results with Google Data Table // Geographic data viz http://1.usa.gov/uIGzZ 54Saturday, September 29, 12
  163. 163. The heatmap from Cypher to Google 55Saturday, September 29, 12
  164. 164. Wanna learn more? 56Saturday, September 29, 12

×