Users as Data    Paul Ingles    @pingles
Users as Data
Everything as Data
Code as Data(conj others me)
Data as Data{:name "Paul Ingles" :mood (constantly :happy)}
Data as Data{:name "Paul Ingles" :mood (constantly :hot)}
Builds as Data(defproject energy-comparison "1.0.0"  :description "pricing service"  :dependencies    [[org.clojure/clojur...
Everything as Data
Everything as Data• Data, in its most essential form
Everything as Data• Data, in its most essential form• Few simple abstractions  • map,   seq, fn
Everything as Data• Data, in its most essential form• Few simple abstractions  • map,   seq, fn• General library  • map,  ...
Logs as Data?
Logs as Data?0.0.0.0 - - [22/May/2012:09:00:03 +0000] "GET / HTTP/1.1" 304 0                        "SomeUserAgent"
Logs as Files/var/log/nginx/access.log
current power < potential power
log crunching   event processing
Users as Data
current power < potential power
Energy             Replication, ETL InsuranceCredit Cards               Session DB           WarehouseBroadband
Data Muddling  Energy             Replication, ETL InsuranceCredit Cards               Session DB           WarehouseBroad...
Source: http://www.irregulartimes.com/panopt.html
Energy               Insurance           Integration DBCredit Cards           Broadband
data in its essential       form?
simple abstractions?
The Joy of Clojure
The Joy of Clojure• Functional • Separation of computation and data
The Joy of Clojure• Functional • Separation of computation and data• Immutable state
The Joy of Clojure• Functional • Separation of computation and data• Immutable state• Data as data
The Joy of Clojure• Functional • Separation of computation and data• Immutable state• Data as data• Time, state and identity
Users as Data?
user = fn(interactions)
fn          fnS0         S1         S2                              User                           (Identity)          Time
User     fn          fnS0         S1         S2                                User                             (Identity)...
data > current state
data outlivesapplications
current state                event processing snapshots
Source: http://www.irregulartimes.com/panopt.html
“It is better to skip the massive up-front time and expense, focusinginstead on making it very fast and easyto add new dat...
data              data ecosystemwarehousing
Data Ecosystem
Producer Producer                Producer            AggregatorConsumer                 Consumer            Consumer
events carrying smaller, simpler pieces of data
{:user-id 1234 :email "paul@forward.co.uk"}                            {:method :GET                             :uri “/” ...
Users as Data
How we’ve used this...
http://github.com/pingles/clj-esper
$ ./print.sh -u <user-id>(->> (events aggregator)     (filter (criteria :user-id "pingles")))
Users as Data
Thanks!
Thanks! @pingles               h irin g!      We ’re
Users as Data
Users as Data
Users as Data
Users as Data
Users as Data
Users as Data
Users as Data
Users as Data
Upcoming SlideShare
Loading in...5
×

Users as Data

1,283

Published on

Published in: Technology, Business
0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
1,283
On Slideshare
0
From Embeds
0
Number of Embeds
4
Actions
Shares
0
Downloads
0
Comments
0
Likes
2
Embeds 0
No embeds

No notes for slide

Users as Data

  1. 1. Users as Data Paul Ingles @pingles
  2. 2. Users as Data
  3. 3. Everything as Data
  4. 4. Code as Data(conj others me)
  5. 5. Data as Data{:name "Paul Ingles" :mood (constantly :happy)}
  6. 6. Data as Data{:name "Paul Ingles" :mood (constantly :hot)}
  7. 7. Builds as Data(defproject energy-comparison "1.0.0" :description "pricing service" :dependencies [[org.clojure/clojure "1.4.0"] ...)
  8. 8. Everything as Data
  9. 9. Everything as Data• Data, in its most essential form
  10. 10. Everything as Data• Data, in its most essential form• Few simple abstractions • map, seq, fn
  11. 11. Everything as Data• Data, in its most essential form• Few simple abstractions • map, seq, fn• General library • map, filter, sort, take
  12. 12. Logs as Data?
  13. 13. Logs as Data?0.0.0.0 - - [22/May/2012:09:00:03 +0000] "GET / HTTP/1.1" 304 0 "SomeUserAgent"
  14. 14. Logs as Files/var/log/nginx/access.log
  15. 15. current power < potential power
  16. 16. log crunching event processing
  17. 17. Users as Data
  18. 18. current power < potential power
  19. 19. Energy Replication, ETL InsuranceCredit Cards Session DB WarehouseBroadband
  20. 20. Data Muddling Energy Replication, ETL InsuranceCredit Cards Session DB WarehouseBroadband
  21. 21. Source: http://www.irregulartimes.com/panopt.html
  22. 22. Energy Insurance Integration DBCredit Cards Broadband
  23. 23. data in its essential form?
  24. 24. simple abstractions?
  25. 25. The Joy of Clojure
  26. 26. The Joy of Clojure• Functional • Separation of computation and data
  27. 27. The Joy of Clojure• Functional • Separation of computation and data• Immutable state
  28. 28. The Joy of Clojure• Functional • Separation of computation and data• Immutable state• Data as data
  29. 29. The Joy of Clojure• Functional • Separation of computation and data• Immutable state• Data as data• Time, state and identity
  30. 30. Users as Data?
  31. 31. user = fn(interactions)
  32. 32. fn fnS0 S1 S2 User (Identity) Time
  33. 33. User fn fnS0 S1 S2 User (Identity) Time
  34. 34. data > current state
  35. 35. data outlivesapplications
  36. 36. current state event processing snapshots
  37. 37. Source: http://www.irregulartimes.com/panopt.html
  38. 38. “It is better to skip the massive up-front time and expense, focusinginstead on making it very fast and easyto add new data sources or newelements to existing sources.” Source: http://thinkrelevance.com/blog/2012/04/04/big-data-reference-model
  39. 39. data data ecosystemwarehousing
  40. 40. Data Ecosystem
  41. 41. Producer Producer Producer AggregatorConsumer Consumer Consumer
  42. 42. events carrying smaller, simpler pieces of data
  43. 43. {:user-id 1234 :email "paul@forward.co.uk"} {:method :GET :uri “/” {:user-id 1234 :user-id 1234} :postcode "NW3 2LB"}{:user-id 1234 :energy-consumption {:gas {:units :kwh :value 20500}}}
  44. 44. Users as Data
  45. 45. How we’ve used this...
  46. 46. http://github.com/pingles/clj-esper
  47. 47. $ ./print.sh -u <user-id>(->> (events aggregator) (filter (criteria :user-id "pingles")))
  48. 48. Users as Data
  49. 49. Thanks!
  50. 50. Thanks! @pingles h irin g! We ’re

×