Interop - Exploring Machine Data

547 views

Published on

Published in: Technology, Business
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
547
On SlideShare
0
From Embeds
0
Number of Embeds
24
Actions
Shares
0
Downloads
23
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Interop - Exploring Machine Data

  1. 1. @michaelwilde, Co-CTO, Splunk ExploringMachine Data
  2. 2. Hi... I work at Splunk.
  3. 3. We stare at data all day.
  4. 4. WTF is Machine Data?!
  5. 5. is it logs?
  6. 6. is it netflow?
  7. 7. is it TWEETS?
  8. 8. Aaaahhh, well... kind of.
  9. 9. a simple way to describethe exhaust from technology *or a big giant pain in the butt.
  10. 10. Machine data is the BIGgest DATA Machine-generated data is one of the GPS, fastest growing, most complex RFID,and most valuable segments of big data Hypervisor, Web Servers, Email, Messaging Clickstreams, Mobile, Telephony, IVR, Databases, Sensors, Telematics, Storage, Servers, Security Devices, Desktops Volume | Velocity | Variety | Variability
  11. 11. no, not uswe’re justnice guyswho wantshow youcool stuff
  12. 12. building a service?you are a producer and consumer of data using an app?
  13. 13. Seth Rabinowitz James Rodmell CEO CTO Location-­‐Based  Messaging   and  Intelligence  For  Your  App   and  Your  Customers
  14. 14. DATE/TIME Data! Good! DEVICE ID2011-11-06 11:57:31,65,00027d27-ae02-627d-a79a-fa0004d3a347,40.75496,-73.963853,602011-11-06 12:17:32,65,00027d27-ae02-627d-a79a-fa0004d3a347,40.755001,-73.963886,702011-11-06 12:37:33,65,00027d27-ae02-627d-a79a-fa0004d3a347,40.754982,-73.963849,75 LAT/LONG2011-11-06 12:57:34,65,00027d27-ae02-627d-a79a-fa0004d3a347,40.754984,-73.963883,852011-11-06 13:17:35,65,00027d27-ae02-627d-a79a-fa0004d3a347,40.754941,-73.9639,902011-11-06 13:37:36,65,00027d27-ae02-627d-a79a-fa0004d3a347,40.754948,-73.963874,902011-11-06 13:57:37,65,00027d27-ae02-627d-a79a-fa0004d3a347,40.754931,-73.963892,95 BATTERY STRENGTH2011-11-06 14:17:38,50,00027d27-ae02-627d-a79a-fa0004d3a347,40.755232,-73.963522,1002011-11-06 14:37:33,65,00027d27-ae02-627d-a79a-fa0004d3a347,40.754979,-73.9639,100
  15. 15. show them something cool already!
  16. 16. Oh, real quick. Did you check in or tweet #splunk #interop ...please
  17. 17. All this data can be pretty cool and empowering
  18. 18. except one littlePROBLEM Text
  19. 19. alot of it looks like this
  20. 20. 0,113/Apr/2011 08:52:53,Info,Teardown,ASA-session-6-302014,TCP,192.168.2.16,192.168.1.6,(empty),(empty),1100,43025,43025_tcp,(empty),0,113/Apr/2011 08:52:55,Info,Teardown,ASA-session-6-302014,TCP,192.168.2.75,192.168.1.6,(empty),(empty),1048,135,epmap,(empty),0,113/Apr/2011 08:52:55,Info,Teardown,ASA-session-6-302014,TCP,192.168.2.75,192.168.1.6,(empty),(empty),1049,43025,43025_tcp,(empty),0,113/Apr/2011 08:52:55,Info,Teardown,ASA-session-6-302014,TCP,192.168.2.75,192.168.1.6,(empty),(empty),1051,135,epmap,(empty),0,113/Apr/2011 08:52:55,Info,Teardown,ASA-session-6-302014,TCP,192.168.2.75,192.168.1.6,(empty),(empty),1052,43025,43025_tcp,(empty),0,113/Apr/2011 08:52:55,Info,Teardown,ASA-session-6-302014,TCP,192.168.2.64,192.168.1.6,(empty),(empty),1694,135,epmap,(empty),
  21. 21. and we’re expected to talk to it like this
  22. 22. select (select max(answer.answer) from answer where answer.member_id in (select member_id from team_members where project_id in ( select project_idfrom project where Business_stream=Upstream and stage=Appraise andproject_id in (select project_id from projectextra where subteam<>1 ) ) ) andanswer.page_id=page.page_id) as thinl, (select max(avgscore) from task_projectwhere task_project.project_id not in (select project_id from projectextrawhere subteam=1 ) and task_project.project_id in (select project_id fromproject where stage=Appraise and Business_stream = Upstream) andtask_project.page_id=page.page_id) as bmax, (select max(answer) from answerwhere answer.page_id=page.page_id) as datamax, (select avg(avgscore) fromtask_project where project_id=1 and task_project.page_id=page.page_id) asprojavg, (select avg(avgscore) from task_project where project_id not in(select project_id from projectextra where subteam=1) andtask_project.page_id=page.page_id) as companyavg, (select avg(avgscore) fromtask_project where project_id not in (select project_id from projectextrawhere subteam=1) and project_id in (select project_id from project whereBusiness_stream = Upstream) and task_project.page_id=page.page_id) asbusinessavg, page.* from page,riverorder where page.category_name=BusinessBoundaries and stage_name=Appraise andriverorder.category_name=page.category_name order byriverorder.riverorder,page.order_id select (select max(answer.answer) fromanswer where answer.member_id in ( select member_id from team_members whereproject_id in ( select project_id from project where
  23. 23. It could be better.yes? better is good!
  24. 24. {[-­‐]    checkin  :  {[-­‐]        badges  :  [],        created  :  1331454784,        geolat  :  "30.2640941786",        geolong  :  "-­‐97.7414819408",        mayor  :  {[-­‐]            type  :  "nochange"        },        primarycategory  :  {[-­‐]            fullpathname  :  "Food:American  Restaurants",            iconurl  :  "https://foursquare.com/img/categories/food/default.png",            id  :  "4bf58dd8d48988d14e941735", Text            nodename  :  "American  Restaurants"        },        timezone  :  "America/Chicago",        user  :  {[-­‐]            gender  :  "male"        },        venue  :  {[-­‐]            id  :  "4d752b1bba682d43e7563876",            name  :  "CNN  Grill  @  SXSW  (Maxs  Wine  Dive)"        }    }} readable, ya think?
  25. 25. failed password | timechart count by client_ip The languages to talk to data are getting better for us humans
  26. 26. Guys.. come on! Goback to the data please.
  27. 27. Need data?a simple way to describe a massive problemA friend in Boulder can help
  28. 28. The Social Media APIJud ValeskiCo-Founder, CEO
  29. 29. Sometimes machine data ishelpful to those OTHER than IT
  30. 30. Someone with a different perspective sees your exhaust as asource of fuel
  31. 31. please, please, please CALL THE VP OF ENGINEERINGat all of your vendors.
  32. 32. DEMAND REALTIME DATAIN A STREAM OVER THE WEB IN JSON FORMAT
  33. 33. Hey audience!We still have a few minutes. What questions might you have been saving untilthis exact moment?
  34. 34. Thanks. @michaelwilde Michael Wilde Splunk Ninja Co-CTO, SplunkWho else sends you on your way with a cute dog photo?

×