Big Data for Everyman

Erik Swan, Michael WildeBig Data for Everyman

Hi... We work at Splunk.

We stare at data all day.

WTF is Big Data?!

larger than small data?

smaller than giant data?

some cool sauce for DBAs?

Aaaahhh, no.

a simple way to describe a massive problem *or opportunity depending on your p.o.v.

Big data comes out of machines GPS, RFID, Hypervisor, Web Servers, Email, Messaging Clickstreams, Mobile, Telephony, IVR, Databases, Sensors, Telematics, Storage, Servers, Security Devices, Desktops Volume | Velocity | Variety | Variability

Big data comes out of machines Machine-generated data is one of the GPS, fastest growing, most complex RFID,and most valuable segments of big data Hypervisor, Web Servers, Email, Messaging Clickstreams, Mobile, Telephony, IVR, Databases, Sensors, Telematics, Storage, Servers, Security Devices, Desktops Volume | Velocity | Variety | Variability

no, not uswe’re justnice guyswho wantshow youcool stuﬀ

building a service?you are a producer and consumer of data using an app?

Seth Rabinowitz James Rodmell CEO CTO Location-‐Based Messaging and Intelligence For Your App and Your Customers

DATE/TIME Data! Good! DEVICE ID2011-11-06 11:57:31,65,00027d27-ae02-627d-a79a-fa0004d3a347,40.75496,-73.963853,602011-11-06 12:17:32,65,00027d27-ae02-627d-a79a-fa0004d3a347,40.755001,-73.963886,702011-11-06 12:37:33,65,00027d27-ae02-627d-a79a-fa0004d3a347,40.754982,-73.963849,75 LAT/LONG2011-11-06 12:57:34,65,00027d27-ae02-627d-a79a-fa0004d3a347,40.754984,-73.963883,852011-11-06 13:17:35,65,00027d27-ae02-627d-a79a-fa0004d3a347,40.754941,-73.9639,902011-11-06 13:37:36,65,00027d27-ae02-627d-a79a-fa0004d3a347,40.754948,-73.963874,902011-11-06 13:57:37,65,00027d27-ae02-627d-a79a-fa0004d3a347,40.754931,-73.963892,95 BATTERY STRENGTH2011-11-06 14:17:38,50,00027d27-ae02-627d-a79a-fa0004d3a347,40.755232,-73.963522,1002011-11-06 14:37:33,65,00027d27-ae02-627d-a79a-fa0004d3a347,40.754979,-73.9639,100

show them something cool already!

Oh, real quick. Did you check in or tweet #splunk #sxsw ...please

All this data can be pretty cool and empowering

except one littlePROBLEM Text

alot of it looks like this

0,113/Apr/2011 08:52:53,Info,Teardown,ASA-session-6-302014,TCP,192.168.2.16,192.168.1.6,(empty),(empty),1100,43025,43025_tcp,(empty),0,113/Apr/2011 08:52:55,Info,Teardown,ASA-session-6-302014,TCP,192.168.2.75,192.168.1.6,(empty),(empty),1048,135,epmap,(empty),0,113/Apr/2011 08:52:55,Info,Teardown,ASA-session-6-302014,TCP,192.168.2.75,192.168.1.6,(empty),(empty),1049,43025,43025_tcp,(empty),0,113/Apr/2011 08:52:55,Info,Teardown,ASA-session-6-302014,TCP,192.168.2.75,192.168.1.6,(empty),(empty),1051,135,epmap,(empty),0,113/Apr/2011 08:52:55,Info,Teardown,ASA-session-6-302014,TCP,192.168.2.75,192.168.1.6,(empty),(empty),1052,43025,43025_tcp,(empty),0,113/Apr/2011 08:52:55,Info,Teardown,ASA-session-6-302014,TCP,192.168.2.64,192.168.1.6,(empty),(empty),1694,135,epmap,(empty),

and we’re expected to talk to it like this

select (select max(answer.answer) from answer where answer.member_id in (select member_id from team_members where project_id in ( select project_idfrom project where Business_stream=Upstream and stage=Appraise andproject_id in (select project_id from projectextra where subteam<>1 ) ) ) andanswer.page_id=page.page_id) as thinl, (select max(avgscore) from task_projectwhere task_project.project_id not in (select project_id from projectextrawhere subteam=1 ) and task_project.project_id in (select project_id fromproject where stage=Appraise and Business_stream = Upstream) andtask_project.page_id=page.page_id) as bmax, (select max(answer) from answerwhere answer.page_id=page.page_id) as datamax, (select avg(avgscore) fromtask_project where project_id=1 and task_project.page_id=page.page_id) asprojavg, (select avg(avgscore) from task_project where project_id not in(select project_id from projectextra where subteam=1) andtask_project.page_id=page.page_id) as companyavg, (select avg(avgscore) fromtask_project where project_id not in (select project_id from projectextrawhere subteam=1) and project_id in (select project_id from project whereBusiness_stream = Upstream) and task_project.page_id=page.page_id) asbusinessavg, page.* from page,riverorder where page.category_name=BusinessBoundaries and stage_name=Appraise andriverorder.category_name=page.category_name order byriverorder.riverorder,page.order_id select (select max(answer.answer) fromanswer where answer.member_id in ( select member_id from team_members whereproject_id in ( select project_id from project where

It could be better.yes? better is good!

{[-‐] checkin : {[-‐] badges : [], created : 1331454784, geolat : "30.2640941786", geolong : "-‐97.7414819408", mayor : {[-‐] type : "nochange" }, primarycategory : {[-‐] fullpathname : "Food:American Restaurants", iconurl : "https://foursquare.com/img/categories/food/default.png", id : "4bf58dd8d48988d14e941735", Text nodename : "American Restaurants" }, timezone : "America/Chicago", user : {[-‐] gender : "male" }, venue : {[-‐] id : "4d752b1bba682d43e7563876", name : "CNN Grill @ SXSW (Maxs Wine Dive)" } }} readable, ya think?

source=foursquare | timechart count by checkin.venue.name The languages to talk to data are getting better for us humans

Guys.. come on! Goback to the data please.

Need data?a simple way to describe a massive problemA friend in Boulder can help

The Social Media APIJud ValeskiCo-Founder, CEO

Just when you think you’re all done, wait. There is another consumer you may have forgotten

Someone with a diﬀerent perspective sees your service asinput to theirs

DEMAND REALTIME DATAIN A STREAM OVER THE WEB IN JSON FORMAT

Hey audience!We still have a few minutes. What questions might you have been saving untilthis exact moment?

Thanks. Erik Swan, CTO Co-Founder, Splunk Michael Wilde Splunk NinjaWho else sends you on your way with a cute dog photo?

Big Data for Everyman

by Michael Wilde

Accessibility

Categories

Upload Details

Usage Rights

2 Embeds 56

Statistics

Big Data for Everyman Presentation Transcript

http://lanyrd.com	54
http://us-w1.rockmelt.com	2