Your SlideShare is downloading. ×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×

Introducing the official SlideShare app

Stunning, full-screen experience for iPhone and Android

Text the download link to your phone

Standard text messaging rates apply

SplunkLive! Customer Presentation - Satcom Direct

581
views

Published on

Published in: Technology, Business

0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
581
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
0
Comments
0
Likes
1
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide
  • Graphics – split into 2 slides.Story: Takes support team 2-3 systems to dig though the data and figure out if they are connected, etc…
  • Transcript

    • 1. Copyright © 2013 Splunk Inc. Splunk… on a Plane? Ken Bantoft VP Satcom Technology & Development Satcom Direct
    • 2. About Me Spent 12 years doing Networking, Linux, High Performance Computing in Finance, Bio-Technology and other sectors Left IT in 2007 to focus on product development Did a 1 week contract fixing Avionics Networking code, and haven’t left Aviation since. Now responsible for Product & Services Development at Satcom Direct
    • 3. About My Company Satcom Direct provides connectivity and communications for Aviation, Maritime and Land Mobile customers. Built around a core focus of support and service, we now serve thousands of customers world wide, including the Fortune 500, NATO & Allied Forces, and various Heads of State.
    • 4. Agenda Splunk – not really on a plane (yet) Data Sources How we use Splunk – Support – Monitoring & Alerting – Business Analytics Tracking Planes – The technican’s flight tracker Splunk Tips
    • 5. Copyright © 2013 Splunk Inc. Data Sources
    • 6. Data Sources We feed Splunk pretty much anything we can get our hands on, both standard IT data, and some more esoteric data – – – – – CDRs for Phone Calls (AudioCodes, Asterisk) Syslogs from network appliances & servers Radius accounting data Logs from Satcom Systems (via email, or mobile apps) Aircraft Position + Status Reports We normalize Aircraft Position reports before feeding them to Splunk – Fields are extremely complex, often missing, sometimes delayed, and come from at least 5 different sources. And they are all totally inconsistent.
    • 7. Data Sources - AudioCodes Max-Forwards: 70 User-Agent: AeroV-Gateway CSeq: 102 OPTIONS Call-ID: 66bac96862403ef05c1aac9922e3d3d2@63.###.###.238 Contact: <sip:AeroV-Gateway@63.###.###.238> To: <sip:63.###.###.241> From: "AeroV-Gateway" <sip:AeroV-Gateway@63.###.###.238>;tag=as7a930744 Via: SIP/2.0/UDP 63.###.###.238:5060;branch=z9hG4bK47c1eef2;rport Sep 14 14:50:02 63.###.###.241 OPTIONS sip:63.###.###.241 SIP/2.0 Sep 14 14:50:02 63.###.###.241 ( lgr_flow)(658474 ) ---- Incoming SIP Message from 63.###.###.238:5060 to SIPInterface #0 UdpTransportObject[#3343] --- [Time: 09-14-2013@14:50:02] Sep 14 14:50:02 63.###.###.241 ( sip_stack)(658473 ) New SIPMessage created - #15 [Time: 09-14-2013@14:50:02] Sep 14 14:49:58 63.###.###.241 ( sip_stack)(658472 ) SIPDialog(#138) changes state from DialogDisconnected to DialogIdle [Time: 09-14-2013@14:49:58] Sep 14 14:49:58 63.###.###.241 ( lgr_flow)(658471 ) | | TransactionUserMngr::ReturnDialog - #138 [Time: 09-14-2013@14:49:58] Sep 14 14:49:53 63.###.###.241 ( sip_stack)(658470 ) Resource SIPMessage deleted - #12 [Time: 09-14-2013@14:49:53] Sep 14 14:49:53 63.###.###.241 ( sip_stack)(658469 ) AcSIPStackAPI::FreeDialogAPI - #34 [Time: 09-14-2013@14:49:53] Sep 14 14:49:53 63.###.###.241 ( sip_stack)(658468 ) SIPDialog(#138) changes state from DialogConnected to DialogDisconnected [Time: 09-142013@14:49:53] Sep 14 14:49:53 63.###.###.241 ( lgr_flow)(658467 ) | |(SIPTU#138)DIALOG_DISCONNECT_REQ State:DialogConnected(370678c35bed1a1c1d2f36a20e0b0fd0@63.###.###.248) [Time: 09-14-2013@14:49:53] Sep 14 14:49:53 63.###.###.241 ( sip_stack)(658466 ) Resource SIPMessage deleted - #70 [Time: 09-14-2013@14:49:53]
    • 8. Data Sources - Expand Sep 14 15:53:07 63.###.###.210 accelerator[4142]: Link ID 115 was Updated Sep 14 15:53:07 63.###.###.210 accelerator[4142]: Link 10.###.###.66 status changed from negotiating to accelerating Sep 14 15:53:07 63.###.###.210 accelerator[4142]: Acceleration enable to peer-10.###.###.66:0, with decore size - 4194304 Sep 14 15:53:07 63.###.###.210 accelerator[4142]: Acceleration enable to peer- 10.###.###.66:0, with core size - 4194304 Sep 14 15:53:07 63.###.###.210 accelerator[4142]: Link ID 115 was Updated Sep 14 15:53:00 63.###.###.210 accelerator[4142]: Link ID 115 was Updated Sep 14 15:53:00 63.###.###.210 accelerator[4142]: Link 10.###.###.66 status changed from drop to negotiating Sep 14 15:53:00 63.###.###.210 accelerator[4142]: Link ID 115 was Updated Sep 14 15:53:00 63.###.###.210 accelerator[4142]: Subnets for Remote link CP Id 115 changed Sep 14 15:53:00 63.###.###.210 accelerator[4142]: Link ID 115 was Updated Sep 14 15:52:34 63.###.###.210 accelerator[4142]: Link ID 115 was Updated Sep 14 15:52:34 63.###.###.210 accelerator[4142]: Link 10.###.###.66 status changed from accelerating to drop Sep 14 15:52:34 63.###.###.210 accelerator[4142]: Update peer failed with code 22. Sep 14 15:52:34 63.###.###.210 accelerator[4142]: Link ID 103 was Updated
    • 9. Copyright © 2013 Splunk Inc. Monitoring & Alerting
    • 10. Support: Monitoring and Alerting • Splunk provides a real-time dashboard in our NOC about the status of several key services • Previously, support techs would need to login to 3-5 different systems to look for faults or errors. Each system had a different UI, different formats and different data. Techs learned, but over long periods as errors were often infrequent and obscure • Now data is in one system, one interface, with intelligence ‘coded in’ by our senior techs
    • 11. Support: Monitoring and Alerting • We merge log data with our Configuration Management database so we can display aircraft Tail Numbers, Phone Numbers and relevant data directly on the dashboard. – Allows our support team to see customers as their aircraft logon to the satellites and move data or make voice calls – Support techs can verify while still on the phone with the customer (data is ~60-90 seconds delayed) CSA Data Entry CM Servers Feed Splunk CSV tables for Lookups indexer
    • 12. Support: Monitoring and Alerting • We can be proactive – Splunk alerting allows us to capture issues immediately – customers unable to connect (incorrect passwords, or invalid settings). We know we’ll get a call, or we can call the customer directly.
    • 13. Support: Monitoring & Alerting • .conf 2013 Stump the Experts Report – counting in-flight (Literally!) transactions over time to gauge volumes
    • 14. Support: Monitoring and Alerting • Alerts help capture out of the ordinary situations • More that # occurrences in a given timespan alerts take 60 seconds to setup – use them • Now when something spirals out of control, you’ll know!
    • 15. Copyright © 2013 Splunk Inc. Business Analytics
    • 16. Business Analytics • We’ve always been a data driven organization – we focus heavily on configuration management for customer avionics • Using Splunk to analyze the data helps us make smart decisions • Each time we deep dive into the data, we learn new things
    • 17. Business Analytics • We used Splunk to determine how to size our new DNS infrastructure • Fed DNS stats (Bind + script + syslog) into Splunk for a few weeks, visualized the results and then were able to do capacity planning
    • 18. Business Analytics – VoIP Call Rates • We can monitor the Country Codes dialed for our Satellite Voice calls in aggregate, so we know what countries our customers call most often. We then push our telecom & VoIP providers to negotiate better rates. • Splunk tells us what countries we need to focus on, so we ignore the long rate cards and get right down to the ones we care about.
    • 19. Business Analytics – VoIP Call Rates • We can then route outbound calls based on destination country code to a different provider, reducing our direct cost per second for call terminations
    • 20. Copyright © 2013 Splunk Inc. Flight Tracking
    • 21. Flight Tracking Where the plane is coming or going isn’t what is important Common problems with Satellite communications are handovers – where you change which satellite you are talking to while in flight Historically it’s hard to correlate events with location visually Google Earth/Google Maps were a major leap, but not automated Enter Splunk w/Google Maps plugin – now we can put all the data in a consistent visual format.
    • 22. Flight Tracking Data FAA ASDI users Other Apps Sat. Provider 1 FT Server Process & Normalize All Data Sat. Provider 2 Satcom Terminal forwarder indexer
    • 23. FAA ASDI Data <trackInformation><nxcm:aircraftId>ACA117</nxcm:aircraftId><nxcm:speed>280</nxcm:speed><nxcm:reportedAltitude><nxce:assignedAltitude><nxce:simpleAltitud e>103</nxce:simpleAltitude></nxce:assignedAltitude></nxcm:reportedAltitude><nxcm:position><nxce:latitude><nxce:latitudeDMS degrees="43" minutes="51" direction="NORTH"/></nxce:latitude><nxce:longitude><nxce:longitudeDMS degrees="079" minutes="50" direction="WEST"/></nxce:longitude></nxcm:position></trackInformation></asdiMessage><asdiMessage sourceFacility="CCZY" sourceTimeStamp="2009-0921T12:34:31Z" trigger="TZ"><trackInformation><nxcm:aircraftId>MES3455</nxcm:aircraftId><nxcm:speed>400</nxcm:speed><nxcm:reportedAltitude><nxce:assignedAltitude><nxc e:simpleAltitude>360</nxce:simpleAltitude></nxce:assignedAltitude></nxcm:reportedAltitude><nxcm:position><nxce:latitude><nxce:latitudeDMS degrees="42" minutes="12" direction="NORTH"/></nxce:latitude><nxce:longitude><nxce:longitudeDMS degrees="076" minutes="16" direction="WEST"/></nxce:longitude></nxcm:position></trackInformation></asdiMessage><asdiMessage sourceFacility="CCZW" sourceTimeStamp="2009-0921T12:34:31Z" trigger="TZ"><trackInformation><nxcm:aircraftId>ACA114</nxcm:aircraftId><nxcm:speed>440</nxcm:speed><nxcm:reportedAltitude><nxce:assignedAltitude><nxce: simpleAltitude>262</nxce:simpleAltitude></nxce:assignedAltitude></nxcm:reportedAltitude><nxcm:position><nxce:latitude><nxce:latitudeDMS degrees="53" minutes="10" direction="NORTH"/></nxce:latitude><nxce:longitude><nxce:longitudeDMS degrees="111" minutes="54" direction="WEST"/></nxce:longitude></nxcm:position></trackInformation></asdiMessage><asdiMessage sourceFacility="CCZY" sourceTimeStamp="2009-0921T12:34:32Z" trigger="TZ"><trackInformation><nxcm:aircraftId>UAL801</nxcm:aircraftId><nxcm:speed>440</nxcm:speed><nxcm:reportedAltitude><nxce:assignedAltitude><nxce: simpleAltitude>340</nxce:simpleAltitude></nxce:assignedAltitude></nxcm:reportedAltitude><nxcm:position><nxce:latitude><nxce:latitudeDMS degrees="42" minutes="59" direction="NORTH"/></nxce:latitude><nxce:longitude><nxce:longitudeDMS degrees="082" minutes="52" direction="WEST"/></nxce:longitude></nxcm:position></trackInformation></asdiMessage><asdiMessage sourceFacility="CCZY" sourceTimeStamp="2009-0921T12:34:32Z" trigger="TZ"><trackInformation><nxcm:aircraftId>EJA802</nxcm:aircraftId><nxcm:speed>370</nxcm:speed><nxcm:reportedAltitude><nxce:assignedAltitude><nxce: simpleAltitude>400</nxce:simpleAltitude></nxce:assignedAltitude></nxcm:reportedAltitude><nxcm:position><nxce:latitude><nxce:latitudeDMS degrees="42" minutes="15" direction="NORTH"/></nxce:latitude><nxce:longitude><nxce:longitudeDMS degrees="078" minutes="52" direction="WEST"/></nxce:longitude></nxcm:position></trackInformation></asdiMessage><asdiMessage sourceFacility="CCZW" sourceTimeStamp="2009-0921T12:34:32Z" trigger="UZ"><boundaryCrossingUpdate><nxcm:aircraftId>PAG113</nxcm:aircraftId><nxcm:flightAircraftSpecs specialAircraftQualifier="B757_TCAS" equipmentQualifier="G">BE99</nxcm:flightAircraftSpecs> * http://www.fly.faa.gov/ASDI/asdi.html
    • 24. Flight Tracker – Post Normalization TimeOfReport 9/8/13 20:21 9/8/13 20:20 9/8/13 20:19 9/8/13 20:19 9/8/13 20:18 9/8/13 20:18 9/8/13 20:17 9/8/13 20:17 9/8/13 20:17 9/8/13 20:16 9/8/13 20:07 9/8/13 19:57 9/8/13 19:47 9/8/13 17:21 9/6/13 19:59 9/6/13 19:49 9/6/13 19:41 Source FaaAsdiFAA FaaAsdiFAA FaaAsdiFAA FaaAsdiFAA FaaAsdiFAA SbbGps FaaAsdiFAA FlightDeckFusion FaaAsdiFAA FaaAsdiFAA FlightDeckFusion FlightDeckFusion SbbGps SbbGps FlightDeckFusion FlightDeckFusion SbbGps Received 9/8/13 20:26 9/8/13 20:25 9/8/13 20:24 9/8/13 20:24 9/8/13 20:23 9/8/13 20:22 9/8/13 20:23 9/8/13 20:17 9/8/13 20:22 9/8/13 20:21 9/8/13 20:07 9/8/13 19:57 9/8/13 19:52 9/8/13 17:28 9/6/13 19:59 9/6/13 19:49 9/6/13 19:47 MessageId Latitude Longitude Altitude Heading FaaAsdi132839420 35.8889 -115.0775 15100 FaaAsdi132839201 35.8986 -115.1664 11800 FaaAsdi132839013 35.9114 -115.2625 9200 FaaAsdi132838985 35.9264 -115.2839 8600 FaaAsdi132838854 35.9797 -115.2719 7200 SbbGps20130908201801000000N651SD 35.9907 -115.253 FaaAsdi132838737 35.9942 -115.2483 7000 SD20130908201716976007N651SD 36.02 -115.2 5900 FaaAsdi132838595 36.0314 -115.1908 5300 FaaAsdi132838463 36.0681 -115.1708 3100 SD20130908200716316162N651SD 36.0967 -115.1517 2000 SD20130908195716125081N651SD 36.0983 -115.16 2000 SbbGps20130908194757000000N651SD 36.0997 -115.1603 SbbGps20130908172106000000N651SD 36.0995 -115.1603 SD20130906195946601934N651SD 36.1 -115.1583 2100 SD20130906194946395228N651SD 36.0983 -115.1583 2100 SbbGps20130906194144000000N651SD 36.0999 -115.1595 Speed 272 285 284 295 272 246 218 204 195 14 0 0 0
    • 25. Flight Tracking
    • 26. Copyright © 2013 Splunk Inc. Splunk Tips
    • 27. Transactions Insanely powerful for gathering statistics. tag="Expand" "status changed" |rex "s.*?Links(?<AircraftIP>S+)" |transaction AircraftIP State startswith="negotiating to accelerating" endswith="accelerating to drop" Sep 14 15:53:07 63.###.###.210 accelerator[4142]: Link ID 115 was Updated Sep 14 15:53:07 63.###.###.210 accelerator[4142]: Link 10.###.###.66 status changed from negotiating to accelerating Sep 14 15:53:07 63.###.###.210 accelerator[4142]: Acceleration enable to peer-10.###.###.66:0, with decore size - 4194304 Sep 14 15:53:07 63.###.###.210 accelerator[4142]: Acceleration enable to peer- 10.###.###.66:0, with core size - 4194304 Sep 14 15:53:07 63.###.###.210 accelerator[4142]: Link ID 115 was Updated Sep 14 15:53:00 63.###.###.210 accelerator[4142]: Link ID 115 was Updated Sep 14 15:53:00 63.###.###.210 accelerator[4142]: Link 10.###.###.66 status changed from drop to negotiating Sep 14 15:53:00 63.###.###.210 accelerator[4142]: Link ID 115 was Updated Sep 14 15:53:00 63.###.###.210 accelerator[4142]: Subnets for Remote link CP Id 115 changed Sep 14 15:53:00 63.###.###.210 accelerator[4142]: Link ID 115 was Updated Sep 14 15:52:34 63.###.###.210 accelerator[4142]: Link ID 115 was Updated Sep 14 15:52:34 63.###.###.210 accelerator[4142]: Link 10.###.###.66 status changed from accelerating to drop Sep 14 15:52:34 63.###.###.210 accelerator[4142]: Update peer failed with code 22. Sep 14 15:52:34 63.###.###.210 accelerator[4142]: Link ID 103 was Updated
    • 28. Transactions Run against a few hours of data, and we see lots of transactions occurring. So we know how long each Aircraft is ‘in session’ for.
    • 29. Transactions Now what? Let’s do some math and get some stats! tag="Expand" "status changed" |rex "s.*?Links(?<AircraftIP>S+)" |transaction AircraftIP State startswith="negotiating to accelerating" endswith="accelerating to drop" | eval ConnectedFor(Mins)=round(duration/60) | lookup taillookup ip as AircraftIP OUTPUT subnet_name as Tail|stats sum(ConnectedFor(Mins)) as TimeOnline by Tail| sort TimeOnline
    • 30. Transaction - Visualizations Once you have the data, visualizations on the dashboard allow us to know at a glance if a service is performing within limits We adjust the gauge colors – in this case, higher is better
    • 31. Don’t Fear CSV KISS – and CSV is certainly that Great for mapping things like IP/Subnets to Customers Easier to manipulate text files to clean them up Great for things that don’t change too often # Sort by IP address so searches are easier sort -t . -k 1,1n -k 2,2n -k 3,3n -k 4,4n ip-customers.in > ip-customers.csv cp ip-customers.csv /opt/splunk/etc/system/lookups/ip-customers.csv CIDR Lookup Scripts: http://answers.splunk.com/answers/5916/using-cidr-in-a-lookup-table
    • 32. Summary Alerting based on frequency of events within a timeframe can be extremely powerful to detect anomalies Sometimes you need to clean up your data before you send it into Splunk – Garbage in, garbage out Adding external lookups can be as simple as CSV files – don’t overthink it ’transaction’ helps make sense of time & duration based data Use Splunk to guide your choices with real data – embrace Empiricism to make good business decisions
    • 33. Q & A Time
    • 34. Copyright © 2013 Splunk Inc. Thank You!