Copyright © 2013 Splunk Inc.

Splunk… on a Plane?
Ken Bantoft
VP Satcom Technology & Development
Satcom Direct
About Me
Spent 12 years doing Networking, Linux, High Performance
Computing in Finance, Bio-Technology and other sectors
L...
About My Company
Satcom Direct provides connectivity and communications for
Aviation, Maritime and Land Mobile customers. ...
Agenda
Splunk – not really on a plane (yet)
Data Sources
How we use Splunk
– Support – Monitoring & Alerting
– Business An...
Copyright © 2013 Splunk Inc.

Data Sources
Data Sources
We feed Splunk pretty much anything we can get our hands
on, both standard IT data, and some more esoteric da...
Data Sources - AudioCodes
Max-Forwards: 70
User-Agent: AeroV-Gateway
CSeq: 102 OPTIONS
Call-ID: 66bac96862403ef05c1aac9922...
Data Sources - Expand
Sep 14 15:53:07 63.###.###.210 accelerator[4142]: Link ID 115 was Updated
Sep 14 15:53:07 63.###.###...
Copyright © 2013 Splunk Inc.

Monitoring & Alerting
Support: Monitoring and Alerting
•

Splunk provides a real-time dashboard in our NOC about the status
of several key servi...
Support: Monitoring and Alerting
•

We merge log data with our Configuration Management database
so we can display aircraf...
Support: Monitoring and Alerting
•

We can be proactive – Splunk alerting allows us to capture issues
immediately – custom...
Support: Monitoring & Alerting
•

.conf 2013 Stump the Experts Report – counting in-flight (Literally!)
transactions over ...
Support: Monitoring and Alerting
•

Alerts help capture out of the ordinary situations

•

More that # occurrences in a gi...
Copyright © 2013 Splunk Inc.

Business Analytics
Business Analytics
•

We’ve always been a data driven organization – we focus heavily on
configuration management for cust...
Business Analytics
•

We used Splunk to determine how to size our new DNS
infrastructure

•

Fed DNS stats (Bind + script ...
Business Analytics – VoIP Call Rates
•

We can monitor the Country Codes dialed for our Satellite Voice
calls in aggregate...
Business Analytics – VoIP Call Rates

•

We can then route outbound calls based on destination country
code to a different...
Copyright © 2013 Splunk Inc.

Flight Tracking
Flight Tracking
Where the plane is coming or going isn’t what is important
Common problems with Satellite communications a...
Flight Tracking Data
FAA ASDI

users
Other Apps

Sat. Provider 1
FT Server
Process & Normalize All Data
Sat. Provider 2

S...
FAA ASDI Data
<trackInformation><nxcm:aircraftId>ACA117</nxcm:aircraftId><nxcm:speed>280</nxcm:speed><nxcm:reportedAltitud...
Flight Tracker – Post Normalization
TimeOfReport
9/8/13 20:21
9/8/13 20:20
9/8/13 20:19
9/8/13 20:19
9/8/13 20:18
9/8/13 2...
Flight Tracking
Copyright © 2013 Splunk Inc.

Splunk Tips
Transactions
Insanely powerful for gathering statistics.
tag="Expand" "status changed" |rex "s.*?Links(?<AircraftIP>S+)" |...
Transactions
Run against a few hours of data, and we see lots of transactions
occurring. So we know how long each Aircraft...
Transactions
Now what? Let’s do some math and get some stats!

tag="Expand" "status changed" |rex
"s.*?Links(?<AircraftIP>...
Transaction - Visualizations
Once you have the data, visualizations on the dashboard allow us
to know at a glance if a ser...
Don’t Fear CSV
KISS – and CSV is certainly that
Great for mapping things like IP/Subnets to Customers
Easier to manipulate...
Summary
Alerting based on frequency of events within a timeframe can be
extremely powerful to detect anomalies
Sometimes y...
Q & A Time
Copyright © 2013 Splunk Inc.

Thank You!
Upcoming SlideShare
Loading in...5
×

SplunkLive! Customer Presentation - Satcom Direct

721

Published on

Published in: Technology, Business
0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
721
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
0
Comments
0
Likes
2
Embeds 0
No embeds

No notes for slide
  • Graphics – split into 2 slides.Story: Takes support team 2-3 systems to dig though the data and figure out if they are connected, etc…
  • Transcript of "SplunkLive! Customer Presentation - Satcom Direct"

    1. 1. Copyright © 2013 Splunk Inc. Splunk… on a Plane? Ken Bantoft VP Satcom Technology & Development Satcom Direct
    2. 2. About Me Spent 12 years doing Networking, Linux, High Performance Computing in Finance, Bio-Technology and other sectors Left IT in 2007 to focus on product development Did a 1 week contract fixing Avionics Networking code, and haven’t left Aviation since. Now responsible for Product & Services Development at Satcom Direct
    3. 3. About My Company Satcom Direct provides connectivity and communications for Aviation, Maritime and Land Mobile customers. Built around a core focus of support and service, we now serve thousands of customers world wide, including the Fortune 500, NATO & Allied Forces, and various Heads of State.
    4. 4. Agenda Splunk – not really on a plane (yet) Data Sources How we use Splunk – Support – Monitoring & Alerting – Business Analytics Tracking Planes – The technican’s flight tracker Splunk Tips
    5. 5. Copyright © 2013 Splunk Inc. Data Sources
    6. 6. Data Sources We feed Splunk pretty much anything we can get our hands on, both standard IT data, and some more esoteric data – – – – – CDRs for Phone Calls (AudioCodes, Asterisk) Syslogs from network appliances & servers Radius accounting data Logs from Satcom Systems (via email, or mobile apps) Aircraft Position + Status Reports We normalize Aircraft Position reports before feeding them to Splunk – Fields are extremely complex, often missing, sometimes delayed, and come from at least 5 different sources. And they are all totally inconsistent.
    7. 7. Data Sources - AudioCodes Max-Forwards: 70 User-Agent: AeroV-Gateway CSeq: 102 OPTIONS Call-ID: 66bac96862403ef05c1aac9922e3d3d2@63.###.###.238 Contact: <sip:AeroV-Gateway@63.###.###.238> To: <sip:63.###.###.241> From: "AeroV-Gateway" <sip:AeroV-Gateway@63.###.###.238>;tag=as7a930744 Via: SIP/2.0/UDP 63.###.###.238:5060;branch=z9hG4bK47c1eef2;rport Sep 14 14:50:02 63.###.###.241 OPTIONS sip:63.###.###.241 SIP/2.0 Sep 14 14:50:02 63.###.###.241 ( lgr_flow)(658474 ) ---- Incoming SIP Message from 63.###.###.238:5060 to SIPInterface #0 UdpTransportObject[#3343] --- [Time: 09-14-2013@14:50:02] Sep 14 14:50:02 63.###.###.241 ( sip_stack)(658473 ) New SIPMessage created - #15 [Time: 09-14-2013@14:50:02] Sep 14 14:49:58 63.###.###.241 ( sip_stack)(658472 ) SIPDialog(#138) changes state from DialogDisconnected to DialogIdle [Time: 09-14-2013@14:49:58] Sep 14 14:49:58 63.###.###.241 ( lgr_flow)(658471 ) | | TransactionUserMngr::ReturnDialog - #138 [Time: 09-14-2013@14:49:58] Sep 14 14:49:53 63.###.###.241 ( sip_stack)(658470 ) Resource SIPMessage deleted - #12 [Time: 09-14-2013@14:49:53] Sep 14 14:49:53 63.###.###.241 ( sip_stack)(658469 ) AcSIPStackAPI::FreeDialogAPI - #34 [Time: 09-14-2013@14:49:53] Sep 14 14:49:53 63.###.###.241 ( sip_stack)(658468 ) SIPDialog(#138) changes state from DialogConnected to DialogDisconnected [Time: 09-142013@14:49:53] Sep 14 14:49:53 63.###.###.241 ( lgr_flow)(658467 ) | |(SIPTU#138)DIALOG_DISCONNECT_REQ State:DialogConnected(370678c35bed1a1c1d2f36a20e0b0fd0@63.###.###.248) [Time: 09-14-2013@14:49:53] Sep 14 14:49:53 63.###.###.241 ( sip_stack)(658466 ) Resource SIPMessage deleted - #70 [Time: 09-14-2013@14:49:53]
    8. 8. Data Sources - Expand Sep 14 15:53:07 63.###.###.210 accelerator[4142]: Link ID 115 was Updated Sep 14 15:53:07 63.###.###.210 accelerator[4142]: Link 10.###.###.66 status changed from negotiating to accelerating Sep 14 15:53:07 63.###.###.210 accelerator[4142]: Acceleration enable to peer-10.###.###.66:0, with decore size - 4194304 Sep 14 15:53:07 63.###.###.210 accelerator[4142]: Acceleration enable to peer- 10.###.###.66:0, with core size - 4194304 Sep 14 15:53:07 63.###.###.210 accelerator[4142]: Link ID 115 was Updated Sep 14 15:53:00 63.###.###.210 accelerator[4142]: Link ID 115 was Updated Sep 14 15:53:00 63.###.###.210 accelerator[4142]: Link 10.###.###.66 status changed from drop to negotiating Sep 14 15:53:00 63.###.###.210 accelerator[4142]: Link ID 115 was Updated Sep 14 15:53:00 63.###.###.210 accelerator[4142]: Subnets for Remote link CP Id 115 changed Sep 14 15:53:00 63.###.###.210 accelerator[4142]: Link ID 115 was Updated Sep 14 15:52:34 63.###.###.210 accelerator[4142]: Link ID 115 was Updated Sep 14 15:52:34 63.###.###.210 accelerator[4142]: Link 10.###.###.66 status changed from accelerating to drop Sep 14 15:52:34 63.###.###.210 accelerator[4142]: Update peer failed with code 22. Sep 14 15:52:34 63.###.###.210 accelerator[4142]: Link ID 103 was Updated
    9. 9. Copyright © 2013 Splunk Inc. Monitoring & Alerting
    10. 10. Support: Monitoring and Alerting • Splunk provides a real-time dashboard in our NOC about the status of several key services • Previously, support techs would need to login to 3-5 different systems to look for faults or errors. Each system had a different UI, different formats and different data. Techs learned, but over long periods as errors were often infrequent and obscure • Now data is in one system, one interface, with intelligence ‘coded in’ by our senior techs
    11. 11. Support: Monitoring and Alerting • We merge log data with our Configuration Management database so we can display aircraft Tail Numbers, Phone Numbers and relevant data directly on the dashboard. – Allows our support team to see customers as their aircraft logon to the satellites and move data or make voice calls – Support techs can verify while still on the phone with the customer (data is ~60-90 seconds delayed) CSA Data Entry CM Servers Feed Splunk CSV tables for Lookups indexer
    12. 12. Support: Monitoring and Alerting • We can be proactive – Splunk alerting allows us to capture issues immediately – customers unable to connect (incorrect passwords, or invalid settings). We know we’ll get a call, or we can call the customer directly.
    13. 13. Support: Monitoring & Alerting • .conf 2013 Stump the Experts Report – counting in-flight (Literally!) transactions over time to gauge volumes
    14. 14. Support: Monitoring and Alerting • Alerts help capture out of the ordinary situations • More that # occurrences in a given timespan alerts take 60 seconds to setup – use them • Now when something spirals out of control, you’ll know!
    15. 15. Copyright © 2013 Splunk Inc. Business Analytics
    16. 16. Business Analytics • We’ve always been a data driven organization – we focus heavily on configuration management for customer avionics • Using Splunk to analyze the data helps us make smart decisions • Each time we deep dive into the data, we learn new things
    17. 17. Business Analytics • We used Splunk to determine how to size our new DNS infrastructure • Fed DNS stats (Bind + script + syslog) into Splunk for a few weeks, visualized the results and then were able to do capacity planning
    18. 18. Business Analytics – VoIP Call Rates • We can monitor the Country Codes dialed for our Satellite Voice calls in aggregate, so we know what countries our customers call most often. We then push our telecom & VoIP providers to negotiate better rates. • Splunk tells us what countries we need to focus on, so we ignore the long rate cards and get right down to the ones we care about.
    19. 19. Business Analytics – VoIP Call Rates • We can then route outbound calls based on destination country code to a different provider, reducing our direct cost per second for call terminations
    20. 20. Copyright © 2013 Splunk Inc. Flight Tracking
    21. 21. Flight Tracking Where the plane is coming or going isn’t what is important Common problems with Satellite communications are handovers – where you change which satellite you are talking to while in flight Historically it’s hard to correlate events with location visually Google Earth/Google Maps were a major leap, but not automated Enter Splunk w/Google Maps plugin – now we can put all the data in a consistent visual format.
    22. 22. Flight Tracking Data FAA ASDI users Other Apps Sat. Provider 1 FT Server Process & Normalize All Data Sat. Provider 2 Satcom Terminal forwarder indexer
    23. 23. FAA ASDI Data <trackInformation><nxcm:aircraftId>ACA117</nxcm:aircraftId><nxcm:speed>280</nxcm:speed><nxcm:reportedAltitude><nxce:assignedAltitude><nxce:simpleAltitud e>103</nxce:simpleAltitude></nxce:assignedAltitude></nxcm:reportedAltitude><nxcm:position><nxce:latitude><nxce:latitudeDMS degrees="43" minutes="51" direction="NORTH"/></nxce:latitude><nxce:longitude><nxce:longitudeDMS degrees="079" minutes="50" direction="WEST"/></nxce:longitude></nxcm:position></trackInformation></asdiMessage><asdiMessage sourceFacility="CCZY" sourceTimeStamp="2009-0921T12:34:31Z" trigger="TZ"><trackInformation><nxcm:aircraftId>MES3455</nxcm:aircraftId><nxcm:speed>400</nxcm:speed><nxcm:reportedAltitude><nxce:assignedAltitude><nxc e:simpleAltitude>360</nxce:simpleAltitude></nxce:assignedAltitude></nxcm:reportedAltitude><nxcm:position><nxce:latitude><nxce:latitudeDMS degrees="42" minutes="12" direction="NORTH"/></nxce:latitude><nxce:longitude><nxce:longitudeDMS degrees="076" minutes="16" direction="WEST"/></nxce:longitude></nxcm:position></trackInformation></asdiMessage><asdiMessage sourceFacility="CCZW" sourceTimeStamp="2009-0921T12:34:31Z" trigger="TZ"><trackInformation><nxcm:aircraftId>ACA114</nxcm:aircraftId><nxcm:speed>440</nxcm:speed><nxcm:reportedAltitude><nxce:assignedAltitude><nxce: simpleAltitude>262</nxce:simpleAltitude></nxce:assignedAltitude></nxcm:reportedAltitude><nxcm:position><nxce:latitude><nxce:latitudeDMS degrees="53" minutes="10" direction="NORTH"/></nxce:latitude><nxce:longitude><nxce:longitudeDMS degrees="111" minutes="54" direction="WEST"/></nxce:longitude></nxcm:position></trackInformation></asdiMessage><asdiMessage sourceFacility="CCZY" sourceTimeStamp="2009-0921T12:34:32Z" trigger="TZ"><trackInformation><nxcm:aircraftId>UAL801</nxcm:aircraftId><nxcm:speed>440</nxcm:speed><nxcm:reportedAltitude><nxce:assignedAltitude><nxce: simpleAltitude>340</nxce:simpleAltitude></nxce:assignedAltitude></nxcm:reportedAltitude><nxcm:position><nxce:latitude><nxce:latitudeDMS degrees="42" minutes="59" direction="NORTH"/></nxce:latitude><nxce:longitude><nxce:longitudeDMS degrees="082" minutes="52" direction="WEST"/></nxce:longitude></nxcm:position></trackInformation></asdiMessage><asdiMessage sourceFacility="CCZY" sourceTimeStamp="2009-0921T12:34:32Z" trigger="TZ"><trackInformation><nxcm:aircraftId>EJA802</nxcm:aircraftId><nxcm:speed>370</nxcm:speed><nxcm:reportedAltitude><nxce:assignedAltitude><nxce: simpleAltitude>400</nxce:simpleAltitude></nxce:assignedAltitude></nxcm:reportedAltitude><nxcm:position><nxce:latitude><nxce:latitudeDMS degrees="42" minutes="15" direction="NORTH"/></nxce:latitude><nxce:longitude><nxce:longitudeDMS degrees="078" minutes="52" direction="WEST"/></nxce:longitude></nxcm:position></trackInformation></asdiMessage><asdiMessage sourceFacility="CCZW" sourceTimeStamp="2009-0921T12:34:32Z" trigger="UZ"><boundaryCrossingUpdate><nxcm:aircraftId>PAG113</nxcm:aircraftId><nxcm:flightAircraftSpecs specialAircraftQualifier="B757_TCAS" equipmentQualifier="G">BE99</nxcm:flightAircraftSpecs> * http://www.fly.faa.gov/ASDI/asdi.html
    24. 24. Flight Tracker – Post Normalization TimeOfReport 9/8/13 20:21 9/8/13 20:20 9/8/13 20:19 9/8/13 20:19 9/8/13 20:18 9/8/13 20:18 9/8/13 20:17 9/8/13 20:17 9/8/13 20:17 9/8/13 20:16 9/8/13 20:07 9/8/13 19:57 9/8/13 19:47 9/8/13 17:21 9/6/13 19:59 9/6/13 19:49 9/6/13 19:41 Source FaaAsdiFAA FaaAsdiFAA FaaAsdiFAA FaaAsdiFAA FaaAsdiFAA SbbGps FaaAsdiFAA FlightDeckFusion FaaAsdiFAA FaaAsdiFAA FlightDeckFusion FlightDeckFusion SbbGps SbbGps FlightDeckFusion FlightDeckFusion SbbGps Received 9/8/13 20:26 9/8/13 20:25 9/8/13 20:24 9/8/13 20:24 9/8/13 20:23 9/8/13 20:22 9/8/13 20:23 9/8/13 20:17 9/8/13 20:22 9/8/13 20:21 9/8/13 20:07 9/8/13 19:57 9/8/13 19:52 9/8/13 17:28 9/6/13 19:59 9/6/13 19:49 9/6/13 19:47 MessageId Latitude Longitude Altitude Heading FaaAsdi132839420 35.8889 -115.0775 15100 FaaAsdi132839201 35.8986 -115.1664 11800 FaaAsdi132839013 35.9114 -115.2625 9200 FaaAsdi132838985 35.9264 -115.2839 8600 FaaAsdi132838854 35.9797 -115.2719 7200 SbbGps20130908201801000000N651SD 35.9907 -115.253 FaaAsdi132838737 35.9942 -115.2483 7000 SD20130908201716976007N651SD 36.02 -115.2 5900 FaaAsdi132838595 36.0314 -115.1908 5300 FaaAsdi132838463 36.0681 -115.1708 3100 SD20130908200716316162N651SD 36.0967 -115.1517 2000 SD20130908195716125081N651SD 36.0983 -115.16 2000 SbbGps20130908194757000000N651SD 36.0997 -115.1603 SbbGps20130908172106000000N651SD 36.0995 -115.1603 SD20130906195946601934N651SD 36.1 -115.1583 2100 SD20130906194946395228N651SD 36.0983 -115.1583 2100 SbbGps20130906194144000000N651SD 36.0999 -115.1595 Speed 272 285 284 295 272 246 218 204 195 14 0 0 0
    25. 25. Flight Tracking
    26. 26. Copyright © 2013 Splunk Inc. Splunk Tips
    27. 27. Transactions Insanely powerful for gathering statistics. tag="Expand" "status changed" |rex "s.*?Links(?<AircraftIP>S+)" |transaction AircraftIP State startswith="negotiating to accelerating" endswith="accelerating to drop" Sep 14 15:53:07 63.###.###.210 accelerator[4142]: Link ID 115 was Updated Sep 14 15:53:07 63.###.###.210 accelerator[4142]: Link 10.###.###.66 status changed from negotiating to accelerating Sep 14 15:53:07 63.###.###.210 accelerator[4142]: Acceleration enable to peer-10.###.###.66:0, with decore size - 4194304 Sep 14 15:53:07 63.###.###.210 accelerator[4142]: Acceleration enable to peer- 10.###.###.66:0, with core size - 4194304 Sep 14 15:53:07 63.###.###.210 accelerator[4142]: Link ID 115 was Updated Sep 14 15:53:00 63.###.###.210 accelerator[4142]: Link ID 115 was Updated Sep 14 15:53:00 63.###.###.210 accelerator[4142]: Link 10.###.###.66 status changed from drop to negotiating Sep 14 15:53:00 63.###.###.210 accelerator[4142]: Link ID 115 was Updated Sep 14 15:53:00 63.###.###.210 accelerator[4142]: Subnets for Remote link CP Id 115 changed Sep 14 15:53:00 63.###.###.210 accelerator[4142]: Link ID 115 was Updated Sep 14 15:52:34 63.###.###.210 accelerator[4142]: Link ID 115 was Updated Sep 14 15:52:34 63.###.###.210 accelerator[4142]: Link 10.###.###.66 status changed from accelerating to drop Sep 14 15:52:34 63.###.###.210 accelerator[4142]: Update peer failed with code 22. Sep 14 15:52:34 63.###.###.210 accelerator[4142]: Link ID 103 was Updated
    28. 28. Transactions Run against a few hours of data, and we see lots of transactions occurring. So we know how long each Aircraft is ‘in session’ for.
    29. 29. Transactions Now what? Let’s do some math and get some stats! tag="Expand" "status changed" |rex "s.*?Links(?<AircraftIP>S+)" |transaction AircraftIP State startswith="negotiating to accelerating" endswith="accelerating to drop" | eval ConnectedFor(Mins)=round(duration/60) | lookup taillookup ip as AircraftIP OUTPUT subnet_name as Tail|stats sum(ConnectedFor(Mins)) as TimeOnline by Tail| sort TimeOnline
    30. 30. Transaction - Visualizations Once you have the data, visualizations on the dashboard allow us to know at a glance if a service is performing within limits We adjust the gauge colors – in this case, higher is better
    31. 31. Don’t Fear CSV KISS – and CSV is certainly that Great for mapping things like IP/Subnets to Customers Easier to manipulate text files to clean them up Great for things that don’t change too often # Sort by IP address so searches are easier sort -t . -k 1,1n -k 2,2n -k 3,3n -k 4,4n ip-customers.in > ip-customers.csv cp ip-customers.csv /opt/splunk/etc/system/lookups/ip-customers.csv CIDR Lookup Scripts: http://answers.splunk.com/answers/5916/using-cidr-in-a-lookup-table
    32. 32. Summary Alerting based on frequency of events within a timeframe can be extremely powerful to detect anomalies Sometimes you need to clean up your data before you send it into Splunk – Garbage in, garbage out Adding external lookups can be as simple as CSV files – don’t overthink it ’transaction’ helps make sense of time & duration based data Use Splunk to guide your choices with real data – embrace Empiricism to make good business decisions
    33. 33. Q & A Time
    34. 34. Copyright © 2013 Splunk Inc. Thank You!

    ×