• Like
  • Save

SplunkLive! Customer Presentation - Satcom Direct

  • 516 views
Uploaded on

 

More in: Technology , Business
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
    Be the first to like this
No Downloads

Views

Total Views
516
On Slideshare
0
From Embeds
0
Number of Embeds
0

Actions

Shares
Downloads
0
Comments
0
Likes
0

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide
  • Graphics – split into 2 slides.Story: Takes support team 2-3 systems to dig though the data and figure out if they are connected, etc…

Transcript

  • 1. Copyright © 2013 Splunk Inc. Splunk… on a Plane? Ken Bantoft VP Satcom Technology & Development Satcom Direct
  • 2. About Me Spent 12 years doing Networking, Linux, High Performance Computing in Finance, Bio-Technology and other sectors Left IT in 2007 to focus on product development Did a 1 week contract fixing Avionics Networking code, and haven’t left Aviation since. Now responsible for Product & Services Development at Satcom Direct
  • 3. About My Company Satcom Direct provides connectivity and communications for Aviation, Maritime and Land Mobile customers. Built around a core focus of support and service, we now serve thousands of customers world wide, including the Fortune 500, NATO & Allied Forces, and various Heads of State.
  • 4. Agenda Splunk – not really on a plane (yet) Data Sources How we use Splunk – Support – Monitoring & Alerting – Business Analytics Tracking Planes – The technican’s flight tracker Splunk Tips
  • 5. Copyright © 2013 Splunk Inc. Data Sources
  • 6. Data Sources We feed Splunk pretty much anything we can get our hands on, both standard IT data, and some more esoteric data – – – – – CDRs for Phone Calls (AudioCodes, Asterisk) Syslogs from network appliances & servers Radius accounting data Logs from Satcom Systems (via email, or mobile apps) Aircraft Position + Status Reports We normalize Aircraft Position reports before feeding them to Splunk – Fields are extremely complex, often missing, sometimes delayed, and come from at least 5 different sources. And they are all totally inconsistent.
  • 7. Data Sources - AudioCodes Max-Forwards: 70 User-Agent: AeroV-Gateway CSeq: 102 OPTIONS Call-ID: 66bac96862403ef05c1aac9922e3d3d2@63.###.###.238 Contact: <sip:AeroV-Gateway@63.###.###.238> To: <sip:63.###.###.241> From: "AeroV-Gateway" <sip:AeroV-Gateway@63.###.###.238>;tag=as7a930744 Via: SIP/2.0/UDP 63.###.###.238:5060;branch=z9hG4bK47c1eef2;rport Sep 14 14:50:02 63.###.###.241 OPTIONS sip:63.###.###.241 SIP/2.0 Sep 14 14:50:02 63.###.###.241 ( lgr_flow)(658474 ) ---- Incoming SIP Message from 63.###.###.238:5060 to SIPInterface #0 UdpTransportObject[#3343] --- [Time: 09-14-2013@14:50:02] Sep 14 14:50:02 63.###.###.241 ( sip_stack)(658473 ) New SIPMessage created - #15 [Time: 09-14-2013@14:50:02] Sep 14 14:49:58 63.###.###.241 ( sip_stack)(658472 ) SIPDialog(#138) changes state from DialogDisconnected to DialogIdle [Time: 09-14-2013@14:49:58] Sep 14 14:49:58 63.###.###.241 ( lgr_flow)(658471 ) | | TransactionUserMngr::ReturnDialog - #138 [Time: 09-14-2013@14:49:58] Sep 14 14:49:53 63.###.###.241 ( sip_stack)(658470 ) Resource SIPMessage deleted - #12 [Time: 09-14-2013@14:49:53] Sep 14 14:49:53 63.###.###.241 ( sip_stack)(658469 ) AcSIPStackAPI::FreeDialogAPI - #34 [Time: 09-14-2013@14:49:53] Sep 14 14:49:53 63.###.###.241 ( sip_stack)(658468 ) SIPDialog(#138) changes state from DialogConnected to DialogDisconnected [Time: 09-142013@14:49:53] Sep 14 14:49:53 63.###.###.241 ( lgr_flow)(658467 ) | |(SIPTU#138)DIALOG_DISCONNECT_REQ State:DialogConnected(370678c35bed1a1c1d2f36a20e0b0fd0@63.###.###.248) [Time: 09-14-2013@14:49:53] Sep 14 14:49:53 63.###.###.241 ( sip_stack)(658466 ) Resource SIPMessage deleted - #70 [Time: 09-14-2013@14:49:53]
  • 8. Data Sources - Expand Sep 14 15:53:07 63.###.###.210 accelerator[4142]: Link ID 115 was Updated Sep 14 15:53:07 63.###.###.210 accelerator[4142]: Link 10.###.###.66 status changed from negotiating to accelerating Sep 14 15:53:07 63.###.###.210 accelerator[4142]: Acceleration enable to peer-10.###.###.66:0, with decore size - 4194304 Sep 14 15:53:07 63.###.###.210 accelerator[4142]: Acceleration enable to peer- 10.###.###.66:0, with core size - 4194304 Sep 14 15:53:07 63.###.###.210 accelerator[4142]: Link ID 115 was Updated Sep 14 15:53:00 63.###.###.210 accelerator[4142]: Link ID 115 was Updated Sep 14 15:53:00 63.###.###.210 accelerator[4142]: Link 10.###.###.66 status changed from drop to negotiating Sep 14 15:53:00 63.###.###.210 accelerator[4142]: Link ID 115 was Updated Sep 14 15:53:00 63.###.###.210 accelerator[4142]: Subnets for Remote link CP Id 115 changed Sep 14 15:53:00 63.###.###.210 accelerator[4142]: Link ID 115 was Updated Sep 14 15:52:34 63.###.###.210 accelerator[4142]: Link ID 115 was Updated Sep 14 15:52:34 63.###.###.210 accelerator[4142]: Link 10.###.###.66 status changed from accelerating to drop Sep 14 15:52:34 63.###.###.210 accelerator[4142]: Update peer failed with code 22. Sep 14 15:52:34 63.###.###.210 accelerator[4142]: Link ID 103 was Updated
  • 9. Copyright © 2013 Splunk Inc. Monitoring & Alerting
  • 10. Support: Monitoring and Alerting • Splunk provides a real-time dashboard in our NOC about the status of several key services • Previously, support techs would need to login to 3-5 different systems to look for faults or errors. Each system had a different UI, different formats and different data. Techs learned, but over long periods as errors were often infrequent and obscure • Now data is in one system, one interface, with intelligence ‘coded in’ by our senior techs
  • 11. Support: Monitoring and Alerting • We merge log data with our Configuration Management database so we can display aircraft Tail Numbers, Phone Numbers and relevant data directly on the dashboard. – Allows our support team to see customers as their aircraft logon to the satellites and move data or make voice calls – Support techs can verify while still on the phone with the customer (data is ~60-90 seconds delayed) CSA Data Entry CM Servers Feed Splunk CSV tables for Lookups indexer
  • 12. Support: Monitoring and Alerting • We can be proactive – Splunk alerting allows us to capture issues immediately – customers unable to connect (incorrect passwords, or invalid settings). We know we’ll get a call, or we can call the customer directly.
  • 13. Support: Monitoring & Alerting • .conf 2013 Stump the Experts Report – counting in-flight (Literally!) transactions over time to gauge volumes
  • 14. Support: Monitoring and Alerting • Alerts help capture out of the ordinary situations • More that # occurrences in a given timespan alerts take 60 seconds to setup – use them • Now when something spirals out of control, you’ll know!
  • 15. Copyright © 2013 Splunk Inc. Business Analytics
  • 16. Business Analytics • We’ve always been a data driven organization – we focus heavily on configuration management for customer avionics • Using Splunk to analyze the data helps us make smart decisions • Each time we deep dive into the data, we learn new things
  • 17. Business Analytics • We used Splunk to determine how to size our new DNS infrastructure • Fed DNS stats (Bind + script + syslog) into Splunk for a few weeks, visualized the results and then were able to do capacity planning
  • 18. Business Analytics – VoIP Call Rates • We can monitor the Country Codes dialed for our Satellite Voice calls in aggregate, so we know what countries our customers call most often. We then push our telecom & VoIP providers to negotiate better rates. • Splunk tells us what countries we need to focus on, so we ignore the long rate cards and get right down to the ones we care about.
  • 19. Business Analytics – VoIP Call Rates • We can then route outbound calls based on destination country code to a different provider, reducing our direct cost per second for call terminations
  • 20. Copyright © 2013 Splunk Inc. Flight Tracking
  • 21. Flight Tracking Where the plane is coming or going isn’t what is important Common problems with Satellite communications are handovers – where you change which satellite you are talking to while in flight Historically it’s hard to correlate events with location visually Google Earth/Google Maps were a major leap, but not automated Enter Splunk w/Google Maps plugin – now we can put all the data in a consistent visual format.
  • 22. Flight Tracking Data FAA ASDI users Other Apps Sat. Provider 1 FT Server Process & Normalize All Data Sat. Provider 2 Satcom Terminal forwarder indexer
  • 23. FAA ASDI Data <trackInformation><nxcm:aircraftId>ACA117</nxcm:aircraftId><nxcm:speed>280</nxcm:speed><nxcm:reportedAltitude><nxce:assignedAltitude><nxce:simpleAltitud e>103</nxce:simpleAltitude></nxce:assignedAltitude></nxcm:reportedAltitude><nxcm:position><nxce:latitude><nxce:latitudeDMS degrees="43" minutes="51" direction="NORTH"/></nxce:latitude><nxce:longitude><nxce:longitudeDMS degrees="079" minutes="50" direction="WEST"/></nxce:longitude></nxcm:position></trackInformation></asdiMessage><asdiMessage sourceFacility="CCZY" sourceTimeStamp="2009-0921T12:34:31Z" trigger="TZ"><trackInformation><nxcm:aircraftId>MES3455</nxcm:aircraftId><nxcm:speed>400</nxcm:speed><nxcm:reportedAltitude><nxce:assignedAltitude><nxc e:simpleAltitude>360</nxce:simpleAltitude></nxce:assignedAltitude></nxcm:reportedAltitude><nxcm:position><nxce:latitude><nxce:latitudeDMS degrees="42" minutes="12" direction="NORTH"/></nxce:latitude><nxce:longitude><nxce:longitudeDMS degrees="076" minutes="16" direction="WEST"/></nxce:longitude></nxcm:position></trackInformation></asdiMessage><asdiMessage sourceFacility="CCZW" sourceTimeStamp="2009-0921T12:34:31Z" trigger="TZ"><trackInformation><nxcm:aircraftId>ACA114</nxcm:aircraftId><nxcm:speed>440</nxcm:speed><nxcm:reportedAltitude><nxce:assignedAltitude><nxce: simpleAltitude>262</nxce:simpleAltitude></nxce:assignedAltitude></nxcm:reportedAltitude><nxcm:position><nxce:latitude><nxce:latitudeDMS degrees="53" minutes="10" direction="NORTH"/></nxce:latitude><nxce:longitude><nxce:longitudeDMS degrees="111" minutes="54" direction="WEST"/></nxce:longitude></nxcm:position></trackInformation></asdiMessage><asdiMessage sourceFacility="CCZY" sourceTimeStamp="2009-0921T12:34:32Z" trigger="TZ"><trackInformation><nxcm:aircraftId>UAL801</nxcm:aircraftId><nxcm:speed>440</nxcm:speed><nxcm:reportedAltitude><nxce:assignedAltitude><nxce: simpleAltitude>340</nxce:simpleAltitude></nxce:assignedAltitude></nxcm:reportedAltitude><nxcm:position><nxce:latitude><nxce:latitudeDMS degrees="42" minutes="59" direction="NORTH"/></nxce:latitude><nxce:longitude><nxce:longitudeDMS degrees="082" minutes="52" direction="WEST"/></nxce:longitude></nxcm:position></trackInformation></asdiMessage><asdiMessage sourceFacility="CCZY" sourceTimeStamp="2009-0921T12:34:32Z" trigger="TZ"><trackInformation><nxcm:aircraftId>EJA802</nxcm:aircraftId><nxcm:speed>370</nxcm:speed><nxcm:reportedAltitude><nxce:assignedAltitude><nxce: simpleAltitude>400</nxce:simpleAltitude></nxce:assignedAltitude></nxcm:reportedAltitude><nxcm:position><nxce:latitude><nxce:latitudeDMS degrees="42" minutes="15" direction="NORTH"/></nxce:latitude><nxce:longitude><nxce:longitudeDMS degrees="078" minutes="52" direction="WEST"/></nxce:longitude></nxcm:position></trackInformation></asdiMessage><asdiMessage sourceFacility="CCZW" sourceTimeStamp="2009-0921T12:34:32Z" trigger="UZ"><boundaryCrossingUpdate><nxcm:aircraftId>PAG113</nxcm:aircraftId><nxcm:flightAircraftSpecs specialAircraftQualifier="B757_TCAS" equipmentQualifier="G">BE99</nxcm:flightAircraftSpecs> * http://www.fly.faa.gov/ASDI/asdi.html
  • 24. Flight Tracker – Post Normalization TimeOfReport 9/8/13 20:21 9/8/13 20:20 9/8/13 20:19 9/8/13 20:19 9/8/13 20:18 9/8/13 20:18 9/8/13 20:17 9/8/13 20:17 9/8/13 20:17 9/8/13 20:16 9/8/13 20:07 9/8/13 19:57 9/8/13 19:47 9/8/13 17:21 9/6/13 19:59 9/6/13 19:49 9/6/13 19:41 Source FaaAsdiFAA FaaAsdiFAA FaaAsdiFAA FaaAsdiFAA FaaAsdiFAA SbbGps FaaAsdiFAA FlightDeckFusion FaaAsdiFAA FaaAsdiFAA FlightDeckFusion FlightDeckFusion SbbGps SbbGps FlightDeckFusion FlightDeckFusion SbbGps Received 9/8/13 20:26 9/8/13 20:25 9/8/13 20:24 9/8/13 20:24 9/8/13 20:23 9/8/13 20:22 9/8/13 20:23 9/8/13 20:17 9/8/13 20:22 9/8/13 20:21 9/8/13 20:07 9/8/13 19:57 9/8/13 19:52 9/8/13 17:28 9/6/13 19:59 9/6/13 19:49 9/6/13 19:47 MessageId Latitude Longitude Altitude Heading FaaAsdi132839420 35.8889 -115.0775 15100 FaaAsdi132839201 35.8986 -115.1664 11800 FaaAsdi132839013 35.9114 -115.2625 9200 FaaAsdi132838985 35.9264 -115.2839 8600 FaaAsdi132838854 35.9797 -115.2719 7200 SbbGps20130908201801000000N651SD 35.9907 -115.253 FaaAsdi132838737 35.9942 -115.2483 7000 SD20130908201716976007N651SD 36.02 -115.2 5900 FaaAsdi132838595 36.0314 -115.1908 5300 FaaAsdi132838463 36.0681 -115.1708 3100 SD20130908200716316162N651SD 36.0967 -115.1517 2000 SD20130908195716125081N651SD 36.0983 -115.16 2000 SbbGps20130908194757000000N651SD 36.0997 -115.1603 SbbGps20130908172106000000N651SD 36.0995 -115.1603 SD20130906195946601934N651SD 36.1 -115.1583 2100 SD20130906194946395228N651SD 36.0983 -115.1583 2100 SbbGps20130906194144000000N651SD 36.0999 -115.1595 Speed 272 285 284 295 272 246 218 204 195 14 0 0 0
  • 25. Flight Tracking
  • 26. Copyright © 2013 Splunk Inc. Splunk Tips
  • 27. Transactions Insanely powerful for gathering statistics. tag="Expand" "status changed" |rex "s.*?Links(?<AircraftIP>S+)" |transaction AircraftIP State startswith="negotiating to accelerating" endswith="accelerating to drop" Sep 14 15:53:07 63.###.###.210 accelerator[4142]: Link ID 115 was Updated Sep 14 15:53:07 63.###.###.210 accelerator[4142]: Link 10.###.###.66 status changed from negotiating to accelerating Sep 14 15:53:07 63.###.###.210 accelerator[4142]: Acceleration enable to peer-10.###.###.66:0, with decore size - 4194304 Sep 14 15:53:07 63.###.###.210 accelerator[4142]: Acceleration enable to peer- 10.###.###.66:0, with core size - 4194304 Sep 14 15:53:07 63.###.###.210 accelerator[4142]: Link ID 115 was Updated Sep 14 15:53:00 63.###.###.210 accelerator[4142]: Link ID 115 was Updated Sep 14 15:53:00 63.###.###.210 accelerator[4142]: Link 10.###.###.66 status changed from drop to negotiating Sep 14 15:53:00 63.###.###.210 accelerator[4142]: Link ID 115 was Updated Sep 14 15:53:00 63.###.###.210 accelerator[4142]: Subnets for Remote link CP Id 115 changed Sep 14 15:53:00 63.###.###.210 accelerator[4142]: Link ID 115 was Updated Sep 14 15:52:34 63.###.###.210 accelerator[4142]: Link ID 115 was Updated Sep 14 15:52:34 63.###.###.210 accelerator[4142]: Link 10.###.###.66 status changed from accelerating to drop Sep 14 15:52:34 63.###.###.210 accelerator[4142]: Update peer failed with code 22. Sep 14 15:52:34 63.###.###.210 accelerator[4142]: Link ID 103 was Updated
  • 28. Transactions Run against a few hours of data, and we see lots of transactions occurring. So we know how long each Aircraft is ‘in session’ for.
  • 29. Transactions Now what? Let’s do some math and get some stats! tag="Expand" "status changed" |rex "s.*?Links(?<AircraftIP>S+)" |transaction AircraftIP State startswith="negotiating to accelerating" endswith="accelerating to drop" | eval ConnectedFor(Mins)=round(duration/60) | lookup taillookup ip as AircraftIP OUTPUT subnet_name as Tail|stats sum(ConnectedFor(Mins)) as TimeOnline by Tail| sort TimeOnline
  • 30. Transaction - Visualizations Once you have the data, visualizations on the dashboard allow us to know at a glance if a service is performing within limits We adjust the gauge colors – in this case, higher is better
  • 31. Don’t Fear CSV KISS – and CSV is certainly that Great for mapping things like IP/Subnets to Customers Easier to manipulate text files to clean them up Great for things that don’t change too often # Sort by IP address so searches are easier sort -t . -k 1,1n -k 2,2n -k 3,3n -k 4,4n ip-customers.in > ip-customers.csv cp ip-customers.csv /opt/splunk/etc/system/lookups/ip-customers.csv CIDR Lookup Scripts: http://answers.splunk.com/answers/5916/using-cidr-in-a-lookup-table
  • 32. Summary Alerting based on frequency of events within a timeframe can be extremely powerful to detect anomalies Sometimes you need to clean up your data before you send it into Splunk – Garbage in, garbage out Adding external lookups can be as simple as CSV files – don’t overthink it ’transaction’ helps make sense of time & duration based data Use Splunk to guide your choices with real data – embrace Empiricism to make good business decisions
  • 33. Q & A Time
  • 34. Copyright © 2013 Splunk Inc. Thank You!