• Save
SplunkLive! Customer Presentation - Satcom Direct
 

SplunkLive! Customer Presentation - Satcom Direct

on

  • 705 views

 

Statistics

Views

Total Views
705
Views on SlideShare
705
Embed Views
0

Actions

Likes
0
Downloads
0
Comments
0

0 Embeds 0

No embeds

Accessibility

Categories

Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment
  • Graphics – split into 2 slides.Story: Takes support team 2-3 systems to dig though the data and figure out if they are connected, etc…

SplunkLive! Customer Presentation - Satcom Direct SplunkLive! Customer Presentation - Satcom Direct Presentation Transcript

  • Copyright © 2013 Splunk Inc. Splunk… on a Plane? Ken Bantoft VP Satcom Technology & Development Satcom Direct
  • About Me Spent 12 years doing Networking, Linux, High Performance Computing in Finance, Bio-Technology and other sectors Left IT in 2007 to focus on product development Did a 1 week contract fixing Avionics Networking code, and haven’t left Aviation since. Now responsible for Product & Services Development at Satcom Direct
  • About My Company Satcom Direct provides connectivity and communications for Aviation, Maritime and Land Mobile customers. Built around a core focus of support and service, we now serve thousands of customers world wide, including the Fortune 500, NATO & Allied Forces, and various Heads of State.
  • Agenda Splunk – not really on a plane (yet) Data Sources How we use Splunk – Support – Monitoring & Alerting – Business Analytics Tracking Planes – The technican’s flight tracker Splunk Tips
  • Copyright © 2013 Splunk Inc. Data Sources
  • Data Sources We feed Splunk pretty much anything we can get our hands on, both standard IT data, and some more esoteric data – – – – – CDRs for Phone Calls (AudioCodes, Asterisk) Syslogs from network appliances & servers Radius accounting data Logs from Satcom Systems (via email, or mobile apps) Aircraft Position + Status Reports We normalize Aircraft Position reports before feeding them to Splunk – Fields are extremely complex, often missing, sometimes delayed, and come from at least 5 different sources. And they are all totally inconsistent.
  • Data Sources - AudioCodes Max-Forwards: 70 User-Agent: AeroV-Gateway CSeq: 102 OPTIONS Call-ID: 66bac96862403ef05c1aac9922e3d3d2@63.###.###.238 Contact: <sip:AeroV-Gateway@63.###.###.238> To: <sip:63.###.###.241> From: "AeroV-Gateway" <sip:AeroV-Gateway@63.###.###.238>;tag=as7a930744 Via: SIP/2.0/UDP 63.###.###.238:5060;branch=z9hG4bK47c1eef2;rport Sep 14 14:50:02 63.###.###.241 OPTIONS sip:63.###.###.241 SIP/2.0 Sep 14 14:50:02 63.###.###.241 ( lgr_flow)(658474 ) ---- Incoming SIP Message from 63.###.###.238:5060 to SIPInterface #0 UdpTransportObject[#3343] --- [Time: 09-14-2013@14:50:02] Sep 14 14:50:02 63.###.###.241 ( sip_stack)(658473 ) New SIPMessage created - #15 [Time: 09-14-2013@14:50:02] Sep 14 14:49:58 63.###.###.241 ( sip_stack)(658472 ) SIPDialog(#138) changes state from DialogDisconnected to DialogIdle [Time: 09-14-2013@14:49:58] Sep 14 14:49:58 63.###.###.241 ( lgr_flow)(658471 ) | | TransactionUserMngr::ReturnDialog - #138 [Time: 09-14-2013@14:49:58] Sep 14 14:49:53 63.###.###.241 ( sip_stack)(658470 ) Resource SIPMessage deleted - #12 [Time: 09-14-2013@14:49:53] Sep 14 14:49:53 63.###.###.241 ( sip_stack)(658469 ) AcSIPStackAPI::FreeDialogAPI - #34 [Time: 09-14-2013@14:49:53] Sep 14 14:49:53 63.###.###.241 ( sip_stack)(658468 ) SIPDialog(#138) changes state from DialogConnected to DialogDisconnected [Time: 09-142013@14:49:53] Sep 14 14:49:53 63.###.###.241 ( lgr_flow)(658467 ) | |(SIPTU#138)DIALOG_DISCONNECT_REQ State:DialogConnected(370678c35bed1a1c1d2f36a20e0b0fd0@63.###.###.248) [Time: 09-14-2013@14:49:53] Sep 14 14:49:53 63.###.###.241 ( sip_stack)(658466 ) Resource SIPMessage deleted - #70 [Time: 09-14-2013@14:49:53]
  • Data Sources - Expand Sep 14 15:53:07 63.###.###.210 accelerator[4142]: Link ID 115 was Updated Sep 14 15:53:07 63.###.###.210 accelerator[4142]: Link 10.###.###.66 status changed from negotiating to accelerating Sep 14 15:53:07 63.###.###.210 accelerator[4142]: Acceleration enable to peer-10.###.###.66:0, with decore size - 4194304 Sep 14 15:53:07 63.###.###.210 accelerator[4142]: Acceleration enable to peer- 10.###.###.66:0, with core size - 4194304 Sep 14 15:53:07 63.###.###.210 accelerator[4142]: Link ID 115 was Updated Sep 14 15:53:00 63.###.###.210 accelerator[4142]: Link ID 115 was Updated Sep 14 15:53:00 63.###.###.210 accelerator[4142]: Link 10.###.###.66 status changed from drop to negotiating Sep 14 15:53:00 63.###.###.210 accelerator[4142]: Link ID 115 was Updated Sep 14 15:53:00 63.###.###.210 accelerator[4142]: Subnets for Remote link CP Id 115 changed Sep 14 15:53:00 63.###.###.210 accelerator[4142]: Link ID 115 was Updated Sep 14 15:52:34 63.###.###.210 accelerator[4142]: Link ID 115 was Updated Sep 14 15:52:34 63.###.###.210 accelerator[4142]: Link 10.###.###.66 status changed from accelerating to drop Sep 14 15:52:34 63.###.###.210 accelerator[4142]: Update peer failed with code 22. Sep 14 15:52:34 63.###.###.210 accelerator[4142]: Link ID 103 was Updated
  • Copyright © 2013 Splunk Inc. Monitoring & Alerting
  • Support: Monitoring and Alerting • Splunk provides a real-time dashboard in our NOC about the status of several key services • Previously, support techs would need to login to 3-5 different systems to look for faults or errors. Each system had a different UI, different formats and different data. Techs learned, but over long periods as errors were often infrequent and obscure • Now data is in one system, one interface, with intelligence ‘coded in’ by our senior techs
  • Support: Monitoring and Alerting • We merge log data with our Configuration Management database so we can display aircraft Tail Numbers, Phone Numbers and relevant data directly on the dashboard. – Allows our support team to see customers as their aircraft logon to the satellites and move data or make voice calls – Support techs can verify while still on the phone with the customer (data is ~60-90 seconds delayed) CSA Data Entry CM Servers Feed Splunk CSV tables for Lookups indexer
  • Support: Monitoring and Alerting • We can be proactive – Splunk alerting allows us to capture issues immediately – customers unable to connect (incorrect passwords, or invalid settings). We know we’ll get a call, or we can call the customer directly.
  • Support: Monitoring & Alerting • .conf 2013 Stump the Experts Report – counting in-flight (Literally!) transactions over time to gauge volumes
  • Support: Monitoring and Alerting • Alerts help capture out of the ordinary situations • More that # occurrences in a given timespan alerts take 60 seconds to setup – use them • Now when something spirals out of control, you’ll know!
  • Copyright © 2013 Splunk Inc. Business Analytics
  • Business Analytics • We’ve always been a data driven organization – we focus heavily on configuration management for customer avionics • Using Splunk to analyze the data helps us make smart decisions • Each time we deep dive into the data, we learn new things
  • Business Analytics • We used Splunk to determine how to size our new DNS infrastructure • Fed DNS stats (Bind + script + syslog) into Splunk for a few weeks, visualized the results and then were able to do capacity planning
  • Business Analytics – VoIP Call Rates • We can monitor the Country Codes dialed for our Satellite Voice calls in aggregate, so we know what countries our customers call most often. We then push our telecom & VoIP providers to negotiate better rates. • Splunk tells us what countries we need to focus on, so we ignore the long rate cards and get right down to the ones we care about.
  • Business Analytics – VoIP Call Rates • We can then route outbound calls based on destination country code to a different provider, reducing our direct cost per second for call terminations
  • Copyright © 2013 Splunk Inc. Flight Tracking
  • Flight Tracking Where the plane is coming or going isn’t what is important Common problems with Satellite communications are handovers – where you change which satellite you are talking to while in flight Historically it’s hard to correlate events with location visually Google Earth/Google Maps were a major leap, but not automated Enter Splunk w/Google Maps plugin – now we can put all the data in a consistent visual format.
  • Flight Tracking Data FAA ASDI users Other Apps Sat. Provider 1 FT Server Process & Normalize All Data Sat. Provider 2 Satcom Terminal forwarder indexer
  • FAA ASDI Data <trackInformation><nxcm:aircraftId>ACA117</nxcm:aircraftId><nxcm:speed>280</nxcm:speed><nxcm:reportedAltitude><nxce:assignedAltitude><nxce:simpleAltitud e>103</nxce:simpleAltitude></nxce:assignedAltitude></nxcm:reportedAltitude><nxcm:position><nxce:latitude><nxce:latitudeDMS degrees="43" minutes="51" direction="NORTH"/></nxce:latitude><nxce:longitude><nxce:longitudeDMS degrees="079" minutes="50" direction="WEST"/></nxce:longitude></nxcm:position></trackInformation></asdiMessage><asdiMessage sourceFacility="CCZY" sourceTimeStamp="2009-0921T12:34:31Z" trigger="TZ"><trackInformation><nxcm:aircraftId>MES3455</nxcm:aircraftId><nxcm:speed>400</nxcm:speed><nxcm:reportedAltitude><nxce:assignedAltitude><nxc e:simpleAltitude>360</nxce:simpleAltitude></nxce:assignedAltitude></nxcm:reportedAltitude><nxcm:position><nxce:latitude><nxce:latitudeDMS degrees="42" minutes="12" direction="NORTH"/></nxce:latitude><nxce:longitude><nxce:longitudeDMS degrees="076" minutes="16" direction="WEST"/></nxce:longitude></nxcm:position></trackInformation></asdiMessage><asdiMessage sourceFacility="CCZW" sourceTimeStamp="2009-0921T12:34:31Z" trigger="TZ"><trackInformation><nxcm:aircraftId>ACA114</nxcm:aircraftId><nxcm:speed>440</nxcm:speed><nxcm:reportedAltitude><nxce:assignedAltitude><nxce: simpleAltitude>262</nxce:simpleAltitude></nxce:assignedAltitude></nxcm:reportedAltitude><nxcm:position><nxce:latitude><nxce:latitudeDMS degrees="53" minutes="10" direction="NORTH"/></nxce:latitude><nxce:longitude><nxce:longitudeDMS degrees="111" minutes="54" direction="WEST"/></nxce:longitude></nxcm:position></trackInformation></asdiMessage><asdiMessage sourceFacility="CCZY" sourceTimeStamp="2009-0921T12:34:32Z" trigger="TZ"><trackInformation><nxcm:aircraftId>UAL801</nxcm:aircraftId><nxcm:speed>440</nxcm:speed><nxcm:reportedAltitude><nxce:assignedAltitude><nxce: simpleAltitude>340</nxce:simpleAltitude></nxce:assignedAltitude></nxcm:reportedAltitude><nxcm:position><nxce:latitude><nxce:latitudeDMS degrees="42" minutes="59" direction="NORTH"/></nxce:latitude><nxce:longitude><nxce:longitudeDMS degrees="082" minutes="52" direction="WEST"/></nxce:longitude></nxcm:position></trackInformation></asdiMessage><asdiMessage sourceFacility="CCZY" sourceTimeStamp="2009-0921T12:34:32Z" trigger="TZ"><trackInformation><nxcm:aircraftId>EJA802</nxcm:aircraftId><nxcm:speed>370</nxcm:speed><nxcm:reportedAltitude><nxce:assignedAltitude><nxce: simpleAltitude>400</nxce:simpleAltitude></nxce:assignedAltitude></nxcm:reportedAltitude><nxcm:position><nxce:latitude><nxce:latitudeDMS degrees="42" minutes="15" direction="NORTH"/></nxce:latitude><nxce:longitude><nxce:longitudeDMS degrees="078" minutes="52" direction="WEST"/></nxce:longitude></nxcm:position></trackInformation></asdiMessage><asdiMessage sourceFacility="CCZW" sourceTimeStamp="2009-0921T12:34:32Z" trigger="UZ"><boundaryCrossingUpdate><nxcm:aircraftId>PAG113</nxcm:aircraftId><nxcm:flightAircraftSpecs specialAircraftQualifier="B757_TCAS" equipmentQualifier="G">BE99</nxcm:flightAircraftSpecs> * http://www.fly.faa.gov/ASDI/asdi.html
  • Flight Tracker – Post Normalization TimeOfReport 9/8/13 20:21 9/8/13 20:20 9/8/13 20:19 9/8/13 20:19 9/8/13 20:18 9/8/13 20:18 9/8/13 20:17 9/8/13 20:17 9/8/13 20:17 9/8/13 20:16 9/8/13 20:07 9/8/13 19:57 9/8/13 19:47 9/8/13 17:21 9/6/13 19:59 9/6/13 19:49 9/6/13 19:41 Source FaaAsdiFAA FaaAsdiFAA FaaAsdiFAA FaaAsdiFAA FaaAsdiFAA SbbGps FaaAsdiFAA FlightDeckFusion FaaAsdiFAA FaaAsdiFAA FlightDeckFusion FlightDeckFusion SbbGps SbbGps FlightDeckFusion FlightDeckFusion SbbGps Received 9/8/13 20:26 9/8/13 20:25 9/8/13 20:24 9/8/13 20:24 9/8/13 20:23 9/8/13 20:22 9/8/13 20:23 9/8/13 20:17 9/8/13 20:22 9/8/13 20:21 9/8/13 20:07 9/8/13 19:57 9/8/13 19:52 9/8/13 17:28 9/6/13 19:59 9/6/13 19:49 9/6/13 19:47 MessageId Latitude Longitude Altitude Heading FaaAsdi132839420 35.8889 -115.0775 15100 FaaAsdi132839201 35.8986 -115.1664 11800 FaaAsdi132839013 35.9114 -115.2625 9200 FaaAsdi132838985 35.9264 -115.2839 8600 FaaAsdi132838854 35.9797 -115.2719 7200 SbbGps20130908201801000000N651SD 35.9907 -115.253 FaaAsdi132838737 35.9942 -115.2483 7000 SD20130908201716976007N651SD 36.02 -115.2 5900 FaaAsdi132838595 36.0314 -115.1908 5300 FaaAsdi132838463 36.0681 -115.1708 3100 SD20130908200716316162N651SD 36.0967 -115.1517 2000 SD20130908195716125081N651SD 36.0983 -115.16 2000 SbbGps20130908194757000000N651SD 36.0997 -115.1603 SbbGps20130908172106000000N651SD 36.0995 -115.1603 SD20130906195946601934N651SD 36.1 -115.1583 2100 SD20130906194946395228N651SD 36.0983 -115.1583 2100 SbbGps20130906194144000000N651SD 36.0999 -115.1595 Speed 272 285 284 295 272 246 218 204 195 14 0 0 0
  • Flight Tracking
  • Copyright © 2013 Splunk Inc. Splunk Tips
  • Transactions Insanely powerful for gathering statistics. tag="Expand" "status changed" |rex "s.*?Links(?<AircraftIP>S+)" |transaction AircraftIP State startswith="negotiating to accelerating" endswith="accelerating to drop" Sep 14 15:53:07 63.###.###.210 accelerator[4142]: Link ID 115 was Updated Sep 14 15:53:07 63.###.###.210 accelerator[4142]: Link 10.###.###.66 status changed from negotiating to accelerating Sep 14 15:53:07 63.###.###.210 accelerator[4142]: Acceleration enable to peer-10.###.###.66:0, with decore size - 4194304 Sep 14 15:53:07 63.###.###.210 accelerator[4142]: Acceleration enable to peer- 10.###.###.66:0, with core size - 4194304 Sep 14 15:53:07 63.###.###.210 accelerator[4142]: Link ID 115 was Updated Sep 14 15:53:00 63.###.###.210 accelerator[4142]: Link ID 115 was Updated Sep 14 15:53:00 63.###.###.210 accelerator[4142]: Link 10.###.###.66 status changed from drop to negotiating Sep 14 15:53:00 63.###.###.210 accelerator[4142]: Link ID 115 was Updated Sep 14 15:53:00 63.###.###.210 accelerator[4142]: Subnets for Remote link CP Id 115 changed Sep 14 15:53:00 63.###.###.210 accelerator[4142]: Link ID 115 was Updated Sep 14 15:52:34 63.###.###.210 accelerator[4142]: Link ID 115 was Updated Sep 14 15:52:34 63.###.###.210 accelerator[4142]: Link 10.###.###.66 status changed from accelerating to drop Sep 14 15:52:34 63.###.###.210 accelerator[4142]: Update peer failed with code 22. Sep 14 15:52:34 63.###.###.210 accelerator[4142]: Link ID 103 was Updated
  • Transactions Run against a few hours of data, and we see lots of transactions occurring. So we know how long each Aircraft is ‘in session’ for.
  • Transactions Now what? Let’s do some math and get some stats! tag="Expand" "status changed" |rex "s.*?Links(?<AircraftIP>S+)" |transaction AircraftIP State startswith="negotiating to accelerating" endswith="accelerating to drop" | eval ConnectedFor(Mins)=round(duration/60) | lookup taillookup ip as AircraftIP OUTPUT subnet_name as Tail|stats sum(ConnectedFor(Mins)) as TimeOnline by Tail| sort TimeOnline
  • Transaction - Visualizations Once you have the data, visualizations on the dashboard allow us to know at a glance if a service is performing within limits We adjust the gauge colors – in this case, higher is better
  • Don’t Fear CSV KISS – and CSV is certainly that Great for mapping things like IP/Subnets to Customers Easier to manipulate text files to clean them up Great for things that don’t change too often # Sort by IP address so searches are easier sort -t . -k 1,1n -k 2,2n -k 3,3n -k 4,4n ip-customers.in > ip-customers.csv cp ip-customers.csv /opt/splunk/etc/system/lookups/ip-customers.csv CIDR Lookup Scripts: http://answers.splunk.com/answers/5916/using-cidr-in-a-lookup-table
  • Summary Alerting based on frequency of events within a timeframe can be extremely powerful to detect anomalies Sometimes you need to clean up your data before you send it into Splunk – Garbage in, garbage out Adding external lookups can be as simple as CSV files – don’t overthink it ’transaction’ helps make sense of time & duration based data Use Splunk to guide your choices with real data – embrace Empiricism to make good business decisions
  • Q & A Time
  • Copyright © 2013 Splunk Inc. Thank You!