• Like

Don't Re-write Code to Get Better Analytics

  • 2,952 views
Uploaded on

Almost all developers face the challenge of reactively debugging failed business transaction processes. Not only does this require extensive navigation of enormous volumes of log data, but determining …

Almost all developers face the challenge of reactively debugging failed business transaction processes. Not only does this require extensive navigation of enormous volumes of log data, but determining root cause becomes a laborious and time-consuming task.

Additionally, business managers often request developers and operations to provide analytics on applications, resulting in the tedious task of charting the information, most usually from intangible data. Learn how to capture, extract and analyze your event data by having analytics embedded in the application. Download the white-paper that details how to gain Application Intelligence through effective logging.
Check out the webinar here: http://www.splunk.com/goto/analytics_webcast

More in: Technology
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
No Downloads

Views

Total Views
2,952
On Slideshare
0
From Embeds
0
Number of Embeds
3

Actions

Shares
Downloads
73
Comments
0
Likes
5

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. Don’t  Rewrite   Code  to  Get   BeCer  AnalyEcs   Archana  Ganapathi   Research  Engineer  Copyright  ©  2012,  Splunk  Inc.   Listen  to  your  data.  
  • 2. AnalyEcs  Can  Be  Challenging!   •  Modern  systems  are  distributed  and  heterogeneous   •  Consolidate  informaEon   •  Analyzing  across  a  distributed  architecture     •  AnalyEcs  is  limited  to  informaEon  that  is  made  “available”  Copyright  ©  2012,  Splunk  Inc.   Listen  to  your  data.  
  • 3. Typical  Architecture   ApplicaEons   Data   Direct  Insert   Warehouse   BI,  AnalyEcs,   ReporEng  Tool   ETL   Database   Connector  Copyright  ©  2012,  Splunk  Inc.   Listen  to  your  data.  
  • 4. Development  Cycle   Early  Structure  Binding   Decide  the  quesEons  you  want  to  ask   Design  the  Schema   Normalize  the  data  and   Write  DB  inserEon  code   SELECT  customers.*  FROM  customers  WHERE   Create  SQL  &  feed  into  AnalyEcs  Tool   customers.customer_id  NOT  IN(SELECT  customer_id  FROM   orders  WHERE  year(orders.order_date)  =  2004)  Copyright  ©  2012,  Splunk  Inc.   Listen  to  your  data.  
  • 5. A  Paradigm  Change:   Use  Your  Log  Files  Copyright  ©  2012,  Splunk  Inc.   Listen  to  your  data.  
  • 6. Using  Log  Files   !Log.debug(“orderstatus=error,errorcode=454,! !user=%s,transactionid=%d”, userId, transId)!ü   You  already  log  key  informaEon  Copyright  ©  2012,  Splunk  Inc.   Listen  to  your  data.  
  • 7. Using  Log  Files   They  contain  a  gold  mine  of  informaEon  •  DefiniEve  record  of  acEvity  and  behavior  •  Ensure  system  security  •  Meet  compliance  mandates   User  IP   AcEon   Login   Result   10.2.1.44 - [25/Sep/2009:09:52:30 -0700] type=USER_LOGIN msg=audit(1253898008.056:199891): auid=4294967295 msg=acct="TAYLOR": exe="/usr/sbi addr=10.2.1.48, terminal=sshd res=failed)Copyright  ©  2012,  Splunk  Inc.   Listen  to  your  data.  
  • 8. Using  Log  Files   They  contain  a  gold  mine  of  informaEon  •  Important  insight  for  IT  and  the  business  •  Customer  behavior  and  experience  •  Product  and  service  usage   User  IP   Product   Category  •  End-­‐to-­‐end  transacEon  visibility   10.2.1.80 - - [25/Jan/2010:09:52:30 -0700] "GET /petstore/product.screen ?product_id=AV-CB-01 HTTP/1.1" 200 9967 "http://10 category.screen?category_id=BIRDS" "Mozilla/5.0 (co Linux)”"JSESSIONID=xZDTK81Gjq9gJLGWnt2NXrJ2tpGZb1Hy Copyright  ©  2012,  Splunk  Inc.   Listen  to  your  data.  
  • 9. They  Help  You  Find  Problems  Apr 29 19:13:01 45.2.98.7 SentriantGenericAlert: Time="04/29/06 07:12 PMPDT",Host="roach_motel.enet.interop.net",Category="fabric_network_activity",Generator="Response:SlowScan",Type="NOTICE",Priority="High",Body="Appliance=roach_motel.enet.interop.net,Reporting Segment=ENET network,Action=Response disabled,Response=Slow Scan,Duration=90seconds,Source Segment=Unprotected,Source IP=88.73.39.200,Source MAC=00:01:30:BC:93:90,Current Target Count=0"! 45.2.98.7Apr 29 19:13:01 45.2.98.7 SentriantGenericAlert: Time="04/29/06 07:12 PM SentriantGePDT",Host="roach_motel.enet.interop.net",Category="fabric_network_activity",Generator="Response:Slow nericAlert: Time="04  Scan",Type="NOTICE",Priority="High",Body="Appliance=roach_motel.enet.interop.net,Reporting Segment=ENET network,Action=Response disabled,Response=Slow Scan,Duration=69seconds,Source Segment=Unprotected,Source IP=68.163.20.95,Source MAC=00:01:30:BC:93:90,Current Target Count=0"!Apr 29 19:13:01 45.2.98.7 SentriantGenericAlert: Time="04/29/06 07:12 PMPDT",Host="roach_motel.enet.interop.net",Category="fabric_network_activity",Generator="Response:Slow! Copyright  ©  2012,  Splunk  Inc.   Listen  to  your  data.  
  • 10. Machine-­‐generated  Events  are  Everywhere   AddiEonal  Sources   Core  IT   Customer-­‐facing  IT  Copyright  ©  2012,  Splunk  Inc.   Listen  to  your  data.  
  • 11. Splunk:  The  Plaiorm  for  Machine  Data  Customer     Outside  the  Facing  Data   Datacenter  "  Click-­‐stream  data   "  Manufacturing,  "  Shopping  cart  data   logisEcs…  "  Online  transacEon   "  CDRs  &  IPDRs   data   "  Power  consumpEon   Logfiles   Configs   Messages   Traps     Metrics   Scripts   Changes   Tickets   "  RFID  data    Alerts   "  GPS  data   Windows   Linux/Unix   Virtualiza7on     Applica7ons   Databases   Networking  "  Registry   "  ConfiguraEons   &  Cloud   "  Web  logs   "  ConfiguraEons   "  ConfiguraEons  "  Event  logs   "  syslog   "  Hypervisor   "  Log4J,  JMS,  JMX   "  Audit/query  logs   "  syslog  "  File  system   "  File  system   "  Guest  OS,  Apps   "  .NET  events   "  Tables   "  SNMP  "  sysinternals   "  ps,  iostat,  top   "  Cloud   "  Code  and  scripts   "  Schemas   "  neilow   Burlingame,  March  8,  2012   Copyright  ©  2011,  Splunk  Inc.   Listen  to  your  data.   Copyright  ©  2012,  Splunk  Inc.  
  • 12. Splunk  Collects  and  Indexes  Any  Machine  Data   Customer     Outside  the   Facing  Data   Datacenter   "  Click-­‐stream  data   "  Manufacturing,   "  Shopping  cart  data   logisEcs…   • Any  amount,  any  locaEon,  any  source.   "  Online  transacEon   "  CDRs  &  IPDRs   data   "  Power  consumpEon   Logfiles   Configs   Messages   Traps     Metrics   Scripts   Changes   Tickets   "  RFID  data   GPS  data   No  upfront  schema    Alerts   "  No  custom  connectors   Windows   Linux/Unix   Virtualiza7on     Applica7ons   Databases   Networking   "  Registry   "  ConfiguraEons   &  Cloud   DBMS   Web  logs   No  R "  ConfiguraEons   "  "  ConfiguraEons   "  Event  logs   "  syslog   " Hypervisor   Log4J,  JMS,  JMX   "  Audit/query  logs   "  "  syslog   "  "  File  system   sysinternals   "  "  File  system   ps,  iostat,  top   No  nS,  Apps   to  filter/forward   Tables   "  "  Guest  O eed   Cloud   .NET  events   "  Code  and  scripts   "  "  Schemas   "  "  "  SNMP   neilow   Burlingame,  March  8,  2012   Copyright  ©  2011,  Splunk  Inc.   Listen  to  your  data.   Copyright  ©  2012,  Splunk  Inc.  
  • 13. A  Single  Plaiorm  for  OperaEonal  Intelligence   Single  Data  Store   Single  UI   Across  Use  Cases   Three  Primary  CapabiliEes   Search  /  Naviga7on   Real-­‐7me  Visibility   Historical  Analy7cs   •  Data  drilldown   •  Live  dashboards   •  Baseline  and  thresholds   •  “Needle  in  a  haystack”   •  Event  correlaEon   •  Trending   •  Root  cause  analysis  /     •  Monitoring  and  alerEng   •  OperaEonal  insights   troubleshooEng   •  Performance  issues   •  Historical  paCerns   •  Incident  invesEgaEons   •  TransacEon  levels   •  Compliance  reports   •  SLA  tracking   Burlingame,  March  8,  2012  Copyright  ©  2011,  Splunk  Inc.   Listen  to  your  data.   Copyright  ©  2012,  Splunk  Inc.  
  • 14. Real  Business  Value  with  OperaEonal  Metrics  Copyright  ©  2012,  Splunk  Inc.   March  8,  2012   Burlingame,   Listen  to  y©  2012,  Splunk  Inc.   Copyright  our  data.  
  • 15. Intelligence  on  your  ApplicaEons  with  Splunk   Log  Files   ApplicaEon   OperaEonal   Database   Intelligence   Java  EE  Server   +   AnalyEcs   Unix  based  OS   Unix  based  OS   +     ReporEng  Copyright  ©  2012,  Splunk  Inc.   March  8,  2012   Burlingame,   Listen  to  y©  2012,  Splunk  Inc.   Copyright  our  data.  
  • 16. An  AlternaEve  Development  Cycle   Apr 29 19:13:01 45.2.98.7 entriantGenericAlert: Late  Structure  Binding   Time="04/29/06 07:12 PM PDT”, Host="roach_motel.enet.interop.net",Category="fabric_ network_activity",Generator="Response:Slow Write  events  to  your  log  files   Scan",Type="NOTICE",Priority="High",Body="Appliance=r oach_motel.enet.interop.net,Reporting Segment=ENET network,Action=Response disabled,Response=Slow Collect  log  files   Scan,Duration=90 seconds,Source Segment=Unprotected,Source IP=88.73.39.200,Source MAC=00:01:30:BC:93:90,Current Target Count=0"! Apr 29 19:13:01 45.2.98.7 SentriantGenericAlert: Create  searches,  graphs  and  reports   Time="04/29/06 07:12 PM PDT",Host="roach_motel.enet.interop.net",Category="fa bric_network_activity",Generator="Response:Slow Scan",Type="NOTICE",Priority="High",Body="Appliance=r oach_motel.enet.interop.net,Reporting!Copyright  ©  2012,  Splunk  Inc.   March  8,  2012   Burlingame,   Listen  to  y©  2012,  Splunk  Inc.   Copyright  our  data.  
  • 17. “SemanEc  Logging”   Events  which  are  wriCen  explicitly  for  the   gathering  of  analyEcs  Copyright  ©  2012,  Splunk  Inc.   March  8,  2012   Burlingame,   Listen  to  y©  2012,  Splunk  Inc.   Copyright  our  data.  
  • 18. A  Simple  Example  void submitPurchase(transctionID) !{ ! log.info("action=submitPurchaseStart, transactionId=%d",transactionID, “ productId=%s”, productId, “ listPrice=%dn”, listPrice)!! //these calls throw an exception on error! submitToCreditCard(...)! generateInvoice(...)! generateFullfillmentOrder(...)!! log.info("action=submitPurchaseStop, transactionID=%dn",transactionID)!} !!!Copyright  ©  2012,  Splunk  Inc.   March  8,  2012   Burlingame,   Listen  to  y©  2012,  Splunk  Inc.   Copyright  our  data.  
  • 19. AnalyEcs  QuesEons  Enabled   ü  Purchase  volume  by  hour,  by  day,  by  month   ü  How  long  are  purchases  taking?   ü  Are  my  purchases  taking  longer  than  they  did  last  month?   ü  Are  my  systems  geong  slower?   ü  How  many  purchases  are  failing?     ü  Which  specific  purchases  are  failing?  Copyright  ©  2012,  Splunk  Inc.   March  8,  2012   Burlingame,   Listen  to  y©  2012,  Splunk  Inc.   Copyright  our  data.  
  • 20. AnalyEcs  Dashboard  Copyright  ©  2012,  Splunk  Inc.   March  8,  2012   Burlingame,   Listen  to  y©  2012,  Splunk  Inc.   Copyright  our  data.  
  • 21. Streaming  Radio  Example  Copyright  ©  2012,  Splunk  Inc.   March  8,  2012   Burlingame,   Listen  to  y©  2012,  Splunk  Inc.   Copyright  our  data.  
  • 22. Group  TransacEons   sourcetype=radiolog | transaction IPAddress startswith="play" endswith="stop"  Copyright  ©  2012,  Splunk  Inc.   March  8,  2012   Burlingame,   Listen  to  y©  2012,  Splunk  Inc.   Copyright  our  data.  
  • 23. Calculate  Concurrency   " sourcetype=radiolog | transaction IPAddress startswith="play" endswith="stop" | concurrency duration=duration  Copyright  ©  2012,  Splunk  Inc.   March  8,  2012   Burlingame,   Listen  to  y©  2012,  Splunk  Inc.   Copyright  our  data.  
  • 24. Add  Lookups  and  StaEsEcs   > sourcetype=radiolog | transaction IPAddress startswith="play" endswith="stop" | concurrency duration=duration | eval key=1 | lookup songs key | stats first(song) as song max(concurrency) as concurrency by id | stats sum(concurrency) by songCopyright  ©  2012,  Splunk  Inc.   Listen  to  your  data.  
  • 25. Developer  Concerns  Copyright  ©  2012,  Splunk  Inc.   March  8,  2012   Burlingame,   Listen  to  y©  2012,  Splunk  Inc.   Copyright  our  data.  
  • 26.   Developer  Concern:  Performance     92  sec   15  sec  Copyright  ©  2012,  Splunk  Inc.   March  8,  2012   Burlingame,   Listen  to  y©  2012,  Splunk  Inc.   Copyright  our  data.  
  • 27. Developer  Concern:  Infrastructure  Cost   ü Splunk  Requires  standard  hardware   ü Start  with  an  easy  download   ü Free  Apps  for  domain  specific  analyEcs   ü Proven  in  Big  Data  Copyright  ©  2012,  Splunk  Inc.   March  8,  2012   Burlingame,   Listen  to  y©  2012,  Splunk  Inc.   Copyright  our  data.  
  • 28. Developer  Concern:  Refactoring  Code   ü Start  gradually  and  grow  organically   ü Develop  future  applicaEons  with  analyEcs  and  Splunk  in  mind   ü Build  closer  relaEonships  with  Ops,  Support  and  QA   ü ROI  can  be  priceless  Copyright  ©  2012,  Splunk  Inc.   March  8,  2012   Burlingame,   Listen  to  y©  2012,  Splunk  Inc.   Copyright  our  data.  
  • 29. Developer  Concern:  How  Much  to  Log   Two  approaches  to  event  logs:   ü Log  what  is  evidently  required   ü Open  the  flood-­‐gates     QuanEty  and  granularity  can  vary  based  on  task:   -­‐  Diagnosis   -­‐  ReporEng   -­‐  AnalyEcs  Copyright  ©  2012,  Splunk  Inc.   March  8,  2012   Burlingame,   Listen  to  y©  2012,  Splunk  Inc.   Copyright  our  data.  
  • 30. Logging  Best  PracEces  Copyright  ©  2012,  Splunk  Inc.   March  8,  2012   Burlingame,   Listen  to  y©  2012,  Splunk  Inc.   Copyright  our  data.  
  • 31. Create  Human  Readable  Events   ü  Log  in  Text   ü  Make  it  easy  for  humans   ü  Categorize   ü  Avoid  XML  or  JSON  Copyright  ©  2012,  Splunk  Inc.   March  8,  2012   Burlingame,   Listen  to  y©  2012,  Splunk  Inc.   Copyright  our  data.  
  • 32. Clearly  Time  Stamp  Every  Event   ü  Do  not  use  Fme  offsets   ü  Use  human  readable  Fmestamps   ü  Favor  the  beginning  of  the  line  Copyright  ©  2012,  Splunk  Inc.   March  8,  2012   Burlingame,   Listen  to  y©  2012,  Splunk  Inc.   Copyright  our  data.  
  • 33. Use  Clear  Key/Value  Pairs   Example  (Bad):   !Log.debug(“error 454 - %s %d”, userId, transId)! Example  (Good):   !Log.debug(“orderstatus=error,errorcode=454,! !user=%s,transactionid=%d”, userId, transId)!Copyright  ©  2012,  Splunk  Inc.   March  8,  2012   Burlingame,   Listen  to  y©  2012,  Splunk  Inc.   Copyright  our  data.  
  • 34. Break  MulE-­‐Value  InformaEon  Into  Separate  Events   Example  (Bad):   <TS>  phonenumber=415-­‐555-­‐1212,app=angrybirds,facebook   Example  (Good):   <TS>  phonenumber=415-­‐555-­‐1212,  app=angrybirds,  installdate=xx/xx/xx   <TS>  phonenumber=415-­‐555-­‐1212,  app=facebook,  installdate=yy/yy/yy   Copyright  ©  2012,  Splunk  Inc.   March  8,  2012   Burlingame,   Listen  to  y©  2012,  Splunk  Inc.   Copyright  our  data.  
  • 35. Log  Unique  IdenEfiers   ü  Allows  to  track  transacEons  in  detail   ü  Use  TransiEve  Closure  if  you  need  to:     transid=abcdef,     Transac7on   transid=abcdef,    otherid=  qrstuv,  .  .  .  .  .   otherid=qrstuv  Copyright  ©  2012,  Splunk  Inc.   March  8,  2012   Burlingame,   Listen  to  y©  2012,  Splunk  Inc.   Copyright  our  data.  
  • 36. Using  Header  Lines  for  Keys   <TS>   USER              PID    %CPU  %MEM            VSZ        RSS      TT    STAT  STARTED            TIME  COMMAND   root                41    21.9    1.7    3233968  143624      ??    Rs        7Jul11    48:09.67  /System/Library/foo   rdas              790      4.5    0.4    4924432    32324      ??    S          8Jul11      9:00.57  /System/Library/baz     .  .  .  .  .  .  .  .  •  Splunk  will  interpret  the  column  headers  as  keys  and  each  line  as  values   Copyright  ©  2012,  Splunk  Inc.   March  8,  2012   Burlingame,   Listen  to  y©  2012,  Splunk  Inc.   Copyright  our  data.  
  • 37. Top  Takeaways   Log  anything  that  can  add   value  when  aggregated   and/or  visualized   Copyright  ©  2012,  Splunk  Inc.   37   Listen  to  your  data.  
  • 38. Top  Takeaways   Simplify  your  life…   Splunk  logs  for  AnalyEcs   Copyright  ©  2012,  Splunk  Inc.   38   Listen  to  your  data.  
  • 39. Thanks!   QuesEons?  Copyright  ©  2012,  Splunk  Inc.   Listen  to  your  data.