Universitat Politecnica de Catalunya, Castelldefels – Jun, 5th 2013
Traffic	
  Matrices	
  and	
  its	
  measurement	
  
	
 ...
Agenda
o  Introduction
o  The tool: pmacct
o  Setting the pitch
o  Case study: peering at AS286
Traffic	
  Matrices	
  and	
...
Why	
  speaking	
  of	
  traffic	
  matrices?	
  
–  A;er	
  focusing	
  on	
  traffic	
  matrices	
  during	
  your	
  
cours...
Why	
  speaking	
  of	
  traffic	
  matrices?	
  
–  So,	
  a;er	
  all,	
  are	
  traffic	
  matrices	
  useful	
  to	
  a	
 ...
Review	
  of	
  traffic	
  matrices:	
  internal	
  
–  POP	
  to	
  POP,	
  AR	
  to	
  AR,	
  CR	
  to	
  CR	
  
CR
CR
CR
...
Review	
  of	
  traffic	
  matrices:	
  external	
  
–  Router	
  (AR	
  or	
  CR)	
  to	
  external	
  AS	
  or	
  external...
Let’s	
  focus	
  on	
  an	
  external	
  traffic	
  
matrix	
  to	
  support	
  peering	
  decisions	
  
–  Analysis	
  of	...
Our	
  traffic	
  matrix	
  visualized	
  
P	
   P	
  
P	
  
PE	
  
A	
  
PE	
  
D	
  
PE	
  
C	
  
PE	
  
B	
  
P3	
  
P1	
...
Our	
  traffic	
  matrix	
  shopping	
  cart	
  
–  What	
  is	
  needed:	
  
§  BGP	
  
§  (IGP)	
  
§  Telemetry	
  dat...
Agenda
o  Introduction
o  The tool: pmacct
o  Setting the pitch
o  Case study: peering at AS286
Traffic	
  Matrices	
  and	
...
pmacct	
  is	
  free	
  open-­‐source	
  GPL’ed	
  so;ware	
  
pmacct	
  
libpcap	
  
BGP	
  
maps	
  
IS-IS
MySQL	
  
Pos...
Why	
  pmacct	
  was	
  started	
  anyway?	
  
–  I	
  was	
  23	
  
–  I	
  had	
  just	
  joined	
  CNR	
  (Na>onal	
  C...
Why	
  pmacct	
  was	
  started	
  anyway?	
  
–  Before	
  turning	
  24	
  I	
  realized	
  I	
  was	
  wrong:	
  
§  T...
Usage	
  scenarios	
  
Key	
  pmacct	
  non-­‐technical	
  facts	
  
§  A	
  name	
  you	
  can’t	
  spell	
  a;er	
  the	
  second	
  drink	
  ...
Some	
  technical	
  facts	
  
§  Pluggable	
  architecture	
  
•  Straighnorward	
  to	
  add	
  support	
  for	
  new	
...
Introducing	
  BGP	
  na>vely	
  into	
  a	
  
NetFlow/sFlow	
  collector	
  
–  pmacct	
  introduced	
  a	
  Quagga-­‐bas...
Agenda
o  Introduction
o  The tool: pmacct
o  Setting the pitch
o  Case study: peering at AS286
Traffic	
  Matrices	
  and	
...
Getng	
  BGP	
  to	
  the	
  collector	
  
–  Let	
  the	
  collector	
  BGP	
  peer	
  with	
  all	
  PE	
  devices:	
  
...
–  Set	
  the	
  collector	
  as	
  iBGP	
  peer	
  at	
  the	
  PE	
  devices:	
  	
  
§  Configure	
  it	
  as	
  a	
  R...
Getng	
  telemetry	
  to	
  the	
  collector	
  
–  Export	
  ingress-­‐only	
  measurements	
  at	
  all	
  PE	
  
device...
Telemetry	
  data/BGP	
  correla>on	
  
Storing	
  data	
  persistently	
  	
  
–  Data	
  need	
  to	
  be	
  aggregated	
  both	
  in	
  spa>al	
  and	
  
tempo...
Briefly	
  on	
  scalability	
  
–  A	
  single	
  collector	
  might	
  not	
  fit	
  it	
  all:	
  
§  Memory:	
  can’t	
...
Agenda
o  Introduction
o  The tool: pmacct
o  Setting the pitch
o  Case study: peering at AS286
Traffic	
  Matrices	
  and	
...
Case-­‐study:	
  peering	
  at	
  AS286	
  
Nego>a>on	
  
Engineering	
  
Produc>on	
  
Op>miza>on	
  
–  Peering	
  as	
 ...
Case-­‐study:	
  peering	
  at	
  AS286	
  
–  250+	
  Gbps	
  rou>ng-­‐domain	
  
–  100+	
  high-­‐end	
  routers	
  aro...
–  AS286	
  backbone	
  routers	
  are	
  first	
  configured	
  from	
  
templates:	
  
§  NetFlow	
  +	
  BGP	
  collecto...
Thanks for your attention!
Questions? Now or later ..
Paolo Lucente
pmacct
<paolo at pmacct dot net>
Keep in touch via Lin...
Backup slides
Traffic	
  Matrices	
  and	
  its	
  measurement	
  
	
  
Universitat Politecnica de Catalunya, Castelldefels ...
IGP	
  (IS-­‐IS)	
  integra>on	
  
–  A	
  Quagga-­‐based	
  IS-­‐IS	
  daemon	
  was	
  introduced:	
  
§  Implemented	
...
Storing	
  data	
  persisently:	
  RDBMS	
  
create table acct_bgp (
agent_id INT(4) UNSIGNED NOT NULL,
as_src INT(4) UNSI...
Storing	
  data	
  persisently:	
  RDBMS	
  
–  Traffic	
  delivered	
  to	
  a	
  BGP	
  peer,	
  per	
  loca>on:
mysql> SE...
Storing	
  data	
  persisently:	
  RDBMS	
  
–  Traffic	
  breakdown,	
  ie.	
  top	
  N	
  grouping	
  BGP	
  peers	
  
of	...
Storing	
  data	
  persisently:	
  MongoDB	
  
–  pmacct	
  opening	
  to	
  noSQL	
  databases	
  
–  noSQL	
  landscape	...
Upcoming SlideShare
Loading in …5
×

Traffic Matrices and its measurement

814 views

Published on

Paolo Lucente, author of the software pmacct.
Traffic matrices can greatly benefit key IP Service Provider activities like capacity planning, traffic engineering, better understand their traffic patterns and take meaningful peering decisions. This talk wants to present a way to build traffic matrices using telemetry data and BGP, leveraging along the way some case-studies with a technical cut. pmacct (http://www.pmacct.net) is a commonly used, free, open-source IPv4/IPv6a ccounting package which integrates a NetFlow/sFlow and a multi-RIB BGP collector in a single piece of software and is authored by the presenter.

Published in: Technology
0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
814
On SlideShare
0
From Embeds
0
Number of Embeds
3
Actions
Shares
0
Downloads
10
Comments
0
Likes
2
Embeds 0
No embeds

No notes for slide

Traffic Matrices and its measurement

  1. 1. Universitat Politecnica de Catalunya, Castelldefels – Jun, 5th 2013 Traffic  Matrices  and  its  measurement     Paolo Lucente pmacct  
  2. 2. Agenda o  Introduction o  The tool: pmacct o  Setting the pitch o  Case study: peering at AS286 Traffic  Matrices  and  its  measurement     Universitat Politecnica de Catalunya, Castelldefels – Jun, 5th 2013
  3. 3. Why  speaking  of  traffic  matrices?   –  A;er  focusing  on  traffic  matrices  during  your   course  you  can  find  this  ques>on  silly.  But  ..     §  In  the  industry,  finding  knowledgeable  people  on  the   subject  is  hard   §  Quite  unspoken  subject  at  technical  conferences   §  Presenta>ons  (nega>vely)  stop  to  the  math  
  4. 4. Why  speaking  of  traffic  matrices?   –  So,  a;er  all,  are  traffic  matrices  useful  to  a   network  operator  in  the  first  place?  Yes  …   §  Capacity  planning  (build  capacity  where  needed)   §  Traffic  Engineering  (steer  traffic  where  capacity  is   available)   §  BeQer  understand  traffic  paQerns  (what  to  expect,   without  a  crystal  ball)   §  Support  peering  decisions  (traffic  insight,  traffic   engineering  at  the  border,  support  what  if  scenarios)  
  5. 5. Review  of  traffic  matrices:  internal   –  POP  to  POP,  AR  to  AR,  CR  to  CR   CR CR CR CR PoP AR AR AR AR AR PoP AR Customers Customers AS1 AS2 AS3 AS4 AS5 Server Farm 1 Server Farm 2 © 2010 Cisco Systems, Inc./Cariden Technologies, Inc.
  6. 6. Review  of  traffic  matrices:  external   –  Router  (AR  or  CR)  to  external  AS  or  external  AS  to   external  AS  (for  IP  transit  providers)   CR CR CR CR PoP AR AR AR AR AR PoP AR Customers Customers AS1 AS2 AS3 AS4 AS5 Server Farm 1 Server Farm 2 © 2010 Cisco Systems, Inc./Cariden Technologies, Inc.
  7. 7. Let’s  focus  on  an  external  traffic   matrix  to  support  peering  decisions   –  Analysis  of  exis>ng  peers  and  interconnects:   §  Support  policy  and  rou>ng  changes   §  Fine-­‐grained  accoun>ng  of  traffic  volumes  and  ra>os   §  Determine  backbone  costs  associated  to  peering   §  Determine  revenue  leaks   –  Planning  of  new  peers  and  interconnects:   §  Who  to  peer  next   §  Where  to  place  next  interconnect   §  Modeling  and  forecas>ng  
  8. 8. Our  traffic  matrix  visualized   P   P   P   PE   A   PE   D   PE   C   PE   B   P3   P1   P2   P4   BZ   A  =  {  src_as,  in_iface,  peer_src_ip,  peer_dst_ip,  as_path,  tstamp,  bytes  }   {  CZ,  X  (-­‐>  CY),  PE  C,  PE  A,  AZ_AY,  <>me>,  <bytes>  }   {  BX,  Y  (-­‐>  BY),  PE  B,  PE  C,  CY_CX,  <>me>,  <bytes>  }   {  AX,  Z  (-­‐>  AZ),  PE  A,  PE  B,  BY_BZ,  <>me>,  <bytes>  }  
  9. 9. Our  traffic  matrix  shopping  cart   –  What  is  needed:   §  BGP   §  (IGP)   §  Telemetry  data:  NetFlow,  sFlow   §  Collector  infrastructure:  tool,  system(s)   §  Storage:  either  of  RDBMS,  RRD,  noSQL   §  Maintenance  and  post-­‐processing  scripts   –  Risks:   §  800  pound  gorilla  project  
  10. 10. Agenda o  Introduction o  The tool: pmacct o  Setting the pitch o  Case study: peering at AS286 Traffic  Matrices  and  its  measurement     Universitat Politecnica de Catalunya, Castelldefels – Jun, 5th 2013
  11. 11. pmacct  is  free  open-­‐source  GPL’ed  so;ware   pmacct   libpcap   BGP   maps   IS-IS MySQL   PostgreSQL   SQLite  3   MongoDB   BerkeleyDB   flat-­‐files   memory   tables   NetFlow   IPFIX   sFlow   sFlow   NetFlow   IPFIX   tee  
  12. 12. Why  pmacct  was  started  anyway?   –  I  was  23   –  I  had  just  joined  CNR  (Na>onal  Council  for  Research)   –  I  thought  you  can  manage  IP  networks  like  that  
  13. 13. Why  pmacct  was  started  anyway?   –  Before  turning  24  I  realized  I  was  wrong:   §  Too  much  data  (ie.  flow-­‐tools)  and  canned  reports  (ie.  ntop)  were   causing  lack  of  visibilty  in  my  backbone.   §  Not  everything  was  invented  yet!  
  14. 14. Usage  scenarios  
  15. 15. Key  pmacct  non-­‐technical  facts   §  A  name  you  can’t  spell  a;er  the  second  drink   §  10  years  old  project   §  Free,  open-­‐source,  independent   §  Under  ac>ve  development   §  Innova>on  being  introduced   §  Well  deployed  around,  also  large  SPs   §  Aims  to  be  the  traffic  accoun>ng  tool  closer  to   the  SP  community  needs  
  16. 16. Some  technical  facts   §  Pluggable  architecture   •  Straighnorward  to  add  support  for  new  collec>on   methods  or  backends   §  An  abstrac>on  layer  allows  out-­‐of-­‐the-­‐box  any   collec>on  method  to  interact  with  any  backend   §  Both  mul>-­‐process  and  (coarse)  mul>-­‐threading   •  Mul>ple  plugins  (of  same  or  different  type)  can  be   instan>ated  at  run>me,  each  with  own  config    
  17. 17. Introducing  BGP  na>vely  into  a   NetFlow/sFlow  collector   –  pmacct  introduced  a  Quagga-­‐based  BGP  daemon   §  Implemented  as  a  parallel  thread  within  the  collector   §  Doesn’t  send  UPDATEs  and  WITHDRAWs  whatsoever   §  Behaves  as  a  passive  BGP  neighbor   §  Maintains  per-­‐peer  BGP  RIBs   §  Supports  32-­‐bit  ASNs;  IPv4  and  IPv6  families     –  Why  BGP  at  the  collector?   §  Telemetry  reports  on  forwarding-­‐plane   §  Telemetry  should  not  move  control-­‐plane   informa>on  over  and  over    
  18. 18. Agenda o  Introduction o  The tool: pmacct o  Setting the pitch o  Case study: peering at AS286 Traffic  Matrices  and  its  measurement     Universitat Politecnica de Catalunya, Castelldefels – Jun, 5th 2013
  19. 19. Getng  BGP  to  the  collector   –  Let  the  collector  BGP  peer  with  all  PE  devices:   facing  peers,  transit  and  customers.   –  Determine  memory  footprint  (below  in  MB/peer)   44.03   22.73   19.97   18.59   18.12   17.89   17.76   17.57   17.48   17.39   50   0   10   20   30   40   50   60   0   200   400   600   800   1000   1200   1400   MB/peer  >=  0.12.4   MB/peer  <  0.12.4   Number  of  BGP  peers   MB/peer   500K  IPv4  routes,  50K  IPv6  routes,  64-­‐bit  executable   ~  9GB  total  memory  @  500  peers  
  20. 20. –  Set  the  collector  as  iBGP  peer  at  the  PE  devices:     §  Configure  it  as  a  RR  client  for  best  results   §  Collector  acts  as  iBGP  peer  across  (sub-­‐)ASes   –  BGP  next-­‐hop  has  to  represent  the  remote  edge   of  the  network  model:   §  Typical  scenario  for  MPLS  networks   §  Can  be  followed  up  to  cover  specific  scenarios  like:   •  BGP  confedera>ons   –  Op>onally  polish  the  AS-­‐Path  up  from  sub-­‐ASNs   •  default  gateway  defined  due  to  par>al  or  default-­‐only   rou>ng  tables     Getng  BGP  to  the  collector  (cont.d)  
  21. 21. Getng  telemetry  to  the  collector   –  Export  ingress-­‐only  measurements  at  all  PE   devices:  facing  peers,  transit  and  customers.   §  Traffic  is  routed  to  des>na>on,  so  plenty  of   informa>on  on  where  it’s  going  to   §  It’s  crucial  instead  to  get  as  much  as  possible  about   where  traffic  is  coming  from           –  Leverage  data  reduc>on  techniques  at  the  PE:   §  Sampling   §  Aggrega>on  (but  be  sure  to  carry  IP  prefixes!)  
  22. 22. Telemetry  data/BGP  correla>on  
  23. 23. Storing  data  persistently     –  Data  need  to  be  aggregated  both  in  spa>al  and   temporal  dimensions  before  being  wriQen  down:   §  Op>mal  usage  of  system  resources   §  Avoids  expensive  consolida>on  of  micro-­‐flows       §  Suitable  for  project-­‐driven  data-­‐sets   –  Open-­‐source  RDBMS  appear  a  natural  choice   §  Able  to  handle  large  data-­‐sets   §  Flexible  and  standardized  query  language   §  Solid  and  scalable  (clustering,  par>>oning)   –  noSQL  DBs  picking  up:  between  myth  and  bust  ..  
  24. 24. Briefly  on  scalability   –  A  single  collector  might  not  fit  it  all:   §  Memory:  can’t  store  all  BGP  full  rou>ng  tables   §  CPU:  can’t  cope  with  the  pace  of  telemetry  export   §  Divide-­‐et-­‐impera  approach  is  valid:   •  Assign  PEs  (both  telemetry  and  BGP)  to  collectors   •  Assign  collectors  to  RDBMSs;  or  cluster  the  RDBMS.     –  The  matrix  can  get  big,  but  can  be  reduced:   §  Keep  smaller  routers  out  of  the  equa>on   §  Filter  out  specific  services/customers  on  dense  routers   §  Focus  on  relevant  traffic  direc>on  (ie.  upstream  if  CDN,   downstream  if  ISP)   §  Sample  or  put  thresholds  on  traffic  relevance  
  25. 25. Agenda o  Introduction o  The tool: pmacct o  Setting the pitch o  Case study: peering at AS286 Traffic  Matrices  and  its  measurement     Universitat Politecnica de Catalunya, Castelldefels – Jun, 5th 2013
  26. 26. Case-­‐study:  peering  at  AS286   Nego>a>on   Engineering   Produc>on   Op>miza>on   –  Peering  as  a  cycle   –  NetFlow  +  BGP  traffic  matrix   steers  peering  op>miza>on:   §  Iden>fy  new  and  “old”  peers   §  Traffic  analysis:  backbone   costs,  95th  percen>les,  ra>os   §  Analysis  of  interconnec>on   density  and  traffic  dispersion   §  Forecas>ng  and  trending   §  Ad-­‐hoc  queries  from  Design   &  Engineering  and  indeed  …   the  IPT  Product  Manager  
  27. 27. Case-­‐study:  peering  at  AS286   –  250+  Gbps  rou>ng-­‐domain   –  100+  high-­‐end  routers  around   the  globe:   §  Export  sampled  NetFlow   §  Adver>se  full  rou>ng  table   §  Mix  of  Juniper  and  Cisco   –  Collector  environment:   §  Runs  on  2  Solaris/SPARC  zones   §  pmacct:    dual-­‐core,  4GB  RAM   §  MySQL:  quad-­‐core,  24GB  RAM,   500  GB  disk   –  Data  reten>on:  6  months     AS286  rou>ng   domain   pmacct   MySQL   Internal  users  
  28. 28. –  AS286  backbone  routers  are  first  configured  from   templates:   §  NetFlow  +  BGP  collector  IP  address  defined  over  there   §  Enabler  for  auto-­‐discovery  of  new  devices   –  Edge  interfaces  are  provisioned  following    service   delivery  manuals:   §  Relevant  manuals  and  TSDs  include  NetFlow  ac>va>on   §  Periodic  checks  NetFlow  is  ac>ve  where  it  should   –  Maps,  ie.  source  peer-­‐AS,  are  re-­‐built  periodically   Case-­‐study:  peering  at  AS286  
  29. 29. Thanks for your attention! Questions? Now or later .. Paolo Lucente pmacct <paolo at pmacct dot net> Keep in touch via LinkedIn Traffic  Matrices  and  its  measurement     Universitat Politecnica de Catalunya, Castelldefels – Jun, 5th 2013
  30. 30. Backup slides Traffic  Matrices  and  its  measurement     Universitat Politecnica de Catalunya, Castelldefels – Jun, 5th 2013
  31. 31. IGP  (IS-­‐IS)  integra>on   –  A  Quagga-­‐based  IS-­‐IS  daemon  was  introduced:   §  Implemented  as  a  parallel  thread  within  the  collector   §  IS-­‐IS  neighborship  over  a  GRE  tunnel   §  Currently  limited  to  single  neighborhip,  single  level,   single  topology   §  Useful  to  look  up  non  BGP-­‐routed  networks   §  Promising,  many  plans:   •  Implement  a  non  real-­‐>me,  IGP-­‐protocol  agnos>c  version   •  LFA  kind  of  computa>ons   •  Collec>ng  feedback  on  interest  about  TRILL  
  32. 32. Storing  data  persisently:  RDBMS   create table acct_bgp ( agent_id INT(4) UNSIGNED NOT NULL, as_src INT(4) UNSIGNED NOT NULL, as_dst INT(4) UNSIGNED NOT NULL, peer_as_src INT(4) UNSIGNED NOT NULL, peer_as_dst INT(4) UNSIGNED NOT NULL, peer_ip_src CHAR(15) NOT NULL, peer_ip_dst CHAR(15) NOT NULL, comms CHAR(24) NOT NULL, as_path CHAR(21) NOT NULL, local_pref INT(4) UNSIGNED NOT NULL, med INT(4) UNSIGNED NOT NULL, packets INT UNSIGNED NOT NULL, bytes BIGINT UNSIGNED NOT NULL, stamp_inserted DATETIME NOT NULL, stamp_updated DATETIME, PRIMARY KEY (…) ); BGP Fields Counters Time Tag shell> cat peers.map id=65534 ip=X in=A id=65533 ip=Y in=B src_mac=J id=65532 ip=Z in=C bgp_nexthop=W [ … ] shell> cat pretag.map id=100 peer_src_as=<customer> id=80 peer_src_as=<peer> id=50 peer_src_as=<IP transit> [ … ] –  In  any  schema  (a  subset  of)  BGP  primi>ves  can  be   freely  mixed  with  (a  subset  of)  L1-­‐L7  primi>ves
  33. 33. Storing  data  persisently:  RDBMS   –  Traffic  delivered  to  a  BGP  peer,  per  loca>on: mysql> SELECT peer_as_dst, peer_ip_dst, SUM(bytes), stamp_inserted FROM acct_bgp WHERE peer_as_dst = <peer | customer | IP transit> AND stamp_inserted = < today | last hour | last 5 mins > GROUP BY peer_as_dst, peer_ip_dst;   –  Aggregate  AS  PATHs  to  the  second  hop:   mysql> SELECT SUBSTRING_INDEX(as_path, ‘.’, 2) AS as_path, bytes FROM acct_bgp WHERE local_pref = < IP transit pref> AND stamp_inserted = < today | yesterday | last week > GROUP BY SUBSTRING_INDEX(as_path, ‘.’, 2) ORDER BY SUM(bytes); –  Focus  peak  hour  (say,  8pm)  data:   mysql> SELECT … FROM … WHERE stamp_inserted LIKE ‘2010-02-% 20:00:00’ …
  34. 34. Storing  data  persisently:  RDBMS   –  Traffic  breakdown,  ie.  top  N  grouping  BGP  peers   of  the  same  kind  (ie.  peers,  customers,  transit):   mysql> SELECT … FROM … WHERE … local_pref = <<peer | customer | IP transit> pref> … –  Download  traffic  matrix  (or  a  subset  of  it)  to  3rd   party  backbone  planning/traffic  engineering   applica>on  (ie.  Cariden,  Wandl,  etc.):   mysql> SELECT peer_ip_src, peer_ip_dst, bytes, stamp_inserted FROM acct_bgp WHERE [ peer_ip_src = <location A> AND peer_ip_dst = <location Z> AND … ] stamp_inserted = < today | last hour | last 5 mins > GROUP BY peer_ip_src, peer_ip_dst;  
  35. 35. Storing  data  persisently:  MongoDB   –  pmacct  opening  to  noSQL  databases   –  noSQL  landscape  difficult  to  move  through,  ie.   fragmented  and  lacks  of  standardiza>on   –  MongoDB  seemed  interes>ng  for:   §  Its  na>ve  grouping  opera>on  (more  performing  and   less  complex  than  map/reduce)   §  Its  horizontal  scaling  concept  (called  sharding)   –  Currently  collec>ng  feedbck:  some  preliminar   tes>ng  at  one  large  operator  looks  promising  ..  

×