SlideShare a Scribd company logo
1 of 59
Download to read offline
QCON	
  
                                              London	
  2013	
  




Copyright	
  Push	
  Technology	
  2012	
  
My	
  ‘number’	
  




#6752	
  
Copyright	
  Push	
  Technology	
  2012	
  
•  Distributed	
  Systems	
  /	
  HPC	
  guy.	
  
                                                 	
  
                                              •  Chief	
  Scien*st	
  :-­‐	
  at	
  Push	
  Technology	
  

                                              •  Alumnus	
  of	
  :-­‐	
  	
  
                                                 	
  
                                                 Motorola,	
  IONA,	
  BeTair,	
  JPMC,	
  StreamBase.	
  

                                              •  School:	
  Trinity	
  College	
  Dublin.	
  	
  
                                                 -­‐	
  BA	
  (Mod).	
  Comp.	
  Sci.+	
  	
  
                                                 -­‐	
  M.Sc.	
  Networks	
  &	
  Distributed	
  Systems	
  	
  
                                                 	
  
                                              •  Responds	
  to:	
  Guinness,	
  Whisky	
  



                                                                                 About	
  me?	
  
Copyright	
  Push	
  Technology	
  2012	
  
                                                          Darach@PushTechnology.com	
  
Highly	
  Available	
  
                                                     Real-­‐Time	
  
                                              Data	
  Distribu`on	
  
                                              at	
  the	
  network’s	
  
                                                       Edge	
  




Copyright	
  Push	
  Technology	
  2012	
  
HA.	
  Black	
  swans	
  &	
  Pigfish!	
  




Copyright	
  Push	
  Technology	
  2012	
  
When	
  this	
  …	
  



                                                          esse	
  hic	
  
                                Service	
                 draconum	
  




Copyright	
  Push	
  Technology	
  2012	
  
…	
  inevitably	
  happens	
  


             HA!	
  HA!	
  
               W8!!	
  




Copyright	
  Push	
  Technology	
  2012	
  
Over	
  the	
  Edge,	
  where	
  Fallacies	
  roam	
  

              - The	
  network	
  is	
  reliable	
  
              - Latency	
  is	
  zero	
  	
  
              - Bandwidth	
  is	
  infinite	
  
              - There	
  is	
  one	
  administrator	
  
              - Topology	
  does	
  not	
  change	
  
              - The	
  network	
  is	
  secure	
  
              - Transport	
  cost	
  is	
  zero	
  
              - The	
  network	
  is	
  homogeneous	
  
              	
  
                                               hgp://en.wikipedia.org/wiki/Fallacies_of_Distributed_Compu`ng	
  	
  
	
  

 Copyright	
  Push	
  Technology	
  2012	
  
&	
  the	
  last	
  mile.	
  Dragons	
  there	
  be.	
  
                              Buffer	
  
                              Bloat	
                                             Cell	
                                  Bagery!	
  
                                                                                  Coverage	
  


                                                                                        Download	
  Bandwidth	
  
   30.00	
  
   20.00	
  
                                                                                                                                                                                                         Expected	
  (Mbps)	
  
   10.00	
  
                                                                                                                                                                                                         Download	
  (Mbps)	
  
    0.00	
  
         Sep-­‐06	
  
  -­‐10.00	
                      Oct-­‐06	
                 Oct-­‐06	
           Nov-­‐06	
             Dec-­‐06	
          Dec-­‐06	
             Jan-­‐07	
                    Feb-­‐07	
  

                                                                                                                    Latency	
  
1.00E+00	
  
1.00E+01	
  
                                                                                                                                                           y	
  =	
  -­‐1.1667x	
  +	
  45703	
                           Latency	
  (ms)	
  
1.00E+02	
                                                                                                                                                             R²	
  =	
  0.02496	
                               Expected	
  (ms)	
  
1.00E+03	
  
                                                                                                                                                                                                                          Linear	
  (Latency	
  (ms))	
  
1.00E+04	
  
          Sep-­‐06	
                   Oct-­‐06	
                  Oct-­‐06	
             Nov-­‐06	
               Dec-­‐06	
               Dec-­‐06	
                    Jan-­‐07	
                Feb-­‐07	
  

               Copyright	
  Push	
  Technology	
  2012	
  
Distributed	
  Systems	
  

“A	
  distributed	
  system	
  is	
  one	
  in	
  
which	
  the	
  failure	
  of	
  a	
  computer	
  
you	
  didn’t	
  even	
  know	
  existed	
  
can	
  render	
  your	
  computer	
  
unusable”	
  
                                                                      -­‐	
  Leslie	
  Lamport	
  

Copyright	
  Push	
  Technology	
  2012	
  
High	
  Availability	
  

“The	
  price	
  of	
  reliability	
  is	
  the	
  
pursuit	
  of	
  the	
  utmost	
  simplicity.”	
  
                                                             -­‐  Sir	
  Tony	
  Hoare,	
  Turing	
  Award	
  speech,	
  1980	
  	
  

	
  
“This	
  is	
  the	
  unfortunate	
  truth:	
  
Simplicity	
  is	
  Hard.”	
  
           	
            	
             	
     	
     	
        	
  -­‐	
  	
  Mosely	
  &	
  Marks,	
  Out	
  of	
  the	
  Tar	
  Pit,	
  2006	
  

Copyright	
  Push	
  Technology	
  2012	
  
High	
  Experiences	
  ~	
  Man	
  vs	
  Machine	
  
                                              Image:	
  hgp://www.phillymarke`nglabs.com/man-­‐vs-­‐machine/	
  	
  




Copyright	
  Push	
  Technology	
  2012	
  
Machines	
  are	
  ‘Easy’	
  

A	
  more	
  unfortunate	
  truth	
  is:	
  
	
  




Humans*	
  are	
  Harder.	
  	
  
	
  


Allegedly	
  Sergey*	
  can	
  be	
  Emasculated	
  
 by	
  Glass.	
  But,	
  I	
  bet	
  he	
  owns	
  a	
  god	
  phone.
  	
   	
   	
   	
   	
   	
   	
  *	
  Humans	
  are	
  grumpy!	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  *	
  Colors	
  chosen	
  at	
  random	
  

	
  
  Copyright	
  Push	
  Technology	
  2012	
  
Once	
  Upon	
  a	
  *me	
  …	
  



                                                                             SPOF




                                              There	
  was	
  a	
  very	
  useful	
  ligle	
  single	
  point	
  of	
  failure.	
  


Copyright	
  Push	
  Technology	
  2012	
  
It	
  was	
  so	
  successful	
  



                                                                                 SPOF




                                              It	
  grew	
  and	
  it	
  grew!	
  So	
  we	
  put	
  it	
  in	
  the	
  cloud.	
  


Copyright	
  Push	
  Technology	
  2012	
  
It	
  grew,	
  and	
  it	
  grew	
  …	
  



                                                                          SPOF




                                                    We	
  touched	
  it	
  with	
  all	
  of	
  our	
  devices.	
  


Copyright	
  Push	
  Technology	
  2012	
  
See	
  SPOF	
  run!	
  



                                                                          SPOF




                                              We	
  all	
  touched	
  it	
  except	
  Sergey	
  ‘the	
  emasculated	
  one’	
  


Copyright	
  Push	
  Technology	
  2012	
  
See	
  SPOF	
  spew	
  L	
  



                                                                   Wat?




            Is	
  there	
  nothing	
  we	
  can	
  do?	
  Esse	
  hic	
  draconum,	
  basically.	
  But,	
  yes	
  we	
  can	
  too	
  …	
  


Copyright	
  Push	
  Technology	
  2012	
  
Bad	
  things	
  happen	
  …	
  


                                                                         RapGenius	
  
                                                                            vs	
  
                                                                          Heroku.	
  
                                                                                     	
  
                                                                                  Anyone?	
  
                                                                                   Ferris?	
  




                                              It’s	
  either	
  dyno	
  park	
  this	
  or	
  whale	
  fail	
  in	
  the	
  zoo	
  …	
  


Copyright	
  Push	
  Technology	
  2012	
  
So	
  we	
  spun	
  up	
  another	
  instance	
  



                                              Master                                 Slave




                                              And	
  master	
  /	
  slave	
  dabbled	
  with	
  us	
  

Copyright	
  Push	
  Technology	
  2012	
  
Ac*ve/Passive	
  –	
  Maybe?	
  



                                                        Active                                   Passive
                                                        (Hot)                                     (Cold)




                                              MTTR	
  rela`vely	
  high.	
  MTBF	
  rela`vely	
  low.	
  	
  Ok	
  for	
  	
  
                                              ‘nice-­‐to-­‐have’	
  or	
  	
  ‘too-­‐late-­‐to-­‐cry-­‐now’	
  	
  facili`es.	
  	
  


Copyright	
  Push	
  Technology	
  2012	
  
Ac*ve/Passive	
  –	
  Hmm?	
  



                                              Active                           Passive
                                              (Hot)                            (Warm)




                    MTTR	
  slightly	
  beger.	
  Can	
  replicate	
  state	
  ac`vely	
  for	
  cheaper	
  recovery.	
  

Copyright	
  Push	
  Technology	
  2012	
  
Ac*ve/Ac*ve	
  –	
  Ugh…	
  



                                                  Active                                Active
                                                  Master                                Slave
                                                   (Hot)                                (Hot)




                           MTTR	
  negligeable.	
  Use	
  2x	
  the	
  bandwidth.	
  Get	
  ‘half’	
  the	
  value.	
  Dups	
  	
  

Copyright	
  Push	
  Technology	
  2012	
  
Ac*ve/Ac*ve	
  –	
  Ugh…	
  



                                                       Active                              Active
                                                       (Hot)                              (Warm)




                                        MTTR	
  negligeable.	
  Minify	
  bandwidth	
  usage	
  mostly.	
  	
  Dups	
  	
  

Copyright	
  Push	
  Technology	
  2012	
  
Ac*ve/Ac*ve	
  –	
  Ugh…	
  



                                                       Active                              Active
                                                       (Hot)                              (Warm)




                                        MTTR	
  negligeable.	
  Minify	
  bandwidth	
  usage	
  mostly.	
  	
  Dups	
  	
  

Copyright	
  Push	
  Technology	
  2012	
  
Spin	
  up	
  another	
  box?	
  


                      Active                                         Passive                                  Standby
                      (Hot)                                           (Cold)                                   (Cold)


                         Speak no evil                                   Hear no evil                              See no evil




                                    Failback,	
  Failover,	
  Takeover?	
  Load	
  Balance?	
  This	
  is	
  gesng	
  costly!	
  

Copyright	
  Push	
  Technology	
  2012	
  
Can	
  implement	
  N-­‐Plex	
  HA	
  with	
  a	
  
                              ‘simple’	
  FSM	
  
                                                                               Initial           Boot/DoBoot



                                                Active
                                                              Active/DoSwtichDone                            Booting


                                                                               Switching/DoSwitch
        Stop/DoStopStart                                                                                               Sync/DoSyncStart
                                                                             Switching

                                                                                                              Sync'ing
                                                   Stopping

                                                                             Swtich/DoStandbyActivate
                                                                                                               Standby/DoSyncEnd
                                              Standby/DoStopDone
                                                                                              Standby




                                          Consider	
  roles	
  for	
  Applica`on,	
  Controller,	
  callbacks,	
  …	
  

Copyright	
  Push	
  Technology	
  2012	
  
Spin	
  up	
  a	
  cluster,	
  ring?	
  
                                                  Master



                Slave                                        Slave           A	
  simple	
  implementa`on	
  can	
  be	
  
                                                                             easier	
  to	
  build	
  than	
  HA	
  pairs.	
  Your	
  
                                                                             mileage	
  may	
  vary.	
  Can	
  be	
  masterless	
  
                                                                             too.	
  
                                                                             	
  	
  
Slave                                                                Slave   Where	
  possible,	
  use	
  exis`ng	
  
                                                                             solu`ons	
  such	
  as	
  Akka,	
  riak_core,	
  
                                                                             doozer,	
  Apache	
  Zookeeper.	
  
                                                                             	
  
                                                                             You’ll	
  s`ll	
  need	
  to	
  roll	
  your	
  own	
  
                Slave                                        Slave           data	
  recovery	
  -­‐	
  mostly.	
  Although	
  
                                                                             CRDT’s	
  are	
  changing	
  that	
  in	
  the	
  
                                                  Slave                      database	
  community.	
  


 Copyright	
  Push	
  Technology	
  2012	
  
One	
  more	
  thing	
  …	
  Real-­‐Time	
  streams	
  


                                              Data!	
  
                                      Meh                                                                   Meh


                       Real-­‐`me	
  data	
  streams	
  are	
  long-­‐lived,	
  conversa`onal,	
  can	
  be	
  stateful.	
  	
  

Copyright	
  Push	
  Technology	
  2012	
  
Streaming	
  Data	
  Recovery	
  
                                                   Type	
       Before	
             A]er	
                   Comments	
  
                                                Precise	
     T T T               T T T Completely	
  masks	
  
                                                              1	
   2	
   3	
     4	
   5	
   6	
   failure.	
  
                                                    Gap	
     T T T               ?	
   T T May	
  result	
  in	
  data	
  loss	
  
                                                              1	
   2	
   3	
           5	
   6	
  
                   Repea*ng	
  Rollback	
                     T T T               T T T IO	
  preserva`on.	
  P/S	
  
IIBECM
                                                              1	
   2	
   3	
     2	
   3	
   4	
   dups	
  are	
  equivalent.	
  
                                         Convergent	
         T T T               X X X IO	
  preserva`on.	
  P/S	
  
                                           Rollback	
         1	
   2	
   3	
     2	
   3	
   4	
   dups	
  may	
  differ	
  ini`ally	
  
                                                                                                    but	
  EC.	
  
                   Divergent	
  Rollback	
                    T T T               X X X	
   IO	
  preserva`on.	
  P/S	
  
                                                              1	
   2	
   3	
     2	
   3	
   4	
   dups	
  will	
  differ	
  

                         hgp://nms.lcs.mit.edu/papers/ha-­‐icde05.pdf	
  -­‐	
  HA	
  algos	
  for	
  DSPs	
  	
  

  Copyright	
  Push	
  Technology	
  2012	
  
But,	
  problem:	
  The	
  bird,	
  basically.	
  
             Immediately	
  Inconsistent.	
  
             But,	
  Eventually	
  Consistent	
  
             …	
  Maybe.	
  
             	
  
             	
  
             	
  
             Humans	
  
             Hate	
  
             Repea*ng	
  
             Rollback	
  
             Recovery	
                       IIBECM
Copyright	
  Push	
  Technology	
  2012	
  
Ugh,	
  just	
  one	
  more	
  thing	
  …	
  


                                              State?	
  
                              Finagle                                                                       Finagle



                                  Data	
  rarely	
  passes	
  through	
  a	
  box	
  without	
  being	
  fiddled	
  with.	
  

Copyright	
  Push	
  Technology	
  2012	
  
Streaming	
  Opera*ons	
  –	
  4	
  types	
  
                                                   Type	
                                        Comments	
  
                                              Arbitrary	
     Completely	
  masks	
  failure.	
  
                                   Determinis*c	
             Given	
  Same	
  Input,	
  Same	
  State,	
  produce	
  Same	
  Output	
  
               Convergent-­‐capable	
                         Given	
  Same	
  Input,	
  Earlier	
  Start,	
  Empty	
  State,	
  converge	
  
                                                              to	
  Same	
  Output	
  as	
  Determinis`c	
  case	
  
                                         Repeatable	
         Given	
  Same	
  Input,	
  Earlier	
  State,	
  Empty	
  State,	
  requires	
  
                                                              at	
  most	
  one	
  input	
  to	
  product	
  at	
  most	
  one	
  output	
  to	
  be	
  
                                                              convergent-­‐capable	
  and	
  same	
  as	
  determinis`c	
  case	
  




                       hgp://nms.lcs.mit.edu/papers/ha-­‐icde05.pdf	
  -­‐	
  HA	
  algos	
  for	
  DSPs	
  	
  

Copyright	
  Push	
  Technology	
  2012	
  
DOH?	
  


                  Data	
  Recovery	
  Type	
  
                  x	
  Opera*on	
  Type	
  =	
  
                        The	
  Hard	
  Part	
  

Copyright	
  Push	
  Technology	
  2012	
  
Just	
  one	
  last	
  one	
  more	
  thing	
  …	
  


                           Human	
  
                         Expecta*ons	
  
                     We	
  are	
  fickle,	
  grumpy,	
  changeable,	
  intolerant	
  …	
  

Copyright	
  Push	
  Technology	
  2012	
  
Let’s	
  see.	
  Trading	
  systems?	
  
                           Market
                            Market
                            Data                  Prices                         Prices
                             Market
                             Data
                               Market
                              Data
                                Data
                                                                  Broker         Orders
                                                                  Dealer
                                                                  Service
                             Market
                            Market
                          Market
                            Execution
                         Market
                          Execution
                         Execution
                                                                                                    Fat Cat
                        Execution                 "Flow"
                                                                                 Trades
                                                                                                    Banker




                                                 It Depends…                Precise Recovery
                                              It's Complicated!
                                                                            Gap Recovery

There	
  are	
  at	
  least	
  26	
  race	
  condi`ons	
  in	
  	
  
FIX	
  4.2	
  Order	
  Cancel/Replace/Reject	
                                     And	
  please.	
  No	
  stale	
  data!	
  

Copyright	
  Push	
  Technology	
  2012	
  
A	
  most	
  inconvenient	
  truth	
  


                                              Simple	
  is	
  
                                                Hard	
  
                                So	
  relax.	
  Let	
  it	
  fail.	
  Let	
  live	
  and	
  let’s	
  learn.	
  

Copyright	
  Push	
  Technology	
  2012	
  
Stop.	
  
             - The	
  network	
  is	
  not	
  reliable	
  	
  
                            nor	
  is	
  it	
  cost	
  free.	
  
             - Latency	
  is	
  not	
  zero	
  
                            nor	
  is	
  it	
  a	
  democracy.	
  
             - Bandwidth	
  is	
  not	
  infinite	
  
                            nor	
  predictable	
  especially	
  the	
  last	
  mile!	
  
             - There	
  is	
  not	
  only	
  one	
  administrator	
  
                  trust,	
  rela`onships	
  are	
  key	
  
             - Topology	
  does	
  change	
  
                            It	
  should,	
  however,	
  converge	
  eventually	
  
             - The	
  network	
  is	
  not	
  secure	
  
                            nor	
  is	
  the	
  data	
  that	
  flows	
  through	
  it	
  
             - Transport	
  cost	
  is	
  not	
  zero	
  
                            but	
  what	
  you	
  don’t	
  do	
  is	
  free	
  
             - The	
  network	
  is	
  not	
  homogeneous	
  
                            nor	
  is	
  it	
  smart	
  




Copyright	
  Push	
  Technology	
  2012	
  
Look.	
  

             - 	
  High	
  Availability	
  is	
  not	
  enough	
  for	
  Machine	
  to	
  Human	
  
             - 	
  High	
  Experience	
  requires	
  more	
  
                            Data	
  has	
  a	
  *me	
  value	
  –	
  out	
  of	
  date	
  data	
  is	
  bad.	
  
                            Data	
  has	
  variable	
  relevance,	
  based	
  on	
  loca*on,	
  device	
  …	
  
                            Data	
  rates	
  are	
  faster	
  than	
  human	
  percep*on	
  
                            You	
  cannot	
  send	
  100mb/sec	
  down	
  3G	
  networks	
  
                            You	
  cannot	
  send	
  100mb/sec	
  down	
  3G	
  networks	
  
                            You	
  cannot	
  …	
  waste	
  is	
  bad,	
  bloats	
  buffers,	
  and	
  ‘slows’	
  data.	
  
                            Currency	
  &	
  (Immediate)	
  Consistency	
  are	
  important	
  in	
  M2H.	
  
             - 	
  M2H	
  ‘High	
  Available	
  Experience’	
  might	
  work	
  for	
  M2M	
  …	
  
                            	
  
                            	
  
Copyright	
  Push	
  Technology	
  2012	
  
Listen.	
  

             - 	
  Every	
  nuance	
  comes	
  with	
  a	
  set	
  of	
  tradeoffs.	
  
             - 	
  Choosing	
  the	
  right	
  ones	
  can	
  be	
  hard,	
  but	
  it	
  pays	
  off.	
  
             - 	
  Context,	
  Environment	
  are	
  cri`cal	
  
             - 	
  Break	
  all	
  the	
  rules,	
  one	
  benchmark	
  at	
  a	
  `me.	
  
             	
  


             - 	
  Benchmark	
  Driven	
  Development	
  


Copyright	
  Push	
  Technology	
  2012	
  
Ac`on:	
  Data	
  Distribu`on	
  
Messaging	
  remixed	
  around:	
  
	
  
Relevance 	
  -­‐ 	
  Queue	
  depth	
  for	
  conflatable	
  data	
  should	
  be	
  0	
  or	
  1.	
  No	
  more	
  
     	
  
Responsiveness	
  -­‐	
  Use	
  HTTP/REST	
  for	
  things.	
  Stream	
  the	
  ligle	
  things	
  
	
  
Timeliness 	
  -­‐ 	
  It’s	
  rela`ve.	
  M2M	
  !=	
  M2H.	
  
     	
  
Context 	
     	
  -­‐ 	
  Packed	
  binary,	
  deltas	
  mostly,	
  snapshot	
  on	
  subscribe.	
  
     	
  
Environment-­‐ 	
  Don’t	
  send	
  1M	
  1K	
  events	
  to	
  a	
  mobile	
  phone	
  with	
  0.5mbps.	
  
               	
  



Copyright	
  Push	
  Technology	
  2012	
  
Ac`on.	
  Virtualize	
  Client	
  Queues	
  
                                              A

                                              B
                                                  ba

                                                  bb


                                                        Buffer
                                      Producers         Bloat
                                                                                       Consumers




       Nuance:	
  Client	
  telemetry.	
  Tradeoff:	
  Durable	
  subscrip`ons	
  harder	
  

Copyright	
  Push	
  Technology	
  2012	
  
Ac`on.	
  Add	
  data	
  caching	
  
                                              A

                                              B
                                                  ba   x


                                                  bb   x




                                      Producers
                                                             Is it a   Consumers
                                                            cache?




       One	
  hop	
  closer	
  to	
  the	
  edge	
  …	
  

Copyright	
  Push	
  Technology	
  2012	
  
Ac`on.	
  Exploit	
  data	
  structure	
  
                                              A

                                              B            Snapshot     Delta
                                                  ba   x


                                                  bb   x




                                      Producers                State!                   Consumers




       Snapshot	
  recovery.	
  Deltas	
  or	
  Changes	
  mostly.	
  Conserves	
  bandwidth	
  

Copyright	
  Push	
  Technology	
  2012	
  
Ac`on.	
  Behaviors	
  
                                              A

                                              B
                                                  ba   x


                                                  bb   x

                   X                                           The
                                      Producers
                                                             topic is                               Consumers
                                                                the
                                                              cloud!




       Extensible.	
  Nuance?	
  Roll	
  your	
  own	
  protocols.	
  Tradeoff?	
  3rd	
  party	
  code	
  in	
  the	
  engine	
  :/	
  

Copyright	
  Push	
  Technology	
  2012	
  
3rd	
  Party	
  Code	
  in	
  the	
  Engine?	
  

•     A	
  bug	
  in	
  3rd	
  party	
  code	
  can	
  take	
  out	
  the	
  system.	
  
•     Is	
  necessary	
  in	
  many	
  environments.	
  
•     Can	
  force	
  low	
  density	
  deployments.	
  
•     Solu`ons	
  are	
  appearing,	
  such	
  as	
  Waratek	
  for	
  the	
  JVM.	
  
        •  You	
  can	
  hive	
  out	
  3rd	
  party	
  code	
  into	
  a	
  lightweight	
  
             fully	
  isolated	
  container.	
  
        •  You	
  can	
  manage	
  the	
  container	
  lifecycle.	
  
        •  You	
  can	
  reallocate	
  memory,	
  CPU,	
  network.	
  
        •  It’s	
  language	
  level	
  virtualiza`on.	
  




 Copyright	
  Push	
  Technology	
  2012	
  
Ac`on.	
  Structural	
  Confla`on	
  
                                              A

                                              B
                                                  ba   x


                                                  bb   x

                   X
                                      Producers
                                                             Current                              Consumers
                                                              data!



                         +




       Ensures	
  only	
  current	
  +	
  consistent	
  data	
  is	
  distributed.	
  Ac`vely	
  soaks	
  up	
  bloat.	
  Extensible!	
  

Copyright	
  Push	
  Technology	
  2012	
  
Ac`on.	
  Structural	
  Confla`on	
  [EEP]	
  
                                         C2       A2   C1   B1   A1   'op'             Cop Bop Aop            ...   ...




                                         C2       A2   C1   B1   A1     R              C2      B1    A2                          Current	
  
                                              5   4    3    2    1                       5      1      4
                                                                                                                                 Current	
  
                                         C2       A2   C1   B1   A1    M+              C2      B1    A+                             +	
  	
  
                                              5   4    3    2    1                       8    1        5
                                                                                                                                Consistent	
  
                                                                                         =             =
                                                                                      C2 + C1       A2 + A1
                                                                                         =             =
                                                                                       5+3           4+1
Replace	
                                                             Merge	
  
•  Replace/Overwrite	
  ‘A1’	
  with	
  ‘A2’	
                        •  Merge	
  A1,	
  A2	
  under	
  some	
  opera`on.	
  
•  ‘Current	
  data	
  right	
  now’	
                                     •  Opera`on	
  is	
  pluggable	
  
•  Fast	
                                                                  •  Tunable	
  consistency	
  
                                                                           •  Performance	
  f(opera`on)	
  

Copyright	
  Push	
  Technology	
  2012	
  
Client?	
  Connec`on	
  &	
  Transport	
  
                                      Cascading	
  

                                                                          diffusion.js




                        native

                                              WebSockets    Flash




                                                            out of the box on the web and mobile web
                                          }




                                  Bespoke cascading -
                                Have your cake and eat it


Copyright	
  Push	
  Technology	
  2012	
  
An	
  extreme	
  example.	
  




                                               Box A                             Box B


            Throughput	
  (Worst	
  case)	
                             Latency	
  (Best	
  case)	
  
           •    Ramp	
  clients	
  con`nuously	
                        •  Really	
  simple.	
  1	
  client	
  
           •    100	
  messages	
  per	
  second	
  per	
  client	
     •  Ping	
  –	
  Pong.	
  Measure	
  RTT	
  `me.	
  
           •    Payload:	
  125	
  ..	
  2000	
  bytes	
                •  Payload:	
  125	
  ..	
  2000	
  bytes	
  
           •    Message	
  style	
  vs	
  with	
  confla`on	
  


Copyright	
  Push	
  Technology	
  2012	
  
Latency	
  in	
  perspec`ve	
  
                                                                                         ~                         ~
                                                                                        125                      1000
                                                                                       bytes                     bytes
                                                                                       (avg)                     (avg)




                      2.4 us                      5.0 us   8.0 us                                   50.0 us



•  2.4	
  us.	
  A	
  tuned	
  benchmark	
  in	
  C	
  with	
  low	
  latency	
  10Ge	
  NIC,	
  with	
  kernel	
  
   bypass,	
  with	
  FPGA	
  accelera`on	
  
•  5.0	
  us.	
  A	
  basic	
  java	
  benchmark	
  –	
  as	
  good	
  as	
  it	
  gets	
  in	
  java	
  
•  Diffusion	
  is	
  measurably	
  ‘moving	
  to	
  the	
  le•’	
  release	
  on	
  release	
  
•  We’ve	
  been	
  ac`vely	
  tracking	
  and	
  con`nuously	
  improving	
  since	
  4.0	
  

Copyright	
  Push	
  Technology	
  2012	
  
Throughput.	
  Server	
  view.	
  Non-­‐
                                 conflated	
  data	
  
                                                   •    Cold	
  start	
  server	
  

                                                   •     Ramp	
  750	
  clients	
  ‘simultaneously’	
  at	
  5	
  
                                                        second	
  intervals	
  
                                                        	
  
                                                   •     5	
  minute	
  benchmark	
  dura`on	
  

                                                   •     Clients	
  onboarded	
  linearly	
  un`l	
  IO	
  
                                                        (here)	
  or	
  compute	
  satura`on	
  occurs.	
  
                                                        	
  
                                                   •     What	
  the	
  ‘server’	
  sees	
  




Copyright	
  Push	
  Technology	
  2012	
  
Throughput.	
  Client	
  view.	
  Non-­‐
                                  conflated	
  data.	
  
                                                     •     What	
  the	
  ‘client’	
  sees	
  
                                                          	
  
                                                     •     At	
  and	
  beyond	
  satura`on	
  of	
  some	
  
                                                          resource?	
  

                                                     •     Things	
  break!	
  
                                                          	
  
                                                     •     New	
  connec`ons	
  fail.	
  Good.	
  
                                                     •     Long	
  established	
  connec`ons	
  ok.	
  
                                                     •     Recent	
  connec`ons	
  `meout	
  and	
  client	
  
                                                          connec`ons	
  are	
  dropped.	
  Good.	
  

                                                     •     Diffusion	
  handles	
  smaller	
  message	
  
                                                          sizes	
  more	
  gracefully	
  
                                                          	
  
                                                     •     Back-­‐pressure	
  ‘waveform’	
  can	
  be	
  
                                                          tuned	
  out.	
  Or,	
  you	
  could	
  use	
  structural	
  
                                                          confla`on!	
  




Copyright	
  Push	
  Technology	
  2012	
  
Throughput.	
  Server	
  view.	
  Replace-­‐
                         conflated	
  
                                              •     Cold	
  start	
  server	
  
                                                   	
  
                                              •     Ramp	
  750	
  clients	
  ‘simultaneously’	
  at	
  5	
  
                                                   second	
  intervals	
  
                                                   	
  
                                              •     5	
  minute	
  benchmark	
  dura`on	
  

                                              •     Clients	
  onboarded	
  linearly	
  un`l	
  IO	
  
                                                   (here)	
  or	
  compute	
  satura`on	
  occurs.	
  
                                                   	
  
                                              •     Takes	
  longer	
  to	
  saturate	
  than	
  non-­‐
                                                   conflated	
  case	
  
                                                   	
  
                                              •     Handles	
  more	
  concurrent	
  connec`ons	
  
                                                   	
  
                                              •     Again,	
  ‘server’	
  view	
  




Copyright	
  Push	
  Technology	
  2012	
  
Throughput.	
  Client	
  view.	
  Replace-­‐
                          Conflated	
  
                                              •     What	
  the	
  ‘client’	
  sees	
  
                                                   	
  
                                              •     Once	
  satura`on	
  occurs	
  Diffusion	
  
                                                   adapts	
  ac`vely	
  by	
  degrading	
  messages	
  
                                                   per	
  second	
  per	
  client	
  
                                                   	
  
                                              •     This	
  is	
  good.	
  Soak	
  up	
  the	
  peaks	
  
                                                   through	
  fairer	
  distribu`on	
  of	
  data.	
  
                                                   	
  
                                              •     Handles	
  spikes	
  in	
  number	
  of	
  
                                                   connec`ons,	
  volume	
  of	
  data	
  or	
  
                                                   satura`on	
  of	
  other	
  resources	
  using	
  a	
  
                                                   single	
  load	
  adap`ve	
  mechanism	
  




Copyright	
  Push	
  Technology	
  2012	
  
Can	
  failure	
  be	
  fun?	
  

•  Where	
  do	
  HA	
  algorithms	
  fail	
  
   humans?	
  
•  Can	
  consistent	
  experiences	
  be	
  
   delivered	
  elapsing	
  short,	
  medium	
  
   and	
  longer	
  term	
  failures?	
  
•  Does	
  an	
  M2H	
  eye	
  for	
  an	
  M2M	
  guy	
  
   even	
  work?	
  	
  
Copyright	
  Push	
  Technology	
  2012	
  
Who	
  knows.	
  Let’s	
  figure	
  it	
  out!	
  J	
  


             HA!	
  HA!	
  
               W8!!	
  




Copyright	
  Push	
  Technology	
  2012	
  
My	
  ‘number’	
  




#6752	
  
Copyright	
  Push	
  Technology	
  2012	
  
•  Thank	
  you	
  for	
  listening	
  to,	
  having	
  me	
  
                                              	
  
                                              •  Le	
  twiger:	
  @darachennis	
  
                                                   	
  
                                              •  Work.	
  hgp://www.pushtechnology.com/	
  

                                              •  Fun.	
  hgp://github.com/darach/	
  

                                              •  Grab	
  me	
  any	
  `me	
  on	
  Weds/Thurs.	
  
                                                 	
  
                                              •  Like	
  Erlang?	
  Big	
  Data?	
  
                                                 London	
  User	
  Group	
  meetup.	
  
                                                 Thurs	
  March	
  7th,	
  QE2	
  18:30	
  –	
  20:00.	
  
                                                 (Google	
  for	
  registra`on	
  details).	
  
                                                 	
  	
  
                                                                                 Ques`ons?	
  
Copyright	
  Push	
  Technology	
  2012	
  
                                                          Darach@PushTechnology.com	
  

More Related Content

Similar to QCON London 2013

Data distribution in the cloud with Node.js
Data distribution in the cloud with Node.jsData distribution in the cloud with Node.js
Data distribution in the cloud with Node.jsdarach
 
Bringing 3D, Ultra-Resolution, and Virtual Reality into the Global LambaGrid ...
Bringing 3D, Ultra-Resolution, and Virtual Reality into the Global LambaGrid ...Bringing 3D, Ultra-Resolution, and Virtual Reality into the Global LambaGrid ...
Bringing 3D, Ultra-Resolution, and Virtual Reality into the Global LambaGrid ...Larry Smarr
 
Achieving genuine elastic multitenancy with the Waratek Cloud VM for Java : J...
Achieving genuine elastic multitenancy with the Waratek Cloud VM for Java : J...Achieving genuine elastic multitenancy with the Waratek Cloud VM for Java : J...
Achieving genuine elastic multitenancy with the Waratek Cloud VM for Java : J...JAX London
 
OFC/NFOEC: Network Transformation
OFC/NFOEC: Network TransformationOFC/NFOEC: Network Transformation
OFC/NFOEC: Network TransformationADVA
 
Capacity planning in mobile data networks experiencing exponential growth in ...
Capacity planning in mobile data networks experiencing exponential growth in ...Capacity planning in mobile data networks experiencing exponential growth in ...
Capacity planning in mobile data networks experiencing exponential growth in ...Dr. Kim (Kyllesbech Larsen)
 
Globální zkušenosti s LTE - Ladislav Možný, T-Mobile
Globální zkušenosti s LTE - Ladislav Možný, T-MobileGlobální zkušenosti s LTE - Ladislav Možný, T-Mobile
Globální zkušenosti s LTE - Ladislav Možný, T-MobileTUESDAY Business Network
 
Bjorn Madsen, Researcher at Lego - The future of supply chain
Bjorn Madsen, Researcher at Lego - The future of supply chainBjorn Madsen, Researcher at Lego - The future of supply chain
Bjorn Madsen, Researcher at Lego - The future of supply chainGlobal Business Events
 
Smart TV content converged service & social media
Smart TV content converged service & social mediaSmart TV content converged service & social media
Smart TV content converged service & social mediafungfung Chen
 
Moving Global Content Exchange to the Cloud
Moving Global Content Exchange to the CloudMoving Global Content Exchange to the Cloud
Moving Global Content Exchange to the CloudSigniantMarketing
 
EVOLVING QUANTUM COMPUTERS: Harnessing a Vast Hidden Reality
EVOLVING QUANTUM COMPUTERS: Harnessing a Vast Hidden Reality EVOLVING QUANTUM COMPUTERS: Harnessing a Vast Hidden Reality
EVOLVING QUANTUM COMPUTERS: Harnessing a Vast Hidden Reality J On The Beach
 
Supelec m2m - iot - update 2013 - part 2
Supelec  m2m - iot - update 2013 - part 2Supelec  m2m - iot - update 2013 - part 2
Supelec m2m - iot - update 2013 - part 2Thierry Lestable
 
Evaluating Approaches to Building DPI into an LTE Network at the PDN Gateway ...
Evaluating Approaches to Building DPI into an LTE Network at the PDN Gateway ...Evaluating Approaches to Building DPI into an LTE Network at the PDN Gateway ...
Evaluating Approaches to Building DPI into an LTE Network at the PDN Gateway ...Continuous Computing
 
Geddes/PNSol - Broadband market evolution
Geddes/PNSol - Broadband market evolutionGeddes/PNSol - Broadband market evolution
Geddes/PNSol - Broadband market evolutionMartin Geddes
 
Larry Smarr - Making Sense of Information Through Planetary Scale Computing
Larry Smarr - Making Sense of Information Through Planetary Scale ComputingLarry Smarr - Making Sense of Information Through Planetary Scale Computing
Larry Smarr - Making Sense of Information Through Planetary Scale ComputingDiamond Exchange
 
Cloud computing: evolution or redefinition
Cloud computing: evolution or redefinitionCloud computing: evolution or redefinition
Cloud computing: evolution or redefinitionPET Computação
 
RTMFP Overview for IETF77
RTMFP Overview for IETF77RTMFP Overview for IETF77
RTMFP Overview for IETF77stoem
 
Towards Abundant Do-it-Yourself (DiY) Service Creativity in the Internet-of-T...
Towards Abundant Do-it-Yourself (DiY) Service Creativity in the Internet-of-T...Towards Abundant Do-it-Yourself (DiY) Service Creativity in the Internet-of-T...
Towards Abundant Do-it-Yourself (DiY) Service Creativity in the Internet-of-T...trappenl
 

Similar to QCON London 2013 (20)

Data distribution in the cloud with Node.js
Data distribution in the cloud with Node.jsData distribution in the cloud with Node.js
Data distribution in the cloud with Node.js
 
Bringing 3D, Ultra-Resolution, and Virtual Reality into the Global LambaGrid ...
Bringing 3D, Ultra-Resolution, and Virtual Reality into the Global LambaGrid ...Bringing 3D, Ultra-Resolution, and Virtual Reality into the Global LambaGrid ...
Bringing 3D, Ultra-Resolution, and Virtual Reality into the Global LambaGrid ...
 
Achieving genuine elastic multitenancy with the Waratek Cloud VM for Java : J...
Achieving genuine elastic multitenancy with the Waratek Cloud VM for Java : J...Achieving genuine elastic multitenancy with the Waratek Cloud VM for Java : J...
Achieving genuine elastic multitenancy with the Waratek Cloud VM for Java : J...
 
OFC/NFOEC: Network Transformation
OFC/NFOEC: Network TransformationOFC/NFOEC: Network Transformation
OFC/NFOEC: Network Transformation
 
Capacity planning in mobile data networks experiencing exponential growth in ...
Capacity planning in mobile data networks experiencing exponential growth in ...Capacity planning in mobile data networks experiencing exponential growth in ...
Capacity planning in mobile data networks experiencing exponential growth in ...
 
Globální zkušenosti s LTE - Ladislav Možný, T-Mobile
Globální zkušenosti s LTE - Ladislav Možný, T-MobileGlobální zkušenosti s LTE - Ladislav Možný, T-Mobile
Globální zkušenosti s LTE - Ladislav Možný, T-Mobile
 
Bjorn Madsen, Researcher at Lego - The future of supply chain
Bjorn Madsen, Researcher at Lego - The future of supply chainBjorn Madsen, Researcher at Lego - The future of supply chain
Bjorn Madsen, Researcher at Lego - The future of supply chain
 
Smart TV content converged service & social media
Smart TV content converged service & social mediaSmart TV content converged service & social media
Smart TV content converged service & social media
 
Moving Global Content Exchange to the Cloud
Moving Global Content Exchange to the CloudMoving Global Content Exchange to the Cloud
Moving Global Content Exchange to the Cloud
 
S series presentation
S series presentationS series presentation
S series presentation
 
Gbell Ra
Gbell RaGbell Ra
Gbell Ra
 
EVOLVING QUANTUM COMPUTERS: Harnessing a Vast Hidden Reality
EVOLVING QUANTUM COMPUTERS: Harnessing a Vast Hidden Reality EVOLVING QUANTUM COMPUTERS: Harnessing a Vast Hidden Reality
EVOLVING QUANTUM COMPUTERS: Harnessing a Vast Hidden Reality
 
Supelec m2m - iot - update 2013 - part 2
Supelec  m2m - iot - update 2013 - part 2Supelec  m2m - iot - update 2013 - part 2
Supelec m2m - iot - update 2013 - part 2
 
Evaluating Approaches to Building DPI into an LTE Network at the PDN Gateway ...
Evaluating Approaches to Building DPI into an LTE Network at the PDN Gateway ...Evaluating Approaches to Building DPI into an LTE Network at the PDN Gateway ...
Evaluating Approaches to Building DPI into an LTE Network at the PDN Gateway ...
 
OMG DDS Interoperability Demo 2009
OMG DDS Interoperability Demo 2009OMG DDS Interoperability Demo 2009
OMG DDS Interoperability Demo 2009
 
Geddes/PNSol - Broadband market evolution
Geddes/PNSol - Broadband market evolutionGeddes/PNSol - Broadband market evolution
Geddes/PNSol - Broadband market evolution
 
Larry Smarr - Making Sense of Information Through Planetary Scale Computing
Larry Smarr - Making Sense of Information Through Planetary Scale ComputingLarry Smarr - Making Sense of Information Through Planetary Scale Computing
Larry Smarr - Making Sense of Information Through Planetary Scale Computing
 
Cloud computing: evolution or redefinition
Cloud computing: evolution or redefinitionCloud computing: evolution or redefinition
Cloud computing: evolution or redefinition
 
RTMFP Overview for IETF77
RTMFP Overview for IETF77RTMFP Overview for IETF77
RTMFP Overview for IETF77
 
Towards Abundant Do-it-Yourself (DiY) Service Creativity in the Internet-of-T...
Towards Abundant Do-it-Yourself (DiY) Service Creativity in the Internet-of-T...Towards Abundant Do-it-Yourself (DiY) Service Creativity in the Internet-of-T...
Towards Abundant Do-it-Yourself (DiY) Service Creativity in the Internet-of-T...
 

More from darach

Thing. An unexpected journey. Devoxx UK 2014
Thing. An unexpected journey. Devoxx UK 2014Thing. An unexpected journey. Devoxx UK 2014
Thing. An unexpected journey. Devoxx UK 2014darach
 
FunctionalJS - May 2014 - Streams
FunctionalJS - May 2014 - StreamsFunctionalJS - May 2014 - Streams
FunctionalJS - May 2014 - Streamsdarach
 
Deconstructing Lambda
Deconstructing LambdaDeconstructing Lambda
Deconstructing Lambdadarach
 
Streams and Things
Streams and ThingsStreams and Things
Streams and Thingsdarach
 
Big Data, Mob Scale.
Big Data, Mob Scale.Big Data, Mob Scale.
Big Data, Mob Scale.darach
 
Meta Programming with Streams and Pipes
Meta Programming with Streams and PipesMeta Programming with Streams and Pipes
Meta Programming with Streams and Pipesdarach
 
Erlang/Sapiens
Erlang/SapiensErlang/Sapiens
Erlang/Sapiensdarach
 
Streamy, Pipy, Analyticy
Streamy, Pipy, AnalyticyStreamy, Pipy, Analyticy
Streamy, Pipy, Analyticydarach
 
Complex Er[jl]ang Processing with StreamBase
Complex Er[jl]ang Processing with StreamBaseComplex Er[jl]ang Processing with StreamBase
Complex Er[jl]ang Processing with StreamBasedarach
 
StreamBase - Embedded Erjang - Erlang User Group London - 20th April 2011
StreamBase - Embedded Erjang - Erlang User Group London - 20th April 2011StreamBase - Embedded Erjang - Erlang User Group London - 20th April 2011
StreamBase - Embedded Erjang - Erlang User Group London - 20th April 2011darach
 

More from darach (10)

Thing. An unexpected journey. Devoxx UK 2014
Thing. An unexpected journey. Devoxx UK 2014Thing. An unexpected journey. Devoxx UK 2014
Thing. An unexpected journey. Devoxx UK 2014
 
FunctionalJS - May 2014 - Streams
FunctionalJS - May 2014 - StreamsFunctionalJS - May 2014 - Streams
FunctionalJS - May 2014 - Streams
 
Deconstructing Lambda
Deconstructing LambdaDeconstructing Lambda
Deconstructing Lambda
 
Streams and Things
Streams and ThingsStreams and Things
Streams and Things
 
Big Data, Mob Scale.
Big Data, Mob Scale.Big Data, Mob Scale.
Big Data, Mob Scale.
 
Meta Programming with Streams and Pipes
Meta Programming with Streams and PipesMeta Programming with Streams and Pipes
Meta Programming with Streams and Pipes
 
Erlang/Sapiens
Erlang/SapiensErlang/Sapiens
Erlang/Sapiens
 
Streamy, Pipy, Analyticy
Streamy, Pipy, AnalyticyStreamy, Pipy, Analyticy
Streamy, Pipy, Analyticy
 
Complex Er[jl]ang Processing with StreamBase
Complex Er[jl]ang Processing with StreamBaseComplex Er[jl]ang Processing with StreamBase
Complex Er[jl]ang Processing with StreamBase
 
StreamBase - Embedded Erjang - Erlang User Group London - 20th April 2011
StreamBase - Embedded Erjang - Erlang User Group London - 20th April 2011StreamBase - Embedded Erjang - Erlang User Group London - 20th April 2011
StreamBase - Embedded Erjang - Erlang User Group London - 20th April 2011
 

QCON London 2013

  • 1. QCON   London  2013   Copyright  Push  Technology  2012  
  • 2. My  ‘number’   #6752   Copyright  Push  Technology  2012  
  • 3. •  Distributed  Systems  /  HPC  guy.     •  Chief  Scien*st  :-­‐  at  Push  Technology   •  Alumnus  of  :-­‐       Motorola,  IONA,  BeTair,  JPMC,  StreamBase.   •  School:  Trinity  College  Dublin.     -­‐  BA  (Mod).  Comp.  Sci.+     -­‐  M.Sc.  Networks  &  Distributed  Systems       •  Responds  to:  Guinness,  Whisky   About  me?   Copyright  Push  Technology  2012   Darach@PushTechnology.com  
  • 4. Highly  Available   Real-­‐Time   Data  Distribu`on   at  the  network’s   Edge   Copyright  Push  Technology  2012  
  • 5. HA.  Black  swans  &  Pigfish!   Copyright  Push  Technology  2012  
  • 6. When  this  …   esse  hic   Service   draconum   Copyright  Push  Technology  2012  
  • 7. …  inevitably  happens   HA!  HA!   W8!!   Copyright  Push  Technology  2012  
  • 8. Over  the  Edge,  where  Fallacies  roam   - The  network  is  reliable   - Latency  is  zero     - Bandwidth  is  infinite   - There  is  one  administrator   - Topology  does  not  change   - The  network  is  secure   - Transport  cost  is  zero   - The  network  is  homogeneous     hgp://en.wikipedia.org/wiki/Fallacies_of_Distributed_Compu`ng       Copyright  Push  Technology  2012  
  • 9. &  the  last  mile.  Dragons  there  be.   Buffer   Bloat   Cell   Bagery!   Coverage   Download  Bandwidth   30.00   20.00   Expected  (Mbps)   10.00   Download  (Mbps)   0.00   Sep-­‐06   -­‐10.00   Oct-­‐06   Oct-­‐06   Nov-­‐06   Dec-­‐06   Dec-­‐06   Jan-­‐07   Feb-­‐07   Latency   1.00E+00   1.00E+01   y  =  -­‐1.1667x  +  45703   Latency  (ms)   1.00E+02   R²  =  0.02496   Expected  (ms)   1.00E+03   Linear  (Latency  (ms))   1.00E+04   Sep-­‐06   Oct-­‐06   Oct-­‐06   Nov-­‐06   Dec-­‐06   Dec-­‐06   Jan-­‐07   Feb-­‐07   Copyright  Push  Technology  2012  
  • 10. Distributed  Systems   “A  distributed  system  is  one  in   which  the  failure  of  a  computer   you  didn’t  even  know  existed   can  render  your  computer   unusable”   -­‐  Leslie  Lamport   Copyright  Push  Technology  2012  
  • 11. High  Availability   “The  price  of  reliability  is  the   pursuit  of  the  utmost  simplicity.”   -­‐  Sir  Tony  Hoare,  Turing  Award  speech,  1980       “This  is  the  unfortunate  truth:   Simplicity  is  Hard.”              -­‐    Mosely  &  Marks,  Out  of  the  Tar  Pit,  2006   Copyright  Push  Technology  2012  
  • 12. High  Experiences  ~  Man  vs  Machine   Image:  hgp://www.phillymarke`nglabs.com/man-­‐vs-­‐machine/     Copyright  Push  Technology  2012  
  • 13. Machines  are  ‘Easy’   A  more  unfortunate  truth  is:     Humans*  are  Harder.       Allegedly  Sergey*  can  be  Emasculated   by  Glass.  But,  I  bet  he  owns  a  god  phone.              *  Humans  are  grumpy!                                  *  Colors  chosen  at  random     Copyright  Push  Technology  2012  
  • 14. Once  Upon  a  *me  …   SPOF There  was  a  very  useful  ligle  single  point  of  failure.   Copyright  Push  Technology  2012  
  • 15. It  was  so  successful   SPOF It  grew  and  it  grew!  So  we  put  it  in  the  cloud.   Copyright  Push  Technology  2012  
  • 16. It  grew,  and  it  grew  …   SPOF We  touched  it  with  all  of  our  devices.   Copyright  Push  Technology  2012  
  • 17. See  SPOF  run!   SPOF We  all  touched  it  except  Sergey  ‘the  emasculated  one’   Copyright  Push  Technology  2012  
  • 18. See  SPOF  spew  L   Wat? Is  there  nothing  we  can  do?  Esse  hic  draconum,  basically.  But,  yes  we  can  too  …   Copyright  Push  Technology  2012  
  • 19. Bad  things  happen  …   RapGenius   vs   Heroku.     Anyone?   Ferris?   It’s  either  dyno  park  this  or  whale  fail  in  the  zoo  …   Copyright  Push  Technology  2012  
  • 20. So  we  spun  up  another  instance   Master Slave And  master  /  slave  dabbled  with  us   Copyright  Push  Technology  2012  
  • 21. Ac*ve/Passive  –  Maybe?   Active Passive (Hot) (Cold) MTTR  rela`vely  high.  MTBF  rela`vely  low.    Ok  for     ‘nice-­‐to-­‐have’  or    ‘too-­‐late-­‐to-­‐cry-­‐now’    facili`es.     Copyright  Push  Technology  2012  
  • 22. Ac*ve/Passive  –  Hmm?   Active Passive (Hot) (Warm) MTTR  slightly  beger.  Can  replicate  state  ac`vely  for  cheaper  recovery.   Copyright  Push  Technology  2012  
  • 23. Ac*ve/Ac*ve  –  Ugh…   Active Active Master Slave (Hot) (Hot) MTTR  negligeable.  Use  2x  the  bandwidth.  Get  ‘half’  the  value.  Dups     Copyright  Push  Technology  2012  
  • 24. Ac*ve/Ac*ve  –  Ugh…   Active Active (Hot) (Warm) MTTR  negligeable.  Minify  bandwidth  usage  mostly.    Dups     Copyright  Push  Technology  2012  
  • 25. Ac*ve/Ac*ve  –  Ugh…   Active Active (Hot) (Warm) MTTR  negligeable.  Minify  bandwidth  usage  mostly.    Dups     Copyright  Push  Technology  2012  
  • 26. Spin  up  another  box?   Active Passive Standby (Hot) (Cold) (Cold) Speak no evil Hear no evil See no evil Failback,  Failover,  Takeover?  Load  Balance?  This  is  gesng  costly!   Copyright  Push  Technology  2012  
  • 27. Can  implement  N-­‐Plex  HA  with  a   ‘simple’  FSM   Initial Boot/DoBoot Active Active/DoSwtichDone Booting Switching/DoSwitch Stop/DoStopStart Sync/DoSyncStart Switching Sync'ing Stopping Swtich/DoStandbyActivate Standby/DoSyncEnd Standby/DoStopDone Standby Consider  roles  for  Applica`on,  Controller,  callbacks,  …   Copyright  Push  Technology  2012  
  • 28. Spin  up  a  cluster,  ring?   Master Slave Slave A  simple  implementa`on  can  be   easier  to  build  than  HA  pairs.  Your   mileage  may  vary.  Can  be  masterless   too.       Slave Slave Where  possible,  use  exis`ng   solu`ons  such  as  Akka,  riak_core,   doozer,  Apache  Zookeeper.     You’ll  s`ll  need  to  roll  your  own   Slave Slave data  recovery  -­‐  mostly.  Although   CRDT’s  are  changing  that  in  the   Slave database  community.   Copyright  Push  Technology  2012  
  • 29. One  more  thing  …  Real-­‐Time  streams   Data!   Meh Meh Real-­‐`me  data  streams  are  long-­‐lived,  conversa`onal,  can  be  stateful.     Copyright  Push  Technology  2012  
  • 30. Streaming  Data  Recovery   Type   Before   A]er   Comments   Precise   T T T T T T Completely  masks   1   2   3   4   5   6   failure.   Gap   T T T ?   T T May  result  in  data  loss   1   2   3   5   6   Repea*ng  Rollback   T T T T T T IO  preserva`on.  P/S   IIBECM 1   2   3   2   3   4   dups  are  equivalent.   Convergent   T T T X X X IO  preserva`on.  P/S   Rollback   1   2   3   2   3   4   dups  may  differ  ini`ally   but  EC.   Divergent  Rollback   T T T X X X   IO  preserva`on.  P/S   1   2   3   2   3   4   dups  will  differ   hgp://nms.lcs.mit.edu/papers/ha-­‐icde05.pdf  -­‐  HA  algos  for  DSPs     Copyright  Push  Technology  2012  
  • 31. But,  problem:  The  bird,  basically.   Immediately  Inconsistent.   But,  Eventually  Consistent   …  Maybe.         Humans   Hate   Repea*ng   Rollback   Recovery   IIBECM Copyright  Push  Technology  2012  
  • 32. Ugh,  just  one  more  thing  …   State?   Finagle Finagle Data  rarely  passes  through  a  box  without  being  fiddled  with.   Copyright  Push  Technology  2012  
  • 33. Streaming  Opera*ons  –  4  types   Type   Comments   Arbitrary   Completely  masks  failure.   Determinis*c   Given  Same  Input,  Same  State,  produce  Same  Output   Convergent-­‐capable   Given  Same  Input,  Earlier  Start,  Empty  State,  converge   to  Same  Output  as  Determinis`c  case   Repeatable   Given  Same  Input,  Earlier  State,  Empty  State,  requires   at  most  one  input  to  product  at  most  one  output  to  be   convergent-­‐capable  and  same  as  determinis`c  case   hgp://nms.lcs.mit.edu/papers/ha-­‐icde05.pdf  -­‐  HA  algos  for  DSPs     Copyright  Push  Technology  2012  
  • 34. DOH?   Data  Recovery  Type   x  Opera*on  Type  =   The  Hard  Part   Copyright  Push  Technology  2012  
  • 35. Just  one  last  one  more  thing  …   Human   Expecta*ons   We  are  fickle,  grumpy,  changeable,  intolerant  …   Copyright  Push  Technology  2012  
  • 36. Let’s  see.  Trading  systems?   Market Market Data Prices Prices Market Data Market Data Data Broker Orders Dealer Service Market Market Market Execution Market Execution Execution Fat Cat Execution "Flow" Trades Banker It Depends… Precise Recovery It's Complicated! Gap Recovery There  are  at  least  26  race  condi`ons  in     FIX  4.2  Order  Cancel/Replace/Reject   And  please.  No  stale  data!   Copyright  Push  Technology  2012  
  • 37. A  most  inconvenient  truth   Simple  is   Hard   So  relax.  Let  it  fail.  Let  live  and  let’s  learn.   Copyright  Push  Technology  2012  
  • 38. Stop.   - The  network  is  not  reliable     nor  is  it  cost  free.   - Latency  is  not  zero   nor  is  it  a  democracy.   - Bandwidth  is  not  infinite   nor  predictable  especially  the  last  mile!   - There  is  not  only  one  administrator   trust,  rela`onships  are  key   - Topology  does  change   It  should,  however,  converge  eventually   - The  network  is  not  secure   nor  is  the  data  that  flows  through  it   - Transport  cost  is  not  zero   but  what  you  don’t  do  is  free   - The  network  is  not  homogeneous   nor  is  it  smart   Copyright  Push  Technology  2012  
  • 39. Look.   -   High  Availability  is  not  enough  for  Machine  to  Human   -   High  Experience  requires  more   Data  has  a  *me  value  –  out  of  date  data  is  bad.   Data  has  variable  relevance,  based  on  loca*on,  device  …   Data  rates  are  faster  than  human  percep*on   You  cannot  send  100mb/sec  down  3G  networks   You  cannot  send  100mb/sec  down  3G  networks   You  cannot  …  waste  is  bad,  bloats  buffers,  and  ‘slows’  data.   Currency  &  (Immediate)  Consistency  are  important  in  M2H.   -   M2H  ‘High  Available  Experience’  might  work  for  M2M  …       Copyright  Push  Technology  2012  
  • 40. Listen.   -   Every  nuance  comes  with  a  set  of  tradeoffs.   -   Choosing  the  right  ones  can  be  hard,  but  it  pays  off.   -   Context,  Environment  are  cri`cal   -   Break  all  the  rules,  one  benchmark  at  a  `me.     -   Benchmark  Driven  Development   Copyright  Push  Technology  2012  
  • 41. Ac`on:  Data  Distribu`on   Messaging  remixed  around:     Relevance  -­‐  Queue  depth  for  conflatable  data  should  be  0  or  1.  No  more     Responsiveness  -­‐  Use  HTTP/REST  for  things.  Stream  the  ligle  things     Timeliness  -­‐  It’s  rela`ve.  M2M  !=  M2H.     Context    -­‐  Packed  binary,  deltas  mostly,  snapshot  on  subscribe.     Environment-­‐  Don’t  send  1M  1K  events  to  a  mobile  phone  with  0.5mbps.     Copyright  Push  Technology  2012  
  • 42. Ac`on.  Virtualize  Client  Queues   A B ba bb Buffer Producers Bloat Consumers Nuance:  Client  telemetry.  Tradeoff:  Durable  subscrip`ons  harder   Copyright  Push  Technology  2012  
  • 43. Ac`on.  Add  data  caching   A B ba x bb x Producers Is it a Consumers cache? One  hop  closer  to  the  edge  …   Copyright  Push  Technology  2012  
  • 44. Ac`on.  Exploit  data  structure   A B Snapshot Delta ba x bb x Producers State! Consumers Snapshot  recovery.  Deltas  or  Changes  mostly.  Conserves  bandwidth   Copyright  Push  Technology  2012  
  • 45. Ac`on.  Behaviors   A B ba x bb x X The Producers topic is Consumers the cloud! Extensible.  Nuance?  Roll  your  own  protocols.  Tradeoff?  3rd  party  code  in  the  engine  :/   Copyright  Push  Technology  2012  
  • 46. 3rd  Party  Code  in  the  Engine?   •  A  bug  in  3rd  party  code  can  take  out  the  system.   •  Is  necessary  in  many  environments.   •  Can  force  low  density  deployments.   •  Solu`ons  are  appearing,  such  as  Waratek  for  the  JVM.   •  You  can  hive  out  3rd  party  code  into  a  lightweight   fully  isolated  container.   •  You  can  manage  the  container  lifecycle.   •  You  can  reallocate  memory,  CPU,  network.   •  It’s  language  level  virtualiza`on.   Copyright  Push  Technology  2012  
  • 47. Ac`on.  Structural  Confla`on   A B ba x bb x X Producers Current Consumers data! + Ensures  only  current  +  consistent  data  is  distributed.  Ac`vely  soaks  up  bloat.  Extensible!   Copyright  Push  Technology  2012  
  • 48. Ac`on.  Structural  Confla`on  [EEP]   C2 A2 C1 B1 A1 'op' Cop Bop Aop ... ... C2 A2 C1 B1 A1 R C2 B1 A2 Current   5 4 3 2 1 5 1 4 Current   C2 A2 C1 B1 A1 M+ C2 B1 A+ +     5 4 3 2 1 8 1 5 Consistent   = = C2 + C1 A2 + A1 = = 5+3 4+1 Replace   Merge   •  Replace/Overwrite  ‘A1’  with  ‘A2’   •  Merge  A1,  A2  under  some  opera`on.   •  ‘Current  data  right  now’   •  Opera`on  is  pluggable   •  Fast   •  Tunable  consistency   •  Performance  f(opera`on)   Copyright  Push  Technology  2012  
  • 49. Client?  Connec`on  &  Transport   Cascading   diffusion.js native WebSockets Flash out of the box on the web and mobile web } Bespoke cascading - Have your cake and eat it Copyright  Push  Technology  2012  
  • 50. An  extreme  example.   Box A Box B Throughput  (Worst  case)   Latency  (Best  case)   •  Ramp  clients  con`nuously   •  Really  simple.  1  client   •  100  messages  per  second  per  client   •  Ping  –  Pong.  Measure  RTT  `me.   •  Payload:  125  ..  2000  bytes   •  Payload:  125  ..  2000  bytes   •  Message  style  vs  with  confla`on   Copyright  Push  Technology  2012  
  • 51. Latency  in  perspec`ve   ~ ~ 125 1000 bytes bytes (avg) (avg) 2.4 us 5.0 us 8.0 us 50.0 us •  2.4  us.  A  tuned  benchmark  in  C  with  low  latency  10Ge  NIC,  with  kernel   bypass,  with  FPGA  accelera`on   •  5.0  us.  A  basic  java  benchmark  –  as  good  as  it  gets  in  java   •  Diffusion  is  measurably  ‘moving  to  the  le•’  release  on  release   •  We’ve  been  ac`vely  tracking  and  con`nuously  improving  since  4.0   Copyright  Push  Technology  2012  
  • 52. Throughput.  Server  view.  Non-­‐ conflated  data   •  Cold  start  server   •  Ramp  750  clients  ‘simultaneously’  at  5   second  intervals     •  5  minute  benchmark  dura`on   •  Clients  onboarded  linearly  un`l  IO   (here)  or  compute  satura`on  occurs.     •  What  the  ‘server’  sees   Copyright  Push  Technology  2012  
  • 53. Throughput.  Client  view.  Non-­‐ conflated  data.   •  What  the  ‘client’  sees     •  At  and  beyond  satura`on  of  some   resource?   •  Things  break!     •  New  connec`ons  fail.  Good.   •  Long  established  connec`ons  ok.   •  Recent  connec`ons  `meout  and  client   connec`ons  are  dropped.  Good.   •  Diffusion  handles  smaller  message   sizes  more  gracefully     •  Back-­‐pressure  ‘waveform’  can  be   tuned  out.  Or,  you  could  use  structural   confla`on!   Copyright  Push  Technology  2012  
  • 54. Throughput.  Server  view.  Replace-­‐ conflated   •  Cold  start  server     •  Ramp  750  clients  ‘simultaneously’  at  5   second  intervals     •  5  minute  benchmark  dura`on   •  Clients  onboarded  linearly  un`l  IO   (here)  or  compute  satura`on  occurs.     •  Takes  longer  to  saturate  than  non-­‐ conflated  case     •  Handles  more  concurrent  connec`ons     •  Again,  ‘server’  view   Copyright  Push  Technology  2012  
  • 55. Throughput.  Client  view.  Replace-­‐ Conflated   •  What  the  ‘client’  sees     •  Once  satura`on  occurs  Diffusion   adapts  ac`vely  by  degrading  messages   per  second  per  client     •  This  is  good.  Soak  up  the  peaks   through  fairer  distribu`on  of  data.     •  Handles  spikes  in  number  of   connec`ons,  volume  of  data  or   satura`on  of  other  resources  using  a   single  load  adap`ve  mechanism   Copyright  Push  Technology  2012  
  • 56. Can  failure  be  fun?   •  Where  do  HA  algorithms  fail   humans?   •  Can  consistent  experiences  be   delivered  elapsing  short,  medium   and  longer  term  failures?   •  Does  an  M2H  eye  for  an  M2M  guy   even  work?     Copyright  Push  Technology  2012  
  • 57. Who  knows.  Let’s  figure  it  out!  J   HA!  HA!   W8!!   Copyright  Push  Technology  2012  
  • 58. My  ‘number’   #6752   Copyright  Push  Technology  2012  
  • 59. •  Thank  you  for  listening  to,  having  me     •  Le  twiger:  @darachennis     •  Work.  hgp://www.pushtechnology.com/   •  Fun.  hgp://github.com/darach/   •  Grab  me  any  `me  on  Weds/Thurs.     •  Like  Erlang?  Big  Data?   London  User  Group  meetup.   Thurs  March  7th,  QE2  18:30  –  20:00.   (Google  for  registra`on  details).       Ques`ons?   Copyright  Push  Technology  2012   Darach@PushTechnology.com