Presented at API World 2016, San Jose, CA

1. LOADBALANCERS: THE
FABRIC FOR YOUR
MICROSERVICES
Chiradeep Vittal
API World 2016

2. About me
¨ Distinguished Engineer
at Citrix
¨ Work on Citrix
Netscaler and
containers

3. Load balancers are important (again)
¨ Containers and Microservices
¨ Background on Load Balancing
¨ Teaching old load balancers new tricks
¨ Emerging patterns for Load Balancing in
microservices
¨ Future directions

4. Information Technology is rapidly
accelerating
Time
Web
Digital
Social
Mobile
Things
Smart Things
RateofChange

5. The challenge for IT
Match the
velocity of
business change

6. Addressing the gap: Agile
© Christopher Little / Wikimedia Commons / CC-BY-3.0

7. ¨ Collaboration between Dev
and Ops teams
¨ Tooling and automation to
squeeze commit-to-deploy
time
Addressing the Gap:
DevOps

8. Agile and DevOps
1. Frequent Iterations
2. All phases all the time
3. Risk-based
1. Shorten time from commit
to production
2. “You build it, You run it”
3. Measure everything
4. Experimentation and
learning
Agile DevOps

9. Agile + DevOps Revolution
10+ Deploys per Day
Flickr, 2009
50 million Deploys Per
Year
Amazon Web Services,
2015

10. Image by BMW Werk Leipzig - http://bmw-werk-leipzig.de, CC BY-SA 2.0 de, https://commons.wikimedia.org/w/index.php?curid=11928438
Key
Mechanisms:
Automation

11. Key Mechanisms: Microservices
LB
Web
Data

12. 1. Small Autonomous processes
2. Loose coupling
3. Bounded contexts
4. Deploy & scale independently
Key Mechanisms: Microservices

13. Containers & speed
Compile (seconds)
VM Build / Configure (minutes)
Deploy VM (minutes)
Compile (seconds)
Container Build (seconds)
Container Deploy (seconds)
Before Containers (virtual machines) With Containers

14. Container Wins
¨ Small & Autonomous
¨ Deploy and scale independently
¨ Increased Velocity
¨ Reduced risk
¨ Efficiency
Features Benefits

15. Containers or Microservices?
¨ Microservices == architecture
¨ Containers == implementation
¨ Containers win over VMs for
Microservices

16. Monoliths vs. Microservices
LB
Web
Data

17. Load Balancing: Traditional vs. Cloud Native
Static Applications, well
defined topology
Dynamic Microservices,
changing topology
W W W
A A A A A
W
M
M
TrafficismostlyN-S
M
M
M
M
M
M
M
M
M
Traffic is mix
of N-S and E-W

18. Cloud-Native
Patterns of
architecture
and
organization
that deliver
software with
speed and
reliability
¨ Auto Scale
¨ Continuous Delivery
¨ Baked-in Resilience
¨ Deep Monitoring
¨ Collaboration

19. Cloud Native + Containers is Network
Intensive
¨ Implications on
¤ Performance
¤ Reliability
¤ Security
¤ Routing
¤ Naming and discovery
¤ Monitoring

20. Cloud Native Landscape - Microservices
Microservices
APIs
Performance
Resilience
Security Visibility
Continuous
Delivery
AutoScal
e
Circuit
Breaker
Load
Balancing
Throttling
Discovery
Audit Segmentation
E2E
Encryption
Routing
Chaos
Monkey
Distributed
Debug
Back-off
Lifecycle
Management
Auth

21. Containers
Container
Networking
Container
Security
Integrity Isolation AuthN/Z
Container
Orchestration
Infrastructure
Orchestration
Network
Orchestration
Cloud
Orchestration
Multi-cloud
Orchestration
Private
Cloud
OrchestrationHypervisor
Orchestration
Network
Orchestration
IPAM / DNS
Overlay /
Underlay
Microservices
Lifecycle
Management
Container
Performance
Network
Perf
scheduler
Container
Storage
Storage
Orchestration
Storage
Orchestration
Cloud Native Landscape - Infrastructure

22. Container Managers to the rescue?
Docker Swarm Kubernetes
(from Google)
Mesosphere
DCOS
(based on Apache
Mesos)
AWS ECS

23. Container Cluster Managers
α1
β1
δ1
β2
α2
δ3
α3
α4
δ2
α5
α6
HostH1
HostH2
HostH3
HostH4
δ4
ClusterManager
Cluster API

24. Cluster Managers – Sweet Spots
¨ Infrastructure independence
¨ Scheduling
¨ Discovery
¨ Scaling (partially)
¨ Security (a little bit)
¨ Load balancing (limited)

25. A brief history of Load balancing
Internet
W W
High Availability
Internet
W W WW
HA + Load Balancing
Internet
Webserver

26. A brief history of Load balancing
Internet
W WW
App 1 App 2
W WW
foo.com/app1 foo.com/app2
L7 routing
Internet
W WW
SSL Traffic
Unencrypted Traffic
Internet
W WW
App Firewall

27. What LB-in-the-wire enables
¨ Resilience
¨ Encryption offload
¨ Application acceleration
¨ Defend L4à L7
¨ L7 routing
¨ Application Performance Analytics

28. Load Balancing Form Factors
Hardware Software
VM
Virtual As-a-service Containerized

29. Physical
Price-Performance
MPX
VPX
Hypervisor
Virtual
Run Anywhere
=
NetScaler CPX: Cloud Native Load
Balancer
CPX (new)
Container
SDX
Platform
Multi-Tenant
=

30. NetScaler CPX
Containerized
NetScaler for
Developers /
Cloud native
deployments
¨ Same hardened code, same features*
as bigger form factors
¨ Unified control plane with Ingress and
other LB (NetScaler MAS)
¨ Unified monitoring, logging and
analytics
* Same as NetScaler Standard Edition minus VPN

31. NetScaler CPX Express
¨ “Developer” Edition*
¨ No license required
¨ Available for download from Docker App Store in Q3
¨ Drop-in replaceable with licensed version
¨ Get it here:
– http://microloadbalancer.com
*Not for production use

32. https://www.microloadbalancer.com

33. Load Balancing for container clusters – Ingress / Edge
α5
α6
HostH4
Scale out
Internet
α1
β1
δ1
HostH1
β2
α2
δ3
HostH2
α3
α4
δ2HostH3
Public Endpoint
https://alpha:443
LB

34. Load Balancing – intra-cluster
α1
β1
HostH1
α2
δ3
HostH2
α4
δ2
HostH3
LB
LB per endpoint

35. Load Balancing – intra-cluster
α1
β1
HostH1
α2
δ3
HostH2
α4
δ2
HostH3
LBLB LB
LB per container host
• Service clients use
an LB local to
their host
• E.g., Kube-proxy,
Swarm, Mesos -
MinuteMan

36. Ingress vs Intra-cluster differences
¨ Performance
¨ Count
¨ Frequency of reconfiguration
¨ L4 (intra-cluster) vs L7 (ingress)
¨ Form factor

37. Reconfiguration of Load Balancer
α1
β1
HostH1
β2
α2
HostH2
α3
α4
HostH3
Ingress LB
α5
α6
HostH4
ClusterManager
LB
Controller
Cluster API
Container
Events Reconfigur
ation
Container
State
Container
State Query

38. Nitrox – Configure Citrix NetScaler for
Cluster Managers
¨ github.com/chiradeep/nitrox
¨ Apache license
¨ Support for integrating NetScaler with
¤ Kubernetes
¤ Swarm
¤ Consul
¤ Marathon

39. Microservices LB patterns
¨ Client-side LB
¨ Zero-downtime deploys
¨ Resilience patterns
¨ Visibility / Insights

40. Client-side LB
¤ Embedded into calling
application
n e.g., Netflix Ribbon
(with Eureka)
¤ Run as side-cars
(alongside each
application or one-
per-host) Eureka
M M
M
M
M
M

41. Client-side LB
¤ Run as side-cars, or
one-per-host
Service
Discovery
M
M
M
M
M
M
Clients
Service
LB
LB

42. Client-side LB Examples
¨ Netflix Prana - Hystrix as side car (deprecated)
¨ Linkerd from Buoyant.io (based on Twitter Finagle)
¨ AirBnB Smartstack/Synapse (uses HAProxy)
¨ Uber Hyperbahn
¤ Like Finagle, switched from HAProxy
¨ Kube Proxy
¤ Initially user-space, now based on IPTables
¨ Docker Swarm LB
¤ Based on IPVS
¨ Mesos LB
¤ Based on IPTables

43. Zero-Downtime Deploys
¨ Canary deploys
¨ Traffic can be selected
based on source
(country), headers
M M M M
Version N
M M M M M
Version N+1
10%90%
CD
Pipeline

44. Zero-Downtime Deploys
¨ Blue-green deploys
M M M M
Version N
M M M M M
Version N+1
100%
CD
Pipeline

45. Zero downtime deploys with NetScaler
HostH1
HostH2
HostH3
Ingress NetScaler
HostH4
ContainerClusterCluster API
Container
Events Reconfig
uration
Controller
NetScaler CPXJenkins

46. Advanced Resilience – Circuit Breaker
¨ Circuit Breaker is a
pattern that prevents
cascading failures due
to unresponsive
downstream services
¨ A load balancer can be
configured to ignore a
slow backend
Credit: http://martinfowler.com/bliki/CircuitBreaker.html

47. Visibility
¨ Compliance
¨ Topology
¨ Analytics
M
M
M M M
M M M M M
M
Analytics

48. Topology and Compliance
Policy
Violation

49. Wrap-up
¨ Load balancing is
different for containers /
microservices
¨ Integration with container
cluster managers is
needed
¨ Consider using the same
LB technology for ingress
and intra-cluster
¨ Emerging patterns
solidify the importance
of the load balancer
¨ LB in the wire brings
¤ Simplicity
¤ Resilience
¤ Future proofing

50. https://www.microloadbalancer.com