The 2nd half. Scaling to the next^2

FULLSTACK TECH RADAR DAY
The #2nd_half
Scaling to the #next2
Haggai Philip Zagury, DevOps Group & Tech Lead

Who we are
Tikal helps ISV’s in Israel & abroad in their technological
challenges.
Our Engineers are Fullstack Developers with expertise in Android,
DevOps, Java, JS, Python, ML
We are passionate about technology and specialise in
OpenSource technologies.
Our Tech and Group leaders help establish & enhance existing
software teams with innovative & creative thinking.
https://www.meetup.com/full-stack-developer-il/

Haggai Philip Zagury - DevOps Group & Tech lead
My open thinking and open techniques ideology is driven by
Open Source technologies and the collaborative manner defining
my M.O.
My solution driven approach is strongly based on hands-on and
deep understanding of Operating Systems, Applications stacks and
Software languages, Networking, Cloud in general and today more
and more Cloud Native solutions.

Planning a data pipeline (with constraints)
● An event driven application

● High Resolution Video Streams from multiple locations
● Plan for 10s of simultaneous sources …
Digest

● Data enrichment pipelines
Digest Enrich

● Data enrichment pipelines
● Output an HD display in near real-time (+- 8sec)
Digest Enrich

@scale - point #1
● The board emphasises a perquisite that
we have unlimited resources to
complete the task …
● Which isn’t the case in real-life …

@scale - point #2
● If you have a consistent measurable unit
● you can plan your journey to the next
square either by:
● Being more Efficient
● Doubling* your compute power
(not accurate will explain later why)

Planning a data pipeline - starting point
● The scale is not predictable …
● As you grow so does your supporting
services …
● Dealing with debug logs in a section
of a cluster means 6 times network
traffic

Planning a data pipeline - reality
● The scale is not predictable …
● As you grow so does your supporting
services …
● Dealing with debug logs in a section
of a cluster means 6 times network
traffic
● Deployment complexity increases …

The cloud is your friend
● Managed Services
● Solving Complexity of operation
● Increases/Decreases Price (in most
cases)
● Pay as You go …
● Polyglottism / “Architecture captivity“

Solution

Solution
How to store data for
optimised
r/w procedures
55,000 read requests per second
Hitting S3 Limit on >=10 simultaneous streams

Solution
How to store data for
optimised
r/w procedures
55,000 read requests per second
Hitting S3 Limit on >=10 simultaneous streams
# Complexitynext2

Solution
Performance ?

Solution
Performance ?
#2nd_half ?

VM vs Container*
Performance ?
Less ComplexityMore Complexity Gpu/Tpu

Solution
* is on the network !

Solve drift … with “money”
● 1 frame (of 45MB) * 30 frames per sec = 1350 MB == 1.35 GB == 10.8 Gb
> 10 Gb

EC2 Network performance comparison
● Worth a read on ec2 network performance
○ https://docs.google.com/spreadsheets/d/1N2xQqry-
zAKnK6FtW8X5zBYhMiFFnuMySMpx7f3K60s/edit?usp=sharing
○ https://cloudonaut.io/ec2-network-performance-cheat-sheet/

Solution
Multi-tenancy ?
who did what, when ?

Resource Tags are your friend !
● Tagging resources
● Putting price tags on operations !
● How much did pipeline1 cost ?
● Can I do it faster ? [ Time ]
● Each enrichment service has it’s own
cluster optimised for Compute unit sizes
staging
stream
infra

#2nd_half or # issues …next2
Justin Ryan
https://qconsf.com/sf2019/presentation/scaling-patterns-netﬂixs-edge
Scaling Patterns for Netﬂix's Edge

#2nd_half or # issues …next2
https://www.infoq.com/presentations/efﬁciency-better-software-faster/
Todd Montgomery
Todd Montgomery explores the everyday things that those with an eye to
performance and efﬁciency do that can be leveraged by anyone to build better software faster.

Compute
Buffet ;)
● Taken @SFO
Nov 2019

So building distributed systems is hard

The #2nd half/ # solution(s)next2
● What is the 2nd half of the table problem(s)
● ***** real-life scenario !
● Scale ability -> Capacity planning in the “Container Orchestration Era”
● (in the past Marathon) Kubernetes
● FaaS (e.g. AWS Lambda)
● Nomad

Lego for the masses
● Solves Complexity (i know debatable …)
● 1 size fit all
background image by @hagzag

Commoditize compute unit management
kubectl get deploy -n kube-system
cert-manager 1/1 1 1 58d
cert-manager-cainjector 1/1 1 1 58d
cert-manager-webhook 1/1 1 1 58d
coredns 2/2 2 2 173d
enternal-dns-prod-external-dns 1/1 1 1 171d
kube-slack 1/1 1 1 173d
tiller-deploy 1/1 1 1 173d

Resource Limits apiVersion: v1
kind: Pod
metadata:
name: frontend
spec:
containers:
- name: db
image: mysql
env:
- name: MYSQL_ROOT_PASSWORD
value: "password"
resources:
requests:
memory: "64Mi"
cpu: "250m"
limits:
memory: "128Mi"
cpu: "500m"

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: test-network-policy
namespace: default
spec:
podSelector:
matchLabels:
role: db
policyTypes:
- Ingress
- Egress
ingress:
- from:
- ipBlock:
cidr: 172.17.0.0/16
except:
- 172.17.1.0/24
- namespaceSelector:
matchLabels:
project: myproject
- podSelector:
matchLabels:
Resource Limits

Time Days Minuets Seconds Milliseconds
Cost
Agility
Scalability
https://www.slideshare.net/BryanMcAninch/the-faas-and-the-curious-86874211

Make the best of what you have ?
● Be more accurate / precise in what you do !
● Measuring techniques (that scale)

Guess what ?
They both get to Eilat !

Keep Efficiency in mind
Time
Price
Complexity
Define your Quantifiable unit !

F
● A Fixed unit size Grid == FaaS

Solving complexity …

Load Shifting & Peak Shaving

Keep Efficiency in mind
Time
Price
Complexity
Define your Quantifiable unit !
Thank you for taking the Time !

The 2nd half. Scaling to the next^2

More Related Content

What's hot

Similar to The 2nd half. Scaling to the next^2

More from Haggai Philip Zagury

Recently uploaded

The 2nd half. Scaling to the next^2