Scaling infrastructure is tricky, I will try to explain what methods I use when dealing with this issue, and demonstrate an approach which can be applied to almost any type of work load.

1. THE #2NDHALF
SCALING

2. tikalk.com
https://www.linkedin.com/in/salo-shp/
#2ndhalf #next2 #scale #distributed #voodoo
#chaos #hashxor #opensource #performance-
tuning #io #bigdata #devops #monitoring
#architecture #orchestra #cloud

3. चतुरङ् ग
https://en.wikipedia.org/wiki/Chaturanga

4. https://www.kurzweilai.net/the-law-of-accelerating-returns

5. 1 2 4 8 16 32 64 128
256 512 1K 2K 4K 8K 16K 32K
64K 128K 256K 512K 1M 2M 4M 8M
16M 32M 64M 128M 256M 512M 1G 2G
4G 8G 16G 32G 64G 128G 256G 512G
1T 2T 4T 8T 16T 32T 64T 128T
256T 512T 1P 2P 4P 8P 16P 32P
64P 128P 256P 512P 1E 2E 4E 8E
“It should be pointed out that as the emperor and the inventor went through
the first half of the chess board, things were fairly uneventful…”

6. 1 2 4 8 16 32 64 128
256 512 1K 2K 4K 8K 16K 32K
64K 128K 256K 512K 1M 2M 4M 8M
16M 32M 64M 128M 256M 512M 1G 2G
4G 8G 16G 32G 64G 128G 256G 512G
1T 2T 4T 8T 16T 32T 64T 128T
256T 512T 1P 2P 4P 8P 16P 32P
64P 128P 256P 512P 1E 2E 4E 8E
“It was as they progressed through the second half of the chessboard
that the situation quickly deteriorated …”
#2NDHALF

7. 1 2 4 8 16 32 64 128
256 512 1K 2K 4K 8K 16K 32K
64K 128K 256K 512K 1M 2M 4M 8M
16M 32M 64M 128M 256M 512M 1G 2G
4G 8G 16G 32G 64G 128G 256G 512G
1T 2T 4T 8T 16T 32T 64T 128T
256T 512T 1P 2P 4P 8P 16P 32P
64P 128P 256P 512P 1E 2E 4E 8E
Marin
Mersenne
>>> '{:,}'.format(sum([2**i for i in range(0, 64)]))
'18,446,744,073,709,551,615'
>>> '{:,}'.format( 2**64 - 1 )
’18,446,744,073,709,551,615’

8. 1 2 4 8 16 32 64 128
256 512 1K 2K 4K 8K 16K 32K
64K 128K 256K 512K 2M 4M 8M
16M 32M 64M 128M 256M 512M 2G
4G 8G 16G 32G 64G 128G 256G 512G
1T 2T 4T 8T 16T 32T 64T 128T
256T 512T 1P 2P 4P 8P 16P 32P
64P 128P 256P 512P 1E 2E 4E 8E
05:16:04
1G
1M
https://www.guinnessworldrecords.com/news/2019/10/jennifer-aniston-makes-friends-on-instagram-breaks-
guinness-world-records-title-595535/
THE FASTEST TIME TO REACH ONE MILLION FOLLOWERS

9. 1 2 4 8 16 32 64 128
256 512 1K 2K 4K 8K 16K 32K
64K 128K 256K 512K 2M 4M 8M
16M 32M 64M 128M 256M 1G 2G
4G 8G 16G 32G 64G 128G 256G 512G
1T 2T 4T 8T 16T 32T 64T 128T
256T 512T 1P 2P 4P 8P 16P 32P
64P 128P 256P 512P 1E 2E 4E 8E
500,000,000 / 24 / 60 * ( 5 * 60 + 16 )
109,722,222 -> 0.911%
512M
https://instagram-press.com/our-story/
https://edition.cnn.com/2019/10/16/entertainment/jennifer-aniston-broke-instagram-trnd/index.html
1M

10. UNIT TESTS
PERFORMANCE TUNING OF A F15 ENGINE

11. 1 2 4 8 16 32 64 128
256 512 1K 2K 4K 8K 16K 32K
64K 128K 256K 512K 1M 2M 4M 8M
16M 32M 64M 128M 256M 512M 1G 2G
4G 8G 16G 32G 64G 128G 256G 512G
1T 2T 4T 8T 16T 32T 64T 128T
256T 512T 1P 2P 4P 8P 16P 32P
64P 128P 256P 512P 1E 2E 4E 8E
WHERE
U AT ?..

12. BACKGROUND - PERMUTATIONS CALCULATION
POSSIBLE PROTEINS FOR 20 AMINO-ACIDS
https://sciencing.com/many-possible-different-amino-acids-5343578.html

13. BACKGROUND - PERMUTATIONS CALCULATION
SCALING REQUIREMENTS - TIME
Number of Hosts
Class size
2553
= 16,581,375
Port Scanning - 10.0.0.0/8
16,581,375 * 65,536 = 1,086,676,992,000 = 12,577,280days
16,581,375 * 1,024 = 16,979,328,000 = 196,520days
16,581,375 * 1 = 16,581,375 = 191.9days

14. BACKGROUND - PERMUTATIONS CALCULATION
SCALING REQUIREMENTS - PRICE
Wardriving
iw dev wlan0 set channel CH

15. THE RIGHT TOOL FOR THE RIGHT JOB
DECREASING WORK UNITS
▸ Monitor everything
▸ Determine your current square
(set the worker unit)
▸ Prepare a quantifiable test
▸ Run the test at increasing orders of
magnitude (2^0, 2^1 … 2^n)
▸ Analyse bottle-necks
▸ Design and deploy the #next2

16. BACKGROUND - DISTRIBUTION CALCULATION
1. import hashlib, itertools, time
2. lower = 'abcdefghijklmnopqrstuvwxyz'
3. start_time = time.perf_counter()
4. [hashlib.md5(bytes(str(x), 'utf-8')).hexdigest() for x in
list(itertools.product(*[lower], repeat=1))]
[‘cfea63879ac12dfbea87b362984813f1','3dfeeb11befd8433f21911388151
d326' ………’9b2c180e2c7ed04cb1bb742e5d112efa’]
5. time.perf_counter() - start_time
A QUANTIFIABLE TEST

17. PERMUTATION CALCULATIONS
UNDERSTANDING THE PROBLEM DOMAIN
? ? ? ? ? ? ? ? ?
WR X
Linux permissions permutations

18. PERMUTATIONS CALCULATION
3+3+3 DIGIT BINARY NUMBER
0 0 0
0 0 1
0 1 0
0 1 1
1 0 0
1 0 1
1 1 0
1 1 1
Number of Options
String Length
? ? ?
23
* 23
* 23
= 512
{0,1} = 2 options
23
= 8

19. 1 2 4 16 32 64 128
256 1K 2K 4K 8K 16K 32K
64K 128K 256K 512K 1M 2M 4M 8M
16M 32M 64M 128M 256M 512M 1G 2G
4G 8G 16G 32G 64G 128G 256G 512G
1T 2T 4T 8T 16T 32T 64T 128T
256T 512T 1P 2P 4P 8P 16P 32P
64P 128P 256P 512P 1E 2E 4E 8E
8
512

20. BACKGROUND - PERMUTATIONS CALCULATION
PERMUTATION CALCULATIONS
A A A
A A B
N N N
N N O
Z Z Y
Z Z Z
Number of Options
String Length
? ? ?
263
= 17,576
{A-Z} = 26 options
.
..
..
.
N N P
3 capital letters WRD

21. 1 2 4 16 32 64 128
256 512 1K 2K 4K 8K 16K 32K
64K 128K 256K 512K 1M 2M 4M 8M
16M 32M 64M 128M 256M 512M 1G 2G
4G 8G 16G 32G 64G 128G 256G 512G
1T 2T 4T 8T 16T 32T 64T 128T
256T 512T 1P 2P 4P 8P 16P 32P
64P 128P 256P 512P 1E 2E 4E 8E
8
263
= 17,576

22. BACKGROUND - PERMUTATIONS CALCULATION
PERMUTATION CALCULATIONS
A A A
A A A
N N N
N N N
Z Z Z
Z Z Z
Number of Options
String Length
? ? ?
264
= 456,976
{A-Z} = 26 options
.
..
..
.
N N N
4 capital letters word
A
B
N
O
Y
Z
?
P

23. 1 2 4 16 32 64 128
256 512 1K 2K 4K 8K 16K 32K
64K 128K 256K 512K 1M 2M 4M 8M
16M 32M 64M 128M 256M 512M 1G 2G
4G 8G 16G 32G 64G 128G 256G 512G
1T 2T 4T 8T 16T 32T 64T 128T
256T 512T 1P 2P 4P 8P 16P 32P
64P 128P 256P 512P 1E 2E 4E 8E
8
456,976

24. BACKGROUND - PERMUTATIONS CALCULATION
? ? ?
528
= 53,459,728,531,456
{a-z,A-Z} = 26 + 26 = 52 options
8 letters word
? ? ? ? ?
SCALING REQUIREMENTS - CPU

25. 1 2 4 16 32 64 128
256 512 1K 2K 4K 8K 16K 32K
64K 128K 256K 512K 1M 2M 4M 8M
16M 32M 64M 128M 256M 512M 1G 2G
4G 8G 16G 32G 64G 128G 256G 512G
1T 2T 4T 8T 16T 32T 64T 128T
256T 512T 1P 2P 4P 8P 16P 32P
64P 128P 256P 512P 1E 2E 4E 8E
8
53,459,728,531,456

26. BACKGROUND - DISTRIBUTION CALCULATION
528
= 53,459,728,531,456
(52^8)b / 8(B) / 1024 / 1024 / 1024
= 6223.5 GiB
Storage
requirement
for a 1 bit task
SCALING REQUIREMENTS - STORAGE

27. BACKGROUND - DISTRIBUTION CALCULATION
>>> hashlib.md5(b"password").hexdigest()
‘5f4dcc3b5aa765d61d8327deb882cf99'
>>> len(hashlib.md5(b”password").digest())
16
>>> len('password'.encode())
8
>>> '{:,}'.format(sum([(52**x * (16+x)) for x in range(1, 9)]))
‘1,307,122,220,563,796' ~= 1.1PiB
< 8 letters word
16
16
8
x
SCALING REQUIREMENTS - STORAGE
password

28. 1b 2b 4b 1B 2B 4B 8B 16B
32B 64B 128B 256B 512B 1K 2K 4K
8K 16K 32K 64K 128K 256K 512K 1M
2M 4M 8M 16M 32M 64M 128M 256M
512M 1G 2G 4G 8G 16G 32G 64G
128G 256G 512G 1T 2T 4T 8T 16T
32T 64T 128T 256T 512T 1P 2P 4P
8P 16P 32P 64P 128P 256P 512P 1E
1,308,191,004,063,840

29. 1 2 4 8 16 32 64 128
256 512 1K 2K 4K 8K 16K 32K
64K 128K 256K 512K 1M 2M 4M 8M
16M 32M 64M 128M 256M 512M 1G 2G
4G 8G 16G 32G 64G 128G 256G 512G
1T 2T 4T 8T 16T 32T 64T 128T
256T 512T 1P 2P 4P 8P 16P 32P
64P 128P 256P 512P 1E 2E 4E 8E
f = 1/t
Chaos &
Overhead

30. BACKGROUND - DISTRIBUTION CALCULATION
528
= 53,459,728,531,456
8 letters word
520
* 528
= 528
= 1 * 528
= 528
Number
Of Workers
Tasks per
worker
= 521
* 527
= 528
xa
* xb
= = x * * (a + b)

31. BACKGROUND - DISTRIBUTION CALCULATION
(26 + 26)8
= 53,459,728,531,456
8 letters word
521
* 527
= 528
= 52 * 1,028,071,702,528 = 119.6GB
Number
Of Workers
Tasks per
worker
Storage
requirement
for a 1 bit task

32. THE RIGHT TOOL FOR THE RIGHT JOB
CHOOSE YOUR FAVOURITE 2 TCP FLAGS ;-)
Time
Complexity Price

33. t1 t2 t3 t4 t5 t7
Questions?