Data gets bigger and bigger and computing gets smaller and smaller. Do we know why?

1. DATA services

2. whoami
● NETOPIA Payments
○ Surprisingly (or not) we do other things than payments
● mobilPay Wallet
● Elrond
● Blockchain Romania

3. The (physical)
universe consists of a
3 x Space + 1 x Time
manifold

4. The computational
universe consists of 1
x Computing + 1 x
Storage + 1 x Time
manifold

5. In the computational
universe Computing
and Storage are
bound together by
Code

6. These days Code
moves/runs further
and further away*
from Computing

7. Id est
● Hardware has morphed into virtual
machines
● VMs have morphed into containers
● Containers are morphing into functions

8. At the same time
● When talking about “pure” data - the
closer to the hardware the faster it
can be accessed - and with smaller
latency

9. Id est
● Files have morphed into databases
● Databases have morphed into distributed
databases
● Distributed DBs have morphed into
○ distributed Ledgers and Blockchains
○ also into Data Lakes and Deltas (not sure yet about Seas and Oceans)

10. Aside: Big(ger) Data Structures
OLAP OLTP
Homogenous
Data Lake, Delta, Sea
Distributed Ledger*
Heterogeneous Blockchain

11. Empirical observation
Storage
Increases in scope
Computing
Decreases in scope

12. Guesstimation
● This happens because data is “inert”
but
● The smaller the unit of code that needs
to run, the more controllable the
output/behavior (we can even formally
verify some - if not all - of it)

13. Microservices unify
interaction (even if
data is also involved)

14. APIs are contracts

15. Destructuring Monoliths
● 3 TA with distributed application layer
and/or data layer
● Request driven (pull) vs Event driven
(push)
● Messaging - Event Sourcing
● Sagas w/ or w/o orchestrators

16. Microservice
Moving away from Monoliths
Monolithic Application
Computing
Storage
Storage
Computing
Microservice
Storage
Computing

17. Option 1
With this approach
● Big Data = Σ Small(ish) data
● Data consistency (say, transactions) is
hard

18. Microservice
Another option
MicroserviceMonolithic Application
Computing
Storage Storage
Computing
Microservice
Computing

19. Option 2
● Storage = House of Big Data
● Does not work all the time. It might
even be an anti-pattern.

20. Modus Operandi
● API Gateway / Service Mesh
● Service Registry
These tend to handle only the computing
side

21. Data side
● Additional concerns: “at rest” behavior
vs “in flight” behavior
● Schemas => APIs
● Stateful serverless?

22. DataOps
● It’s not just DevOps for Data
● Intersection of Computing and Storage
w/ Organization
○ Human Resource 4-dimensions
■ Wants
■ Knows
■ Time / Availability
■ Privileges

23. Storage and
Computing are just as
important as
metadata (including
processes, people ...)

24. Conway’s law
Any organization that designs a system (defined broadly)
will produce a design whose structure is a copy of the
organization's communication structure
Melvin Conway, 1967

25. But even if we only
talk about technology
things are
complicated

26. CNCF Landscape
Img Src https://landscape.cncf.io/

30. YMMV: just because
you can do it, it
doesn’t mean that you
should do it

31. Thanks!
Follow me: @fixone