Presented at Hyperledger Summit, Basel, 2018.

1. Instrumentation, Resource Allocation and Monitoring for smart
contracts on the Blockchain
Vijayendra Bhamidipati (eBay)
Michael Chan (eBay)
Arpit Jain (eBay)
Hyperledger Summit 2018, The Linux Foundation

2. Agenda
Visibility and Metrics
Instrumenting Systems and Blockchain Frameworks
Billing for Smart Contracts
Resource Allocation
Monitoring and Alerting
Security
Q&A

3. Agenda
Visibility and Metrics
Instrumenting Systems and Blockchain Frameworks
Billing for Smart Contracts
Resource Allocation
Monitoring and Alerting
Security
Q&A

4. Visibility
Like an X-Ray - how easy is it to dissect/analyze the ecosystem?
Examples of visibility -
- Components that make up the system
- Core and Perimeter
- Data Flows
- Control Flows
- System Policies
- Behavioral patterns

5. Visibility in a Blockchain Ecosystem
- What validators are running?
- What permissioned “views” are currently configured?
- Which smart contracts have been deployed?
- What tokens have been created?
- What consensus algorithm(s) is/are in use?
- What is the current scale of deployment?
- And more..

6. Metrics
Health
Deadlocked/Blocked processes,
Deviations from expected states,
etc.
Resource Utilization
CPU, Memory, Disk I/O, Network
I/O, etc.
Blockchain
Framework
Smart Contracts
Infra
Hashrate, # of blocks mined per
period, Block read times, Block
write times, Consensus Algo, etc.
Business Logic
Overall
Latency
Oraclized
Call Perf

7. A note on Debuggability
Closely tied to visibility and metrics
- Logging and Log Analysis infrastructure
- Debug Builds
- Live and offline debugging capabilities
- Robust test frameworks

8. Agenda
Visibility and Metrics
Instrumenting Systems and Blockchain Frameworks
Billing for Smart Contracts
Resource Allocation
Monitoring and Alerting
Security
Q&A

9. Frameworks that facilitate metric creation, emission, collection and analysis
Traditional resource utilization measurement/profiling techniques -
- Language specific profiling - Go profiling, Python profiling, etc.
- System level profiling - SAR, vmstat, iostat, etc.
Instrumentation

10. - Not granular enough
- For function level metrics, need modification of code.
- Should be planned ahead of time
- May not be ideal in all cases of production deployment
- Not easily modifiable
- In case of blockchain, immutability is a major blocker
Disadvantages of traditional techniques

11. A note on Hyperledger Caliper
• In Hyperledger incubation
• Requires code modification for instrumentation

12. Is there an alternative?
eBPF! (Extended Berkeley Packet Filter)

13. Highly programmable
Highly granular
Can be inline, but doesn’t require code modification
Ideal for blockchain and smart contracts
- Linux Perf can run eBPF
- iovisor bcc is preferred over Linux Perf
eBPF

14. Brings in visibility at userland function level
Couple this with call graphing
Enables measurement of -
- Latency of functions in a smart contract
- Resource utilization of functions in a smart contract
More about eBPF

15. eBPF VM
Demo
Hyperledger Fabric
ChainCode
(MyEchoAsset)
ElasticSearch
Kibana
MyEchoAsset.Invoke() Probe
Stats
Emitter

17. Video

18. Drawbacks of eBPF
Available only on Linux 4.4+
Scalability
○ Alternative is to build similar functionality into the Virtual Machine executing
the smart contracts
■ Will require blockchain framework support

19. Agenda
Visibility and Metrics
Instrumenting Systems and Blockchain Frameworks
Billing for Smart Contracts
Resource Allocation
Monitoring and Alerting
Security
Q&A

20. Billing
Public Blockchains
Drivers
- Next few minutes
- Next few blocks
Variability is ok
Permissioned/Private
Blockchains
Drivers
- QoS
- OpEx
Variability is not
always ok

21. Enterprise billing must be multi variate. Examples:
- Specific function chain latency
- Partnership service charges
- IP valued smart contract services
- Demand based contract/service pricing
- Resource utilization based pricing
- Sub-licensing or multi party service provider based pricing
Blockchain frameworks in general can help build Automated Billing Policy Engines
Billing (contd)..

22. Agenda
Visibility and Metrics
Instrumenting Systems and Blockchain Frameworks
Billing for Smart Contracts
Resource Allocation
Monitoring and Alerting
Security
Q&A

23. Two parts to this -
- Engines that dictate resource allocation policies
- Planes that realize the actual resource allocation/allotment.
Resource Allocation

24. Resource Allocation Plane
Containerization
- Readily available Linux native construct
- Allows for resource limits and requests
- Not too granular - system specific and based on Linux namespace
- Multiple blockchain logical constructs in a single namespace
- How do you allot resources between them within a namespace?
Blockchain Aware Native Scheduling (BANS)

25. Resource Allocation Constructs for the Blockchain
Business Logic
Specific Smart Contracts
CoS
Service Differentiation
- IoT
- DRM
- Financial
transactionsQoS
Latency requirements
Environments
- Mobile
- Desktop
- Native

26. Blockchain aware Native Scheduling
Smart contract awareness as kernel constructs and modules into the OS
- Like Linux containers did with namespaces
Build Consensus Algorithms as Protocols and Protocol Suites into the Linux ecosystem
- Allows for -
- Lean implementations for embedded systems (IoT)
- Highly efficient mobile blockchain modules/ecosystems

27. Resource Allocation Engine
Is the Control Plane
- Basic
- Simple policy driven allocation
- Advanced
- Leverage ML/statistical methods to identify patterns of usage and drive scheduling of
resources.

28. Agenda
Visibility and Metrics
Instrumenting Systems and Blockchain Frameworks
Billing for Smart Contracts
Resource Allocation
Monitoring and Alerting
Security
Q&A

29. Monitoring and
Alerting
Popular enterprise monitoring
frameworks -
- Prometheus
- statsd
- Filebeat
- Logstash
- ElasticSearch
- Kibana

30. Agenda
Visibility and Metrics
Instrumenting Systems and Blockchain Frameworks
Billing for Smart Contracts
Resource Allocation
Monitoring and Alerting
Security
Q&A

31. Security
Again, two parts to this -
- Engines that dictate security policies
- Planes that realize the security policies

32. Security enforcement plane
We propose
• “Function Based Access Control” (FBAC)
• “Data Based Access Control” (DBAC)

33. FBAC
Block/allow invocations of specific function chains.
• Userspace functions both in the blockchain framework and Smart contracts
Whitelist/Blacklists are FBAC policies that are baked into the Blockchain itself.
A scalable way of implementing Permissioned Blockchains

34. DBAC
Block/allow invocations of specific function chains based on function parameter values.
Block/allow reads of specific sets of data.
• Again, Whitelist/Blacklists are DBAC policies that are baked into the Blockchain itself.

35. FBAC/DBAC implementation
Two approaches -
- eBPF
- Blockchain Virtual Machine (EVM (ethereum virtual machine) for example)

36. Blockchain Team @ eBay
Michael Chan (michachan@ebay.com)
Vijayendra Bhamidipati (vbhamidipati@ebay.com)
Arpit Jain (arpijain2@ebay.com)
Brian Haslam (bhaslam@ebay.com)
Ashok Murthy (asmurthy@ebay.com)
Derek Chamorro (dchamorro@ebay.com)

37. A note of thanks..
Venkatesh Palani (kupalani@ebay.com)
Brian Haslam (bhaslam@ebay.com)
Qiu Yu (qiuyu@ebay.com)

38. References
https://github.com/hyperledger/caliper/blob/master/docs/Architecture.md
http://www.brendangregg.com/perf.html
http://www.brendangregg.com/blog/2015-05-15/ebpf-one-small-step.html
http://www.brendangregg.com/blog/2017-01-31/golang-bcc-bpf-function-traci
ng.html
https://github.com/iovisor/bcc
https://github.com/cilium/cilium/blob/master/Documentation/bpf.rst

39. Q&A