This keynote talk covered research and insights from the speaker’s last 7 years of research on two topics of relevance to the RSDA workshop. The first part was about behavioural data analytics for reliability and dependability purposes, i.e., with a focus on sequences of events, rather than static structures or the presence of certain events in a log. Specifically, the approach of using Process Mining techniques to gain insights into traces of logged events was presented. The second part of the talk was centred around a summary of insights into security, dependability, and reliability aspects of blockchain applications. With the introduction of smart contracts, blockchain technology has become a general-purpose code execution framework, and architects and developers need to understand the specific implications on security and dependability when building applications on blockchain.

1. Behavioral Analytics and Blockchain
Applications – a Reliability View
Keynote, RSDA 2019
Prof. Dr. Ingo Weber | Oct. 2019
ingo.weber@tu-berlin.de | linkedin.com/in/ingomweber/ | Twitter: @ingomweber

2. Process-Oriented Dependability
(POD) framework
Website:
https://research.csiro.au/data61/process-oriented-dependability/

3. Book
DevOps: A Software Architect’s Perspective
3
Len Bass, Ingo Weber, Liming Zhu.
DevOps: A Software Architect’s
Perspective.
Addison Wesley, June 2015
http://dx.doi.org/10.1007/978-0-134-04984-7

4. • Gartner predicts:
• “… 80% of outages impacting mission-critical services will be caused by people and
process issues, and more than 50% of those outages will be caused by change /
configuration / release integration and hand-off issues.”
• The case of Knight Capital – from Wikipedia:
• The Knight Capital Group, the largest trader in U.S. equities on NYSE and NASDAQ,
made a trading error lost $460 million, which basically bankrupted them
• This took 45 minutes and was an upgrade error
• Confirmed by empirical studies
• Big data analytics application (Yuan, OSDI14) and cloud application (Gunawi, SoCC14)
performance issues are largely caused by operational protocols
Motivation: Dependability of Cloud
Operations
4

5. • Significantly shorter release cycles
• Continuous delivery/deployment: from months / scheduled downtime to hours / any time
• Etsy.com: 25 full deployments per day at 10 commits per deploy
• Baseline-based anomaly detection no longer works!
• Continuous changes
• Multiple sporadic operations at all times
• Scaling in/out, snapshot, migration, reconfiguration, (rolling) upgrade, cron-jobs, backup, recovery…
• Cloud uncertainty
• Limited visibility/monitoring, indirect control, small-scale failures are the norm
Challenges: Continuous Changes &
Uncertainty
5

6. A previous study found:
• Around 50% of application failures are not identifiable by
scanning logs for keywords / events that explicitly denote the
occurrence of failures [Pecchia, ISSRE 2012]
Using logs for error detection?
6

7. • Increasing dependability during Operation time e.g., through:
• More accurate performance monitoring
• Faster error detection
• Fast or autonomous healing (quick fix)
• Root cause diagnosis
• Guided or autonomous recovery
• Incorporating change-related knowledge into system management
• Knowledge about sporadic operation in Process-Oriented Dependability (POD): error detection and
diagnosis using process model & context
Our Approach – High-level View
7

8. POD: Process-Oriented Dependability
[insert hero image][insert hero image]
Start rolling upgrade task
Update launch
configuration
Sort instances
Status info
Rolling upgrade task
completed
Remove and deregister
old instance from ELB
New instance ready and
registered with ELB
Terminate old instance
Wait for ASG to start new
instance
Process Model
Conformance
Checking
Service
Assertion
Evaluation
Service
POD-Detection
Log line
Alert POD-
Diagnosis
Operation
tools
Log
agent
POD-
RecoveryAlert
MetricsMonitoring
Tools
Cause
Process Context
Discover
Log lines
POD-
Discovery
POD-Viz
Offline: training Online: error detection / diagnosis / recovery
8

9. Error detection visualized: POD-Viz
12

10. Error detection visualized: POD-Viz
13

11. Sporadic Operation Example:
Rolling Upgrade
Rolling upgrade: upgrade software on many virtual
machines without downtime or significant additional
cost
Example:
- 100 servers running in the cloud with version 1
software
- Upgrade 10 servers at a time to version 2 software
Potentially takes a long time to complete with errors
during the operation / other interfering operations
14
Start rolling upgrade task
Update launch
configuration
Sort instances
Status info
Rolling upgrade task
completed
Remove and deregister
old instance from ELB
New instance ready and
registered with ELB
Terminate old instance
Wait for ASG to start new
instance

12. System Monitoring During Rolling Upgrade
Standard anomaly detection raises lots of alerts
→ Operators switch it off during sporadic operations or
ignore the alerts
• Not a good idea if done 25x a day
15

13. POD-Detection

14. POD-Detection: finding errors & anomalies
[insert hero image][insert hero image]
Start rolling upgrade task
Update launch
configuration
Sort instances
Status info
Rolling upgrade task
completed
Remove and deregister
old instance from ELB
New instance ready and
registered with ELB
Terminate old instance
Wait for ASG to start new
instance
Process Model
Conformance
Checking
Service
Assertion
Evaluation
Service
POD-Detection
Log line
Alert POD-
Diagnosis
Operation
tools
Log
agent
POD-
RecoveryAlert
MetricsMonitoring
Tools
Cause
Process Context
Discover
Log lines
POD-
Discover
POD-Viz
Log agent forwards log lines as they appear.
Monitoring metrics and Cloud APIs can be accessed
live.
Two error detection services:
• Assertion Evaluation
• Conformance Checking
17

15. POD-Detection: assertion evaluation
[insert hero image][insert hero image]
Start rolling upgrade task
Update launch
configuration
Sort instances
Status info
Rolling upgrade task
completed
Remove and deregister
old instance from ELB
New instance ready and
registered with ELB
Terminate old instance
Wait for ASG to start new
instance
Process Model
Conformance
Checking
Service
Assertion
Evaluation
Service
POD-Detection
Log line
Alert POD-
Diagnosis
Operation
tools
Log
agent
POD-
RecoveryAlert
MetricsMonitoring
Tools
Cause
Process Context
Discover
Log lines
POD-
Discover
POD-Viz
18

16. • Assertions check if the actual state at some point is the expected
state
• Coded against Cloud APIs – can find out the true state of resources directly
• Originally, assertions were coded manually based on our
understanding of the operations
• Low level assertion
• Instance i is terminated successfully
• High level assertion
• There are n instances running version x
Creating Assertions
19

17. Assertion Evaluation: how it works
Log line:
• Remove ...
• Terminate ...
• Wait ...
• New instance ...
Assertions:
• i has been de-registered from
ELB
• i has been removed from ASG
• i successfully terminated• i‘ successfully launched and is
registered with ELB
20

18. To identify and leverage the correlation between state of resources
with changes in cloud resources metrics in order to:
• Identify the logs that are handling changes in resources
• Predict the expected behavior of changes in system.
• Exploit above outcome for runtime anomaly detection
➔ ISSRE Best Paper Award
Automatic Assertion Derivation
21

19. Logs and resources metrics observations
22

20. • Identify the main factors affecting a resource
• Identify the log events that have the most important influence on
changing the state of a system resource.
• Metric selection: which of the 100s of metrics are relevant?
• Extract a prediction formula
• Derived a formula that could be used to estimate the value of a
variable that associated to a system’s resource with an acceptable
range of error
• Derive assertion specification based on above
Correlation between Logs and Resources
23

21. POD-Detection: conformance checking
[insert hero image][insert hero image]
Start rolling upgrade task
Update launch
configuration
Sort instances
Status info
Rolling upgrade task
completed
Remove and deregister
old instance from ELB
New instance ready and
registered with ELB
Terminate old instance
Wait for ASG to start new
instance
Process Model
Conformance
Checking
Service
Assertion
Evaluation
Service
POD-Detection
Log line
Alert POD-
Diagnosis
Operation
tools
Log
agent
POD-
RecoveryAlert
MetricsMonitoring
Tools
Cause
Process Context
Discover
Log lines
POD-
Discover
POD-Viz
24

22. • From logs to models
• Example:
Log:
• (a,b,c,d)
• (a,c,b,d)
• (a,e,d)
Process Mining Basics:
Process Discovery
Model:
25

23. • Compare: log vs. model
• Base criterion: fitness
• “Does the log fit the model and vice versa?”
Process Mining Basics:
Conformance Checking
?
26

24. • 3 levels of CC:
• Basic CC
• Detecting numerical invariants
• Detecting timing anomalies
• Any errors / anomalies detected: raise alert
• All results visualized through POD-Viz
Conformance Checking (CC) approach
Retrieve log
data
Basic
conformance
checking
Detect num.
invariant
violations
Detect
timing
anomalies
Visualize
results
(POD-Viz)
27

25. Basic CC: how it works
Log lines:
• Remove ...
• Terminate ...
• Wait ...
• Terminate ...???
Raise alert
Error count +1
28

26. Basic conformance checking: outcomes
• Conformance checking can detect the following types of errors:
• Unknown / error log line: a log line that corresponds to a known error, or is
simply unknown.
• Unfit: a log line corresponds to a known activity, but said activity should not
happen in the current execution state of the process instance.
• All other log lines are deemed fit
• Goal: 100% fit, else raise alert
• Learn from false alerts → improve classification and/or model
29

27. Advanced CC 2: timing anomalies
• Offline:
• Build precise timing profiles for activities
• Derive anomaly intervals from timing profiles for y% most unlikely timing
• Import into CC for timing anomaly checking
• Online:
• CC checks timing & raises alerts
31
X

28. • Non-intrusive, near real time error detection
• API-based assertion evaluation uses AWS services so it takes some overhead – in the order of sec
• Conformance checking is very fast, ~10ms
• Log-metric correlation accuracy may increase over time – best trade-off for AWS we found 1 min
TW
• API-based assertion evaluation detects previously discovered / known errors
(regression)
• Conformance checking detects any error that causes non-conformance, numerical
invariant violation, or timing anomalies
• Previously discovered or not
• Per subprocess instance, highly precise
• Behavioral log analysis, taken to a new level
• Log-metric correlation is based on historic data
• No labels needed, but needs # of no-failure cases >> # of failure cases
Error Detection Discussion
32

29. Recent study: process mining for error
detection in presence of noise
https://doi.org/10.1016/j.knosys.2019.105054
• Applying POD for application monitoring
• Core idea: low-effort application of POD to application logs for error detection
• “Blindly” learn process models
• At runtime: check if behavior conforms to the models
• Test impact of uncertainty of logs, e.g., missing events and residual noise
33

30. Recent study: process mining for error
detection in presence of noise
• Evaluation on 55,462 execution traces from three independent real-life
applications: Apache Web Server, Open DDS, and MySQL DBMS
• Comparative evaluation with expert system (baseline)
• Main findings:
• All failures detected by the expert system are detected also by conformance checking
• Conformance checking infers
many additional failures that
are missed by the expert system
• Correct executions of an
application might not conform
to its normative model
• Noise in process models
contributes to improved
precision of conformance
checking, mostly with an
acceptable loss of recall
34

31. Blockchain Applications:
Dependability, Reliability, Security

32. Book:
Architecture for Blockchain Applications
Xiwei Xu, Ingo Weber, Mark Staples.
Architecture for Blockchain Applications.
Springer, 2019.
http://dx.doi.org/10.1007/978-3-030-03035-3
36

33. Blockchain 2nd gen – Smart Contracts
37
• 1st gen blockchains: transactions are financial transfers
• From 2nd gen: blockchains can do that, and
store/transact any kind of data
• Blockchains can deploy and execute programs: Smart Contracts
• User-defined code, deployed on and executed by whole
network
• Can enact decisions on complex business conditions
• Can hold and transfer assets, managed by the contract itself
• Ethereum: pay per assembler-level instruction

34. Decentralised Applications and Smart
Contracts Defined
• Smart contracts
• …are programs deployed as data and executed in transactions on the blockchain
• Blockchain can be a computational platform (more than a simple distributed database)
• Code is deterministic and immutable once deployed
• Can invoke other smart contracts
• Can hold and transfer digital assets
• Decentralized applications or dapps
• Main functionality is implemented through smart contracts
• Backend is executed in a decentralized environment
• Frontend can be hosted as a web site on a centralized server
• Interact with its backend through an API
• Could use decentralized data storage such as IPFS
• “State of the dapps” is a directory recorded on blockchain:
https://www.stateofthedapps.com/
38

35. Blockchain-based Application
• A blockchain-based application (or just blockchain application)
makes significant use of blockchain
• dapps are an example, but the concept is far broader
• significant portions of such applications can be based on traditional systems.
• Globally, many financial services companies, enterprises, startups,
and governments are exploring suitable applications
• Areas include supply chain, electronic health records, voting, energy
supply, ownership management, and protecting critical civil
infrastructure
• By now, almost all industry sectors have explored blockchain use
39

36. https://medium.com/fluree/blockchain-for-2018-and-beyond-a-growing-list-of-blockchain-use-cases-37db7c19fb99

37. We (Will) Rely on Blockchain-Based Systems
• Smart contract bugs: DAO ($60M); Parity ($280M)
• Cryptographic key loss
• Hacking: Mt Gox ($450M), Bitfinex ($72M), total Jan-Sep 2018 ($927M);
UK rubbish tip ($146M); guns e.g. NYC ($1.8M)
• Huge future economic value (the main point!)
• e.g. supply chain, asset registries, settlement, energy, …
• Security-critical and Safety-critical use cases
• e.g. e-health records, food safety, pharma supply chain,
IoT management, cybersecurity, law enforcement, …

38. Dependability and Security Attributes
Source: “Basic concepts and taxonomy of dependable and secure computing”
https://www.nasa.gov/pdf/636745main_day_3-algirdas_avizienis.pdf
Dependability Security
Confidentiality
Maintainability
Integrity
Safety
Availability
Reliability
42

39. Non-Functional Trade-Offs
• Compared to conventional databases & script engines,
blockchains have:
(-) Confidentiality, Privacy
(+) Integrity, Non-repudiation
(+ read/ - write) Availability
(-) Modifiability
(-) Throughput / Scalability / Big Data
(+ read/ - write) Latency
Security: combination of
CIA properties

40. Functional
suitability
Functional
correctness
Functional
complete-
ness
Functional
appropriate
-ness
Performance
efficiency
Capacity
Resource
utilization
Time
behavior
Compatib-
ility
Interop-
erability
Co-
existence
Usability
Operability
User error
protection
Reliability
Availability
Recoverab-
ility
Maturity
Fault
tolerance
Security
Integrity
Confidential
-ity
Non-
repudiation
Accountab-
ility
Authent-
icity
Maintain-
ability
Modularity
Reuseability
Modifiab-
ility
Testability
Analyzab-
ility
Portability
Installability
Replace-
ability
Adaptability
ISO/IEC 25010:2011 Quality Model
44

41. ISO/IEC 25010:2011
Security Characteristics
• Integrity
• degree to which a system, product or component prevents unauthorized access to,
or modification of, computer programs or data
• Confidentiality
• degree to which a product or system ensures that data are accessible only to those
authorized to have access
• Non-repudiation
• degree to which actions or events can be proven to have taken place, so that the
events or actions cannot be repudiated later
• Accountability
• degree to which the actions of an entity can be traced uniquely to the entity
• Authenticity
• degree to which the identity of a subject or resource can be proved to be the one
claimed
(ISO/IEC 5010:2011 treats Availability as a “Reliability” characteristic)
Only good
writes & deletes
Only good reads
Undeniable
You did it!
Not fake

42. Integrity for Blockchain Platforms
• Clark-Wilson perspective on blockchain integrity
• Blockchain ledger is the system state and the audit log
• Blocks and their transactions are the “Constrained Data Items”
• Initial state: genesis block is well-formed & cross-checked by all miners
• Mining and cross-checking other miners’ blocks is the “Transformation Procedure”
• Miners’ block validation checks are the “Integrity Validation Procedure”
• Authorisation is checked using signatures from public-key cryptography
• No authentication on public blockchains! Often important on private blockchains
• No separation of duty enforced?
• “Admin changes” are by the mining collective, e.g. hard forks
• Example integrity conditions
• Were transactions against an account address signed by corresponding private key?
• Does the sending address/account have enough cryptocurrency?
• Some platforms support other crypto-assets with different integrity conditions
• Did a miner give themselves the right mining reward?
• Did the execution recorded for a smart contract give the same result I get?

43. Confidentiality
• ISO/IEC 25010:2011: “degree to which a product or system ensures that data
are accessible only to those authorized to have access”
• Blockchain platforms are not good for confidentiality, because mining nodes
cross-check contents of all transactions
• Not just the plain data, also need to be concerned with re-identification
attacks, patterns from transaction analytics, and graph datamining
• Identity of parties?
• Transaction volume?
• Transaction meta-data?
• e.g. characteristic patterns in time-of-day for transactions, geolocation of source IP
addresses of submitted transactions

44. Non-Repudiation
• ISO/IEC 25010:2011: “degree to which actions or events can be proven to
have taken place, so that the events or actions cannot be repudiated later”
• Main support is from blockchain’s immutable ledger
• But be careful of probabilistic immutability in Nakamoto consensus; use the right
number of confirmation blocks for your application’s risk profile
• Some support from public key signatures for transactions
• Just because data is recorded on-chain, doesn’t mean it’s true!
• Someone might have stolen my private key?
• Sender might have fraudulently recorded false data in a transaction
• Blockchain only provides non-repudiation that the transaction happened
• If using hashes on-chain for off-chain data, need to retain original data

45. ISO/IEC 25010:2011
Reliability Characteristics
• Reliability
• degree to which a system, product or component performs specified functions under
specified conditions for a specified period of time
• Availability
• degree to which a system, product or component is operational and accessible when
required for use
• Recoverability
• degree to which, in the event of an interruption or a failure, a product or system can recover
the data directly affected and re-establish the desired state of the system
• Maturity
• degree to which a system, product or component meets needs for reliability under normal
operation
• Fault-Tolerance
• degree to which a system, product or component operates as intended despite the presence
of hardware or software faults

46. Availability
• ISO/IEC 25010:2011: “degree to which a system, product or
component is operational and accessible when required for use”
• A measure could be something like “probability of being able to provide
service at any given time” or “
• Affected by the other reliability characteristics, such as probability of failure,
fault tolerance, time to recovery, etc
• Blockchain platform is highly redundant (many nodes)
• Easy to subscribe to updates to get replicas of the ledger
• An application can run many redundant blockchain nodes locally
→ high read availability
• Write availability is a different story…

47. Availability (SRDS 2017)
• Potential issue: block gas limit (≈size of a block on Ethereum)
• Gas limit is set by miners through “voting”
• The sum of Gas of all transactions in a block must be less than the limit
• Response to DDoS attack: lower block gas limit

48. Availability (SRDS 2017)
• Potential issue: block gas limit (≈size of a block on Ethereum)
• Gas limit is set by miners through “voting”
• The sum of Gas of all transactions in a block must be less than the limit
• Response to DDoS attack: lower block gas limit
• Who would be affected by that?

49. Availability for Blockchain-Based
Applications
• A blockchain-based application has many components
• Even if the blockchain platform works, your other components may fail
• Use normal availability-increasing design strategies for the architecture,
for example…
• Increase quality of each component & connector
• High quality software and hardware (!)
• Eliminate single points of failure/ increase redundancy
• Load balancing/failover monitoring and routing
• Stateless server components
• Blockchain can help enable “stateless” server component (use blockchain to store the
state)
• Detect and recover from failures
• Hot backup/failover servers

50. Maturity (“Reliability”)
• ISO/IEC 25010:2011: “degree to which a system, product or
component meets needs for reliability under normal operation”
• My opinion: not a great name for this…
• Availability is about readiness for correct service
vs.
• Reliability is about continuity of correct service
• Previous discussion about availability for blockchain-based
applications applies here too

51. Process Mining / Analytics
• Process mining can be used to understand how clients and software
interact
• Also for blockchain applications and dapps
• But: understanding log data from blockchain is hard
• Two independent approaches, both presented at BPM 2019, tackle
the problem
• “Mining Blockchain Processes: Extracting Process Mining Data from
Blockchain Applications” → BPM Blockchain Forum 2019, best paper
• Can be used on any blockchain application, designed with process-awareness or not
• “Extracting Event Logs for Process Mining from Data Stored on the
Blockchain” → SPBP Workshop 2019
• Makes the assumption that the blockchain data was generated from executing a process
model

52. BloXES Framework (from “Mining Blockchain Processes[…]”)
IEEE 1849-2016BloXESEthereum
Manifest
Extractor
Logs
Validator
Smart
Contract
Generator

53. Thank you for your attention!
Behavioral Analytics and Blockchain
Applications – a Reliability View
Keynote, RSDA 2019
Prof. Dr. Ingo Weber | Oct. 2019
ingo.weber@tu-berlin.de | linkedin.com/in/ingomweber/ | Twitter: @ingomweber