Christian Sailer, R3, presents on DevDay at CordaCon 2019.

1. 11
Reliable Finality,
Lessons From the Field
Christian Sailer – R3
christian.sailer@r3.com

2. And why is it important?Quick recap:
What is finality
The concept of Finality in Corda:
Ensure a transaction gets notarised and recorded with all relevant parties

3. And why is it important?Quick recap:
What is finality
The concept of Finality in Corda:
Ensure a transaction gets notarised and recorded with all relevant parties
“I see what you see” – the central promise of a distributed ledger

4. And why is it important?Quick recap:
What is finality
The concept of Finality in Corda:
Ensure a transaction gets notarised and recorded with all relevant parties
“I see what you see” – the central promise of a distributed ledger
• Verify transaction
• Notarise state spending if required (check for double spend)
• Record transaction, consumed and created states
• Broadcast transaction to all involved parties

5. What it isn’tQuick recap:
What is finality
Finality takes not care of the following – this has to happen before invoking finality!
• Agreement between counterparties
• Collection of signatures

6. Evolution of finality
• Original Finality flow has been part of the Corda API since very first prototypes
• Reworked for Corda 4:
• Modified to take relevance of states into consideration
• Explicit receive finality flow so only nodes waiting for broadcast will store states
• Use inline flows with explicit list of sessions to broadcast to
• Current changes:
• Verify/ensure ledger consistency if something goes wrong.

7. How does finality work?
The initiating flow invokes the finality subflow
subFlow(FinalityFlow(counterSignedTx, recipientSession))

8. How does finality work?
The initiating flow invokes the finality subflow
subFlow(FinalityFlow(counterSignedTx, recipientSession))
Transaction to be
finalised

9. How does finality work?
The initiating flow invokes the finality subflow
subFlow(FinalityFlow(counterSignedTx, recipientSession))
Transaction to be
finalised
Sessions for
broadcast

10. How does finality work?
The initiating flow invokes the finality subflow
Finality flow (simplified):
subFlow(FinalityFlow(counterSignedTx, recipientSession))
val ledgerTransaction = verifyTx()
val notarised = notariseAndRecord()
for (session in sessions) {
subFlow(SendTransactionFlow(session, notarised))
}

11. How does finality work?
The initiating flow invokes the finality subflow
The initialised flows invoke the receive finality subflow
subFlow(FinalityFlow(counterSignedTx, recipientSession))
// Not ideal that we have to do this check, but we must as FinalityFlow
// does not send locally
if (!serviceHub.myInfo.isLegalIdentity(otherSide.counterparty)) {
subFlow(ReceiveFinalityFlow(otherSide))
}

12. What can go wrong?
• Transaction verification can fail
• That is fine – will just roll back

13. What can go wrong?
• Transaction verification can fail
• That is fine – will just roll back
• Notarisation can fail
• Not a problem for consistency – can still just roll back

14. What can go wrong?
• Transaction verification can fail
• That is fine – will just roll back
• Notarisation can fail
• Not a problem for consistency – can still just roll back
• Once a transaction is notarised, it must be recorded
• The input states are recorded as spent on the notary
• Rolling back would leave them as unspent in the local vault
• The ledger would be inconsistent!

15. Failure after notarisationFinality errors
• Database errors while recording
• Database connection failure
• Database out of space
• Other transient database errors

16. Failure after notarisationFinality errors
• Database errors while recording
• Database connection failure
• Database out of space
• Other transient database errors
• Network errors when broadcasting

17. Failure after notarisationFinality errors
• Database errors while recording
• Database connection failure
• Database out of space
• Other transient database errors
• Network errors when broadcasting
• Database errors on the counterparty

18. Failure after notarisationFinality errors
• Database errors while recording
• Database connection failure
• Database out of space
• Other transient database errors
• Network errors when broadcasting
• Database errors on the counterparty
• Errors in CorDapp code

19. Failure after notarisationFinality errors
• Database errors while recording
• Database connection failure
• Database out of space
• Other transient database errors
• Network errors when broadcasting
• Database errors on the counterparty
• Errors in CorDapp code Wait, what?!

20. Raw vault observerExcursion
• Service gets instantiated on every node where the CorDapp installed
• Sets up observer on raw vault feed
• The subscribed lambda gets run when vault states get added in the context of the
flow fiber before database commit
@CordaService
class DbListenerService(services: AppServiceHub) : SingletonSerializeAsToken() {
init {
services.vaultService.rawUpdates.subscribe { (consumed, produced) ->
…
}
}
}

21. Raw vault observersFinality errors
User (i.e. CorDapp) code can be injected e.g. via a raw VaultObserver
• Raw updates are processed before recording DB transaction is committed
• Can interact with database
• Useful if additional changes need to be committed to the database atomically
• Can throw exceptions that interfere with committing the DB transaction
• Can alter the state of the node (e.g. unsubscribe observers)

22. Raw vault observersFinality errors
User (i.e. CorDapp) code can be injected e.g. via VaultObserver
• Raw updates are processed before recording DB transaction is processed
• Can interact with database
• Can throw exceptions that interfere with committing the DB transaction
• Can alter the state of the node (e.g. unsubscribe observers)
This has been observed by users

23. Raw vault observersFinality errors
User (i.e. CorDapp) code can be injected e.g. via VaultObserver
• Raw updates are processed before recording DB transaction is processed
• Can interact with database
• Can throw exceptions that interfere with committing the DB transaction
• Can alter the state of the node (e.g. unsubscribe observers)
This has been observed by users
Investigation into what happens in error cases

24. Finality errors in
action
• CorDapp that can force a number of errors
• Database or user exception in VaultObserver
• On created or consumed state
• Forcing data base exceptions on transaction commit

25. Finality Errors in
action
• CorDapp that can force a number of errors
• Database or user exception in VaultObserver
• On created or consumed state
• Forcing data base exceptions on transaction commit
• Integration tests using this CorDapp
• Checking contents of vault and notary log
• Checking whether flows finish (successfully or error state) or get stuck
• Checking whether consistency can be recovered via restart

26. Database errors during recording locallyHandling errors
during finality
Notarised
Recorded locally
Distributed to participants
• When throwing an exception, the flow will be passed to the flow hospital
• In case of certain exceptions (e.g. transient exception, connection exception), the
flow might be retried from the last checkpoint
• Other DB exceptions/if retry fails, the flow will be kept dormant

27. Database errors during recording locallyHandling errors
during finality
Notarised
Recorded locally
Distributed to participants
• When throwing an exception, the flow will be passed to the flow hospital
• In case of certain exceptions (e.g. transient exception, connection exception), the
flow might be retried from the last checkpoint
• Other DB exceptions/if retry fails, the flow will be kept dormant
• Fixing the database problem and restarting the node should make finality complete.

28. User error from raw VaultObserverHandling errors
during finality
Notarised
Recorded locally
Distributed to participants
• Exception will stop recording of transactions
• The exception might escape and kill the flow

29. User error from raw VaultObserverHandling errors
during finality
Notarised
Recorded locally
Distributed to participants
• Exception will stop recording of transactions
• The exception might escape and kill the flow
The ledger can end up inconsistent between the notary and the nodes!

30. User error from raw VaultObserverHandling errors
during finality
Notarised
Recorded locally
Distributed to participants
• Exception will stop recording of transactions
• The exception might escape and kill the flow
The ledger can end up inconsistent between the notary and the nodes!
Don’t do this!

31. Network error during broadcastHandling errors
during finality
Notarised
Recorded locally
Distributed to participants - will be delivered eventually
• Initiating flow will keep retrying to broadcast
• If flows get interrupted/nodes restarted, persistent queues will redeliver the
message

32. Network error during broadcastHandling errors
during finality
Notarised
Recorded locally
Distributed to participants - will be delivered eventually
• Initiating flow will keep retrying to broadcast
• If flows get interrupted/nodes restarted, persistent queues will redeliver the
message

33. Database error during recording on counterpartyHandling errors
during finality
Notarised
Recorded locally
Distributed to participants
• Failure to record on the counterparty does not affect initiating flow
• Flow gets kept – fixing database and restarting counterparty will finalise correctly

34. Database error during recording on counterpartyHandling errors
during finality
Notarised
Recorded locally
Distributed to participants
• Failure to record on the counterparty does not affect initiating flow
• Flow gets kept – fixing database and restarting counterparty will finalise correctly

35. User exception from VaultObserver on counterpartyHandling errors
during finality
Notarised
Recorded locally
Distributed to participants
• Failure to record on the counterparty does not affect initiating flow
• The exception might escape and kill the flow

36. User exception from VaultObserver on counterpartyHandling errors
during finality
Notarised
Recorded locally
Distributed to participants
• Failure to record on the counterparty does not affect initiating flow
• The exception might escape and kill the flow
The ledger will be inconsistent between the two nodes!

37. User exception from VaultObserver on counterpartyHandling errors
during finality
Notarised
Recorded locally
Distributed to participants
• Failure to record on the counterparty does not affect initiating flow
• The exception might escape and kill the flow
The ledger will be inconsistent between the two nodes!
Don’t do this!

38. Recap
• Corda handles database exceptions during finality
• Either retry
• Or keeping flows in the flow hospital
Flows in the hospital need attention, the ledger might be inconsistent until they are
resolved

39. Recap
• Corda handles database exceptions during finality
• Either retry
• Or keeping flows in the flow hospital
Flows in the hospital need attention, the ledger might be inconsistent until they are
resolved
• Corda handles connection errors during finality
• Retry and persistent queues ensure eventual consistency

40. Recap
• Throwing exceptions from raw vault observers can be dangerous!
• Exceptions from raw vault observer can threaten integrity of the ledger (this is a bug that
is scheduled to be fixed)
• Exceptions from any vault observer can change the behaviour of the node (also a bug)

41. Recap
• Throwing exceptions from raw vault observers can be dangerous!
• Exceptions from raw vault observer can threaten integrity of the ledger (this is a bug that
is scheduled to be fixed)
• Exceptions from any vault observer can change the behaviour of the node (also a bug)
• Workaround: don’t let exceptions escape your code inside observers
Be careful what code to run in a vault observer!
When using vault observers make sure to handle exceptions!

42. Recap
• Or even better, use an alternative, e.g. VaultService.trackBy()
val updates = database.transaction {
val (snapshot, updates) = vaultService.trackBy<LinearState>()
updates
}

43. Conclusions
• With the latest changes, Corda ensures consistent finality even in case of:
• database exceptions
• connection exceptions
• Only user exceptions from raw VaultObservers are problematic
• Only use raw updates if you absolutely must – often VaultService.trackBy is the
better alternative (e.g. if you just need to track/notify external services)
• If you must use raw observables, be sure to catch and handle all exceptions

44. London
2 London Wall Place,
London, EC2Y 5AU
Hong Kong
Bonham Strand, 7F Office 18-
121
Hong Kong
www.r3.com
Thank you
Brazil
Av. Angélica, 2529 - Bela Vista
6th Floor
São Paulo - SP, 01153-000, Brazil
New York
11 West 42nd Street, 8th Floor
New York, NY 10036
Singapore
18 Robinson Road, Level
#14-02
Singapore, 048547