Blockchain has become a buzzword: people are excited about distributed ledgers and cryptocurrencies, but these technologies are shrouded in myths, and misunderstanding. This talk will shed some light into how this awesome technology is actually used in practice by using Apache Spark to analyze blockchain transactions. We’ll start with a brief introduction to blockchain transactions and how we can ETL transaction graph data obtained from the public binary format. Then we will look at how to model graph data in Spark, briefly comparing GraphFrames and GraphX. The majority of the presentation will be a live demo, running on Spark in the cloud, showing how we can run various queries on the transaction graph data, solve graph algorithms such as PageRank for identifying significant BTC addresses, observe network evolution, and more. All of the work described in this talk is published as open source code and all of the data are available in public and available for community experimentation as well as all the containers. You will leave this talk with a better understanding of blockchain technology and graph processing in Spark and you will have the concrete tools to reproduce my research or start answering your own questions.

1. 1

2. Outline
Blockchain 101
Blockchain to graph
Graph data in Spark
DEMO
2

3. What is Blockchain
Distributed ledger
Linked list of blocks
Trust stems from Merkle trees and proof of work (aka mining)
Cryptography
3

4. header
What is Block
Set of transactions approved at once
Metadata
Hard limit 1 MB ( )*
4

5. Alice Bob
What is Transaction
(INs, OUTs)
sum of INs ≥ sum of OUTs
confirming ~ including it to a new block and finding the "nonce"
5

6. 6

7. What is Transaction(more general case)
7

8. Transaction to Graph
M:N transactions produces a lot of edges
Apache Parquet
blockchain binary data -> parquet converter
8

9. Transaction to Graph
# of Satoshis sent on the edges of type [T→A] and [A→T]
timestamp on the block nodes
more suitable for querying the graph
9

10. Graphs and Spark
GraphX
GraphFrames
built-ins (label propagation, pagerank, triangles, bfs, etc.)
motif ~ cypher
Pregel
10

11. Demo time
Talk is Cheap
11

12. Page Rank
12

13. Takeways
Blockchain is out there
GraphFrames vs GraphX
Reproducible experiments with notebooks and containers
13

14. More projects, tutorials and examples can be found at
radanalytics.io
How to get started
14

15. This presentation
http://bit.ly/sais18
@JirkaKremser
Thank You!
Jiri-Kremser/bitcoin-insights
15