This talk is about a Big Data use case requiring 100K transactions per second. It briefly cover the architecture of MarkLogic, an enterprise-grade NoSQL Document Database and why it's ideal for this case. The talk concludes with a detailed discussion of the techniques used, along with related performance analyses, to achieve the requirements.
2. A bit about me
ā¢ Iāve written software used by millions of people.
Apps, libraries, compilers, device drivers, operating systems
ā¢ This is my third Gluecon and my first talk
ā¢ Iām the Community Director at MarkLogic, last 2 years.
ā¢ I survived having 3 kids in less than 2 years.
7. ļ§ Itās about money.
ļ§ Top 5 bank needed to manage trades
ļ§ Trades look more like documents than tables
ļ§ Schemas for trades change all the time
ļ§ Transactions
ļ§ Scale and velocity (āBig Dataā)
8. Trades and Positions
ļ§ 1 million trades per day
ļ§ Followed by 1 million position reports at end of day
ļ§ Roll up trades of current date for each ābook, instrumentā pair
ļ§ Group-by, with key = ādate, book, instrumentā
1M positions
1M trades
Day Day Day Day Day ...
1 2 3 4 5
16. Key (URI) Value (Document)
/trade/153748994 <trade>
<id>8</id>
<time>2012-02-20T14:00:00</time>
<instrument>BYME AAA</instrument>
<price cur=āusdā>600.27</price>
</trade>
/user/eedeebee {
ānameā : āEric Blochā,
āageā : 47,
āhairā : āgrayā,
ākidsā : [ āGraceā, āRyanā, āOwenā ]
}
/book5293 It was the best of times, it was the worst of times, it
was the age of wisdom, ...
/2012-02-20T14:47:53/01445 .mp3
.avi
[your favorite binary format]