Who Is This Guy?

I’m Brian!

1

Confidential Information of Talksum, Inc.
Who Is This Guy?

• Principal Architect at Talksum
• Open Source
Contributor
ZeroMQ
Rsyslog

2

Confidential Information o...
Talksum Data Stream Router
Next Age of Data Management

November 2013
3

Confidential Information of Talksum, Inc.
The Talksum Data Stream Router

Refined Data Stream

Refined Data Stream

Application
Logs

Unix Logs – RFC3164 UDP/TCP
Ne...
In Other Words…

We turn alphabet soup into
coherent streams of
information so that you can
get to work solving problems.
...
Cross Domain Routing with ZeroMQ

November 2013
6

Confidential Information of Talksum, Inc.
Domains

“A territory over which rule or
control is exercised”
“A sphere of activity, concern, or
function”
- freedictiona...
Some Domains!

•Energy

•Transportation
•Biomedical
•Information Systems
8

Confidential Information of Talksum, Inc.
Some More Domains!

•Application Threads

•Application Processes
•Systems on a network
•Networks themselves
9

Confidentia...
What is ZeroMQ
ØMQ zeromq:
• A network socket library that acts as a concurreny framework

• Carries messages across multi...
What is ZeroMQ

WHAT IF I TOLD YOU
Here there be dragons

WE CAN HAVE LOCK-FREE IN MEMORY
MESSAGE PASSING BETWEEN THREADS?...
What Is ZeroMQ To Us

ZeroMQ is a tool which facilitates the
construction of cross domain
communication patterns

12

Conf...
Patterns!

When we can apply a pattern from
one domain to a different domain,
we can do more with less work.

13

Confiden...
Network Monitoring & Optimization

Customer: Large European ISP/Email
Communications Provider
Use Case: Ingest Netflow dat...
Financial

Customer: Major Financial Stock Exchange
Use Case: Ingest unstructured financial
market data, parse and filter ...
Request Reply

16

Confidential Information of Talksum, Inc.
Round Robin Load Balancing

17

Confidential Information of Talksum, Inc.
Unicast

18

Confidential Information of Talksum, Inc.
Multicast

19

Confidential Information of Talksum, Inc.
Fair Queuing

20

Confidential Information of Talksum, Inc.
Explicit Routing

21

Confidential Information of Talksum, Inc.
Inter Domain Messaging

• Within a thread
• Between threads
• Between processes on the same system
• Between processes on ...
Why Talksum Contributes to Zeromq

• We are all facing a lot of problems.
• We believe in the value of empirical evidence....
Resources
• http://www.zeromq.org/ - Main ZeroMQ Web Site
• http://zguide.zeromq.org/ - ZeroMQ User Guide

• http://github...
Upcoming SlideShare
Loading in …5
×

Brian Knox, Principal Architect, Talksum - 21st BDL meetup

3,010 views

Published on

"Cross-domain Data Management"

Brian Knox is the principal architect at Talksum where he focuses on real-time data routing and analysis. Brian has over 20 years of experience as a systems/data architect with expertise in high-speed messaging, service delivery, operations management, systems integration, and distributed processing. Most recently, Brian has served in the roles as principal architect and database engineer for large-scale infrastructures and service delivery companies, including AOL. In addition, he has also focused on large-scale filtering technologies and data management for other successful start-up companies such as SurfControl and others.

Published in: Technology
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
3,010
On SlideShare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
4
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide
  • Other than “Hi,I am Dale” and throw up on the stage…
  • Other than “Hi,I am Dale” and throw up on the stage…
  • Brian Knox, Principal Architect, Talksum - 21st BDL meetup

    1. 1. Who Is This Guy? I’m Brian! 1 Confidential Information of Talksum, Inc.
    2. 2. Who Is This Guy? • Principal Architect at Talksum • Open Source Contributor ZeroMQ Rsyslog 2 Confidential Information of Talksum, Inc.
    3. 3. Talksum Data Stream Router Next Age of Data Management November 2013 3 Confidential Information of Talksum, Inc.
    4. 4. The Talksum Data Stream Router Refined Data Stream Refined Data Stream Application Logs Unix Logs – RFC3164 UDP/TCP Netflow – UDP – NG v.5, 8, 9, 10 Application Data Patient Records (HL7) XML/ASN.1 Transportation (BSM) SAE J2735 3rd Party Data B2B/M2M Social and Public Data 4 Customer B: Aggregated Data Refined Data Stream Customer C: Dynamic Stream Apache Common Logging – Files SNMP - UDP System Logs Sensor and Industrial Data Customer A: Summarized Data I2C, CAN, SNMP, Serial XML, JSON, File, HTTP REST Twitter, RSS, CAP (Weather Alerts) Confidential Information of Talksum, Inc. Talksum Data Stream Router (TDSR) • Data Normalization • Parsers • Filters • Metrics and Counts • Inline ETL/PTL • Asynchronous Outputs • Protocol Verification Indexed, Mapped, Reduced Ordered, Sorted Data Streams Bulk Data Streams (Lightly Ordered and Filtered) • Object Data Stores • Indexed Data Caches • NoSQL Data Warehouses • SQL Warehouse • Bulk Data Stores • File Storage
    5. 5. In Other Words… We turn alphabet soup into coherent streams of information so that you can get to work solving problems. 5 Confidential Information of Talksum, Inc.
    6. 6. Cross Domain Routing with ZeroMQ November 2013 6 Confidential Information of Talksum, Inc.
    7. 7. Domains “A territory over which rule or control is exercised” “A sphere of activity, concern, or function” - freedictionary.com 7 Confidential Information of Talksum, Inc.
    8. 8. Some Domains! •Energy •Transportation •Biomedical •Information Systems 8 Confidential Information of Talksum, Inc.
    9. 9. Some More Domains! •Application Threads •Application Processes •Systems on a network •Networks themselves 9 Confidential Information of Talksum, Inc.
    10. 10. What is ZeroMQ ØMQ zeromq: • A network socket library that acts as a concurreny framework • Carries messages across multiple transports • Enables networking patterns • Provides Asyncronous I/O • Large and active open source community • Bindings for 40+ programming languages • Works on most operating systems 10 Confidential Information of Talksum, Inc.
    11. 11. What is ZeroMQ WHAT IF I TOLD YOU Here there be dragons WE CAN HAVE LOCK-FREE IN MEMORY MESSAGE PASSING BETWEEN THREADS? 11 Confidential Information of Talksum, Inc.
    12. 12. What Is ZeroMQ To Us ZeroMQ is a tool which facilitates the construction of cross domain communication patterns 12 Confidential Information of Talksum, Inc.
    13. 13. Patterns! When we can apply a pattern from one domain to a different domain, we can do more with less work. 13 Confidential Information of Talksum, Inc.
    14. 14. Network Monitoring & Optimization Customer: Large European ISP/Email Communications Provider Use Case: Ingest Netflow data, parse and aggregate in real time, monitors and alerts, optimize network topology Status: Deploying beta appliance System Logs 14 Unix Logs – RFC3164 UDP/TCP Netflow – UDP – NG v.5, 8, 9, 10 Confidential Information of Talksum, Inc. Refined Data Stream Talksum Data Stream Router (TDSR) • Data Normalization • Parsers • Filters • Metrics and Counts • Inline ETL/PTL • Asynchronous Outputs • Protocol Verification Existing BI Tools Refined Data Stream NOC Alerting Indexed, Mapped, Reduced Ordered, Sorted Data Streams Bulk Data Streams (Lightly Ordered and Filtered) • Object Data Stores • Indexed Data Caches • NoSQL Data Warehouses • SQL Warehouse • Bulk Data Stores • File Storage
    15. 15. Financial Customer: Major Financial Stock Exchange Use Case: Ingest unstructured financial market data, parse and filter for quality, aggregate, integrate with existing data warehouse Status: Acquiring data sample for POC 3rd Party Data Trading Desks Social and Public Data 15 XML, JSON, File, HTTP REST Twitter, RSS, CAP (Weather Alerts) Confidential Information of Talksum, Inc. Refined Data Stream Talksum Data Stream Router (TDSR) • Data Normalization • Parsers • Filters • Metrics and Counts • Inline ETL/PTL • Asynchronous Outputs • Protocol Verification Alerting & Notification Refined Data Stream Market Dashboard Indexed, Mapped, Reduced Ordered, Sorted Data Streams Bulk Data Streams (Lightly Ordered and Filtered) • Object Data Stores • Indexed Data Caches • NoSQL Data Warehouses • SQL Warehouse • Bulk Data Stores • File Storage
    16. 16. Request Reply 16 Confidential Information of Talksum, Inc.
    17. 17. Round Robin Load Balancing 17 Confidential Information of Talksum, Inc.
    18. 18. Unicast 18 Confidential Information of Talksum, Inc.
    19. 19. Multicast 19 Confidential Information of Talksum, Inc.
    20. 20. Fair Queuing 20 Confidential Information of Talksum, Inc.
    21. 21. Explicit Routing 21 Confidential Information of Talksum, Inc.
    22. 22. Inter Domain Messaging • Within a thread • Between threads • Between processes on the same system • Between processes on different systems 22 Confidential Information of Talksum, Inc.
    23. 23. Why Talksum Contributes to Zeromq • We are all facing a lot of problems. • We believe in the value of empirical evidence. • Solutions emerge from all of us working together across our various domain boundaries. • Participate! 23 Confidential Information of Talksum, Inc.
    24. 24. Resources • http://www.zeromq.org/ - Main ZeroMQ Web Site • http://zguide.zeromq.org/ - ZeroMQ User Guide • http://github.com/zeromq - ZeroMQ Git Repository • http://api.zeromq.org/ - ZeroMQ Reference Manual • http://zeromq.org/whitepapers:architecture – Architecture Whitepaper • http://czmq.zeromq.org/ - ZeroMQ C Binding • http://zeromq.github.io/pyzmq/ - ZeroMQ Python Binding 24 Confidential Information of Talksum, Inc.

    ×