Your SlideShare is downloading. ×
0
Big Data & Messaging

with Artem Bilan
by Pivotal
© 2013 Pivotal Inc. All rights reserved. Do not distribute without permi...
Artem Bilan
Spring Integration Team

abilan@gopivotal.com

Spring Framework
Spring AMQP
Spring XD
Reactor

https://spring....
Messaging and why do I care?

© 2013 Pivotal Inc. All rights reserved. Do not distribute without permission.
Messaging is very simple

Headers

Payload

public static void main(String[] args) {
...
}
© 2013 Pivotal Inc. All rights ...
Messaging in Spring IO

© 2013 Pivotal Inc. All rights reserved. Do not distribute without permission.
Spring Integration
Written
Uses

Meet IoC!

© 2013 Pivotal Inc. All rights reserved. Do not distribute without permission.
The heart of Spring Integration
–
–
–

Endpoints (Filters) connected through
Channels (Pipes) exchanging
Message

© 2013 P...
Spring Integration Message Flow

© 2013 Pivotal Inc. All rights reserved. Do not distribute without permission.
How does it work?

© 2013 Pivotal Inc. All rights reserved. Do not distribute without permission.
What is a Big Data application?
 Not just writing a few MR or Hive/Pig jobs.
 The full lifecycle involves
• Ingestion
• ...
Spring XD
• Unified Platform
• Developer Productivity
• Modular Extensibility
• Distributed Architecture
• Portable Runtim...
Spring XD
Files

Sensors

Mobile

Social

Spring XD Shell
Spring XD Runtime

Taps

Jobs

Compute

Workflow

Redis

Ingest
...
Stream Processing Model

How can
http |we make this|easier?
filter
file

Non-linear stream definitions also supported
© 20...
Runtimes
XD Admin

http | filter | file

CLUSTERED NODE

CLUSTERED NODE

CLUSTERED NODE

SINGLE
NODE

HTTP
Module

Filter
...
Streams
MessageStore

HTTP
Tail
File
Mail
Twitter
Gemfire
Syslog
TCP
JMS
RabbitMQ
MQTT

© 2013 Pivotal Inc. All rights res...
Taps
 “Listen” to data from another stream
• Other stream is unaffected by the tap and unaware of its presence
• EAI ‘Wir...
Analytics
• Simple Counter
• Field Value Counter
– Count occurrences of named
fields

• Aggregate Counter
– Pre-aggregate ...
Jobs
 Jobs are a directed graph of steps
 Steps
• Copy or Process data
• Files, Databases, MR, Pig, Hive, Cascading

 S...
SPRING XD

Demo

© 2013 Pivotal Inc. All rights reserved. Do not distribute without permission.
Learn More. Stay Connected.
•
•
•
•
•
•
•
•

Spring IO: https://spring.io/platform
GitHub: https://github.com/spring-proje...
ありがとう
© 2013 Pivotal Inc. All rights reserved. Do not distribute without permission.
Upcoming SlideShare
Loading in...5
×

Big data and messaging with Spring

874

Published on

Published in: Technology
0 Comments
5 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
874
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
36
Comments
0
Likes
5
Embeds 0
No embeds

No notes for slide
  • Unified platform across big data domainsStream processing: ingestion & analyticsBatch processing workflow orchestration & exportProductivityHigh level DSL for managing streams and jobsProven foundation Built on existing assets: Spring Batch, Integration, DataExtensibleDI, Test friendly…DistributedA | B | C - Pluggable transports: Rabbit, Redis, …Portable Runtime Standalone – Simplicity and testabilityYARN –Fault Tolerance and ScalabilityIn-memory data grids – co-location of reference data.PAASHadoop Distribution Agnostic
  • Now have a unified platform that handle the whole thing.TODO – get rid of mobile?....XD runtime address both stream and batch processing in a unfired manner.XD runtime address both stream and batch processing in a unfired manner.
  • Distributed mode, process boundaries
  • Now have a unified platform that handle the whole thing.TODO – get rid of mobile?....XD runtime address both stream and batch processing in a unfired manner.XD runtime address both stream and batch processing in a unfired manner.
  • Now have a unified platform that handle the whole thing.TODO – get rid of mobile?....XD runtime address both stream and batch processing in a unfired manner.XD runtime address both stream and batch processing in a unfired manner.
  • Now have a unified platform that handle the whole thing.TODO – get rid of mobile?....XD runtime address both stream and batch processing in a unfired manner.XD runtime address both stream and batch processing in a unfired manner.
  • http | filehttp | hdfstwittersearch | hdfsaggregatecounter on hashtags
  • Transcript of "Big data and messaging with Spring"

    1. 1. Big Data & Messaging with Artem Bilan by Pivotal © 2013 Pivotal Inc. All rights reserved. Do not distribute without permission.
    2. 2. Artem Bilan Spring Integration Team abilan@gopivotal.com Spring Framework Spring AMQP Spring XD Reactor https://spring.io/team/artembilan http://stackoverflow.com/users/2756547 http://www.linkedin.com/in/cleric https://github.com/artembilan © 2013 Pivotal Inc. All rights reserved. Do not distribute without permission.
    3. 3. Messaging and why do I care? © 2013 Pivotal Inc. All rights reserved. Do not distribute without permission.
    4. 4. Messaging is very simple Headers Payload public static void main(String[] args) { ... } © 2013 Pivotal Inc. All rights reserved. Do not distribute without permission.
    5. 5. Messaging in Spring IO © 2013 Pivotal Inc. All rights reserved. Do not distribute without permission.
    6. 6. Spring Integration Written Uses Meet IoC! © 2013 Pivotal Inc. All rights reserved. Do not distribute without permission.
    7. 7. The heart of Spring Integration – – – Endpoints (Filters) connected through Channels (Pipes) exchanging Message © 2013 Pivotal Inc. All rights reserved. Do not distribute without permission.
    8. 8. Spring Integration Message Flow © 2013 Pivotal Inc. All rights reserved. Do not distribute without permission.
    9. 9. How does it work? © 2013 Pivotal Inc. All rights reserved. Do not distribute without permission.
    10. 10. What is a Big Data application?  Not just writing a few MR or Hive/Pig jobs.  The full lifecycle involves • Ingestion • Stream Processing • Workflow Orchestration • Enterprise Integration • Export • Horizontal scalable deployment  How do you write one of these? • Status quo is to combine different projects – not ideal © 2013 Pivotal Inc. All rights reserved. Do not distribute without permission.
    11. 11. Spring XD • Unified Platform • Developer Productivity • Modular Extensibility • Distributed Architecture • Portable Runtime • Hadoop Distribution Agnostic • Proven Foundation • XD = ‘eXtreme Data’ © 2013 Pivotal Inc. All rights reserved. Do not distribute without permission.
    12. 12. Spring XD Files Sensors Mobile Social Spring XD Shell Spring XD Runtime Taps Jobs Compute Workflow Redis Ingest Streams Export Export RDBMS Gemfire HDFS NoSQL R, SAS Predictive modeling © 2013 Pivotal Inc. All rights reserved. Do not distribute without permission.
    13. 13. Stream Processing Model How can http |we make this|easier? filter file Non-linear stream definitions also supported © 2013 Pivotal Inc. All rights reserved. Do not distribute without permission.
    14. 14. Runtimes XD Admin http | filter | file CLUSTERED NODE CLUSTERED NODE CLUSTERED NODE SINGLE NODE HTTP Module Filter Module File Module All Modules Rabbit, Redis, (Pluggable) © 2013 Pivotal Inc. All rights reserved. Do not distribute without permission. In Memory Transport http | filter | file
    15. 15. Streams MessageStore HTTP Tail File Mail Twitter Gemfire Syslog TCP JMS RabbitMQ MQTT © 2013 Pivotal Inc. All rights reserved. Do not distribute without permission. Filter Transformer Splitter Aggregator Groovy Script Counters HTTP JSON Java Code File HDFS JDBC TCP Mail RabbitMQ Gemfire Splunk MQTT Dynamic Router
    16. 16. Taps  “Listen” to data from another stream • Other stream is unaffected by the tap and unaware of its presence • EAI ‘Wiretap’ © 2013 Pivotal Inc. All rights reserved. Do not distribute without permission.
    17. 17. Analytics • Simple Counter • Field Value Counter – Count occurrences of named fields • Aggregate Counter – Pre-aggregate counts in time buckets • Gauge – last value • Rich Gauge – Last value, running average, min/max © 2013 Pivotal Inc. All rights reserved. Do not distribute without permission. • Abstract API • Implementations for – Memory – Redis • Looking at – Pattern – JPMML
    18. 18. Jobs  Jobs are a directed graph of steps  Steps • Copy or Process data • Files, Databases, MR, Pig, Hive, Cascading  Step executions are persisted • Checkpointing with restart • Rich error handling capabilities  Single node or distributed with data partitioning  Jobs can be Triggered from streams  Executing jobs generate a stream of event data © 2013 Pivotal Inc. All rights reserved. Do not distribute without permission.
    19. 19. SPRING XD Demo © 2013 Pivotal Inc. All rights reserved. Do not distribute without permission.
    20. 20. Learn More. Stay Connected. • • • • • • • • Spring IO: https://spring.io/platform GitHub: https://github.com/spring-projects Spring Integration: http://projects.spring.io/spring-integration Spring XD: http://projects.spring.io/spring-xd Reactor: https://github.com/reactor EIP: http://www.eaipatterns.com Spring Batch: http://projects.spring.io/spring-batch Spring for Hadoop: http://projects.spring.io/spring-hadoop © 2013 Pivotal Inc. All rights reserved. Do not distribute without permission.
    21. 21. ありがとう © 2013 Pivotal Inc. All rights reserved. Do not distribute without permission.
    1. A particular slide catching your eye?

      Clipping is a handy way to collect important slides you want to go back to later.

    ×