Integration and Batch Processing on Cloud Foundry
Upcoming SlideShare
Loading in...5
×
 

Like this? Share it with your network

Share

Integration and Batch Processing on Cloud Foundry

on

  • 1,931 views

This talk explores the new possibilities for scale by using Spring Integration, Spring Batch and RabbitMQ on Cloud Foundry, the open source PaaS from VMWare.

This talk explores the new possibilities for scale by using Spring Integration, Spring Batch and RabbitMQ on Cloud Foundry, the open source PaaS from VMWare.

Statistics

Views

Total Views
1,931
Views on SlideShare
1,931
Embed Views
0

Actions

Likes
2
Downloads
61
Comments
0

0 Embeds 0

No embeds

Accessibility

Categories

Upload Details

Uploaded via as Apple Keynote

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • the toolchain is STS, java compiler (no startup scripts required!) \nintroduce Spring Roo\nintroduce the Appcontexts\nintroduce the various Di configuration types \n\n
  • \n
  • the toolchain is STS, java compiler (no startup scripts required!) \nintroduce Spring Roo\nintroduce the Appcontexts\nintroduce the various Di configuration types \n\n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • A better approach is decompose the war into a set of services.\nYou can do that either by nouns or by verbs.\n\n
  • A better approach is decompose the war into a set of services.\nYou can do that either by nouns or by verbs.\n\n
  • Example of what can happen if you run processes more than correct number of times.\n
  • A better approach is decompose the war into a set of services.\nYou can do that either by nouns or by verbs.\n\n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • A better approach is decompose the war into a set of services.\nYou can do that either by nouns or by verbs.\n\n
  • A better approach is decompose the war into a set of services.\nYou can do that either by nouns or by verbs.\n\n
  • \n
  • \n
  • \n
  • \n
  • \n
  • talk about how convenient the messaging support is, then roll back and start looking at how u might do the same thing manually. Explain that many of the pices are already there, and then seguqe into a discussion about the core Spring APIs. Lets imagine we’re going to build ourselves a file system poller to notify us of when something’s happened on the file system\n\n
  • \n
  • \n
  • \n
  • talk about how convenient the messaging support is, then roll back and start looking at how u might do the same thing manually. Explain that many of the pices are already there, and then seguqe into a discussion about the core Spring APIs. Lets imagine we’re going to build ourselves a file system poller to notify us of when something’s happened on the file system\n\n
  • \n
  • \n

Integration and Batch Processing on Cloud Foundry Presentation Transcript

  • 1. Enterprise Application Integration and Batch Processing in the Cloud Josh Long, Spring Developer Advocate josh.long@springsource.com @starbuxman© 2012 SpringOne 2GX. All rights reserved. Do not distribute without permission.
  • 2. Josh LongSpring Developer Advocatejosh.long@springsource.com @starbuxman 2
  • 3. Worker processes on Cloud Foundry
  • 4. Lots of Potential http://blog.springsource.org/2012/05/09/using-cloud-foundry-workers-with-spring/ lots of use cases --- applications: • background jobs target/appassembler/: • headless applications name: integration-service • batch jobs framework: • embedded web servers name: standalone info: mem: 64M description: Standalone Application exec: runtime: java command: bin/main url: mem: 512M instances: 1 services: stock_rabbitmq: type: rabbitmq 4
  • 5. Easy to Package Multi-component Java applications <build> <plugins> <plugin> <groupId>org.codehaus.mojo</groupId> <artifactId>appassembler-maven-plugin</artifactId> <executions> <execution> <phase>package</phase> <goals> <goal>assemble</goal> </goals> <configuration> <assembledirectory>target</assembledirectory> <programs> <program> <mainClass>org.cloudfoundry.workers.stocks.integration.service.Main</mainClass> </program> </programs> </configuration> </execution> </executions> </plugin> </plugins> </build> 5
  • 6. Cloud Scale Messaging with RabbitMQ
  • 7. First, an Introduction to AMQP & RabbitMQ
  • 8. Messaging Use Cases Decoupling producer consumer shopping cart sending request CC merchant NOT CONFIDENTIAL -- TELL EVERYONE 8
  • 9. Messaging Use Cases Bidirectional Decoupling producer consumer eg: remote procedure call NOT CONFIDENTIAL -- TELL EVERYONE 9
  • 10. Messaging Use Cases Bidirectional Decoupling producer producer consumer eg: place order and wait for confirmation NOT CONFIDENTIAL -- TELL EVERYONE 10
  • 11. Messaging Use Cases Bidirectional Decoupling producer consumer producer consumer eg: place order and wait for confirmation NOT CONFIDENTIAL -- TELL EVERYONE 11
  • 12. Messaging Use Cases work distribution and decoupling consumer producer consumer distribution can be duplicate or round-robin: - duplication for information (logging, auditing, etc) - round robin for scaling and load balancing NOT CONFIDENTIAL -- TELL EVERYONE 12
  • 13. Messaging Use Cases Other Things a Message Broker Can Do • Store-and-forward of messages • Absorbing spikes of activity (again, decoupling) • Routing to different consumers based on message properties NOT CONFIDENTIAL -- TELL EVERYONE 13
  • 14. AMQP and JMS How does AMQP compare to JMS? When would I use one or the other? NOT CONFIDENTIAL -- TELL EVERYONE 14
  • 15. AMQP and JMS AMQP JMS Protocol API No language Java Industry committee, mainly finance JEE Interoperability from client libraries Interoperability from proprietary features Resources can be managed by clients Resources often centrally managed Not ubiquitous in enterprise Ubiquitous through JEE Flexible usage patterns, emphasise routing Two basic producer patterns (Queue and Topic) Designed for large numbers of queues and exchanges Hard to manage large numbers of Destinations NOT CONFIDENTIAL -- TELL EVERYONE 15
  • 16. AMQP and JMS JMS • standard in the Java space • lots of open source options • provided with Java EE application servers • Spring JMS Java producers Message broker Java consumers C P queue C P queue C 16
  • 17. JMS  Sending JMS Messages • Inject an instance of Springs JmsTemplate. • Provide the JMS ConnectionFactory in the JmsTemplate bean definition.@Componentpublic class MessageSender { @Autowired private volatile JmsTemplate jmsTemplate; public void send(String message) { this.jmsTemplate.convertAndSend("example.queue", message); }} @Bean public ConnnectionFactory cf (){ return new CachingConnectionFactory(...); } @Bean public JmsTemplate template(){ return new JmsTemplate(this.cf()) ; } 17
  • 18. AMQP and JMS AMQP • real standard • a product of the the companies with the most mission critical requirements • Spring AMQP AMQP producers exchanges Message broker AMQP consumers C P X queue C P X queue C 1
  • 19. AMQP and JMS JMS Topic Explosion: • Messages with fine-grained routing, e.g. • Market data: region, country, exchange, ticker, … • Online gambling: sport, match, event-type, … • Message selectors are inefficient and fiddly to work with in the API • Exponential explosion of possible combinations of interesting subscriptions • Can implement as hierarchy of Topics • 1000s or 10000s of Destinations • Management nightmare • Sweet spot for AMQP NOT CONFIDENTIAL -- TELL EVERYONE 19
  • 20. AMQP  Sending AMQP Messages • Use AmqpTemplate instead of JmsTemplate (accepts exchange and routingKey). • Nothing changes on the listener side (just a POJO).@Component public class MessageSender { @Autowired private volatile AmqpTemplate amqpTemplate; public void send(String message) { this.amqpTemplate.convertAndSend( "myExchange", "some.routing.key", message); }} 20
  • 21. Spring AMQP  Java and .NET  AMQP core abstraction plus RabbitMQ implementation (built on rabbit client libraries)  Higher level patterns for clients: admin, producers and consumers  RabbitAdmin – declare exchanges, queues, bindings  RabbitTemplate – convenience methods for send and receive  MessageListenerContainer – POJO message handler, asynchronous  Spring Integration support http://www.springsource.org/spring-amqp NOT CONFIDENTIAL -- TELL EVERYONE 21
  • 22. Spring AMQP: RabbitTemplatepublic class MyComponent { private RabbitTemplate rabbitTemplate; public MyComponent(ConnectionFactory connectionFactory) { this.rabbitTemplate = new RabbitTemplate(connectionFactory); } public void read() throws Exception { ... String value = rabbitTemplate.receiveAndConvert("myQueueName"); ... }} Convenience methods NOT CONFIDENTIAL -- TELL EVERYONE 22
  • 23. Spring AMQP: SimpleMessageListenerContainer  Asynchronous message receiver  POJO handlers  Handles re-connection and listener failure (rollback, redelivery)  Message conversion and error handling strategies <listener-container connection-factory="connectionFactory"> <listener ref="handler" method="handle" queue-names="my.queue"> </listener-container> NOT CONFIDENTIAL -- TELL EVERYONE 23
  • 24. Messaging Use Cases Patterns: spatial decoupling (aka communication) temporal decoupling (aka buffering) logical decoupling (aka routing) Promote: flexibility, resilience, performance, and scale Conclusion: There is an enormous range of applications and problems to which messaging is an effective solution. NOT CONFIDENTIAL -- TELL EVERYONE 24
  • 25. Not confidential. Tell everyone. 25
  • 26. Not confidential. Tell everyone. 26
  • 27. Batch Processing in the Cloud Not confidential. Tell everyone. 27
  • 28. First, an Introduction to Spring Batch Not confidential. Tell everyone. 28
  • 29. Why we’re here... Not confidential. Tell everyone. 29
  • 30. ...So, What’s the Problem? Not confidential. Tell everyone. 30
  • 31. Job and Step Job * Step * JobInstance Step Scope * JobExecution * StepExecution Not confidential. Tell everyone. 31
  • 32. Getting Started Application Developer implements ItemProcessor (optional) input output (optional) configures ItemReader ItemWriter Job * StepExecutor concerns Step RepeatOperations ExceptionHandler Not confidential. Tell everyone. 32
  • 33. Spring Batch Supports Batch API... • Jobs have Steps • Steps have Readers, and optional Processors and Writers • Readers read data • Processors process data coming into them, optionally transforming it. Optional. • Writers write data out Not confidential. Tell everyone. 33
  • 34. ItemReader public interface ItemReader<T> { T read() throws Exception, UnexpectedInputException, ParseException, NonTransientResourceException; }Returns null at end of dataset delegate Exception handling to framework Not confidential. Tell everyone. 34
  • 35. ItemProcessor (Optional) public interface ItemProcessor<I, O> { O process(I item) throws Exception; }Delegate Exception handling to framework Not confidential. Tell everyone. 35
  • 36. ItemWriter public interface ItemWriter<T> { void write(List<? extends T> items) throws Exception; }expects a “chunk” delegate Exception handling to framework Not confidential. Tell everyone. 36
  • 37. All Together... <job id="skipJob" incrementer="incrementer" xmlns="http://www.springframework.org/schema/batch"> <step id="step1"> <tasklet> <chunk reader="fileItemReader" processor="tradeProcessor" writer="tradeWriter" commit-interval="3" skip-limit="10"> </chunk> </tasklet> <next on="*" to="step2" /> <next on="COMPLETED WITH SKIPS" to="errorPrint1" /> <fail on="FAILED" exit-code="FAILED" /> </step> ... </job> Not confidential. Tell everyone. 37
  • 38. Partitioning Overview Job Step Step Step Not confidential. Tell everyone. 38
  • 39. Partitioning Overview Job Slave Step Slave Slave Master Slave Slave Step Slave Not confidential. Tell everyone. 39
  • 40. RequestsWe Have a Cloud, Let’s Scale! Spring Batch supports Remote Chunking over AMQP • http://labs.bsb.com/2011/11/remote-partitioning-with-spring-batch/ • http://bit.ly/wrb8Ch master step execution results RabbitMQ Step Execution Requests slave 1 slave 2 slave 3 Not confidential. Tell everyone. 40
  • 41. Simple Strategy SPI for Distribution and Aggregation Not confidential. Tell everyone. 41
  • 42. Application Integration (in the Cloud) Not confidential. Tell everyone. 42
  • 43. Integrations More Common Than You Think channel System A System A System A System B Not confidential. Tell everyone. 43 6
  • 44. Nobody Builds Systems in a Vacuum do NOT reinvent the Wheel! Not confidential. Tell everyone. 44
  • 45. Dont over extend your architecture! Not confidential. Tell everyone. 45
  • 46. Rooted in Classic Patterns • By Gregor Hohpe & Bobby Woolf • Published 2003 • Collection of well-known patterns • http://www.eaipatterns.com/eaipatterns.html • Icon library provided Not confidential. Tell everyone. 46
  • 47. What is Spring Integration?• Light-weight messaging framework• Provides an adapter-based platform• Pipes and Filters at the core of Spring Integration’s architecture – Endpoint (Filter) – Channel (Pipe) – Message Not confidential. Tell everyone. 47
  • 48. DEMO• Spring Integration Primer @starbuxman slideshare.net/joshlong
  • 49. Spring Integration @Gateway Gateways... • hide one system from the messaging semantics of another • Support all the message exchange patterns • Request • Request asynchronous (fire-n-forget) • Request/Reply • Request/Reply asynchronous (fire-n-forget) public class CustomerService { void addCustomer( Customer customer ) throws CrmException; } Not confidential. Tell everyone. 49
  • 50. Spring Integration @Gateway MessageChannel crmChannel = ...; MessagingOperations operations = .... ; operations.sendAndReceive( crmChannel , MessageBuilder.withPayload(new Customer(..)).build() ); Not confidential. Tell everyone. 50
  • 51. Spring Integration @Gateway public interface CustomerService { @Gateway void addCustomer( Customer customer ) throws CrmException; } Not confidential. Tell everyone. 51
  • 52. DEMO• using Spring Integration gateways @starbuxman slideshare.net/joshlong
  • 53. Summary RabbitMQ is readily available Messaging in the Cloud Spring Integration makes integration over AMQP easy Spring Batch Makes Batch Jobs Easy • Remote Chunking Makes Using AMQP as the Coordination Fabric Easy Not confidential. Tell everyone. 53
  • 54. springsource.org | cloudfoundry.org Questions? Say hi! @starbuxman slideshare.net/joshlong@starbuxman josh.long@springsource.com