Batch Processing - Processamento em Lotes no Mundo Corporativo

1,019 views

Published on

Palestra apre

Published in: Software
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
1,019
On SlideShare
0
From Embeds
0
Number of Embeds
7
Actions
Shares
0
Downloads
10
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

Batch Processing - Processamento em Lotes no Mundo Corporativo

  1. 1. Globalcode  –  Open4education Batch Processing: Processamento em Lotes no Mundo Corporativo Rodrigo Cândido da Silva @rcandidosilva
  2. 2. Globalcode  –  Open4education Agenda !   Conceitos !   Batch Domain Language !   Chunk vs. Batchlet !   Partitioned Step !   Flow, Split e Decision !   Listeners e Exceptions !   Execution !   Integration
  3. 3. Globalcode  –  Open4education Porque Batch? !   É muito comum em aplicações !   Várias soluções “personalizadas” !   Produtos começaram a surgir !  Spring Batch !  WebSphere Compute Grid !   Ideal para sistemas ETL
  4. 4. Globalcode  –  Open4education Batch API !   Chunk / Batchlet ! Implementação de um Step !   Contexts !  Job e Step at runtime ! Persistência de metadados !   Listeners !  Callback lifecycle events !   Partitioning ! Processamento paralelo
  5. 5. Globalcode  –  Open4education Batch Domain Language !   Batch job XML definition !   Descreve os steps como um agrupamento de batch artifacts
  6. 6. Globalcode  –  Open4education Batch Domain Language <job id="adressJob” version="1.0"> <listeners> <listener ref="MyJobListener"/> </listeners> <step id="buildingData" next="adressStep"> <batchlet ref="GenerateDataBatchlet" /> </step> <step id="adressStep"> <listeners> <listener ref="MyStepListener"/> </listeners> <chunk item-count="10"> <reader ref="adressItemReader" /> <processor ref="adressItemProcessor" /> <writer ref="adressItemWriter" /> </chunk> </step> </job>
  7. 7. Globalcode  –  Open4education Chunk vs. Batchlet !   Implementam step dentro do job !   Chunk !  Encapsula padrão ETL !  Single Reader, Processor e Writer !  Executado por pedaços dados (chunk) !  Chunk output é escrito unitariamente !   Batchlet !  Promove a execução de um único e simples processo !  Executado até o fim produzindo um código de retorno
  8. 8. Globalcode  –  Open4education Chunk vs. Batchlet !   Chunk ! Batchlet
  9. 9. Globalcode  –  Open4education Batchlet @Named public class MyBatchlet { @Process public String process() throws Exception {..} @Stop public void stopMe() throws Exception {..} } //For a job <step id=”step1”> <batchlet ref=“MyBatchlet”/> </step> public class MyBatchlet implements Batchlet {..}
  10. 10. Globalcode  –  Open4education Chunk !   Step Job //For a job <step id=”sendStatements”> <chunk reader=”accountReader” processor=”accountProcessor” writer=”emailWriter” item-count=”10” /> </step> @Named(“accountReader") ...implements ItemReader... { public Account readItem() { // read account using JPA @Named(“accountProcessor") ...implements ItemProcessor... { public Statement processItems(Account account) { // read Account, return Statement @Named(“emailWriter") ...implements ItemWriter... { public void writeItems(List<Statements> statements) { // use JavaMail to send email
  11. 11. Globalcode  –  Open4education Chunk public interface ItemReader<T> { public void open(Externalizable checkpoint); public T readItem(); public Externalizable checkpointInfo(); public void close(); } public interface ItemWriter<T> { public void open(Externalizable checkpoint); public void writeItems(List<T> items); public Externalizable checkpointInfo(); public void close(); } public interface ItemProcessor<T, R> { public R processItem(T item); }
  12. 12. Globalcode  –  Open4education Checkpoint !   Para tarefas intensivas, longo período de tempo !  Checkpoint/restart é um bastante utilizado !   Basicamente… !  Armazena estado do ItemReader, ItemWriter !   Métodos chamados !   reader.checkpointInfo()! !   writer.checkpointInfo() public interface ItemReader<T> { public void open(Externalizable checkpoint); public Externalizable checkpointInfo(); } public interface ItemWriter<T> { public void open(Externalizable checkpoint); public Externalizable checkpointInfo(); } <chunk checkpoint-policy="item" commit-interval="10" item-count="10">
  13. 13. Globalcode  –  Open4education Partitioned Step !   Step pode rodar particionado !  [N] instâncias do mesmo step em [N] Threads !  Uma partição por Thread <step id="step1" > <chunk ...> <partition> <plan partitions=“10" threads="2"/> <reducer .../> </partition> </chunk> </step>
  14. 14. Globalcode  –  Open4education Partitioned Step !   Partition Mapper !  Decide dinamicamente o número de partições !  Partition Plan !   Partition Reducer !  Demarca a unidade lógica de trabalho !   Partition Collector !  Enviar resultados de processamento das partições !   Partition Analyzer !  Ponto de controle e análise dos resultados enviados
  15. 15. Globalcode  –  Open4education Flow, Split e Decision Flow Step I Task Step II Chunk ItemReader ItemWriter Step III Chunk Deci- sion ItemReader ItemWriter Step IV Chunk ItemReader ItemWriter EndStart ItemProces sor ItemProces sor ItemProces sor
  16. 16. Globalcode  –  Open4education Flow !   Define a lista de steps a ser executado (unitário) <flow id=”flow-1" next=“{flow, step, decision}-id” > <step id=“flow_1_step_1”> </step> <step id=“flow_1_step_2”> </step> </flow>
  17. 17. Globalcode  –  Open4education Split !   Define a lista de flows a serem executados (paralelo) !   Coletores e analisadores para monitoramento <split …> <flow …./> <!– each flow runs on a separate thread --> <flow …./> </split>
  18. 18. Globalcode  –  Open4education Decision !   Possibilita a implementação de workflows
  19. 19. Globalcode  –  Open4education Decision @Named public class Decider { public String decide(BatchContext context) throws Exception { String exit = context.getExitStatus(); if (“SUCCESS”.equals(exit)) { return “SKIP”; } return exit; } } //For a job <step id=”step1”> <decision id=“decision1” ref=“Decider”> <next on=“SKIP” to=“step3”/> <next on=“*” to=“step2”/> </decision> </step> <step id=“step2” next=“step3”/> <step id=“step3”/>
  20. 20. Globalcode  –  Open4education Lifecycle STOPPED 20 STARTING STARTED COMPLETED FAILED STOPPING ABANDONED stop() start() abandon() abandon() abandon() restart() restart()
  21. 21. Globalcode  –  Open4education Listeners !   Step !   StepListener, ItemReadListener, ItemProcessListener, ItemWriterListener, ChunkListener, RetryReadListener, RetryProcessListener, RetryWriteListener, SkipReadListener, SkipProcessListener, SkipWriteListener !   Job !   JobListener @Named public class StepListener { @BatchContext StepContext context; @BeforeStep public void beforeStep() {..} @AfterStep public void afterStep() {..} } //For a job <step id=”step1”> <listeners> <listener ref=“StepListener”/> </listeners> </step>
  22. 22. Globalcode  –  Open4education Exceptions <job id=...> ... <chunk skip-limit=”5” retry-limit=”5”> <skippable-exception-classes> <include class="java.lang.Exception"/> <exclude class="java.io.FileNotFoundException"/> </skippable-exception-classes> <retryable-exception-classes> ... </retryable-exception-classes> <no-rollback-exception-classes> ... </no-rollback-exception-classes> </chunk> ... </job>
  23. 23. Globalcode  –  Open4education JobOperator e Repository !   JobOperator !  Runtime interface para gerenciamento !  start, stop, restart !  JobRepository interface commands !   JobRepository !  Contém informações sobre os jobs !  Completos e em execução
  24. 24. Globalcode  –  Open4education Execution !   JobInstance !  Representação lógica de um job runtime !   JobExecution !  Tentativa de executar um JobInstance !   StepExecution !  Tentativa de rodar um step de um job
  25. 25. Globalcode  –  Open4education Integration !   Suporte ao Java SE !   Application Server Runtime !  Suporte clustering, segurança, gerenciamento de recursos !   Dependency Injection com CDI !   XML descriptors !  META-INF/batch-jobs/myJob.xml !   Empacotamento !  JAR, WAR, EJB
  26. 26. Globalcode  –  Open4education DEMO http://goo.gl/wst7M7
  27. 27. Globalcode  –  Open4education Perguntas ?
  28. 28. Globalcode  –  Open4education Referências ! https://jcp.org/en/jsr/detail?id=352 ! https://java.net/projects/jbatch ! http://projects.spring.io/spring-batch/ ! http://docs.oracle.com/javaee/7/tutorial/doc/batch-processing.htm ! http://www.oracle.com/technetwork/articles/java/batch-1965499.html ! https://github.com/javaee-samples/javaee7-samples/ ! http://blog.arungupta.me/2014/07/schedule-javaee7-batch-jobs- techtip36/ ! http://www.planetjones.co.uk/blog/25-05-2013/introducing-jsr-352-java- batch-ee-7.html
  29. 29. Globalcode  –  Open4education Obrigado! @rcandidosilva rodrigocandido.me

×