Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Batch Processing - Processamento em Lotes no Mundo Corporativo

Palestra apre

Batch Processing - Processamento em Lotes no Mundo Corporativo

  1. 1. Globalcode  –  Open4education Batch Processing: Processamento em Lotes no Mundo Corporativo Rodrigo Cândido da Silva @rcandidosilva
  2. 2. Globalcode  –  Open4education Agenda !   Conceitos !   Batch Domain Language !   Chunk vs. Batchlet !   Partitioned Step !   Flow, Split e Decision !   Listeners e Exceptions !   Execution !   Integration
  3. 3. Globalcode  –  Open4education Porque Batch? !   É muito comum em aplicações !   Várias soluções “personalizadas” !   Produtos começaram a surgir !  Spring Batch !  WebSphere Compute Grid !   Ideal para sistemas ETL
  4. 4. Globalcode  –  Open4education Batch API !   Chunk / Batchlet ! Implementação de um Step !   Contexts !  Job e Step at runtime ! Persistência de metadados !   Listeners !  Callback lifecycle events !   Partitioning ! Processamento paralelo
  5. 5. Globalcode  –  Open4education Batch Domain Language !   Batch job XML definition !   Descreve os steps como um agrupamento de batch artifacts
  6. 6. Globalcode  –  Open4education Batch Domain Language <job id="adressJob” version="1.0"> <listeners> <listener ref="MyJobListener"/> </listeners> <step id="buildingData" next="adressStep"> <batchlet ref="GenerateDataBatchlet" /> </step> <step id="adressStep"> <listeners> <listener ref="MyStepListener"/> </listeners> <chunk item-count="10"> <reader ref="adressItemReader" /> <processor ref="adressItemProcessor" /> <writer ref="adressItemWriter" /> </chunk> </step> </job>
  7. 7. Globalcode  –  Open4education Chunk vs. Batchlet !   Implementam step dentro do job !   Chunk !  Encapsula padrão ETL !  Single Reader, Processor e Writer !  Executado por pedaços dados (chunk) !  Chunk output é escrito unitariamente !   Batchlet !  Promove a execução de um único e simples processo !  Executado até o fim produzindo um código de retorno
  8. 8. Globalcode  –  Open4education Chunk vs. Batchlet !   Chunk ! Batchlet
  9. 9. Globalcode  –  Open4education Batchlet @Named public class MyBatchlet { @Process public String process() throws Exception {..} @Stop public void stopMe() throws Exception {..} } //For a job <step id=”step1”> <batchlet ref=“MyBatchlet”/> </step> public class MyBatchlet implements Batchlet {..}
  10. 10. Globalcode  –  Open4education Chunk !   Step Job //For a job <step id=”sendStatements”> <chunk reader=”accountReader” processor=”accountProcessor” writer=”emailWriter” item-count=”10” /> </step> @Named(“accountReader") ...implements ItemReader... { public Account readItem() { // read account using JPA @Named(“accountProcessor") ...implements ItemProcessor... { public Statement processItems(Account account) { // read Account, return Statement @Named(“emailWriter") ...implements ItemWriter... { public void writeItems(List<Statements> statements) { // use JavaMail to send email
  11. 11. Globalcode  –  Open4education Chunk public interface ItemReader<T> { public void open(Externalizable checkpoint); public T readItem(); public Externalizable checkpointInfo(); public void close(); } public interface ItemWriter<T> { public void open(Externalizable checkpoint); public void writeItems(List<T> items); public Externalizable checkpointInfo(); public void close(); } public interface ItemProcessor<T, R> { public R processItem(T item); }
  12. 12. Globalcode  –  Open4education Checkpoint !   Para tarefas intensivas, longo período de tempo !  Checkpoint/restart é um bastante utilizado !   Basicamente… !  Armazena estado do ItemReader, ItemWriter !   Métodos chamados !   reader.checkpointInfo()! !   writer.checkpointInfo() public interface ItemReader<T> { public void open(Externalizable checkpoint); public Externalizable checkpointInfo(); } public interface ItemWriter<T> { public void open(Externalizable checkpoint); public Externalizable checkpointInfo(); } <chunk checkpoint-policy="item" commit-interval="10" item-count="10">
  13. 13. Globalcode  –  Open4education Partitioned Step !   Step pode rodar particionado !  [N] instâncias do mesmo step em [N] Threads !  Uma partição por Thread <step id="step1" > <chunk ...> <partition> <plan partitions=“10" threads="2"/> <reducer .../> </partition> </chunk> </step>
  14. 14. Globalcode  –  Open4education Partitioned Step !   Partition Mapper !  Decide dinamicamente o número de partições !  Partition Plan !   Partition Reducer !  Demarca a unidade lógica de trabalho !   Partition Collector !  Enviar resultados de processamento das partições !   Partition Analyzer !  Ponto de controle e análise dos resultados enviados
  15. 15. Globalcode  –  Open4education Flow, Split e Decision Flow Step I Task Step II Chunk ItemReader ItemWriter Step III Chunk Deci- sion ItemReader ItemWriter Step IV Chunk ItemReader ItemWriter EndStart ItemProces sor ItemProces sor ItemProces sor
  16. 16. Globalcode  –  Open4education Flow !   Define a lista de steps a ser executado (unitário) <flow id=”flow-1" next=“{flow, step, decision}-id” > <step id=“flow_1_step_1”> </step> <step id=“flow_1_step_2”> </step> </flow>
  17. 17. Globalcode  –  Open4education Split !   Define a lista de flows a serem executados (paralelo) !   Coletores e analisadores para monitoramento <split …> <flow …./> <!– each flow runs on a separate thread --> <flow …./> </split>
  18. 18. Globalcode  –  Open4education Decision !   Possibilita a implementação de workflows
  19. 19. Globalcode  –  Open4education Decision @Named public class Decider { public String decide(BatchContext context) throws Exception { String exit = context.getExitStatus(); if (“SUCCESS”.equals(exit)) { return “SKIP”; } return exit; } } //For a job <step id=”step1”> <decision id=“decision1” ref=“Decider”> <next on=“SKIP” to=“step3”/> <next on=“*” to=“step2”/> </decision> </step> <step id=“step2” next=“step3”/> <step id=“step3”/>
  20. 20. Globalcode  –  Open4education Lifecycle STOPPED 20 STARTING STARTED COMPLETED FAILED STOPPING ABANDONED stop() start() abandon() abandon() abandon() restart() restart()
  21. 21. Globalcode  –  Open4education Listeners !   Step !   StepListener, ItemReadListener, ItemProcessListener, ItemWriterListener, ChunkListener, RetryReadListener, RetryProcessListener, RetryWriteListener, SkipReadListener, SkipProcessListener, SkipWriteListener !   Job !   JobListener @Named public class StepListener { @BatchContext StepContext context; @BeforeStep public void beforeStep() {..} @AfterStep public void afterStep() {..} } //For a job <step id=”step1”> <listeners> <listener ref=“StepListener”/> </listeners> </step>
  22. 22. Globalcode  –  Open4education Exceptions <job id=...> ... <chunk skip-limit=”5” retry-limit=”5”> <skippable-exception-classes> <include class="java.lang.Exception"/> <exclude class="java.io.FileNotFoundException"/> </skippable-exception-classes> <retryable-exception-classes> ... </retryable-exception-classes> <no-rollback-exception-classes> ... </no-rollback-exception-classes> </chunk> ... </job>
  23. 23. Globalcode  –  Open4education JobOperator e Repository !   JobOperator !  Runtime interface para gerenciamento !  start, stop, restart !  JobRepository interface commands !   JobRepository !  Contém informações sobre os jobs !  Completos e em execução
  24. 24. Globalcode  –  Open4education Execution !   JobInstance !  Representação lógica de um job runtime !   JobExecution !  Tentativa de executar um JobInstance !   StepExecution !  Tentativa de rodar um step de um job
  25. 25. Globalcode  –  Open4education Integration !   Suporte ao Java SE !   Application Server Runtime !  Suporte clustering, segurança, gerenciamento de recursos !   Dependency Injection com CDI !   XML descriptors !  META-INF/batch-jobs/myJob.xml !   Empacotamento !  JAR, WAR, EJB
  26. 26. Globalcode  –  Open4education DEMO http://goo.gl/wst7M7
  27. 27. Globalcode  –  Open4education Perguntas ?
  28. 28. Globalcode  –  Open4education Referências ! https://jcp.org/en/jsr/detail?id=352 ! https://java.net/projects/jbatch ! http://projects.spring.io/spring-batch/ ! http://docs.oracle.com/javaee/7/tutorial/doc/batch-processing.htm ! http://www.oracle.com/technetwork/articles/java/batch-1965499.html ! https://github.com/javaee-samples/javaee7-samples/ ! http://blog.arungupta.me/2014/07/schedule-javaee7-batch-jobs- techtip36/ ! http://www.planetjones.co.uk/blog/25-05-2013/introducing-jsr-352-java- batch-ee-7.html
  29. 29. Globalcode  –  Open4education Obrigado! @rcandidosilva rodrigocandido.me

×