Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Hadoop                                  Asakusa                                                                           ...
Asakusa             	                   Hadoop     Hadoop      →                           13           100         ...
     Hadoop                         BI           4-5                                  20-30                          ...
Hadoop                                                           	              BI                     MapReduce      ...
Asakusa                           	                               Hadoop                                            	    ...
Hadoop                                                    Asakusa	       Hadoop          	                                ...
Asakusa                                                                                	                                  ...
Asakusa                          	                                                                                 Mode...
Asakusa                                  	                          Java                          Eclipse/Maven         ...
ModelGenerator	     MapReduce          Java       writable             ModelGenerator          MySQL   DDL SQL) Table...
Ashigel                        	         Hadoop                                                                          ...
DAG                                     DSL                            	                                        DSL      ...
BatchDSL         	                   BatchDSL                                                                          ...
FlowDSL                         	     Flowpart                                 FlowDSL	      @Override        protected v...
FlowDSL                             	     Flowpart                                     FlowDSL                           ...
OperatorDSL                                             	                       DSL              DAG              Opera...
OperatorDSL                                   	 public abstract class ExampleOperator { /**                         	    @...
OperatorDSL	     CoGroup                                               		     Confluent                  	     Convert    ...
Copyright 2011(C) OSS Laboratories Inc. All Rights Reserved 	   19
DSL                                                     	                        Excel                                 ...
           DAG                    STS                    SPF              Shortest Path First –         GRASP    ...
                                                                     	                                                   ...
                                                               	4             In	                                        ...
     Asakusa                                          	                        MapReduce                                 ...
     TX @JobFlow(name = "stock") public class StockJob extends FlowDescription { private In<Shipment> shipmentIn; private...
ThunderGate	          RDBMS Hadoop                                                                       DB             ...
                                                    MonkeyMagic               1. Hadoop               2.              ...
      Ashigel          –                     MonkeyMagic                          rb                	# workflowjobnet("#...
Upcoming SlideShare
Loading in …5
×

110701 asakusa説明資料

1,595 views

Published on

Japanese presentation for Asakusa Framework

Published in: Technology

110701 asakusa説明資料

  1. 1. Hadoop Asakusa Rev. 1.0 2011/7/1 OSS 110701 Copyright 2011(C) OSS Laboratories Inc. All Rights Reserved
  2. 2. Asakusa   Hadoop  Hadoop →       13 100     Copyright 2011(C) OSS Laboratories Inc. All Rights Reserved 2
  3. 3.   Hadoop BI   4-5 20-30             IF     Copyright 2011(C) OSS Laboratories Inc. All Rights Reserved 3
  4. 4. Hadoop BI       MapReduce       Copyright 2011(C) OSS Laboratories Inc. All Rights Reserved 4
  5. 5. Asakusa   Hadoop BI/ MonkeyMagic Oozie Pig Hive MapReduce Java Core HDFS MapReduce Copyright 2011(C) OSS Laboratories Inc. All Rights Reserved 5
  6. 6. Hadoop Asakusa Hadoop Asakusa     DAG    DSL Ashigel ModelGenarator  MapReduce     Ashigel     MonkeMagic    ThunderGate MySQL Copyright 2011(C) OSS Laboratories Inc. All Rights Reserved 6
  7. 7. Asakusa DSL I/F DB Ashigel ThunderGate Importer/ Asakusa Exporter/ MapReduce MonkeyMagic Recoverer JOB JOB Model Generator Hadoop Hadoop MonkeyMagic HDFS MapReduce MySQL Template Generator Test Driver Copyright 2011(C) OSS Laboratories Inc. All Rights Reserved 7
  8. 8. Asakusa       ModelGenerator       Writable DSL     ThunderGate     Ashigel   Import/Export         API   DSL MapReduce        DAG DSL   DSL       Copyright 2011(C) OSS Laboratories Inc. All Rights Reserved 8
  9. 9. Asakusa   Java Eclipse/Maven  MapReduce Model Generator  DSL Domain Specific Language Ashigel Compiler  Rumtime Library  ThunderGate Copyright 2011(C) OSS Laboratories Inc. All Rights Reserved 9
  10. 10. ModelGenerator   MapReduce   Java writable    ModelGenerator   MySQL DDL SQL) Table View   HadoopIO   Eclipse DSL   Table Hadoop   Hadoop Table Copyright 2011(C) OSS Laboratories Inc. All Rights Reserved 10
  11. 11. Ashigel   Hadoop DSL BatchDSL FlowDSL DSL OperatorDSL Map/Reduce Map/Reduce Map Reduce Map Reduce Copyright 2011(C) OSS Laboratories Inc. All Rights Reserved 11
  12. 12. DAG DSL   DSL   3 DSL   BatchDSL   FlowDSL   OperatorDSL      Replace          TX DSL DB     MR   Copyright 2011(C) OSS Laboratories Inc. All Rights Reserved 12
  13. 13. BatchDSL   BatchDSL     DSL job Copyright 2011(C) OSS Laboratories Inc. All Rights Reserved 13
  14. 14. FlowDSL   Flowpart FlowDSL @Override protected void describe() { // Join join = op.join(itemIn, orderIn); // SetStatus missing = op.setStatus(join.missed, " "); orderOut.add(missing.out); Java DAG // Sum sum = op.sum(join.joined); Operator Obj // ToAmount result = op.toAmount(sum.out); resultOut.add(result.out); // core.stop(result.original); } Copyright 2011(C) OSS Laboratories Inc. All Rights Reserved 14
  15. 15. FlowDSL   Flowpart FlowDSL 2 @Override Operator protected void describe() { ReNewTxApMachingOperatorFactory f = new ReNewTxApMachingOperatorFactory(); UpdateTransactionsOperatorFactory f1 = new UpdateTransactionsOperatorFactory(); CoreOperatorFactory core = new CoreOperatorFactory(); // No Ope edge BranchSlipNoWithWithout bra11 = f.branchSlipNoWithWithout(inApMaching); edge // No BranchRecGapAndUnmatch bra12 = f.branchRecGapAndUnmatch(bra11.out2); // edge InitSlipInfo upd11 = f.initSlipInfo(bra12.out1); // InitBillInfo upd12 = f.initBillInfo(bra12.out1); // No Confluent<TxApMaching> cfl11 = core.confluent(bra12.out2, upd12.out); edge // GroupSortBranchDeficitSurplusDiv grs11 = f1.groupSortBranchDeficitSurplusDiv(cfl11.out); Copyright 2011(C) OSS Laboratories Inc. All Rights Reserved 15
  16. 16. OperatorDSL   DSL   DAG   Operator Operator   MapReduce /** * @param info * @param order @return / @MasterJoin public abstract JoinOrder join(ItemInfo info, OrderDetail order); Asakusa /** @param each @return / @Summarize public abstract SumOrder sum(JoinOrder each); /** * @param total * @return / @Convert public OrderAmount toAmount(SumOrder total) { amount.setAmount(total.getAmount()); amount.setOrderId(total.getOrderId()); return amount; } 16 Copyright 2011(C) OSS Laboratories Inc. All Rights Reserved
  17. 17. OperatorDSL public abstract class ExampleOperator { /** @param masters @param tx @return null   / @MasterSelection public ItemMst selectItemMst(List<ItemMst> masters, HogeTrn tx) { for (ItemMst mst : masters) { if (mst.getStart() <= tx.getDate() &&   tx.getDate() <= mst.getEnd()) { return mst; } } return null; }   /** @param master @param tx / @MasterJoinUpdate(selection = "selectItemMst") public void updateWithMaster( @Key(group = "id") ItemMst master, @Key(group = "itemId") HogeTrn tx) { tx.setPrice(master.getPrice()); } } Copyright 2011(C) OSS Laboratories Inc. All Rights Reserved 17
  18. 18. OperatorDSL CoGroup Confluent Convert Duplicate GroupSort Split MasterBranch MasterCheck MasterJoin MasterJoinUpdate Summarize Branch Checkpoint Empty Identity DAG Logging Stop Copyright 2011(C) OSS Laboratories Inc. All Rights Reserved 18
  19. 19. Copyright 2011(C) OSS Laboratories Inc. All Rights Reserved 19
  20. 20. DSL   Excel     BatchDSL FlowDSL OperatorDSL JUnit     JUnit Copyright 2011(C) OSS Laboratories Inc. All Rights Reserved 20
  21. 21.      DAG     STS     SPF   Shortest Path First –   GRASP   General Responsibility Assignment Software Principle      TX   Join   No_SQL   Copyright 2011(C) OSS Laboratories Inc. All Rights Reserved 21
  22. 22.   TRN TRN TRN TRN 4 In TRN TRN 9 In 5 In TRN TRN 5 Out TRN TRN TRN TRN 4 Out 7 In TRN TRN TRN TRN TRN TRN TRN TRN TRN TRN Copyright 2011(C) OSS Laboratories Inc. All Rights Reserved 22
  23. 23.   4 In 5 Out Level-0 TRN TRN Level-1 TRN TRN TRN Level-2 Level-3 4-5 Copyright 2011(C) OSS Laboratories Inc. All Rights Reserved 23
  24. 24.   Asakusa MapReduce JarFile HadoopJob Job TRX Hadoop job IO Copyright 2011(C) OSS Laboratories Inc. All Rights Reserved 24
  25. 25.   TX @JobFlow(name = "stock") public class StockJob extends FlowDescription { private In<Shipment> shipmentIn; private In<Stock> stockIn; private Out<Shipment> shipmentOut; private Out<Stock> stockOut; /** In Out * @param shipmentIn * @param stockIn * @param shipmentOut * @param stockOut / public StockJob( @Import(name = "shipment", description = ShipmentFromDb.class) In<Shipment> shipmentIn, @Import(name = "stock", description = StockFromDb.class) In<Stock> stockIn, @Export(name = "shipment", description = ShipmentToDb.class) TX Out<Shipment> shipmentOut, @Export(name = "stock", description = StockToDb.class) Out<Stock> stockOut) { this.shipmentIn = shipmentIn; this.stockIn = stockIn; this.shipmentOut = shipmentOut; this.stockOut = stockOut; } Copyright 2011(C) OSS Laboratories Inc. All Rights Reserved 25
  26. 26. ThunderGate    RDBMS Hadoop DB HDFS  RDBMS MySQL Oracle RDBMS Copyright 2011(C) OSS Laboratories Inc. All Rights Reserved 26
  27. 27.     MonkeyMagic   1. Hadoop   2.   Ruby   Asakusa experimental shell script     OSS   job   Copyright 2011(C) OSS Laboratories Inc. All Rights Reserved 27
  28. 28.    Ashigel –  MonkeyMagic rb # workflowjobnet("#{BATCH_ID_QUALIFIER}.tutorial", :instance_name => "master", :credential_name => "asakusa") do boot_jobs("order") # jobflow – order jobnet("order", :to => []) do auto_sequence jobnet("__IMPORT__") do boot_jobs("bulkloader") jobnet("bulkloader") do auto_sequence job("bulkloader.asakusa", ""$ASAKUSA_HOME/"bulkloader/bin/importer.sh primary asakusa tutorial order "$($ASAKUSA_HOME/monkeymagic/bin/execution_id.sh)" 20380101000000 "#{BATCH_ARGUMENTS}"", :instance_name => "asakusa", :credential_name => "asakusa") end end jobnet("__STAGE_GRAPH__") do boot_jobs("stage0001") # stage - tutorial / order / stage0001 hadoop_job_run("stage0001", "$ASAKUSA_HOME/monkeymagic/bin/hadoop_job_run.sh com.example.tutorial.batch.tutorial.order.stage0001.StageClient "$ASAKUSA_HOME/batchapps/tutorial/lib/jobflow-order.jar" -D "com.asakusafw.executionId=$($ASAKUSA_HOME/monkeymagic/bin/execution_id.sh)" -D "com.asakusafw.user=$USER" -D "com.asakusafw.batchArgs=#{BATCH_ARGUMENTS}"", :to => ["stage0002"]) do hadoop_job("tutorial.order.stage0001") end # stage - tutorial / order / stage0002 hadoop_job_run(“stage0002”, “$ASAKUSA_HOME/monkeymagic/bin/hadoop_job_run.sh Copyright 2011(C) OSS Laboratories Inc. All Rights Reserved 28

×