Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Jute rc

753 views

Published on

A RC extension to Jute compiler

Published in: Technology, Business
  • Be the first to comment

  • Be the first to like this

Jute rc

  1. 1. Hadoop Record RC Compiler Tanping Wang
  2. 2. Motivation Automatically generate serialization/de- serialization code for any give primitive or composite data type. Directly plug serialization/de-serialization code to generate MapReduce output file that supports RC storage format. An extension to the existing Hadoop Record Compiler (Jute) package – So is named JuteRc.
  3. 3. Haoop Record Compiler Two important packages:  Hadoop.record.compiler.generated – rcc.jj  Hadoop.record.compiler  CodeGenerator  JType : Jboolean, Jstring, JCompType (JMap, Jvector..) Execution path: bin/rcc -> rcc#JFile.genCode -> CodeGenerator#genCode
  4. 4. Haoop Record Compiler cont. Inside CondeGenerator JRecord .codeGen :  set up members,  set up member functions:  setter/getter,  serialization/de-serialization (JType.getRead/WriteMethod)  abastract JType (JavaType, CppType)  JBoolean, JString, JMap, Jvector  JType.getRead/WriteMethod ( important)
  5. 5. Haoop Record Compiler cont. JType.getReadMethod _rio_a.startRecord(this,_rio_tag); _rio_a.writeBuffer(bcookie,"bcookie"); _rio_a.writeLong(timestamp,"timestamp"); _rio_a.endRecord(this,_rio_tag);
  6. 6. Hadoop Record RC Compiler• Execution Path bin/rcc => bin/rc-rcc rcc#JFile.genCode => JRcFile.genCode CodeGenerator#genCode => JRcCodeGenerator
  7. 7. Hadoop Record RC Compiler cont. Inside CondeGenerator JRcRecord .codeGen :  set up members,  set up memberfunctions:  setter/getter,  serialization/de-serialization (JRcType.getRcRead/WriteRcMethod  abastract JRcType (JavaType only)  JRcBoolean, JRcString, JRcMap, JRcVector .. etc.  JType.getRcReadMethod
  8. 8. Hadoop Record RC Compiler cont. JType.getRcReadMethod com.yahoo.ccdi.fetl.RcUtil.writeBuffer(this, bcookie, writeIndx++); com.yahoo.ccdi.fetl.RcUtil.writeLong(this, timestamp, writeIndx++);
  9. 9. How to use the JuteRC Maven packaged README 1) generate JuteRC.jar % mvn install 2) run rcc against DDL file % ./rcc --language javarc something.jr

×