• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
Jute rc
 

Jute rc

on

  • 686 views

A RC extension to Jute compiler

A RC extension to Jute compiler

Statistics

Views

Total Views
686
Views on SlideShare
683
Embed Views
3

Actions

Likes
0
Downloads
0
Comments
0

1 Embed 3

http://paper.li 3

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

    Jute rc Jute rc Presentation Transcript

    • Hadoop Record RC Compiler Tanping Wang
    • Motivation Automatically generate serialization/de- serialization code for any give primitive or composite data type. Directly plug serialization/de-serialization code to generate MapReduce output file that supports RC storage format. An extension to the existing Hadoop Record Compiler (Jute) package – So is named JuteRc.
    • Haoop Record Compiler Two important packages:  Hadoop.record.compiler.generated – rcc.jj  Hadoop.record.compiler  CodeGenerator  JType : Jboolean, Jstring, JCompType (JMap, Jvector..) Execution path: bin/rcc -> rcc#JFile.genCode -> CodeGenerator#genCode
    • Haoop Record Compiler cont. Inside CondeGenerator JRecord .codeGen :  set up members,  set up member functions:  setter/getter,  serialization/de-serialization (JType.getRead/WriteMethod)  abastract JType (JavaType, CppType)  JBoolean, JString, JMap, Jvector  JType.getRead/WriteMethod ( important)
    • Haoop Record Compiler cont. JType.getReadMethod _rio_a.startRecord(this,_rio_tag); _rio_a.writeBuffer(bcookie,"bcookie"); _rio_a.writeLong(timestamp,"timestamp"); _rio_a.endRecord(this,_rio_tag);
    • Hadoop Record RC Compiler• Execution Path bin/rcc => bin/rc-rcc rcc#JFile.genCode => JRcFile.genCode CodeGenerator#genCode => JRcCodeGenerator
    • Hadoop Record RC Compiler cont. Inside CondeGenerator JRcRecord .codeGen :  set up members,  set up memberfunctions:  setter/getter,  serialization/de-serialization (JRcType.getRcRead/WriteRcMethod  abastract JRcType (JavaType only)  JRcBoolean, JRcString, JRcMap, JRcVector .. etc.  JType.getRcReadMethod
    • Hadoop Record RC Compiler cont. JType.getRcReadMethod com.yahoo.ccdi.fetl.RcUtil.writeBuffer(this, bcookie, writeIndx++); com.yahoo.ccdi.fetl.RcUtil.writeLong(this, timestamp, writeIndx++);
    • How to use the JuteRC Maven packaged README 1) generate JuteRC.jar % mvn install 2) run rcc against DDL file % ./rcc --language javarc something.jr