This document discusses building a Hadoop Record Reader in Python. It summarizes that Hadoop has a native data storage format called Hadoop Record or "Jute" that has a Data Definition Language and compiler. It then discusses how the author built a Python parsing library to parse this format into generic Python data types by using Ply (lex and yacc for Python). Future work mentioned includes building a DDL translator and integrating feedback.

1. Hadoop Record Reader in PythonHUG: Nov 18 2009Paul Tarjanhttp://paulisageek.com@ptarjanhttp://github.com/ptarjan/hadoop_record

2. Hey Jute…Tabs and newlines are good and allFor lots of data, don’t do that

3. don’t make it bad...Hadoop has a native data storage format called Hadoop Record or “Jute”org.apache.hadoop.recordhttp://en.wikipedia.org/wiki/Jute

4. take a data structure…There is a Data Definition Language!module links { class Link {ustringURL;booleanisRelative;ustringanchorText; };}

5. and make it better…And a compiler$ rcc -lc++ inclrec.jrtestrec.jr namespace inclrec { class RI : public hadoop::Record { private: int32_t I32; double D;std::string S;

6. remember, to only use C++/Java$rcc--help Usage: rcc --language[java|c++] ddl-files

7. then you can start to make it better…I wanted it in pythonNeed 2 parts:Parsing library and DDL translatorI only did the first partIf you need second part, let me know

8. Hey Jute don't be afraid…

9. you were made to go out and get her…http://github.com/ptarjan/hadoop_record

10. the minute you let her under your skin…I bet you thought I was done with “Hey Jude” references, eh?How I built itPly == lex and yaccParser == 234 lines including tests!Outputs generic data typesYou have to do the class transform yourselfYou can use my lex and yacc stuff in your language of choice

11. and any time you feel the pain…Parsing the binary format is hardVector vsstruct???struct= "s{" record *("," record) "}"vector = "v{" [record *("," record)] "}"LazyString – don’t decode if not needed99% of my hadoop time was decoding strings I didn’t needBinary on disk -> CSV -> python == wastefulHadoopupacks zip files – name it .mod

12. nananananaFuture workDDL ConverterIntegrate it officiallyRecord writer (should be easy)SequenceFileAsOutputFormatIntegrate your feedback