Download as PDF, PPTX
Uploaded bySpark Summit
Spark Summit EU talk by Oscar Castaneda
The document discusses transforming Excel spreadsheets into Spark DataFrames by automatically translating Excel formulas into Spark code. It presents a program transformation pipeline that takes Excel formulas, parses them using a grammar and parser to generate a parse tree, and then generates Spark code from the parse tree. Key aspects covered include using an existing grammar and parser called XLParser to parse Excel formulas, treating Excel as a domain-specific language, and generating code by writing a pretty printer for the target Spark language. The talk concludes with a demonstration of the code generation approach.
More Related Content
![제 23회 보아즈(BOAZ) 빅데이터 컨퍼런스 - [MBOAX] : ABSA를 활용한 소비자 반응 분석 기반 운영 효율화 대시보드 설계](https://cdn.slidesharecdn.com/ss_thumbnails/3-1boaz23rdconferencemboax-260203102709-9d519923-thumbnail.jpg?width=640&height=640&fit=bounds)
Spark Summit EU talk by Oscar Castaneda
- 1. Sparksheet - Transforming Spreadsheets intoSpark Data Frames Oscar Castañeda-Villagrán Universidad del Valle de Guatemala
- 2. About • Researcher atUniversidad del Valle de Guatemala. • Research Interests: • Program Transformation, • Programming Education Research, • Online Learning to Rank.
- 3.
- 4. Prototyping Spark programswith … http://bit.ly/2e5GmyY http://bit.ly/2edYfMs
- 5.
- 6. Agenda • Problem Statementand Motivation • Architecture • Program Transformation • Pipeline • Code-to-Code Transformation • Parsing Excel Formulas • Grammar • Parse Tree • XLParser • Excel as a DSL • Generating Code • Demo • Q&A
- 7. Disclaimer(s) • Ongoing research… • We will focus on how to create a Program Transformation Pipeline.
- 8. Problem Statement Spark programscan be prototyped in Excel but manually translating Excel formulas to Spark programs is tedious and error-prone.
- 9. Motivation • “Straight path”between column-oriented Excel “programs” and Spark programs that make use of the DataFrame API. • But, manually translating Excel formulas to Spark is tedious and error-prone. • What if: Excel compiler?
- 10. Problem Statement Given thatcolumn-oriented Excel applications can be manually translated to Spark programs, … … find a way to automate translation of Excel formulas so that data pipelines can be prototyped in Excel … … and Scala/Python code generated to run in Spark.
- 11. Motivation Automatically translate Excel Formulasto … Spark. Data! http://bit.ly/2expmoF
- 12. Architecture Excel Formulas Columnar Data TakeES snapshot Restore ES snapshot Black Box Data! Refine Code http://bit.ly/2em6RUK http://bit.ly/2e5H1jL
- 13. Excel Formulas Columnar Data TakeES snapshot Restore ES snapshot Black Box Data! Refine Code Architecture http://bit.ly/2em6RUK http://bit.ly/2e5H1jL
- 14. Excel Formulas Columnar Data TakeES snapshot Restore ES snapshot Black Box Data! Refine Code Tomorrow: Spark Cluster with Elasticsearch InsideArchitecture http://bit.ly/2em6RUK http://bit.ly/2e5H1jL
- 15.
- 16. Program Transformation “A programtransformation is any operation that takes a computer program and generates another program.” https://en.wikipedia.org/wiki/Program_transformation
- 17. Architecture Excel Formulas Columnar Data TakeES snapshot Restore ES snapshot Black Box Data! Refine Code
- 18.
- 19. Code-to-Code Transformation “The inputto the code generator typically consists of a parse tree or an abstract syntax tree.” https://en.wikipedia.org/wiki/Code_generation_(compiler)
- 20. http://bit.ly/2dH0ybF We need aGrammar!
- 21. We need aParse Tree! Parse Excel Formulas
- 22. We need aParse Tree! Parse Excel Formulas http://bit.ly/2e0TZA8
- 23. We need aParse Tree! Generate Scala CodeParse Excel Formulas http://bit.ly/2e0TZA8
- 24. We need aGrammar! http://bit.ly/2dH0ybF
- 25. XLParser "If I haveseen further, it is by standing upon the shoulders of giants" — Sir Isaac Newton A Grammar for Spreadsheet Formulas Evaluated on Two Large Datasets – Efthimia Aivaloglou, David Hoepelman & Felienne Hermans, Proceedings of SCAM ’15
- 26. XLParser "If I haveseen further, it is by standing upon the shoulders of giants" — Sir Isaac Newton A Grammar for Spreadsheet Formulas Evaluated on Two Large Datasets – Efthimia Aivaloglou, David Hoepelman & Felienne Hermans, Proceedings of SCAM ’15
- 27. XLParser "If I haveseen further, it is by standing upon the shoulders of giants" — Sir Isaac Newton Excel Formula SUM(A,C) http://xlparser.perfectxl.nl/demo
- 28. XLParser "If I haveseen further, it is by standing upon the shoulders of giants" — Sir Isaac Newton Parse Tree! SUM(A,C) Excel Formula http://xlparser.perfectxl.nl/demo
- 29. Excel as aDSL • External DSL: parsed independently. • XLParser gives us a Parse Tree from an Excel Formula. • Given the Parse Tree, generate code!
- 30. How do yougenerate code from parsed Excel Formulas? ?
- 31. Generating Code “An elegantway to generate code from an AST is to write a class for each non-terminal node in the tree, and then each node in the tree simply generates the piece of code that it is responsible for.” http://www.codeproject.com/Articles/26975/Writing-Your-First-Domain-Specific-Language-Part
- 32. Generating Code A practicalway to generate code is to take a Parse Tree and write a pretty printer for the target language. http://bit.ly/2em73DM
- 33. Generating Code froman AST SUM(A,C)
- 34. Generating Code froman AST
- 35. Generating Code froman AST
- 36.
- 37. What have weseen? • Column-Oriented Excel Applications as Prototypes for Spark programs • Program Transformation. • How to model as a Pipeline. • Why considered a Code-to-Code Transformation. • How to Parse Excel Formulas. • Grammar • Parse Tree • XLParser • Excel as a DSL. • How can we Generate Code? • Demo.
- 38. Next Steps • Translate~500 Excel Formulas. • Modeling Machine Learning in Excel. • Prototype D|’s and ML|’s in Excel.
- 39.
- 40. Q&A Tomorrow: Spark Clusterwith Elasticsearch Inside
- 41. THANK YOU. Email: ofcastaneda@uvg.edu.gt Twitter:@oscar_castaneda