Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

January 2015 HUG: Efficient user environment to facilitate working with Big Data

1,367 views

Published on

We are developing solutions that make it easier to work with data on Hadoop regardless the skill level of the user (whether a developer, an analyst or something else.) The core engine we use and the user-interface have been in research and under development for several years and we are only now starting to “expose” it to the world at large. To give you a sense for our approach, we will provide a variety of demos and discuss the underlying technology to demonstrate the time savings when working with Hadoop.

Speaker:

Shevek, CompilerWorks

Shevek is an expert programmer with a strong interest in parallel and distributed systems. He has worked on cutting edge research in compilers and language design, algorithmic optimization, systems and security. He is capable of maintaining a very straight face under questioning on topics including “Why is our printer playing ‘happy birthday’?” or “What is that message doing on the side of that building?” He received a Doctorate in Computing on the Formalization of Protection Systems from the University of Bath, England. He also holds a Masters in Pure Mathematics and an epee.

Published in: Technology
  • Be the first to comment

  • Be the first to like this

January 2015 HUG: Efficient user environment to facilitate working with Big Data

  1. 1. Compiler Works Our Goal: To make working with data on Hadoop easy Who are we? A group of backend engineers attempting to add a UI to some technology we've built. January 2015
  2. 2. Compiler Works Development Process Flow Jan 2015 2 Write Code (1 second) Compile Code (5 minutes) Test Code (1 hour) Integrate Code (1 day – 1 week) Deploy to Customer (3 months)
  3. 3. Compiler Works Problems Jan 2015 3 Write Code (1 second) Compile Code (5 minutes) Test Code (1 hour) Integrate Code (1 day – 1 week) Deploy to Customer (3 months) IDE highlights bug(s) with static analysis Code leaves developer desktop Say prayers Kind of expensive Runtime errors detected
  4. 4. Compiler Works Writing Code with Fast Feedback? Jan 2015 4 DEMO
  5. 5. Compiler Works What did we solve? (multi-stage compiler) Jan 2015 5
  6. 6. Compiler Works What did we solve? (algebraic optimization) Jan 2015 6 • Join A, B, C, D • These are the object plans – they are leaf nodes in the algebraic tree – they are themselves trees
  7. 7. Compiler Works What did we solve? (scheduling) Jan 2015 7 • In one cycle, a CPU can execute: – Ten adds – Four multiplies – Two divides • Schedule this:
  8. 8. Compiler Works More Demo Jan 2015 8 • Complex joins • Optimizations • e.t.c.
  9. 9. Compiler Works Process Management Jan 2015 9 • Development • Experimentation • Production (export) • Maintenance (import) • Re-experimentation • Reproduction • Re-export
  10. 10. Compiler Works More Demo Jan 2015 10 • Export • What else are you interested in?
  11. 11. Compiler Works Questions Jan 2015 11 • I will answer on stage (if time) • I will answer afterwards (if time) • Email: info@compilerworks.com

×