Successfully reported this slideshow.
Your SlideShare is downloading. ×

How Apache Spark Changed the Way We Hire People with Tomasz Magdanski

Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad

Check these out next

1 of 17 Ad

How Apache Spark Changed the Way We Hire People with Tomasz Magdanski

Download to read offline

As big data technology matures, you’d think there would be more talent available to hire. Although the number of people interested and engaged in the big data world has dramatically increased, job demand is far ahead.

Companies like Google or Facebook have access to the best talent — thousands of engineers with PhDs from the best schools, which is why they are able to innovate. How can a company close the skills gap while innovating and creating product advantage?

This talk highlights how the right technology can allow you to compete without having an army of PhDs at your disposal. At iPass, we’ve created an environment where our engineers can be empowered to create value without getting bogged down by big data and Ops challenges. As a result, we have been able to more easily recruit internal engineers to our big data team, leveraging their current expertise, while bringing them up to speed on big data projects much faster. Join this talk to learn how you can do the same for your organization.

As big data technology matures, you’d think there would be more talent available to hire. Although the number of people interested and engaged in the big data world has dramatically increased, job demand is far ahead.

Companies like Google or Facebook have access to the best talent — thousands of engineers with PhDs from the best schools, which is why they are able to innovate. How can a company close the skills gap while innovating and creating product advantage?

This talk highlights how the right technology can allow you to compete without having an army of PhDs at your disposal. At iPass, we’ve created an environment where our engineers can be empowered to create value without getting bogged down by big data and Ops challenges. As a result, we have been able to more easily recruit internal engineers to our big data team, leveraging their current expertise, while bringing them up to speed on big data projects much faster. Join this talk to learn how you can do the same for your organization.

Advertisement
Advertisement

More Related Content

Slideshows for you (20)

Similar to How Apache Spark Changed the Way We Hire People with Tomasz Magdanski (20)

Advertisement

More from Databricks (20)

Advertisement

How Apache Spark Changed the Way We Hire People with Tomasz Magdanski

  1. 1. Tomasz Magdanski, iPass HOW APACHE SPARK CHANGED THE WAY WE HIRE PEOPLE #EntSAIS17
  2. 2. What if the war for talent ended and your company lost? 2#EntSAIS17 • War for talent – Late ’90s warning from McKinsey about talent shortage – Urged companies to prioritize strategies around recruiting, retaining and developing key employees • One percent problem
  3. 3. Hiring is tough 3#EntSAIS17 Source: edureka
  4. 4. And its going to get worse 4#EntSAIS17 Source: Hour of Code
  5. 5. Since war for talent started we have made a full circle 5#EntSAIS17 • Apart from hiring skilled engineers companies look inside to fill in the gap • Create path to grow within your organization • But wait a minute ? Didn't we just say there is a big skills gap ?
  6. 6. What are we building ? 6#EntSAIS17
  7. 7. Goals • Scalable platform • Cost effective • No data loss • Code portability • Easy R&D • Extendable • Support many languages • Support batch, stream • ML enabled • Collaborative 7#EntSAIS17
  8. 8. Who we were looking for ? • MapReduce • Hadoop / HDFS • Hive / Pig • Storm • Caching • Avro / Parquet • Distributed Computing • Manage Clusters and Infrastructure • Integrate tools • Data Warehousing and Modeling 8#EntSAIS17
  9. 9. Who we were looking for ? • CAP Theorem • Data Transformation • Data Collection • SQL • Cassandra / Hbase / MongoDB / mysql • Kafka • AWS • Scala / Java / Python • Understanding data structures and algorithms • Visualization and Data Analysis • Team player • SPECIFIC INDUSTRY KNOWLEDGE 9#EntSAIS17
  10. 10. Spark changed what we are looking for 10#EntSAIS17
  11. 11. Spark • Simple • Easy to learn • High abstraction API • Build in connectors to major data sources • Supports Batch, Stream • Highly optimized and extendible • ML library to run at scale • Spark provides transactional writes and exact once semantic 11#EntSAIS17
  12. 12. Spark and Databricks • A single platform that unifies data engineering and data science • Automated cluster management / zero-management infrastructure • Intuitive notebooks supporting multiple programming languages • Makes collaboration easy • Blends Data Engineering and Data Science workloads • APIs to integrate with other tools 12#EntSAIS17
  13. 13. Spark and Databricks • Integrated meta store • Integrated Managed and unmanaged tables • Workspace API • Engineering and Customer Support including Solution Architects • Easy dashboards • DbUtils • SBT tools for easy deployment 13#EntSAIS17
  14. 14. Who we are looking for now? • Experience in programming using APIs • Understanding data structures and algorithms • Scala / Java / Python / R • Visualization and Data Analysis • Team player • SPECIFIC INDUSTRY KNOWLEDGE 14#EntSAIS17
  15. 15. Business needs • Wi-Fi connectivity patterns • Our business and customers • Existing system architecture • Skip lengthy onboarding process • One stack to learn 15#EntSAIS17
  16. 16. How did that change the way we hire ? • We have internally hired: – QA engineer – App developers – Ex developer / product manager – Backend engineer • Externally hired: – One very experienced senior Data Engineer – Few collage grads – Junior data engineers 16#EntSAIS17
  17. 17. Summary • Thanks to Databricks we didn’t have to build a platform • Hired mixed of internal and external candidates • Focus on business needs • Created 6 data products • New seven digit revenue stream for our company • Continue to innovate 17#EntSAIS17

×