Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Databricks と周辺プロジェクト

23 views

Published on

Databricks と周辺プロジェクト紹介

Published in: Data & Analytics
  • Be the first to comment

  • Be the first to like this

Databricks と周辺プロジェクト

  1. 1. Databricks First Impression 2018.08.26 入江敦央
  2. 2. 2 Python Tableau etc. Keras w/ TensorFlow 2016 keras SOLEIL DATA DOJO Data Visualization
  3. 3. 3 Databricks Databricks
  4. 4. Databricks
  5. 5. 5 Databricks Apache Spark Spark
  6. 6. 6 Apache Spark
  7. 7. 7 Databricks Notebook Jupyter Notebook Jupyter
  8. 8. 8 Databricks Apache Spark Databricks Apache Spark AWS Azure Apache Spark Databricks Apache Spark Databricks Databricks Apache Spark
  9. 9. 9 Databricks Community Edition Databricks AWS/Azure Databricks Community Edition Databricks Notebook
  10. 10. 10 Databricks Notebook
  11. 11. 11 Databricks Notebook Notebook Notebook Community Edition
  12. 12. 12 Databricks Table / DataFrame Tableau BI
  13. 13. 13 Databricks TensorFlow GPU TensorFlow Deep Learning Databricks Runtime 4.1ML Beta TensorFlow keras, sklearn spark-deep-learning Horovod
  14. 14. 14 Horovod Horovod Uber TensorFlow / Keras / PyTorch Open MPI https://www.youtube.com/watch?v=-h0cWBiQ8s8 TensorFlow Dev Dummit 2018 Distributed TensorFlow Horovod TF Horovod mpirun
  15. 15. 15 HorovodEstimator Horovod Spark HorovodEstimator TensorFlow Estimator model_fn HorovodDistributedOptimizer optimizer wrap Horovod HorovodEstimator model_fn https://databricks.com/blog/2018/06/05/distributed-deep-learning-made-simple.html https://docs.databricks.com/applications/deep-learning/distributed-deep-learning/horovod-estimator.html Beta Runtime
  16. 16. 16 Distributed TensorFlow experimental tf.contrib.distribute GPU Horovod TensorFlow Dev Summit https://www.youtube.com/watch?v=-h0cWBiQ8s8 https://github.com/tensor ow/tensor ow/blob/master/tensor ow/contrib/distribute/README.md
  17. 17. Databricks
  18. 18. 18 Project Hydrogen Apache Spark Deep Learning Deep Learning Deep Learning 貢 GPU / FPGA https://www.slideshare.net/databricks/project-hydrogen-stateoftheart-deep-learning-on-apache-spark https://databricks.com/session/databricks-keynote-2 https://www.datanami.com/2018/06/05/project-hydrogen-unites-apache-spark-with-dl-frameworks/ https://www.kdnuggets.com/2018/08/databricks-project-hydrogen-apache-spark.html
  19. 19. 19 MLFlow Databricks MLFlow alpha https://databricks.com/blog/2018/06/05/introducing-ml ow https://databricks.com/blog/2018/08/08/ml ow-0-4-2-released.html
  20. 20. 20 MLFlow https://databricks.com/blog/2018/06/05/introducing-ml ow Tracking Notebook Projects Models 

  21. 21. 21 Hidden Technical Debt in Machine Learning Systems, Google (2015) Databricks
  22. 22. 22 Databricks Databricks Databricks Notebook ETL OK Spark
  23. 23. 23 Databricks AWS / Azure

×