Uploaded byLINE Corporation
Machine Learning Platform in LINE Fukuoka
Data Labs supports LINE services by performing high-level data analysis and machine learning model development using their Hadoop data lake. The machine learning lifecycle involves many steps beyond just model training, including data collection, preprocessing, deployment, and monitoring. LINE's platform provides the necessary infrastructure to efficiently perform each step of the lifecycle, allowing for rapid continuous development and experimentation through tools like HDFS, Kubernetes, Jupyter notebooks, and CI/CD pipelines.
More Related Content
![[Lakmal] Automate Microservice to API](https://cdn.slidesharecdn.com/ss_thumbnails/lakmalautomatemicroservicetoapi-210512215049-thumbnail.jpg?width=640&height=640&fit=bounds)
![[Spark Summit 2017 NA] Apache Spark on Kubernetes](https://cdn.slidesharecdn.com/ss_thumbnails/apachesparkonkubernetespublic1-170929072840-thumbnail.jpg?width=640&height=640&fit=bounds)
Similar to Machine Learning Platform in LINE Fukuoka
Machine Learning Platform in LINE Fukuoka
- 2. Machine Learning Platform inLINE Fukuoka Kengo Tateishi, LFK Data Labs Team
- 3. LINE Fukuoka DataLabs. FromFukuoka, Japan. 立石 賢吾 TATEISHI KENGO @tkengo @tkengo
- 4. • Introduction toData Labs • Machine learning lifecycle • Platform for machine learning Agenda
- 5.
- 6. • Support ourvarious services from the view point “Data” • High level analysis for the services. • Machine learning model development. • Hadoop clusters as a Data Lake storing huge data. What is Data Labs?
- 7.
- 8. Data Scientist ML Engineer • 2 rolesin LFK Data Labs Team. What is Data Labs?
- 9.
- 10. What is necessaryfor machine learning? Machine learning lifecycle
- 11. • Build amodel to predict something. • Learn a new algorithm to improve accuracy. • Read papers to get more knowledge. • Understand what happens inside of the machine learning model. • And so on… What is necessary for machine learning? Machine learning lifecycle
- 12. • Build amodel to predict something. • Learn a new algorithm to improve accuracy. • Read papers to get more knowledge. • Understand what happens inside of the machine learning model. • And so on… What is necessary for machine learning? ALL STEPS ARE IMPORTANT Machine learning lifecycle
- 13. • Build amodel to predict something. • Learn a new algorithm to improve accuracy. • Read papers to get more knowledge. • Understand what happens inside of the machine learning model. • And so on… What is necessary for machine learning? However… Machine learning lifecycle
- 14. Machine learning lifecycle Thereare so many other things we have to do.
- 15. • Collect dataused for model training. • Prepare a datastore to store the collected data. • Do pre-processing such as cleaning, normalization, completion, and so on. • Create a training environment (Distributed? Single?) • Expose our prediction model as an API. • Build an infrastructure to serve APIs. • Monitor accuracy online and offline regularly. • And others… Machine learning lifecycle There are so many other things we have to do.
- 16.
- 17. Data Collection Data Storing Pre- Processing API Development Deployment Accuracy Monitoring Feedback • Everyone tendsto learn these 2 parts. • But other parts are still important. Basic Flow Feedbac k Basic Flow Data Collection Data Storing Pre- Processing API Development Deployment Accuracy Monitoring Parameter Tuning Model Training Machine learning lifecycle
- 18. Data Collection Data Storing Pre- Processing API Development Deployment Accuracy Monitoring Feedback • Everyone tendsto learn these 2 parts. • But other parts are still important. • In addition… Model Training Parameter Tuning Basic Flow Prepare a storage Code review Test Prepare a dashboard CI Prepare servers Data Collection Data Storing Pre- Processing API Development Deployment Accuracy Monitoring Feedback Basic Flow Feedbac k Basic Flow Data Collection Data Storing Pre- Processing API Development Deployment Accuracy Monitoring Parameter Tuning Model Training Machine learning lifecycle
- 19. Data Collection Data Storing Pre- Processing API Development Deployment Accuracy Monitoring Feedback • Everyone tendsto learn these 2 parts. • But other parts are still important. • In addition… Model Training Parameter Tuning Basic Flow • We need to iterate this process as a lifecycle continuously. Data Collection Data Storing Pre- Processing API Development Deployment Accuracy Monitoring Feedback Model Training Parameter Tuning Basic Flow Prepare a storage Code review Test Prepare a dashboard CI Prepare servers Data Collection Data Storing Pre- Processing API Development Deployment Accuracy Monitoring Feedback Basic Flow Feedbac k Basic Flow Data Collection Data Storing Pre- Processing API Development Deployment Accuracy Monitoring Parameter Tuning Model Training Machine learning lifecycle
- 20.
- 21. As you mayknow. To achieve rapid continuous development in machine learning, we need an environment for machine learning. Platform for machine learning
- 22. HDFS Sqoop Fluentd Flask App Hive PySpark Training Servers Our Platform Application side Coderepository (Github) Object Storage Drone as a CI Jupyter Notebook Pre- Processing (ETL) sklearn Tensorflow Kubernetes Cluster
- 23. HDFS Sqoop Fluentd Jupyter Notebook sklearn Tensorflow Flask App Hive PySpark Kubernetes Cluster Training Servers Our Platform Application side Pre- Processing (ETL) Object Storage Droneas a CI Importing HDFS Sqoop Fluentd Flask Hive PySpark Training Servers Object Storage Drone as a CI Jupyter Notebook Pre- Processing (ETL) sklearn Tensorflow Kubernetes Cluster Code repository (Github)
- 24. HDFS Sqoop Fluentd Jupyter Notebook sklearn Tensorflow Flask App Hive PySpark Kubernetes Cluster Training Servers Our Platform Application side Pre- Processing (ETL) Object Storage Droneas a CI Storing HDFS Sqoop Fluentd Flask Hive PySpark Training Servers Object Storage Drone as a CI Jupyter Notebook Pre- Processing (ETL) sklearn Tensorflow Kubernetes Cluster Importing Code repository (Github)
- 25. HDFS Sqoop Fluentd Jupyter Notebook sklearn Tensorflow Flask App Hive PySpark Kubernetes Cluster Training Servers Our Platform Application side Pre- Processing (ETL) Object Storage Droneas a CI Data understanding Experimental modeling HDFS Sqoop Fluentd Flask Hive PySpark Training Servers Object Storage Drone as a CI Jupyter Notebook Pre- Processing (ETL) sklearn Tensorflow Kubernetes Cluster Storing Importing Code repository (Github)
- 26. HDFS Sqoop Fluentd Jupyter Notebook sklearn Tensorflow Flask App Hive PySpark Kubernetes Cluster Training Servers Our Platform Application side Pre- Processing (ETL) Object Storage Droneas a CI Developing something great… HDFS Sqoop Fluentd Flask Hive PySpark Training Servers Object Storage Drone as a CI Jupyter Notebook Pre- Processing (ETL) sklearn Tensorflow Kubernetes Cluster Data understanding Experimental modeling Storing Importing Code repository (Github)
- 27. HDFS Sqoop Fluentd Jupyter Notebook sklearn Tensorflow Flask App Hive PySpark Kubernetes Cluster Training Servers Our Platform Application side Pre- Processing (ETL) Object Storage Droneas a CI Deployment Deployment Deployment HDFS Sqoop Fluentd Flask Hive PySpark Training Servers Object Storage Drone as a CI Jupyter Notebook Pre- Processing (ETL) sklearn Tensorflow Kubernetes Cluster Data understanding Experimental modeling Storing Importing Code repository (Github)
- 28. HDFS Sqoop Fluentd Jupyter Notebook sklearn Tensorflow Flask App Hive PySpark Kubernetes Cluster Training Servers Our Platform Application side Pre- Processing (ETL) Object Storage Droneas a CI ETL Training HDFS Sqoop Fluentd Flask Hive PySpark Training Servers Object Storage Drone as a CI Jupyter Notebook Pre- Processing (ETL) sklearn Tensorflow Kubernetes Cluster Deployment Deployment Deployment Data understanding Experimental modeling Storing Importing Code repository (Github)
- 29. HDFS Sqoop Fluentd Jupyter Notebook sklearn Tensorflow Flask App Hive PySpark Kubernetes Cluster Training Servers Our Platform Application side Pre- Processing (ETL) Object Storage Droneas a CI Uploading Trained Model HDFS Sqoop Fluentd Flask Hive PySpark Training Servers Object Storage Drone as a CI Jupyter Notebook Pre- Processing (ETL) sklearn Tensorflow Kubernetes Cluster ETL Training Deployment Deployment Deployment Data understanding Experimental modeling Storing Importing Code repository (Github)
- 30. HDFS Sqoop Fluentd Jupyter Notebook sklearn Tensorflow Flask Hive PySpark Kubernetes Cluster Training Servers Our Platform Application side Pre- Processing (ETL) Object Storage Droneas a CI Serving our model HDFS Sqoop Fluentd Flask Hive PySpark Training Servers Object Storage Drone as a CI Jupyter Notebook Pre- Processing (ETL) sklearn Tensorflow Kubernetes Cluster ETL Training Deployment Deployment Deployment Data understanding Experimental modeling Storing Importing Uploading Trained Model Code repository (Github) App
- 31.
- 32. What is necessaryfor machine learning? Summary
- 33. • Of course,we need knowledge about training algorithms, parameter tuning, and so on. • In addition, the platform for machine learning is also important. • It makes our development very rapid, robust, and easy. What is necessary for machine learning? Summary
- 34.