Data science and cloud computing
•
1 like•425 views
This presentation discusses key topics in data science and cloud computing including: 1. Data storage and processing resources in the cloud that help simplify and reduce costs of data science projects. 2. Machine learning services that provide automated algorithms and managed infrastructure as a service. 3. How cloud computing helps data science practitioners by simplifying access to resources and tools for tasks like data storage, processing, and applying machine learning models in applications through API services.
More Related Content
What's hot
What's hot (20)
Similar to Data science and cloud computing
Similar to Data science and cloud computing (20)
Recently uploaded
Recently uploaded (20)
Data science and cloud computing
- 1. All images in this presentation are subject to copyright and belong to respective Hands-on hack session Data Science & Cloud Computing
- 2. All images in this presentation are subject to copyright and belong to respective DISCLAIMER 2: The opinions expressed in this presentation are my own views and not those of JITHENDRA BALAKRISHNAN Technical Leader, Cloud Product Solutions Head of Technology, 47Line Technologies @jitcompil e /jithendrabalakrishn an DISCLAIMER 1: All copyrights and trademarks of images belong to their respective IP owners and are used under Fair Use for educational
- 3. All images in this presentation are subject to copyright and belong to respective AGENDA Cloud Computing Storage Data Science Compute Learning Hands on Hack
- 4. All images in this presentation are subject to copyright and belong to respectiveAll images in this presentation are subject to copyright and belong to respective Harvard Business Review “Data Scientist: The Sexiest Job of the 21st Century”
- 5. All images in this presentation are subject to copyright and belong to respective Data Science Process
- 6. All images in this presentation are subject to copyright and belong to respectiveAll images in this presentation are subject to copyright and belong to respective Paul Maritz, Pivotal “Cloud is about how you do computing, not where you do computing”
- 7. All images in this presentation are subject to copyright and belong to respective Storage Compute Learning CLOUD COMPUTING SERVICES
- 8. All images in this presentation are subject to copyright and belong to respective AMAZONWEB SERVICES
- 9. All images in this presentation are subject to copyright and belong to respectiveAll images in this presentation are subject to copyright and belong to respective W. Edwards Deming, Scholar & Teacher “In God we trust. All others must bring data”
- 11. All images in this presentation are subject to copyright and belong to respective DATA IS THE NEW OIL Value Variety Velocity Volume
- 12. All images in this presentation are subject to copyright and belong to respective AMAZON S3 Object storage to store and retrieve any amount of data from anywhere. AMAZON REDSHIFT Fully managed petabyte scale data warehouse. AMAZON NEPTUNE Fully managed graph database engine. AMAZON RDS Fully managed relational database service. AMAZON DYNAMODB Fast & Flexible NoSQL database service. AMAZON ELASTICACHE Managed Redis & MemCached as a Service. AMAZON AURORA Fully managed MySQL & PostgreSQL compliant cloud database. AMAZON GLACIER Secure, durable & low cost data archival & long term backup service. AMAZON SIMPLEDB Highly available, secure & inexpensive NoSQL data store.
- 13. All images in this presentation are subject to copyright and belong to respectiveAll images in this presentation are subject to copyright and belong to respective Peter Norvig, Google Research “More data beats clever algorithms, but better data beats more data.”
- 14. All images in this presentation are subject to copyright and belong to respective SCALABLE PROCESSING ELASTICIT Y SCALABILI TY COST
- 15. All images in this presentation are subject to copyright and belong to respective Secure resizable elastic compute capacity in the cloud. EC2 Managed Hadoop framework for easy, fast and cost-effective cluster for processing large amounts of data Interactive SQL query service to analyze data in S3. ATHENA Fully managed ETL service to prepare and load data for analytics. EMRGLUE COMPUTE SERVICES
- 16. All images in this presentation are subject to copyright and belong to respective COST OPTIONS SPOT INSTANCES Spare AWS capacity available at up to 90% discount. Recommended for stateless, low cost and flexible timed applications. RESERVED INSTANCES Provides up to 75% discount on committed usage over 1 or 3 year period. Recommended for Steady state and planned capacity needs. SPOT BLOCK Spare AWS capacity available at up to 40% discount on committed usage of 6 hours. Recommended for low cost, low risk and known duration workloads. 02 03 01
- 17. All images in this presentation are subject to copyright and belong to respectiveAll images in this presentation are subject to copyright and belong to respective Andrew Ng, Chairman, Coursera “Artificial Intelligence is the new Electricity”
- 18. All images in this presentation are subject to copyright and belong to respective
- 19. All images in this presentation are subject to copyright and belong to respective o Machine Learning for everyone o API-driven ML services o GPU Instances o Powerful Compute o FPGA Hardware Acceleration MACHINE LEARNING AS A SERVICE
- 20. All images in this presentation are subject to copyright and belong to respective
- 21. All images in this presentation are subject to copyright and belong to respective
- 22. All images in this presentation are subject to copyright and belong to respective SUMMARY 1 DATA SCIENCE Inter-disciplinary field that involves the entire technology organization 2 CLOUD COMPUTING Helps data science practitioners by simplifying usage of resources & tools 3 DATA STORAGE Data is collected at volume and clear storage plan helps in reducing costs 4 DATA PROCESSING Cheap compute resources helps in cleaning & extracting value from data 5 MACHINE LEARNING Automated algorithms available as service with managed infrastructure 6 MODEL USAGE API services to apply machine learning models in real world applications
Editor's Notes
- Understand audience distribution Set the context Basics of Data Science How Cloud Computing helps doing Data Science
- Introduction Explain cmpute.io & the data science work done there Cisco acquisition of cmpute.io Cisco Disclaimer Image Fair Use Disclaimer
- Agenda for the workshop What data science process looks like How cloud computing has changed the way things are done today Storage concepts from data science perspective Compute specific services for data science ML & Deep Learning Explain one problem of cmpute.io & walk through how it was resolved
- Joke: LinkedIn inclusion of “Data Science” as core skill increased after this article Data Science history Costly and difficult skill Very niche and not available everywhere Increase in data storage increased need to find value in them Data Science is a must have skill in today’s information age
- Explain the Process Continuous Learning model – Similarities to cmpute.io bid model
- Cloud brings the best processes into organization Design for failure Unlimited Scale
- Data Science specific topics in Cloud Storage – Store information Compute – Clean and Process information Learning – Ready to use services for AI & Deep Learning
- Showcasing AWS to demonstrate Cloud Computing Early innovator in Cloud space Has multiple choices of Services for each of the previous areas Fit for Beginners to Expert level Presenter is familiar with this cloud
- Data is the starting point for all analysis Collect as much as you can Collect in native forms and then transpose them for analysis
- Companies now need varied storage choices Structured – Traditional Relational Storage - SQL Unstructured – Modern Storage – NOSQL Graph – Significant focus on Relationships – Social information Time Series – Streaming data – Metrics
- Data is classified based on origin and scale Variety Twitter feed is saved to MongoDB Website form information saved to RDBMS Velocity Downstream mainframes which drop files once a day Twitter sending unending stream of requests for support to company social media handle Volume IOT devices sending many metrics every seconds Leave Management System receiving a few requests per day Value Finding value in all information is the goal of Data Science
- Storage Choices on Amazon Relational Amazon RDS Amazon Aurora Amazon Redshift NoSQL Amazon SimpleDb Amazon DynamoDB UnStructured Amazon S3 Amazon Glacier Graph Amazon Neptune Metrics Amazon CloudWatch Speciality Amazon ElastiCache
- Collecting data is important Processing the collected data to make meaningful training sets is primary Computers work on the principle of GIGO Garbage In Garbage Out Gold In Gold Out
- Cloud Computing solves 3 important needs of data science Elasticity Scale up and down based on your needs Scalability Aim for any size cluster and cloud makes it available Cost Cost conscious computing choices available based on needs
- Basic services for processing and querying large data sets EC2 Write processing and scale based on your own framework EMR Process and scale on top of Hadoop, Pig, Hive models Glue Managed ETL without any code Athena Query data directly without any servers
- Available cost choices Reserved For predictable work loads Spot Block For checkpoint based time limited work loads Spot For interrupt tolerant processing
- Machine Learning became a widely discussed topic due to the free AI course from Coursera.
- Machine Learning, a niche skill has became a common skill due to commoditization and open source
- Ready to use Machine Learning Services and comparison with other clouds. Tensorflow is backed by Google Glucon is deep learning project backed by Amazon & Microsoft MxNet is open source and supported by Amazon Amazon ML has limited choices Built for beginners Proprietary engine Supports only 3 algorithms – Binary Classification, Multi Classification and Regression Amazon SageMaker has both ready made algorithms and support for custom algorithms Built for data scientists Uses TensorFlow and MxNet Azure and GCP have much advanced support for ML, AI and Deep Learning Cognitive services are outside of the scope of this presentation We are focused only on Data Speech, Image and Other recognition services are considered cognitive All clouds offer ready to use services which have advanced automation and are available over API
- Cmpute.io Initial days What we did How we did Issues we faced Why we turned to data science Need for predictions Need for classification How we went about solving our problems Explain flowchart Demo Problem Predict spot prices using historical data Disclaimer: Cmpute.io used multiple sources of data and not just historical information
- A simple Real time Spot prediction System Infrastructure Amazon RDS Aurora – Store information AWS Fargate – Scheduled Container execution AWS S3 – Training data storage AWS ML – Machine Learning model and evaluation AWS Api Gateway – REST Service AWS Lambda – Actual functions for API React – Front end Background Services Spot fetcher – Fetch prices every 5 minutes Training data – Convert daily data into training data every day – 1 file per day Machine Learning Create Data source from S3 Training Data Create Model using Regression Create Evaluation using Model Create Real time API for evaluation API Get Current prices – Fetch information from aws api and save to database Get Prediction – Call AWS ML Real time prediction API Front end Simple grid that shows the matrix of region, availability zone, instance type, platform ,current price and predicted price
- Data science is a inter-disciplinary process that involves the entire organization Cloud computing is here to stay and offers significant advances to the data science process Storage management solutions allows any type of data and is built for volume, variety, velocity Cleaning and extraction brings out value in data Democratization of AI has made it easy for data processing API based models help in real world usage