Leveraging Data Driven Research Through Microsoft Azure

•

2 likes•482 views

In this talk I gave an overview of some of the tools that Microsoft Azure offers to researchers. I spoke about Microsoft's Big Data platform, called HDInsight, that allows for creating Spark and Hadoop applications; about Azure ML Studio, a GUI for developing machine learning models very quickly; and about the Data Science Virtual Machine (DSVM), a VM targeted to data scientists and machine learning professionals, which contains all the needed software to create any machine learning system.

Technology

LEVERAGING DATA DRIVEN RESEARCH
THROUGH MICROSOFT AZURE
Dr. Miguel Fierro
Data Scientist at Microsoft
@miguelgfierro
miguel.gonzalezfierro@microsoft.com
https://miguelgfierro.com
Plymouth University | Jan 27, 2017 | Plymouth, UK

AZURE FOR RESEARCH AWARD
Plymouth University January 2017 - Dr. Miguel Fierro @miguelgfierro
azurerfp@microsoft.com
Free Azure resources if awarded
Areas: data science, climate, health…
Ex: Alan Turing Institute got $5M

D a t a S c i e n ce V i r t ua l
M a chi ne
A z u re M L S t u d io
S p a r k a n d H a d o o p
w i t h A z u re
OUTLINE

WHAT IS HDINSIGHT
Plymouth University January 2017 - Dr. Miguel Fierro @miguelgfierro
HDInsight
Managed Service

MANAGER GUI: AMBARI
Plymouth University January 2017 - Dr. Miguel Fierro @miguelgfierro

APACHE HADOOP
Plymouth University January 2017 - Dr. Miguel Fierro @miguelgfierro
Software for storing and analysing
massive amounts (~Tb) of
structured and unstructured data

APACHE SPARK
Plymouth University January 2017 - Dr. Miguel Fierro @miguelgfierro
Framework that runs large-scale data analytics applications
pySpark, Spark (Scala), SparkR
100x faster than Hadoop (processing in memory)

APACHE KAFKA
Plymouth University January 2017 - Dr. Miguel Fierro @miguelgfierro
Stream processing for real time apps
Publisher & subscriber messaging system
Millions of messages per second

APACHE STORM
Plymouth University January 2017 - Dr. Miguel Fierro @miguelgfierro
Distributed framework for real-time applications
ETL, continuous computation, online machine learning
Million of operations per second in each node

APACHE HBASE
Plymouth University January 2017 - Dr. Miguel Fierro @miguelgfierro
Non-relational database (NoSQL) for Big Data applications
Distributed, fast tolerant and scalable
Built on top of HDFS (Hadoop Distributed File System)

APACHE HIVE
Plymouth University January 2017 - Dr. Miguel Fierro @miguelgfierro
SQL-like language to query data in Hadoop systems
Word count program

EXAMPLE OF ARCHITECTURE
Plymouth University January 2017 - Dr. Miguel Fierro @miguelgfierro

DEMO: PYSPARK APPLICATION
Plymouth University January 2017 - Dr. Miguel Fierro @miguelgfierro
Log analysis with PySpark Predictive analysis on food inspection with PySpark
source: https://docs.microsoft.com/en-us/azure/hdinsight/hdinsight-apache-spark-
machine-learning-mllib-ipython
source: https://docs.microsoft.com/en-us/azure/hdinsight/hdinsight-apache-spark-
custom-library-website-log-analysis

WHAT IS AZURE ML STUDIO
Plymouth University January 2017 - Dr. Miguel Fierro @miguelgfierro
GUI for Machine Learning

DATA INPUT/OUTPUT
Plymouth University January 2017 - Dr. Miguel Fierro @miguelgfierro

DATA TRANSFORMATION
Plymouth University January 2017 - Dr. Miguel Fierro @miguelgfierro

DATA MANIPULATION
Plymouth University January 2017 - Dr. Miguel Fierro @miguelgfierro

FEATURE SELECTION
Plymouth University January 2017 - Dr. Miguel Fierro @miguelgfierro

CLASSIFICATION & REGRESSION
Plymouth University January 2017 - Dr. Miguel Fierro @miguelgfierro

TRAINING & SCORING
Plymouth University January 2017 - Dr. Miguel Fierro @miguelgfierro

PYTHON & R SCRIPTS
Plymouth University January 2017 - Dr. Miguel Fierro @miguelgfierro

AUTOMATIC API
Plymouth University January 2017 - Dr. Miguel Fierro @miguelgfierro

DEMO: CREDIT RISK ANOMALY DETECTION
Plymouth University January 2017 - Dr. Miguel Fierro @miguelgfierro
source: https://gallery.cortanaintelligence.com/Experiment/1219e87f8fb84e88a2e1b54256808bb3

WHAT IS THE DSVM
Plymouth University January 2017 - Dr. Miguel Fierro @miguelgfierro
Windows:
- Anaconda with python Jupyter notebooks
- Microsoft R Server
- Visual Studio
- SQL Server
- Azure SDK
- Deep learning: CNTK & MXNet
- Machine Learning: XGBoost
Linux:
- Anaconda with python Jupyter notebooks
- Microsoft R Server
- PyCharm
- Azure SDK
- Deep learning: CNTK & MXNet
- Machine Learning: XGBoost, Weka

DEEP LEARNING DSVM
Plymouth University January 2017 - Dr. Miguel Fierro @miguelgfierro
Libs:
- CNTK
- MXNet
- TensorFlow
- Keras
Digit recognition Image recognitionExamples:

NVIDIA TESLA K80
Plymouth University January 2017 - Dr. Miguel Fierro @miguelgfierro

AI LANDSCAPE: IMAGES
Plymouth University January 2017 - Dr. Miguel Fierro @miguelgfierro
15.4%
7.3%
6.7%
3.6%
3.1%
5.1% (human)
error (%)
ImageNet (image recognition competition) top-5 error
AlexNet
(2012)
VGG
(2014)
Inception
(2015)
ResNet
(2015)
Inception-
ResNet
(2016)

AI LANDSCAPE: SPEECH
Plymouth University January 2017 - Dr. Miguel Fierro @miguelgfierro
Microsoft Research achieves parity with
human speech level
source: http://blogs.microsoft.com/next/2016/10/18/historic-achievement-microsoft-researchers-reach-human-parity-conversational-speech-recognition
CNN
(VGG, ResNet, LACE)
RNN
(Bi-LSTM)
Multi-GPU and multi server
(1-bit Stochastic Gradient Descent)

IMAGE CLASSIFICATION
Plymouth University January 2017 - Dr. Miguel Fierro @miguelgfierro
1.
2.
3.
4.
5.
source: https://blogs.technet.microsoft.com/machinelearning/2016/11/15/imagenet-deep-neural-
network-training-using-microsoft-r-server-and-azure-gpu-vms/

IMAGE CLASSIFICATION IMAGENET
Plymouth University January 2017 - Dr. Miguel Fierro @miguelgfierro
source: https://blogs.technet.microsoft.com/machinelearning/2016/11/15/imagenet-deep-neural-network-training-using-microsoft-r-server-and-azure-gpu-vms/
Real class
Predicted class

TEXT CLASSIFICATION
Plymouth University January 2017 - Dr. Miguel Fierro @miguelgfierro
Train
Backend
Dataset
Azure NC24 VM with 4 K80 GPUs
.R
model.params
Azure Cloud Services
.py
.js
.html
Score
Web app
API
DNN
input text

DEMO: TEXT CLASSIFICATION WEB APP
Plymouth University January 2017 - Dr. Miguel Fierro @miguelgfierro

This document discusses techniques for improving machine learning model predictions, including stacking and hyper-parameter tuning. It uses an animal shelter data set to predict adoption outcomes as an example. Basic feature engineering is applied to reformat predictors before using common machine learning techniques like random forests. More advanced methods like model stacking and hyper-parameter tuning are shown to further improve predictions. H2O is presented as a tool for scalable machine learning that supports these advanced methods.

Jeremy_Wan_Resume

Jeremy Wan

Jeremy Wan is a 2017 graduate of UC Berkeley with a B.S. in Electrical Engineering and Computer Science. He has experience in information security, Python and Java programming, and robotics research. His career experiences include roles as an Information Security Consultant for UC Berkeley Residential Computing and as a Python Developer for the UC Berkeley Psychology Department. He is currently involved in tensegrity robotics research with Berkeley Emergent Space Technologies.

KunGao_Resume.

Gao Kun

Kun Gao is seeking a full time job in computer science or a related field. He has a Master's degree in Electrical and Computer Engineering from UC Santa Barbara and a Bachelor's degree from Ohio State University. For internships, he completed research on die-to-die interconnections at AMD China and built models for interconnection simulation. Some of his class projects include building a fault tolerant secure data storage system on AWS and developing a multi-person chat room web application using Java and sockets.

Scalable Machine Learning

Mikio L. Braun

The document discusses scalable machine learning techniques for analyzing large datasets. It explains that while parts of the machine learning pipeline like data preparation are easily parallelizable, training steps involving gradient descent are more difficult to parallelize. However, there are approaches for scalable training such as stochastic gradient descent, parameter servers, and feature hashing that approximate the model to make distributed optimization feasible. The key aspects of scalable machine learning involve faster learning algorithms, approximating the optimization problem and features, and asynchronous distributed techniques rather than just relying on parallelization alone.

Graph Computing with JanusGraph

Jason Plurad

Start Flying with Python & Apache TinkerPop

Jason Plurad

This document summarizes a presentation about using Python and Apache TinkerPop to work with graph databases. It discusses Gremlin, a graph traversal language, and how Gremlin has been incorporated into Python through Gremlin-Python. It provides an example of building a small web application and APIs to work with an air routes graph stored in a graph database, and deploying the application and database to the cloud.

SDCSB CYTOSCAPE AND NETWORK ANALYSIS WORKSHOP at Sanford Consortium

Keiichiro Ono

This document provides an overview and update on Cytoscape, an open source platform for biological network analysis and visualization. Key points discussed include: - Cytoscape 3.2.1 is the latest desktop application release with new features like a chart editor and exporting visualizations as web applications. - Cytoscape.js is a JavaScript library for building web applications that visualize networks, and there are examples of web apps built with it. - Cytoscape's cyberinfrastructure initiative aims to make the software more accessible and integratable for computational biologists through services, apps, and repositories.

Resume_Weixiang Ding

Weixiang Ding

Weixiang Ding is seeking a full-time position in software engineering or data science starting in Fall 2017. He has a Master's degree in Computer Science from Carnegie Mellon University and a Bachelor's degree from Sun Yat-sen University in China. He has internship experience at Microsoft Corporation and research experience at Brigham Young University. His skills include programming languages like Java, C#, Python, and MATLAB.

Graph databases are relative newcomers in the NoSQL database landscape. What are some graph model and design considerations when choosing a graph database in your architecture? Let's take a tour of a couple graph use cases that we've collaborated on recently with our clients to help you better understand how and why a graph database can be integrated to help solve problems found with connected data. Presented at DataWorks Summit San Jose - IBM Meetup on June 18, 2018. https://www.meetup.com/BigDataDevelopers/events/251307524/

Python in Data Science Work

Rick. Bahague

Official resume titash_mandal_

Titash Mandal

Titash Mandal is a graduate student pursuing a Master's degree in Computer Science from New York University with extensive experience in programming, technical skills, and academic projects. He has worked as a teaching assistant and lead instructor for K-12 STEM programs where he designed curriculum and taught topics in computer science and cybersecurity. His research experience includes developing algorithms for image analysis and retinal disease classification.

Introduction to Data Science

Ncib Lotfi

This document provides an introduction to data science. It discusses the rapid growth of data and defines data science as extracting insights from vast amounts of data using scientific methods. The document outlines the typical steps in the data science process: acquire, prepare, analyze, report and act on data. It also discusses career opportunities in data science and common tools used, including programming languages, mathematics/statistics foundations and visualization/modeling tools.

Project 03 Student Data Files

Angela Edel

Neu-IR 2017: welcome

Bhaskar Mitra

The document welcomes attendees to Neu-IR'17, the Second SIGIR Workshop on Neural Information Retrieval. It provides details on the organizers and notes that almost 1 in 4 papers at SIGIR 2017 were related to neural IR, showing the growth of interest in the topic. Statistics on the Neu-IR'17 workshop are given, including 178 registrations and a 76% acceptance rate for the 19 accepted papers out of 25 submissions. The day will include oral presentations, posters, panels and discussions to discuss, share and learn about neural information retrieval.

Python PPT

Edureka!

Python for Big Data Analytics

Edureka!

This document provides an overview and objectives of a Python course for big data analytics. It discusses why Python is well-suited for big data tasks due to its libraries like PyDoop and SciPy. The course includes demonstrations of web scraping using Beautiful Soup, collecting tweets using APIs, and running word count on Hadoop using Pydoop. It also discusses how Python supports key aspects of data science like accessing, analyzing, and visualizing large datasets.

Delivering Agile Data Science on Openshift - Red Hat Summit 2019

John Archer

Resume Yash Tanna

Yash Tanna

Yash Tanna is a Masters student in Computer Science at USC with a 3.35 GPA. He has experience in programming languages like Java, Python, and C++. Some of his projects include building a popular movies Android app using an API, creating a web app to analyze emotions using video and Microsoft's Emotion API, and developing a predictive engine to suggest careers based on a user's online behavior. Currently he works as a Student Technical Assistant at USC ITS providing tech support. Past experience includes internships building a mobile event app and a task management portal.

Tds — big science dec 2021

Gérard Dupont

BigScience is a one-year research workshop involving over 800 researchers from 60 countries to build and study very large multilingual language models and datasets. It was granted 5 million GPU hours on the Jean Zay supercomputer in France. The workshop aims to advance AI/NLP research by creating shared models and data as well as tools for researchers. Several working groups are studying issues like bias, scaling, and engineering challenges of training such large models. The first model, T0, showed strong zero-shot performance. Upcoming work includes further model training and papers.

Living Outside the Comfort Zone - Daron green florianopolis 5-7-2014

Microsoft Azure for Research

How Cloud is Affecting Data Scientists

CCG

This document discusses how data science models have transitioned to the cloud to take advantage of greater computing resources. It notes that data science models are resource-intensive and traditionally required powerful local machines. The cloud allows data scientists to run models on cloud infrastructure for lower costs than high-end laptops and with access to many GPUs. Several major cloud platforms - Azure, AWS, and Google Cloud - are discussed and compared in terms of their machine learning offerings. The document also introduces Microsoft's Team Data Science Process, which aims to help data science teams collaborate more effectively on projects in the cloud.

Python webinar 4th june

Edureka!

Programmers love Python because of how fast and easy it is to use. Python cuts development time in half with its simple to read syntax and easy compilation feature. Debugging your programs is a breeze in Python with its built in debugger. Using Python makes Programmers more productive and their programs ultimately better. Python is continued to be a favourite option for data scientists who use it for building and using Machine learning applications and other scientific computations. Python runs on Windows, Linux/Unix, Mac OS and has been ported to Java and .NET virtual machines. Python is free to use, even for the commercial products, because of its OSI-approved open source license. Python has evolved as the most preferred Language for Data Analytics and the increasing search trends on python also indicates that Python is the next "Big Thing" and a must for Professionals in the Data Analytics domain.

Big(ger) Data in Software Engineering

Mehdi Mirakhorli

Data has always been used in every company irrespective of its domain to improve the operational efficiency and the products themselves. However, analyzing and extracting information from “Big Data” is the next revolution in technology, since previously unknown nuggets of information are now made visible. In fact, over 90% of the data available in the world has been generated in the last two years. “Big Data” analytics has become the next hot topic for most companies - from financial institutions to technology companies to service providers. Likewise in software engineering, data collected about the development of software, the operation of the software in the field, and the users feedback on software have been used before. However, collecting and analyzing this information across hundreds of thousands or millions of software projects gives us the unique ability to reason about the ecosystem at large, and software in general. At no time in history has there been easier access to extremely powerful computational resources as it is today, thanks to the advances in cloud computing, both from the technology and business perspectives. Therefore, it is easier today than ever before to analyze big data. In this technical briefing, we will present the state-of-the-art with respect to the research carried out in the area of big data analytics in software engineering research. We will present the research along three dimensions: 1) What are the software engineering problems being solved? Examples of problems include: How much source code is newly written and how much is reused from past projects? Can we recommend best practices to developers by observing the development of software among hundreds of thousands of software projects? 2) What are the datasets that are being used? Examples of my datasets include: all the mobile apps in the Google Play store, all of the world's Open Source projects, and hundreds of gigabytes of execution logs. Such large datasets provide us with a unique view into the SE field. 3) What are the tools and techniques available to analyze the large datasets? We intend to present generic software solutions that have been applied to big datasets in other areas of research, and the tools and techniques created by software engineering researchers. In the end we will present the challenges inherently present in large datasets - volume, variety, velocity, and veracity. Such challenges often complicate the analysis of the data and can invalidate the interpretation of the results. We will conclude with the future opportunities that are present in big data analytics for software engineering research.

President Election of Korea in 2017

Jongwook Woo

The Future of Data Science

DataWorks Summit

Resume prashant

Prashant Kumar

Prashant Kumar is a software engineer with a Master's degree in Computer Science from Iowa State University and a Bachelor's degree in Information Technology from GGSIP University in New Delhi, India. He has over 4 years of experience as a software engineer at companies including PlaceIQ, Google India, and Pearson Education. His technical skills include programming languages like Java, Python, and JavaScript as well as tools such as Google Cloud, Hadoop, Docker, and AWS. He has worked on projects involving data processing, web development, mobile applications, and machine learning.

Kaushik shakkari internship - resume

Kaushik Shakkari

Kaushik Shakkari's resume summarizes his education and qualifications for data science roles. He received a Master's in Computer Science from USC with a focus on data science and holds a B.Tech in Computer Science from Amrita University in India. His skills include Python, Java, SQL, Hadoop, and data visualization tools like Tableau. He has research experience analyzing user browsing behavior and work experience with grading, database design, and a big data project using Hadoop. His projects involve predicting customer churn, data cleaning, and data analysis of weather datasets.

Eecs6893 big dataanalytics-lecture1

Aravindharamanan S

This document provides an overview of a lecture on big data analytics given by Dr. Ching-Yung Lin. The key points covered in the lecture include: - Definitions and characteristics of big data based on the 3V's of volume, velocity and variety. - Techniques used for big data such as massive parallelism, distributed storage and processing, machine learning and data visualization. - Factors that have enabled big data to become prominent in recent years like greater data collection, open source software and commodity hardware. - Examples of big data platforms, databases and analytics techniques including Hadoop, Spark, NoSQL databases and graph databases. - The large and growing market for big data

Presentation on Big Data Hadoop (Summer Training Demo)

Ashok Royal

This document summarizes a practical training presentation on Big Data Hadoop. It was presented by Ashutosh Tiwari and Ashok Rayal from Poornima Institute of Engineering & Technology, Jaipur under the guidance of Dr. E.S. Pilli from MNIT Jaipur. The training took place from May 28th to July 9th 2014 at MNIT Jaipur and consisted of studying Hadoop and related papers, building a Hadoop cluster, and implementing a near duplicate detection project using Hadoop MapReduce. The near duplicate detection project aimed to comparatively analyze documents to find similar ones based on a predefined threshold. Snapshots of the HDFS, MapReduce processing, and output of the project are

Resume Diego Marinho de Oliveira

Diego Marinho de Oliveira

This resume summarizes Diego Marinho de Oliveira's experience as a data scientist and AI manager. He has worked at SEEK, Catho Online, RBS Group's Appus startup, Zunnit Technologies, Visagio, and TOTVS in data science roles, accumulating many successfully delivered projects involving machine learning, natural language processing, and recommender systems. He leads a team of global data scientists at SEEK and has contributed exceptional results improving AI to enhance users' experiences.

What's hot

Exploring Graph Use Cases with JanusGraph

Jason Plurad

Python in Data Science Work

Rick. Bahague

Official resume titash_mandal_

Titash Mandal

Introduction to Data Science

Ncib Lotfi

Project 03 Student Data Files

Angela Edel

Neu-IR 2017: welcome

Bhaskar Mitra

What's hot (6)

Exploring Graph Use Cases with JanusGraph

Python in Data Science Work

Official resume titash_mandal_

Introduction to Data Science

Project 03 Student Data Files

Neu-IR 2017: welcome

Similar to Leveraging Data Driven Research Through Microsoft Azure

Python PPT

Edureka!

Python for Big Data Analytics

Edureka!

Delivering Agile Data Science on Openshift - Red Hat Summit 2019

John Archer

Resume Yash Tanna

Yash Tanna

Tds — big science dec 2021

Gérard Dupont

Living Outside the Comfort Zone - Daron green florianopolis 5-7-2014

Microsoft Azure for Research

How Cloud is Affecting Data Scientists

CCG

Python webinar 4th june

Edureka!

Big(ger) Data in Software Engineering

Mehdi Mirakhorli

President Election of Korea in 2017

Jongwook Woo

The Future of Data Science

DataWorks Summit

Resume prashant

Prashant Kumar

Kaushik shakkari internship - resume

Kaushik Shakkari

Eecs6893 big dataanalytics-lecture1

Aravindharamanan S

Presentation on Big Data Hadoop (Summer Training Demo)

Ashok Royal

Resume Diego Marinho de Oliveira

Diego Marinho de Oliveira

Detailed presentation on big data hadoop +Hadoop Project Near Duplicate Detec...

Ashok Royal

Webinar: Mastering Python - An Excellent tool for Web Scraping and Data Anal...

Edureka!

Maximising (Re)Usability of Resources using Linked Data

Asuncion Gomez-Perez

BUILDING BETTER PREDICTIVE MODELS WITH COGNITIVE ASSISTANCE IN A DATA SCIENCE...

Alex Liu

Similar to Leveraging Data Driven Research Through Microsoft Azure (20)

Python PPT

Python for Big Data Analytics

Delivering Agile Data Science on Openshift - Red Hat Summit 2019

Resume Yash Tanna

Tds — big science dec 2021

Living Outside the Comfort Zone - Daron green florianopolis 5-7-2014

How Cloud is Affecting Data Scientists

Python webinar 4th june

Big(ger) Data in Software Engineering

President Election of Korea in 2017

The Future of Data Science

Resume prashant

Kaushik shakkari internship - resume

Eecs6893 big dataanalytics-lecture1

Presentation on Big Data Hadoop (Summer Training Demo)

Resume Diego Marinho de Oliveira

Detailed presentation on big data hadoop +Hadoop Project Near Duplicate Detec...

Webinar: Mastering Python - An Excellent tool for Web Scraping and Data Anal...

Maximising (Re)Usability of Resources using Linked Data

BUILDING BETTER PREDICTIVE MODELS WITH COGNITIVE ASSISTANCE IN A DATA SCIENCE...

More from Miguel González-Fierro

Los retos de la inteligencia artificial en la sociedad actual

Miguel González-Fierro

Knowledge Graph Recommendation Systems For COVID-19

Miguel González-Fierro

The objective of this tutorial is to give the audience hands-on experience to work through the basics about knowledge graph and recommender technology, and how to use them for building an article recommender for COVID-19 research. Particularly, we used a dataset of COVID research-related papers. We consider two tasks, author to paper recommendation and paper to paper recommendation. This work is interesting in the sense that it touches several areas of ML such as recommendation systems, knowledge graphs and NLP. This work has been done with my amazing collaborators Zhihong (Iris) Shen, Jianxun Lian and Le Zhang. The talk was presented at the Toronto Machine Learning Summit on November 16th 2020

Thesis dissertation: Humanoid Robot Control of Complex Postural Tasks based o...

Miguel González-Fierro

The document discusses control of complex postural tasks for humanoid robots based on learning from demonstration. It covers several topics: imitation learning and skill innovation for single skills; planning and control of sequential skills; postural control using fractional calculus controllers; and high-level control architecture for executing complex tasks. The key contributions are in postural motion planning and control using reward functions; modeling sequential skills as reward transition matrices; and applying fractional control to reduce model disturbances. Future work areas include goal emulation, reward profile selection using machine learning, and implementing fractional control on a real humanoid.

Best practices in coding for beginners

Miguel González-Fierro

Distributed training of Deep Learning Models

Miguel González-Fierro

In the last year there have been a number of attempts to train deep CNNs on the ImageNet dataset in the shortest time possible, with the most recent attempt managing to do it in 15 minutes. All of these attempts happen on custom clusters which are out of the reach of most data scientists. One of the key advantages of the cloud is being able to scale out compute resources as required. In this talk we will present two platforms for running distributed deep learning in the cloud which are within the reach of every data scientist. The first is a service called Batch AI which uses the Azure Batch infrastructure to easily run Deep Learning jobs at scale across GPUs. The second is an open source toolkit that allows data scientists to spin up clusters in turn-key fashion. It utilises Kubernetes and Grafana for easy job scheduling and monitoring. It has been used in daily production for Microsoft internal groups. Both utilise Docker containers making it possible to run any deep learning framework on them. We will use the aforementioned training platforms to train a ResNet network on ImageNet dataset using each of the following frameworks: CNTK, Tensorflow (Horovod), PyTorch, MxNet and Chainer. We will then compare and contrast the performance improvement as we scale the number of nodes as well as provides tips and details of the pitfalls of each framework and platform. The examples presented can also be used as templates so that others can utilise these for their own deep learning problems. Presented at Strata London 2018: https://conferences.oreilly.com/strata/strata-eu/public/schedule/detail/65426

Running Intelligent Applications inside a Database: Deep Learning with Python...

Miguel González-Fierro

In this talk we present a new paradigm of computation where the intelligence is computed inside the database. Standard software systems must get the data from the database to execute a routine. If the size of the data is big, there are inefficiencies due to the data movement. Store procedures tried to solve this issue in the past, allowing for computing simple functions inside the database. However, only simple routines can be executed. To showcase the capabilities of our new system, we created a lung cancer detection algorithm using Microsoft’s Cognitive Toolkit, also known as CNTK. We used transfer learning between ImageNet dataset, which contains natural images, and a lung cancer dataset, which contains scans of horizontal sections of the lung for healthy and sick patients. Specifically, a pretrained Convolutional Neural Network on ImageNet is used on the lung cancer dataset to generate features. Once the features are computed, a boosted tree is applied to predict whether the patient has cancer or not. All this process is computed inside the database, so the data movement is minimized. We are even able to execute the algorithm using the GPU of the virtual machine that hosts the database. Using a GPU, we can compute the featurization in less than 1h, in contrast to using a CPU, that would take up to 32h. Finally, we set up an API to connect the solution to a web app, where a doctor can analyze the images and get a prediction of a patient.

Deep Learning for Sales Professionals

Miguel González-Fierro

AI is considered the biggest commercial opportunity in the changing economy. It is expected to generate $15.7T by 2030 (according to PwC). In this talk I explained what are some of the top business opportunities for cloud sales professionals in the field of deep learning and artificial intelligence. I addressed the most important principles of deep learning techniques and finally, I studied two deep learning problems (image recognition with CNNs and sentiment analysis with RNNs) with 5 different deep learning frameworks.

Deep Learning for Lung Cancer Detection

Miguel González-Fierro

Since its launch in mid-January, the Data Science Bowl Lung Cancer Detection Competition has attracted more than 1,000 submissions. To be successful in this competition, data scientists need to be able to get started quickly and make rapid iterative changes. In this talk, we show how to compute features of the scanned images in the competition with a pre-trained Convolutional Neural Network (CNN) with Cognitive Toolkit (previously named CNTK), and use these features to classify the scans into cancerous or not cancerous, using a boosted tree with Light-GBM library, all in one hour. Blog post: https://blogs.technet.microsoft.com/machinelearning/2017/02/17/quick-start-guide-to-the-data-science-bowl-lung-cancer-detection-challenge-using-deep-learning-microsoft-cognitive-toolkit-and-azure-gpu-vms/

Mastering Computer Vision Problems with State-of-the-art Deep Learning

Miguel González-Fierro

Deep learning has been especially successful in computer vision tasks such as image classification because convolutional neural nets (CNNs) can create hierarchical representations in an image. One of the most remarkable advances is ResNet, the CNN that surpassed human-level accuracy for the first time in history. ImageNet competition has become the de facto benchmark for image classification in the research community. The “small” ImageNet data contains more than 1.2 million images distributed in 1,000 classes. Miguel González-Fierro explains how to train a state of the art deep neural network, ResNet, using Microsoft RSever and MXNet with the ImageNet dataset. (While most of the deep learning libraries are programmed in C++ and Python, only MXNet offers an API for R programmers.) Miguel then demonstrates how to operationalize this training for real-world business problems related to image classification. This talk was presented at Strata London 2017: https://conferences.oreilly.com/strata/strata-eu/public/schedule/detail/57428

Speeding up machine-learning applications with the LightGBM library

Miguel González-Fierro

This document compares the XGBoost and LightGBM machine learning libraries and discusses their applications and ability to handle concept drift. It finds that LightGBM typically trains models faster than XGBoost and performs competitively or better on several datasets. The document explores concept drift, how it can occur gradually, suddenly, or cyclically over time, and presents examples of conceptual drift in BCI, flight delay, and football datasets and how the two libraries handle drift.

Empowering every person on the planet to achieve more

Miguel González-Fierro

1. Dr. Miguel Fierro gave an industry talk at King's College London on November 30, 2016 about his work as a data scientist at Microsoft. 2. He discussed recent advances in deep learning and machine learning, including convolutional neural networks and their application to image recognition, sentiment analysis, and speech recognition. 3. He also talked about using cloud computing resources from Microsoft Azure to efficiently train large neural networks on big data and deploy intelligent applications.

Deep Learning for NLP

Miguel González-Fierro

More from Miguel González-Fierro (12)

Los retos de la inteligencia artificial en la sociedad actual

Knowledge Graph Recommendation Systems For COVID-19

Thesis dissertation: Humanoid Robot Control of Complex Postural Tasks based o...

Best practices in coding for beginners

Distributed training of Deep Learning Models

Running Intelligent Applications inside a Database: Deep Learning with Python...

Deep Learning for Sales Professionals

Deep Learning for Lung Cancer Detection

Mastering Computer Vision Problems with State-of-the-art Deep Learning

Speeding up machine-learning applications with the LightGBM library

Empowering every person on the planet to achieve more

Deep Learning for NLP

Recently uploaded

How to use Firebase Data Connect For Flutter

Daiki Mogmet Ito

Artificial Intelligence for XMLDevelopment

Octavian Nadolu

In the rapidly evolving landscape of technologies, XML continues to play a vital role in structuring, storing, and transporting data across diverse systems. The recent advancements in artificial intelligence (AI) present new methodologies for enhancing XML development workflows, introducing efficiency, automation, and intelligent capabilities. This presentation will outline the scope and perspective of utilizing AI in XML development. The potential benefits and the possible pitfalls will be highlighted, providing a balanced view of the subject. We will explore the capabilities of AI in understanding XML markup languages and autonomously creating structured XML content. Additionally, we will examine the capacity of AI to enrich plain text with appropriate XML markup. Practical examples and methodological guidelines will be provided to elucidate how AI can be effectively prompted to interpret and generate accurate XML markup. Further emphasis will be placed on the role of AI in developing XSLT, or schemas such as XSD and Schematron. We will address the techniques and strategies adopted to create prompts for generating code, explaining code, or refactoring the code, and the results achieved. The discussion will extend to how AI can be used to transform XML content. In particular, the focus will be on the use of AI XPath extension functions in XSLT, Schematron, Schematron Quick Fixes, or for XML content refactoring. The presentation aims to deliver a comprehensive overview of AI usage in XML development, providing attendees with the necessary knowledge to make informed decisions. Whether you’re at the early stages of adopting AI or considering integrating it in advanced XML development, this presentation will cover all levels of expertise. By highlighting the potential advantages and challenges of integrating AI with XML development tools and languages, the presentation seeks to inspire thoughtful conversation around the future of XML development. We’ll not only delve into the technical aspects of AI-powered XML development but also discuss practical implications and possible future directions.

“Building and Scaling AI Applications with the Nx AI Manager,” a Presentation...

Edge AI and Vision Alliance

For the full video of this presentation, please visit: https://www.edge-ai-vision.com/2024/06/building-and-scaling-ai-applications-with-the-nx-ai-manager-a-presentation-from-network-optix/ Robin van Emden, Senior Director of Data Science at Network Optix, presents the “Building and Scaling AI Applications with the Nx AI Manager,” tutorial at the May 2024 Embedded Vision Summit. In this presentation, van Emden covers the basics of scaling edge AI solutions using the Nx tool kit. He emphasizes the process of developing AI models and deploying them globally. He also showcases the conversion of AI models and the creation of effective edge AI pipelines, with a focus on pre-processing, model conversion, selecting the appropriate inference engine for the target hardware and post-processing. van Emden shows how Nx can simplify the developer’s life and facilitate a rapid transition from concept to production-ready applications.He provides valuable insights into developing scalable and efficient edge AI solutions, with a strong focus on practical implementation.

Microsoft - Power Platform_G.Aspiotis.pdf

Uni Systems S.M.S.A.

Goodbye Windows 11: Make Way for Nitrux Linux 3.5.0!

SOFTTECHHUB

As the digital landscape continually evolves, operating systems play a critical role in shaping user experiences and productivity. The launch of Nitrux Linux 3.5.0 marks a significant milestone, offering a robust alternative to traditional systems such as Windows 11. This article delves into the essence of Nitrux Linux 3.5.0, exploring its unique features, advantages, and how it stands as a compelling choice for both casual users and tech enthusiasts.

Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slack

shyamraj55

GraphSummit Singapore | Graphing Success: Revolutionising Organisational Stru...

Neo4j

Sudheer Mechineni, Head of Application Frameworks, Standard Chartered Bank Discover how Standard Chartered Bank harnessed the power of Neo4j to transform complex data access challenges into a dynamic, scalable graph database solution. This keynote will cover their journey from initial adoption to deploying a fully automated, enterprise-grade causal cluster, highlighting key strategies for modelling organisational changes and ensuring robust disaster recovery. Learn how these innovations have not only enhanced Standard Chartered Bank’s data infrastructure but also positioned them as pioneers in the banking sector’s adoption of graph technology.

Observability Concepts EVERY Developer Should Know -- DeveloperWeek Europe.pdf

Paige Cruz

Monitoring and observability aren’t traditionally found in software curriculums and many of us cobble this knowledge together from whatever vendor or ecosystem we were first introduced to and whatever is a part of your current company’s observability stack. While the dev and ops silo continues to crumble….many organizations still relegate monitoring & observability as the purview of ops, infra and SRE teams. This is a mistake - achieving a highly observable system requires collaboration up and down the stack. I, a former op, would like to extend an invitation to all application developers to join the observability party will share these foundational concepts to build on:

みなさんこんにちはこれ何文字まで入るの？40文字以下不可とか本当に意味わからないけどこれ限界文字数書いてないからマジでやばい文字数いけるんじゃないの？えこ...

名前です男

20240605 QFM017 Machine Intelligence Reading List May 2024

Matthew Sinclair

GenAI Pilot Implementation in the organizations

kumardaparthi1024

Mariano G Tinti - Decoding SpaceX

Mariano Tinti

Serial Arm Control in Real Time Presentation

tolgahangng

UiPath Test Automation using UiPath Test Suite series, part 6

DianaGray10

Welcome to UiPath Test Automation using UiPath Test Suite series part 6. In this session, we will cover Test Automation with generative AI and Open AI. UiPath Test Automation with generative AI and Open AI webinar offers an in-depth exploration of leveraging cutting-edge technologies for test automation within the UiPath platform. Attendees will delve into the integration of generative AI, a test automation solution, with Open AI advanced natural language processing capabilities. Throughout the session, participants will discover how this synergy empowers testers to automate repetitive tasks, enhance testing accuracy, and expedite the software testing life cycle. Topics covered include the seamless integration process, practical use cases, and the benefits of harnessing AI-driven automation for UiPath testing initiatives. By attending this webinar, testers, and automation professionals can gain valuable insights into harnessing the power of AI to optimize their test automation workflows within the UiPath ecosystem, ultimately driving efficiency and quality in software development processes. What will you get from this session? 1. Insights into integrating generative AI. 2. Understanding how this integration enhances test automation within the UiPath platform 3. Practical demonstrations 4. Exploration of real-world use cases illustrating the benefits of AI-driven test automation for UiPath Topics covered: What is generative AI Test Automation with generative AI and Open AI. UiPath integration with generative AI Speaker: Deepak Rai, Automation Practice Lead, Boundaryless Group and UiPath MVP

GraphSummit Singapore | Neo4j Product Vision & Roadmap - Q2 2024

Neo4j

Building Production Ready Search Pipelines with Spark and Milvus

Zilliz

20240607 QFM018 Elixir Reading List May 2024

Matthew Sinclair

TrustArc Webinar - 2024 Global Privacy Survey

TrustArc

How does your privacy program stack up against your peers? What challenges are privacy teams tackling and prioritizing in 2024? In the fifth annual Global Privacy Benchmarks Survey, we asked over 1,800 global privacy professionals and business executives to share their perspectives on the current state of privacy inside and outside of their organizations. This year’s report focused on emerging areas of importance for privacy and compliance professionals, including considerations and implications of Artificial Intelligence (AI) technologies, building brand trust, and different approaches for achieving higher privacy competence scores. See how organizational priorities and strategic approaches to data security and privacy are evolving around the globe. This webinar will review: - The top 10 privacy insights from the fifth annual Global Privacy Benchmarks Survey - The top challenges for privacy leaders, practitioners, and organizations in 2024 - Key themes to consider in developing and maintaining your privacy program

Uni Systems Copilot event_05062024_C.Vlachos.pdf

Uni Systems S.M.S.A.

Presentation of the OECD Artificial Intelligence Review of Germany

innovationoecd

Recently uploaded (20)

How to use Firebase Data Connect For Flutter

Artificial Intelligence for XMLDevelopment

“Building and Scaling AI Applications with the Nx AI Manager,” a Presentation...

Microsoft - Power Platform_G.Aspiotis.pdf

Goodbye Windows 11: Make Way for Nitrux Linux 3.5.0!

Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slack

GraphSummit Singapore | Graphing Success: Revolutionising Organisational Stru...

Observability Concepts EVERY Developer Should Know -- DeveloperWeek Europe.pdf

20240605 QFM017 Machine Intelligence Reading List May 2024

GenAI Pilot Implementation in the organizations

Mariano G Tinti - Decoding SpaceX

Serial Arm Control in Real Time Presentation

UiPath Test Automation using UiPath Test Suite series, part 6

GraphSummit Singapore | Neo4j Product Vision & Roadmap - Q2 2024

Building Production Ready Search Pipelines with Spark and Milvus

20240607 QFM018 Elixir Reading List May 2024

TrustArc Webinar - 2024 Global Privacy Survey

Uni Systems Copilot event_05062024_C.Vlachos.pdf

Presentation of the OECD Artificial Intelligence Review of Germany

Leveraging Data Driven Research Through Microsoft Azure

1. LEVERAGING DATA DRIVEN RESEARCH THROUGH MICROSOFT AZURE Dr. Miguel Fierro Data Scientist at Microsoft @miguelgfierro miguel.gonzalezfierro@microsoft.com https://miguelgfierro.com Plymouth University | Jan 27, 2017 | Plymouth, UK

2. AZURE FOR RESEARCH AWARD Plymouth University January 2017 - Dr. Miguel Fierro @miguelgfierro azurerfp@microsoft.com Free Azure resources if awarded Areas: data science, climate, health… Ex: Alan Turing Institute got $5M

3. D a t a S c i e n ce V i r t ua l M a chi ne A z u re M L S t u d io S p a r k a n d H a d o o p w i t h A z u re OUTLINE

4. SPARK & HADOOP WITH AZURE

5. WHAT IS HDINSIGHT Plymouth University January 2017 - Dr. Miguel Fierro @miguelgfierro HDInsight Managed Service

6. MANAGER GUI: AMBARI Plymouth University January 2017 - Dr. Miguel Fierro @miguelgfierro

7. APACHE HADOOP Plymouth University January 2017 - Dr. Miguel Fierro @miguelgfierro Software for storing and analysing massive amounts (~Tb) of structured and unstructured data

8. APACHE SPARK Plymouth University January 2017 - Dr. Miguel Fierro @miguelgfierro Framework that runs large-scale data analytics applications pySpark, Spark (Scala), SparkR 100x faster than Hadoop (processing in memory)

9. APACHE KAFKA Plymouth University January 2017 - Dr. Miguel Fierro @miguelgfierro Stream processing for real time apps Publisher & subscriber messaging system Millions of messages per second

10. APACHE STORM Plymouth University January 2017 - Dr. Miguel Fierro @miguelgfierro Distributed framework for real-time applications ETL, continuous computation, online machine learning Million of operations per second in each node

11. APACHE HBASE Plymouth University January 2017 - Dr. Miguel Fierro @miguelgfierro Non-relational database (NoSQL) for Big Data applications Distributed, fast tolerant and scalable Built on top of HDFS (Hadoop Distributed File System)

12. APACHE HIVE Plymouth University January 2017 - Dr. Miguel Fierro @miguelgfierro SQL-like language to query data in Hadoop systems Word count program

13. EXAMPLE OF ARCHITECTURE Plymouth University January 2017 - Dr. Miguel Fierro @miguelgfierro

14. DEMO: PYSPARK APPLICATION Plymouth University January 2017 - Dr. Miguel Fierro @miguelgfierro Log analysis with PySpark Predictive analysis on food inspection with PySpark source: https://docs.microsoft.com/en-us/azure/hdinsight/hdinsight-apache-spark- machine-learning-mllib-ipython source: https://docs.microsoft.com/en-us/azure/hdinsight/hdinsight-apache-spark- custom-library-website-log-analysis

15. AZURE ML STUDIO

16. WHAT IS AZURE ML STUDIO Plymouth University January 2017 - Dr. Miguel Fierro @miguelgfierro GUI for Machine Learning

17. DATA INPUT/OUTPUT Plymouth University January 2017 - Dr. Miguel Fierro @miguelgfierro

18. DATA TRANSFORMATION Plymouth University January 2017 - Dr. Miguel Fierro @miguelgfierro

19. DATA MANIPULATION Plymouth University January 2017 - Dr. Miguel Fierro @miguelgfierro

20. FEATURE SELECTION Plymouth University January 2017 - Dr. Miguel Fierro @miguelgfierro

21. CLASSIFICATION & REGRESSION Plymouth University January 2017 - Dr. Miguel Fierro @miguelgfierro

22. TRAINING & SCORING Plymouth University January 2017 - Dr. Miguel Fierro @miguelgfierro

23. PYTHON & R SCRIPTS Plymouth University January 2017 - Dr. Miguel Fierro @miguelgfierro

24. AUTOMATIC API Plymouth University January 2017 - Dr. Miguel Fierro @miguelgfierro

25. DEMO: CREDIT RISK ANOMALY DETECTION Plymouth University January 2017 - Dr. Miguel Fierro @miguelgfierro source: https://gallery.cortanaintelligence.com/Experiment/1219e87f8fb84e88a2e1b54256808bb3

26. DATA SCIENCE VIRTUAL MACHINE

27. WHAT IS THE DSVM Plymouth University January 2017 - Dr. Miguel Fierro @miguelgfierro Windows: - Anaconda with python Jupyter notebooks - Microsoft R Server - Visual Studio - SQL Server - Azure SDK - Deep learning: CNTK & MXNet - Machine Learning: XGBoost Linux: - Anaconda with python Jupyter notebooks - Microsoft R Server - PyCharm - Azure SDK - Deep learning: CNTK & MXNet - Machine Learning: XGBoost, Weka

28. DEEP LEARNING DSVM Plymouth University January 2017 - Dr. Miguel Fierro @miguelgfierro Libs: - CNTK - MXNet - TensorFlow - Keras Digit recognition Image recognitionExamples:

29. NVIDIA TESLA K80 Plymouth University January 2017 - Dr. Miguel Fierro @miguelgfierro

30. AI LANDSCAPE: IMAGES Plymouth University January 2017 - Dr. Miguel Fierro @miguelgfierro 15.4% 7.3% 6.7% 3.6% 3.1% 5.1% (human) error (%) ImageNet (image recognition competition) top-5 error AlexNet (2012) VGG (2014) Inception (2015) ResNet (2015) Inception- ResNet (2016)

31. AI LANDSCAPE: SPEECH Plymouth University January 2017 - Dr. Miguel Fierro @miguelgfierro Microsoft Research achieves parity with human speech level source: http://blogs.microsoft.com/next/2016/10/18/historic-achievement-microsoft-researchers-reach-human-parity-conversational-speech-recognition CNN (VGG, ResNet, LACE) RNN (Bi-LSTM) Multi-GPU and multi server (1-bit Stochastic Gradient Descent)

32. IMAGE CLASSIFICATION Plymouth University January 2017 - Dr. Miguel Fierro @miguelgfierro 1. 2. 3. 4. 5. source: https://blogs.technet.microsoft.com/machinelearning/2016/11/15/imagenet-deep-neural- network-training-using-microsoft-r-server-and-azure-gpu-vms/

33. IMAGE CLASSIFICATION IMAGENET Plymouth University January 2017 - Dr. Miguel Fierro @miguelgfierro source: https://blogs.technet.microsoft.com/machinelearning/2016/11/15/imagenet-deep-neural-network-training-using-microsoft-r-server-and-azure-gpu-vms/ Real class Predicted class

34. TEXT CLASSIFICATION Plymouth University January 2017 - Dr. Miguel Fierro @miguelgfierro Train Backend Dataset Azure NC24 VM with 4 K80 GPUs .R model.params Azure Cloud Services .py .js .html Score Web app API DNN input text

35. DEMO: TEXT CLASSIFICATION WEB APP Plymouth University January 2017 - Dr. Miguel Fierro @miguelgfierro

36. LEVERAGING DATA DRIVEN RESEARCH THROUGH MICROSOFT AZURE Dr. Miguel Fierro Data Scientist at Microsoft @miguelgfierro miguel.gonzalezfierro@microsoft.com https://miguelgfierro.com Plymouth University | Jan 27, 2017 | Plymouth, UK

Leveraging Data Driven Research Through Microsoft Azure

Recommended

Recommended

More Related Content

What's hot

What's hot (6)

Similar to Leveraging Data Driven Research Through Microsoft Azure

Similar to Leveraging Data Driven Research Through Microsoft Azure (20)

More from Miguel González-Fierro

More from Miguel González-Fierro (12)

Recently uploaded

Recently uploaded (20)

Leveraging Data Driven Research Through Microsoft Azure