Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

AI Developments Aug 2017

768 views

Published on

Latest developments including hardware and algorithm updates presented at the London Deep Learning Lab meetup https://www.meetup.com/Deep-Learning-Lab/

Published in: Technology

AI Developments Aug 2017

  1. 1. AI Developments 1 AI Developments v0.10 Peter Morgan August 2017
  2. 2. Outline • Concepts • AI Market • Data • Algorithms • Hardware • Conferences • New Developments AI Developments v0.10 Peter Morgan August 2017 2
  3. 3. AI Developments v0.10 Peter Morgan August 2017 3
  4. 4. Types of Intelligence AI Developments v0.10 Peter Morgan August 2017 4
  5. 5. The Intelligence Revolution 5AI Developments v0.10 Peter Morgan August 2017
  6. 6. Deep Learning/AI Frameworks - The Big Picture AI Developments v0.10 Peter Morgan August 2017 6 AI Frameworks Cognitive Architectures ML Frameworks Supervised, Unsupervised & Reinforcement Deep Learning Frameworks Neural Networks
  7. 7. What is Intelligence? • Intelligence is an Agent’s ability to adapt to and to achieve goals within its Environment • Human vs machine intelligence – ultimately the same • “The term artificial intelligence is somewhat nonsensical. Something is either intelligent or it isn’t. Just as something either flies or it doesn’t. We don’t talk about artificial flying” - Zoubin Ghahramani, Cambridge University • Information processing, computation, physics, hardware • Exploration vs exploitation • Biological (any species) versus machine (any type) •Does substrate matter - carbon vs silicon? 7AI Developments v0.10 Peter Morgan August 2017
  8. 8. What is Learning? • Learning algorithms – system gets better with more data until no further improvement • Train the system – just like animals learn • Supervised, unsupervised and reinforcement learning • Physically, it is the strengthening of connections (synapses) between nodes (neurons) • Memory (short and long term) is involved • Deep learning is a step towards the goal of artificial general intelligence (AGI) • Ensemble of techniques 8AI Developments v0.10 Peter Morgan August 2017
  9. 9. What is Deep Learning? 9AI Developments v0.10 Peter Morgan August 2017
  10. 10. Deep Learning = Neural Networks • Refers to systems that learn from data • These systems are based on artificial neural networks (ANNs), which in turn are based on biological neural networks (BNN), such as the human brain • In practice such learning systems consist of data, multiple layers, nodes, weights and optimisation algorithms AI Developments v0.10 Peter Morgan August 2017 10
  11. 11. Biological Neuron AI Developments v0.10 Peter Morgan August 2017 11
  12. 12. Deep Learning Is Eating the World • What about the “AI winters”? 1974–80 and 1987–93, where AI companies over-promised and under-delivered https://en.wikipedia.org/wiki/AI_winter • Due to more labeled data, more compute power, better optimization algorithms, and better neural net models and architectures, deep learning has started to supersede humans when it comes to image recognition and classification • Work is being done to obtain similar levels of performance in natural language processing and understanding • According to Jeff Dean in a recent interview, Google have implemented DL in over one hundred of their products and services including search and photos • AI is enjoying a renaissance now, not simply because of the promise it holds for the future but because of the impact it is having on businesses today AI Developments v0.10 Peter Morgan August 2017 12
  13. 13. AI Market AI Developments v0.10 Peter Morgan August 2017 13
  14. 14. AI Developments v0.10 Peter Morgan August 2017 14
  15. 15. Deep Learning Startups AI Developments v0.10 Peter Morgan August 2017 15
  16. 16. AI Developments v0.10 Peter Morgan August 2017 16 • Hardware (compute) – Nvidia GPU, Intel (Nervana), AMD Radeon • Data available - structured and unstructured • Research activity • Conference attendance (e.g., NIPS) • Meetup groups • PhD enrolments in CS and machine learning • Performance measures - chess, Jeopardy, Go, computer vision, language processing, ... • Number of papers being published in AI/ML AI Trends 1
  17. 17. AI Developments v0.10 Peter Morgan August 2017 17 • Availability of open source deep learning frameworks, including TensorFlow, Mxnet, etc. - Number of packages - GitHub commits - Contributors etc. • For example, TF is most downloaded repo from GitHub in under a year • Fact that major corporates open sourced their AI frameworks, starting with Google (TF, etc.) • Number of AI related jobs on job boards • Salaries for AI experts AI Trends 2
  18. 18. AI Developments v0.10 Peter Morgan August 2017 18 • Backing and realignment by corporates to rebrand as AI companies - Microsoft, IBM, Amazon (Google and Facebook were already there) • For example: - IBM Watson HQ in NYC - Microsoft announcing 5000 strong AI division - Apple announcing at NIPS that it would be open sourcing its AI research - Siri, Cortana, Alexa are all NLP apps using neural nets • Number of AI products and apps • Number of AI/deep learning startups AI Trends 3
  19. 19. AI Developments v0.10 Peter Morgan August 2017 19 • Venture capital investment in AI startups • Number of press/news articles • Announcements from AI experts with 30 years experience confirming that this time is for real - there will be no more AI winters • Number of professors being hired way from academia to join AI companies , e.g. Uber and CMU, Google and Oxford, Facebook, Apple, etc. • Government level panels on the development and impact of AI on jobs, society and policy, e.g., Whitehouse and U.K. parliament • AI Safety consortium announced last month between Google, Microsoft, IBM and Amazon to track developments in AI • Recent books published by professors and engineers on AI development AI Trends 4
  20. 20. AI Developments v0.10 Peter Morgan August 2017 20
  21. 21. Image classification Progress in machine classification of images - error rate by year. Red line is the error rate of a trained human. 2.25% as of July 2017 (ImageNet). 21AI Developments v0.10 Peter Morgan August 2017
  22. 22. DL Outperforms ML 22AI Developments v0.10 Peter Morgan August 2017
  23. 23. Computer Vision Accuracy AI Developments v0.10 Peter Morgan August 2017 2323
  24. 24. GPU Faster than Moore’s Law AI Developments v0.10 Peter Morgan August 2017 2424
  25. 25. 25AI Developments v0.10 Peter Morgan August 2017
  26. 26. AI Developments v0.10 Peter Morgan August 2017 26
  27. 27. As of May 2016, Aug 2017 > 66,000 AI Developments v0.10 Peter Morgan August 2017 27
  28. 28. Growth of Deep Learning atGoogle and many more . . .. Directories containing model description files AI Developments v0.10 Peter Morgan August 2017 28
  29. 29. Data AI Developments v0.10 Peter Morgan August 2017 29
  30. 30. Where does the data come from? • Science – particle, astrophysics • Industry – oil, finance, telecom (all verticals) • Social – Facebook, LinkedIn, Twitter • Medicine – genome, neuroscience • Government – census, education, police • Sports – statistics • Environment – weather, sensors 30AI Developments v0.10 Peter Morgan August 2017
  31. 31. Data Sets • Raw data input into the neural network can originate from any environmental source • It can be recorded and stored in a database (e.g., text, images, audio, video), or live (incident directly from the environment) streaming data • Examples of recorded data sets include MNIST, Labeled Faces in the Wild (LFW), ImageNet, CIFAR and YouTube-8M AI Developments v0.10 Peter Morgan August 2017 31 MNIST LFW
  32. 32. Data Sets • Images: MNIST, CIFAR-10, ImageNet, PASCAL VOC, Mini-Places2, Food 101 • Text: IMDB, Penn Treebank, Shakespeare Text, bAbI, Hutter-prize • Video: UCF101, Kinetics, YouTube-8M • Others: flickr8k, flickr30k, COCO AI Developments v0.10 Peter Morgan August 2017 32
  33. 33. Algorithms AI Developments v0.10 Peter Morgan August 2017 33
  34. 34. Deep Learning Evolution 34AI Developments v0.10 Peter Morgan August 2017
  35. 35. Convolutional Neural Networks • First developed in 1970’s • Widely used for image recognition and classification • Inspired by biological processes, CNN’s are a type of feed-forward ANN • The individual neurons are tiled in such a way that they respond to overlapping regions in the visual field AI Developments v0.10 Peter Morgan August 2017 35
  36. 36. Recurrent Neural Networks • First developed in 1970’s • RNN’s are neural networks that are used to predict the next element in a sequence or time series • This could be, for example, words in a sentence or letters in a word • Applications include predicting or generating music, stories, news, code, financial instrument pricing, text, speech, in fact the next element in any event stream AI Developments v0.10 Peter Morgan August 2017 36
  37. 37. LSTM and NTM • Long Short Term Memory (LSTM) • LSTM (Schmidhuber, 1997) is an RNN architecture that contains blocks that can remember a value for an arbitrary length of time • It solves the vanishing or exploding gradient problem when calculating back propagation • An LSTM network is universal in the sense that given enough network units it can compute anything a conventional computer can compute, provided it has the proper weight matrix • LSTM outperforms alternative RNNs and Hidden Markov Models and other sequence learning methods in numerous applications, e.g., in handwriting recognition, speech recognition and music composition • Neural Turing Machines (NTM) • NTMs are a method of extending the capabilities of recurrent neural networks by coupling them to external memory resources 37AI Developments v0.10 Peter Morgan August 2017
  38. 38. Optimizations • Initializers Constant, Uniform, Gaussian, Glorot Uniform, Xavier, Kaiming, IdentityInit, Orthonormal • Optimizers Gradient Descent with Momentum, RMSProp, Adadelta, Adam, Adagrad, MultiOptimizer • Activations Rectified Linear, Softmax, Tanh, Logistic, Identity, ExpLin • Layers Linear, Convolution, Pooling, Deconvolution, Dropout, Recurrent, Long Short- Term Memory, Gated Recurrent Unit, BatchNorm, LookupTable, Local Response Normalizat ion, Bidirectional-RNN, Bidirectional-LSTM • Costs Binary Cross Entropy, Multiclass Cross Entropy, Sum of Squares Error • Metrics, Misclassification (Top1, TopK), LogLoss, Accuracy, PrecisionRecall, ObjectDetection AI Developments v0.10 Peter Morgan August 2017 38
  39. 39. GANs • Introduced by Ian Goodfellow et al in 2014 (see references) •A class of artificial intelligence algorithms used in unsupervised deep learning • A theory of adversarial examples, resembling what we have for normal supervised learning • Implemented by a system of two neural networks, a discriminator, D and a generator, G • D & G contest with each other in a zero-sum game framework • Generator generates candidate networks and the discriminator evaluates them AI Developments v0.10 Peter Morgan August 2017 39
  40. 40. Stacked Generative Adversarial Networks https://arxiv.org/abs/1612.04357v1AI Developments v0.10 Peter Morgan August 2017 40
  41. 41. Monet Paintings to Photos AI Developments v0.10 Peter Morgan August 2017 41
  42. 42. Collection Style Transfer AI Developments v0.10 Peter Morgan August 2017 42
  43. 43. Object Transfiguration AI Developments v0.10 Peter Morgan August 2017 43
  44. 44. Season Transfer AI Developments v0.10 Peter Morgan August 2017 44
  45. 45. Deep Learning Frameworks AI Developments v0.10 Peter Morgan August 2017 45
  46. 46. TensorFlow • TensorFlow is the newly (Nov 2015) open sourced deep learning library from Google • It is their second generation system for the implementation and deployment of large-scale machine learning models • Written in C++ with a python interface, it is borne from research and deploying machine learning projects throughout a wide range of Google products and services • Initially TF ran only on a single node (your laptop, say), but Google have now released a version that runs on a distributed cluster • Available in the cloud on GCP • https://www.tensorflow.org/ AI Developments v0.10 Peter Morgan August 2017 46
  47. 47. AI Developments v0.10 Peter Morgan August 2017 47
  48. 48. TensorFlow supports manyplatforms Raspberry Pi AndroidiOS 1st-gen TPU GPUCPU Cloud TPU AI Developments v0.10 Peter Morgan August 2017 48
  49. 49. TensorFlow supports manylanguages Java AI Developments v0.10 Peter Morgan August 2017 49
  50. 50. Cognitive Toolkit • Microsoft open source deep learning framework (Jan 25, 2016) • Version 2.0 released Oct 25, major upgrade • Renamed CNTK to Microsoft Cognitive Toolkit • Announced partnership with Nvidia and OpenAI (Elon Musk backed AI startup), Nov 16 • Languages are Python, C++ or BrainScript • Can run on Azure GPU’s • https://www.microsoft.com/en-us/research /product/cognitive-toolkit/ AI Developments v0.10 Peter Morgan August 2017 50
  51. 51. Torch • First released in 2000, with over 50,000 downloads, company users include Google, Facebook, Twitter • The goal of Torch is to have maximum flexibility and speed in building scientific algorithms while making the process extremely simple • Torch is a neural network library written in Lua with a C/CUDA interface originally developed by a team from the Swiss institute EPFL • At the heart of Torch are popular neural network and optimization libraries which are simple to use, while being flexible in implementing different complex neural network topologies • http://torch.ch/ 51AI Developments v0.10 Peter Morgan August 2017
  52. 52. Hardware is Hot Again AI Developments v0.10 Peter Morgan August 2017 52
  53. 53. Types of Hardware • Sensors, processors, storage, memory, network • Processors - CPU, GPU, FPGA, ASIC, NPU, QPU • GPU - Graphics Processing Units were first brought to market by Nvidia in 2007 to meet the demands of the gaming market • Massively parallel processing (MPP) • 100 x speedup compared with CPU’s • Widespread application – science, industry, government • Nvidia www.nvidia.com • Intel Xeon Phi http://www.intel.com/content/www/us/en/processors/xeon/xeon- phi-detail.html • AMD Radeon www.amd.com 53AI Developments v0.10 Peter Morgan August 2017
  54. 54. CPU v GPU Architecture AI Developments v0.10 Peter Morgan August 2017 5454
  55. 55. CPU – Intel Xeon AI Developments v0.10 Peter Morgan August 2017 55
  56. 56. CPU – AMD RyZen AI Developments v0.10 Peter Morgan August 2017 56
  57. 57. GPU’s AI Developments v0.10 Peter Morgan August 2017 5757
  58. 58. Nvidia GPU Exponentials AI Developments v0.10 Peter Morgan August 2017 5858
  59. 59. New Hardware – Nvidia Volta V100 AI Developments v0.10 Peter Morgan August 2017 59
  60. 60. Volta V100 Specs •Pairs NvidiaCUDA and Tensor Cores to deliver the performance of an AI supercomputer •Over 21 billion transistors •With 640 Tensor Cores, Volta delivers over 100 TFLOPS of deep learning performance •Over a 5X increase compared to prior generation NVIDIA Pascal architecture (last year) •Next generation of Nvidia NVLink connects multiple V100 GPUs at up to 300 GB/s AI Developments v0.10 Peter Morgan August 2017 60
  61. 61. Volta V100 Benchmarks AI Developments v0.10 Peter Morgan August 2017 61
  62. 62. HGX-1 - For AI Cloud Computing •Purpose-built for cloud computing •Eight NVIDIA Tesla GPUs interconnected with an NVLink hybrid cube •Applications include including deep learning training, inference, advanced analytics, and HPC •Faster and cheaper than legacy CPU-based servers •Extract the full AI performance that Tesla V100 provides AI Developments v0.10 Peter Morgan August 2017 62
  63. 63. Self-driving Cars – Nvidia Drive PX2 • Delivers 20 TFLOPS of performance • Consumes only 20W of power • Packed with 7 billion transistors AI Developments v0.10 Peter Morgan August 2017 63
  64. 64. AMD Radeon Vega GPU AI Developments v0.10 Peter Morgan August 2017 64
  65. 65. Intel Nervana Hardware AI Developments v0.10 Peter Morgan August 2017 65
  66. 66. Google TPU v1 AI Developments v0.10 Peter Morgan August 2017 66
  67. 67. Google TPU v2 AI Developments v0.10 Peter Morgan August 2017 67
  68. 68. Fujitsu DLU AI Developments v0.10 Peter Morgan August 2017 68
  69. 69. Graphcore - IPU AI Developments v0.10 Peter Morgan August 2017 69
  70. 70. Graphcore - IPU AI Developments v0.10 Peter Morgan August 2017 70
  71. 71. Neuromorphic – HBP Spinnaker AI Developments v0.10 Peter Morgan August 2017 71
  72. 72. Neuromorphic – IBM True North AI Developments v0.10 Peter Morgan August 2017 72
  73. 73. Quantum - DWave AI Developments v0.10 Peter Morgan August 2017 73
  74. 74. Deep Learning in the Cloud AI Developments v0.10 Peter Morgan August 2017 74
  75. 75. DLaaS - Cloud Services • DL/ML as a Service is offered by all the major cloud providers • AWS https://aws.amazon.com/machine-learning/ • Azure https://azure.microsoft.com/en-us/services/machine-learning/ • GCP https://cloud.google.com/products/machine-learning/ • Bluemix https://www.ibm.com/cloud-computing/bluemix/watson AI Developments v0.10 Peter Morgan August 2017 75
  76. 76. TPU as a Service AI Developments v0.10 Peter Morgan August 2017 76
  77. 77. AI Conferences – Business Focussed • O’Reilly AI http://conferences.oreilly.com/artificial-intelligence/ai-ny • AI Frontiers http://www.aifrontiers.com/ • AI World http://aiworldexpo.com/program/ • AI Europe http://ai-europe.com/ • AI Summit https://theaisummit.com/london/ • Re:Work https://www.re-work.co/events/ • World of Watson https://www-01.ibm.com/software/events/wow/ AI Developments v0.10 Peter Morgan August 2017 77
  78. 78. AI Conferences – Research Focussed • NIPS = Neural Information Processing Systems https://nips.cc/ • IJCNN = International Joint Conference on Neural Networks http://www.ijcnn.org/ • IJCAI = International Joint Conference on Artificial Intelligence http://ijcai.org/ • ICANN = International Conference on Artificial Neural Networks http://www.icann2017.org/ • IWANN = International Work-Conference on Artificial Neural Networks http://iwann.uma.es/ • ICONIP = International Conference on Neural Information Processing http://www.iconip2017.org/papers.html • ICAART = International Conference on Agents and Artificial Intelligence http://www.icaart.org/ • ISIS = International Symposium on Advanced Intelligent Systems http://isis2017.org/ • AAAI = Association of Advancement of Artificial Intelligence http://www.aaai.org/Conferences/conferences.php • ACM = Association of Computing Machinery https://www.acm.org/conferences • AGI = Artificial General Intelligence Conference http://agi-conf.org/ • TensorCon = TensorFlow Conference https://ti.to/TensorCon/ 78AI Developments v0.10 Peter Morgan August 2017
  79. 79. AI Research Centers • Nvidia AI Lab - Donated DGX-1’s plus research funding to twenty universities • MIT CSAIL – Image classification • Stanford SAIL – Robotics • Toronto – Self-driving cars • Montreal MILA – Disease prediction • Berkeley BAIR – Robotics planning • NYU – Cancer screening • Oxford – Lip reading • IDSIA – AGI AI Developments v0.10 Peter Morgan August 2017 79
  80. 80. New Developments • Multi-modal learning, Transfer learning, One-shot learning, GANs • Better reinforcement learning / integration of deep learning and reinforcement learning • Better generative models. Algorithms that can reliably learn how to generate images, speech, text that humans can’t tell apart from the real thing • Learning to learn and ubiquitous deep learning: algorithms that redesign their own architecture, tune their own hyperparameters, etc. Right now it still takes a human expert to run the learning-to- learn algorithm, but in the future it will be easier to deploy, and all kinds of business that don’t specialize in AI will be able to leverage deep learning AI Developments v0.10 Peter Morgan August 2017 80
  81. 81. New Developments (cont.) • Sample-efficient learning algorithms that learn from as few labeled examples as humans do • Semi-supervised learning and one-shot learning will reduce the amount of data needed to train several kinds of models and make AI use more widespread • Research will focus on making extremely robust models that almost never make a mistake, for use in safety-critical applications • Deep learning will continue to spread out into general culture and we’ll see artists and meme creators using it to do things that we never would have anticipated AI Developments v0.10 Peter Morgan August 2017 81
  82. 82. Where are we headed? AI Developments v0.10 Peter Morgan August 2017 82 World Economic Forum (WEF) Report, 2016: Today, we are at the beginning of a Fourth Industrial Revolution. Developments in genetics, artificial intelligence, robotics, nanotechnology, 3D printing and biotechnology, to name just a few, are all building on and amplifying one another. This will lay the foundation for a revolution more comprehensive and all-encompassing than anything we have ever seen Deepmind Mission: Solve intelligence then use it to solve everything else
  83. 83. AI Related Books • Bengio, Yoshua et al, Deep Learning, MIT Press, 2016 • Brynjolfsson, Erik and Andrew McAfee, The Second Machine Age, W.W. Norton & Co., 2014 • Chollet, Francois, Deep Learning with Python, Manning, Oct 2017 • Domingos, Pedro, The Master Algorithm, Basic Books, 2015 • Ford, Martin, Rise of the Robots: Technology and the Threat of a Jobless Future, Basic Books, 2015 • Kaku, Michio, The Future of the Mind, Doubleday, 2014 • Kurzweil, Ray, How to Create a Mind, Penguin Books, 2013 • Russell and Norvig, Artificial Intelligence, A Modern Approach, Pearson, 2009 • Shanahan, Murray, The Technological Singularity, MIT Press, 2015 • Yampolskiy, Roman, Artificial Superintelligence, A Futuristic Approach, CRC, 2015 83AI Developments v0.10 Peter Morgan August 2017
  84. 84. References • LeCunn, Y., Unsupervised Learning: the Next Frontier in AI [video], Nov 2016 https://www.aices.rwth-aachen.de/charlemagne-distinguished-lecture-series • LeCun, Y., Bengio, Y., and Hinton, G., Deep Learning, Nature, v.521, p.436–444, May 2016 http://www.nature.com/nature/journal/v521/n7553/abs/nature14539.html • Goodfellow, I. et al, Generative Adversarial Networks, in NIPS 2014 • Radford, A., Metz, L., and Chintala, S., Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks, Jan 2016 https://arxiv.org/abs/1511.06434 • CycleGAN https://arxiv.org/abs/1703.10593 • Mathieu, M., Couprie, C., and LeCun, Y., Deep Multi-Scale Video Prediction Beyond Mean Square Error, ICLR 2016 conference paper, Feb 2016 https://arxiv.org/abs/1511.05440 AI Developments v0.10 Peter Morgan August 2017 84
  85. 85. References • Brtiz, D. et al, Massive Exploration of Neural Machine Translation Architectures, Mar 2017 https://arxiv.org/abs/1703.03906 • Johnson, M. et al, Google's Multilingual Neural Machine Translation System: Enabling Zero-Shot Translation, Nov 2016 https://arxiv.org/abs/1611.04558 • Gehring, J. et al, Convolutional Sequence to Sequence Learning, May 2017 https://arxiv.org/abs/1705.03122 • Feedback Networks http://feedbacknet.stanford.edu/feedback_networks_2016.pdf • AI safety discussion https://www.facebook.com/groups/467062423469736/ • Google ICML 2017 Publications https://research.googleblog.com/2017/08/google-at-icml-2017.html AI Developments v0.10 Peter Morgan August 2017 85
  86. 86. Questions? 86AI Developments v0.10 Peter Morgan August 2017 86

×