Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Kristian Kersting, Associate Professor for Computer Science, TU Dortmund University, Germany at MLconf SEA - 5/20/16

648 views

Published on

Declarative Programming for Statistical ML: The democratization of complex data does not mean dropping the data on everyone’s desk and saying, “good luck”! It means to make machine learning methods usable in such a way that people can easily instruct machines to have a “look” at complex data and help them to understand and act on it. Existing statistical relational learning and probabilistic programming languages only provide partial solutions; most of them do not support convex optimization commonly used in machine learning.

In this talk I will present RELOOP, a declarative mathematical programming language embedded into Python. It allows the user to specify mathematical programs before she knows what individuals are in the domain and, therefore, before she knows what variables and constraints exist. It facilitates the formulation of abstract, general knowledge. And, it reveals the rich logical structure underlying many machine learning problems to the solver and, turn, may make it go faster.

With this, people can start to rapidly develop statistical machine learning approaches for complex data. For instance, adding just three lines of RELOOP code makes a linear support vector machines aware of any underlying network that connects the objects to be classified.

Joint work with Martin Mladenov and many others.

Published in: Technology
  • Be the first to comment

Kristian Kersting, Associate Professor for Computer Science, TU Dortmund University, Germany at MLconf SEA - 5/20/16

  1. 1. Declarative Programming for Statistical ML Kristian Kersting Martiin Mladenov TUD Babak Ahmadi PicoEgo Amir Globerson HUJI Martin Grohe RWTH Sriraam Natarajan U. Indiana Leonard Kleinhans TUD Danny Heinrich TUD andmany more… Pavel Tokmakov INRIA Grenoble
  2. 2. Is there a “-01” flag for Statistical ML? Kristian Kersting Martiin Mladenov TUD Babak Ahmadi PicoEgo Amir Globerson HUJI Martin Grohe RWTH Sriraam Natarajan U. Indiana Leonard Kleinhans TUD Danny Heinrich TUD andmany more… Pavel Tokmakov INRIA Grenoble
  3. 3. Kristian Kersting - Declarative Programming for Statistical ML
  4. 4. There is an arms race to “deeply” understand data Kristian Kersting - Declarative Programming for Statistical ML
  5. 5. Kristian Kersting - Declarative Programming for Statistical ML Take your spreadsheet … Features Objects
  6. 6. Latent Dirichlet AllocationMatrix Factorization Features Objects … and apply some ML Gaussian Processes Decision Trees/Boosting Autoencoder/Deep Learning and many more … Kristian Kersting - Declarative Programming for Statistical ML Support Vector Machines
  7. 7. IS IT REALLY THAT SIMPLE? Kristian Kersting - Declarative Programming for Statistical ML
  8. 8. Guy van den Broeck UCLA
  9. 9. Guy van den Broeck UCLA card (1,d2) card (1,d3) card (1,pAce) card (52,d2) card (52,d3) card (52,pAce) … … … …
  10. 10. Guy van den Broeck UCLA card (1,d2) card (1,d3) card (1,pAce) card (52,d2) card (52,d3) card (52,pAce) … … … …
  11. 11. Guy van den Broeck UCLA No independencies. Fully connected. 22704 states card (1,d2) card (1,d3) card (1,pAce) card (52,d2) card (52,d3) card (52,pAce) … … … … Kristian Kersting - Declarative Programming for Statistical ML
  12. 12. Guy van den Broeck UCLA A machine will not solve the problem card (1,d2) card (1,d3) card (1,pAce) card (52,d2) card (52,d3) card (52,pAce) … … … …
  13. 13. Faster modelling Faster inference and learning
  14. 14. Kristian Kersting - Declarative Programming for Statistical ML Symmetry-Aware Message Passing Compress the model Run message passing inference on the smaller model [Singla, Domingos AAAI’08; Kersting, Ahmadi, Natarajan UAI’09; Ahmadi, Kersting, Mladenov, Natarajan MLJ’13; Mladenov, Globerson, Kersting AISTATS `14, UAI `14; Mladenov, Kersting UAI´15; ...] Big model small model
  15. 15. Kristian Kersting - Declarative Programming for Statistical ML De Raedt, Kersting, Natarajan, Poole “Statistical Relational Artificial Intelligence”,2016 … the study and design of intelligent agents that act in noisy worlds composed of objects and relations among the objects Statistical Relational AI ScalingUncertainty Logic Graphs Trees Mining And Learning [Getoor, Taskar MIT Press ’07; De Raedt, Frasconi, Kersting, Muggleton, LNCS’08; Domingos, Lowd Morgan Claypool ’09; Natarajan, Kersting, Khot, Shavlik Springer Brief’15; Russell CACM 58(7): 88-97 ’15]
  16. 16. BUT WAIT A MINUTE! WE WANT TO USE SOME ML, NOT JUST GRAPHICAL MODELS! Kristian Kersting - Declarative Programming for Statistical ML Latent Dirichlet Allocation Matrix Factorization Gaussian Processes Decision Trees/Boosting Autoencoder/Deep Learning and many more … Support Vector Machines
  17. 17. Let’s say we want to classify publications that cite each other
  18. 18. This is a quadratic program. If you replace l2- by l1-,l∞-norm you get a linear program Standard ML approach: Support Vector Machines Kristian Kersting - Declarative Programming for Statistical ML [Vapnik ´79; Bennett´99; Mangasarian´99; Zhou, Zhang, Jiao´02, ... ]
  19. 19. Write down the problem in „paper form“. The machine then compiles automatically into algebraic solver form. Statistical Machine Learning via Declarative Programming [Kersting, Mladenov, Tokmakov AIJ´15, Mladenov, Heinrich, Kleinhans, Gonsio, Kersting DeLBP´16] Logically parameterized variable (set of ground variables) Logically parameterized constraint Logically parameterized objective Data stored externally
  20. 20. Program1 Data1 Program2 Data2 Program3 Data3 ... Kristian Kersting - Declarative Programming for Statistical ML [Kersting, Mladenov, Tokmakov AIJ´15, Mladenov, Heinrich, Kleinhans, Gonsio, Kersting DeLBP´16] Captures the essence of a problem and can be reused for several problems
  21. 21. MP1 Declarative Program MP2 MPn Data1 Data2 Datan ... Kristian Kersting - Declarative Programming for Statistical ML [Kersting, Mladenov, Tokmakov AIJ´15, Mladenov, Heinrich, Kleinhans, Gonsio, Kersting DeLBP´16] Captures the essence of a problem and can be reused for several problems
  22. 22. But wait, publications are citing each other. OMG, I have to use graph kernels! REALLY?
  23. 23. Simply program some additional constraints [Kersting, Mladenov, Tokmakov AIJ´15, Mladenov, Heinrich, Kleinhans, Gonsio, Kersting DeLBP´16] Kristian Kersting - Declarative Programming for Statistical ML
  24. 24. http://www-ai.cs.uni-dortmund.de/weblab/static/RLP/html/ … Kristian Kersting - Declarative Programming for Statistical ML Loops and relations get interwined, and models can refer to each other DBMS Interface Using a probabilistic programming language we can even get stochastic relational mathematical programs
  25. 25. Finally, the „-O1“ flag [Kersting, Mladenov, Tokmakov AIJ 2015, Mladenov, Kleinhans, Kersting 2016] Kristian Kersting - Declarative Programming for Statistical ML (1) Reduce the QP via symmetries (2) Run any solver on the reduced QP
  26. 26. … and the “-02” flag Kristian Kersting - Declarative Programming for Statistical ML Algebraic Decision Diagrams Formulae parse trees Matrix Free Optimization (  )+ = Optimization with 60 Millions of non-zeros with 12 minutes per log-barrier iteration and actually sublinear in the number of non-zeros
  27. 27. HIGH-LEVEL LANGUAGES FOR MACHINE LEARNING AND OPTIMIZATION ARE A STEP TOWARDS THE … Conclusions Kristian Kersting - Declarative Programming for Statistical ML
  28. 28. DEMOCRATIZATION OF MACHINE LEARNING Kristian Kersting - Declarative Programming for Statistical ML  Reduces the level of expertise necessary to build optimization applications, makes models faster to write and easier to communicate  Facilitate the construction of sophisticated models with rich domain knowledge  Speed up solvers by exploiting language properties, compression, and compilation
  29. 29. Kristian Kersting - Declarative Programming for Statistical ML
  30. 30. Kristian Kersting - Declarative Programming for Statistical ML

×