This document discusses tensors and their use in machine learning. It explains that tensors were originally developed for physics but are now commonly used in computing to represent important patterns of computation. Tensors make it easier to code numerical algorithms by capturing operations like element-wise computations, outer products, reductions, and matrix/vector products. Additionally, automatic differentiation is now possible using tensor frameworks, which allows gradients to be computed automatically rather than derived by hand. This has significantly advanced machine learning by enabling new optimization algorithms and the training of complex neural networks. Tensor systems also allow the same code to run on CPUs, GPUs, and clusters, improving productivity.

1. © 2017 MapR TechnologiesMapR Confidential 1
Tensor Abuse
In the Workplace

2. © 2017 MapR TechnologiesMapR Confidential 2
Agenda
• What are tensors?
• How modern tensor systems work
• It's not just machine learning
• It’s also not magic
• Machine learning as optimization
• Animation of optimization algorithm

3. © 2017 MapR TechnologiesMapR Confidential 3
Spoilers
• Trigger warning: Greek letters ahead
– AND lots of animations for developing intuitions
• Tensors have a history in physics
• We don’t use them that way in machine learning
• Tensors are cool because they make it easier to code
numerical algorithms
• But auto-derivatives are as big a deal
• And we can abuse this machinery

4. © 2017 MapR TechnologiesMapR Confidential 4
Linear Operators on To the N-th Degree
• Tensors were originally invented for differential geometry
• That gave us relativity

5. © 2017 MapR TechnologiesMapR Confidential 5
But This Isn’t Physics
• In computing, we use some of the same words
• But they don’t mean the same thing
• We don’t need the Greek letters
• The point is that we have important patterns of computation
– We mostly don’t care about changing coordinate systems

6. © 2017 MapR TechnologiesMapR Confidential 6
Basic Operations
• Element-wise operations
• Outer products
• Reductions
• Matrix and vector products are special cases

7. © 2017 MapR TechnologiesMapR Confidential 7
This is news?!?

8. © 2017 MapR TechnologiesMapR Confidential 8
This is news?!?
Sounds like APL in sheep’s clothing

9. © 2017 MapR TechnologiesMapR Confidential 9
But it really is important
because
loop structuring is critical

10. © 2017 MapR TechnologiesMapR Confidential 10
Why This Matters for Machine Learning
• Sums of products and element-wise evaluations are ubiquitous
in machine learning
• And this is often surrounded by an outer loop
• This is susceptible to pipelining in GPU’s, but only if you can
see the large-scale patterns in the code

11. © 2017 MapR TechnologiesMapR Confidential 11
But, tensors are only
half of the story

12. © 2017 MapR TechnologiesMapR Confidential 12
Machine Learning Has Changed
• Machine learning used to be really hard
– Numeric performance was zilch
– Training data was poor and small
– Learning many layers of a MLP was impossible
– Productivity for new approaches hampered by code complexity
• Recent advances have changed things (a lot)
– Important new regularization techniques
– Per coefficient learning rate techniques
– New gradient based optimization algorithms
– Automated differentiation

13. © 2017 MapR TechnologiesMapR Confidential 13

14. © 2017 MapR TechnologiesMapR Confidential 14

15. © 2017 MapR TechnologiesMapR Confidential 15

16. © 2017 MapR TechnologiesMapR Confidential 17
Gradients are a Big Deal
• In low dimensions, simple search techniques work well
– Line search, evolutionary processes, polyhedron warping all work
• It’s different in high dimensions
– Need some guide about which direction to go
– Too many ways to get sideways

17. © 2017 MapR TechnologiesMapR Confidential 18

18. © 2017 MapR TechnologiesMapR Confidential 19

19. © 2017 MapR TechnologiesMapR Confidential 20
Yeah, But …
• Gradients are also a pain in the ***
– Traditionally you needed to derive them by hand
– For some problems like robot kinematics or astrodynamics, equations
would cover many pages
• The big change in recent years is automatic differentiation
– Some limits on how problems are formulated
– But pretty complicated forms can be allowed easily

20. © 2017 MapR TechnologiesMapR Confidential 21

21. © 2017 MapR TechnologiesMapR Confidential 22

22. © 2017 MapR TechnologiesMapR Confidential 23
hidden = tf.nn.elu(x - biases)
y_pred = tf.matmul(hidden, weights)
loss = tf.reduce_mean((y - y_pred) ** 2)

23. © 2017 MapR TechnologiesMapR Confidential 24
The Lessons to Learn
• These new systems are astounding
– Tensors allow us to encode very complex algorithms
– Auto-differentiation allows us to use gradients
– New optimizers solve very nasty problems
– The same code can drive CPU, GPU or clusters
– The code is strange and abstract, but not that hard
• Not just for breakfast any more
– Machine learning isn’t the only thing you can do with these systems
– Go for it!

24. © 2017 MapR TechnologiesMapR Confidential 25
Q&A
@mapr
maprtechnologies
yourname@mapr.com
ENGAGE WITH US