Successfully reported this slideshow.

# Designing linear algebra into Julia

Presented at PASC 2017, Lugano, Switzerland

• Full Name
Comment goes here.

Are you sure you want to Yes No
• Be the first to comment

### Designing linear algebra into Julia

1. 1. Designing linear algebra into Julia Jiahao Chen MIT CSAIL ⟶ Capital One New York PASC 2017, 06-26 Alan Edelman MIT Andreas Noack MIT/J.C. Jake Bolewski TileDB Sacha Verweij Stanford Jeff Bezanson Julia Computing ...and many others! Viral B Shah Julia Computing Simon Byrne Julia Computing
2. 2. Claim 1. Julia's generic function system (multimethods/ multiple dispatch) is ergonomically designed to capture mathematical abstraction. Claim 2. We're only just learning how to explain abstractions clearly to a computer. Claim 3. The future of high performance lies in composable abstraction.
3. 3. New mathematical ideas extend existing abstractions, e.g. by overloading * What does it mean to multiply A*B? It depends on what A and B are. Many people learn to multiply in this order: 1. Int * Int 2. Rational * Int 3. Real * Real 4. Matrix * Matrix, Matrix * Vector,  Complex * Complex,...
4. 4. julia> methods(*) # 181 methods for generic function "*": *(x::Bool, z::Complex{Bool}) in Base at complex.jl:236...*(x::Bool, y::Bool) in Base at bool.jl:91 *(x::Bool, y::T) where T<:Unsigned in Base at bool.jl:104 *(x::Bool, z::Complex) in Base at complex.jl:243 *(x::Bool, y::Irrational) in Base at irrationals.jl:111 *(x::Bool, y::T) where T<:Number in Base at bool.jl:101 *(a::Float16, b::Float16) in Base at float.jl:372 *(x::Float32, y::Float32) in Base at float.jl:378 *(x::Float64, y::Float64) in Base at float.jl:379 *(z::Complex{Bool}, x::Bool) in Base at complex.jl:237 *(z::Complex{Bool}, x::Real) in Base at complex.jl:255 *(this::Base.Grisu.Float, other::Base.Grisu.Float) in Base.Grisu at grisu/float.jl:138 *(c::BigInt, x::BigFloat) in Base.MPFR at mpfr.jl:312 *(a::BigInt, b::BigInt, c::BigInt, d::BigInt, e::BigInt) in Base.GMP at gmp.jl:334 *(a::BigInt, b::BigInt, c::BigInt, d::BigInt) in Base.GMP at gmp.jl:327 *(a::BigInt, b::BigInt, c::BigInt) in Base.GMP at gmp.jl:321 *(x::BigInt, y::BigInt) in Base.GMP at gmp.jl:289 *(x::BigInt, c::Union{UInt16, UInt32, UInt64, UInt8}) in Base.GMP at gmp.jl:368 *(x::BigInt, c::Union{Int16, Int32, Int64, Int8}) in Base.GMP at gmp.jl:374 *(a::BigFloat, b::BigFloat, c::BigFloat, d::BigFloat, e::BigFloat) in Base.MPFR at mpfr.jl:460 *(a::BigFloat, b::BigFloat, c::BigFloat, d::BigFloat) in Base.MPFR at mpfr.jl:453 *(a::BigFloat, b::BigFloat, c::BigFloat) in Base.MPFR at mpfr.jl:447 *(x::BigFloat, c::BigInt) in Base.MPFR at mpfr.jl:308 *(x::BigFloat, y::BigFloat) in Base.MPFR at mpfr.jl:277 *(x::BigFloat, c::Union{UInt16, UInt32, UInt64, UInt8}) in Base.MPFR at mpfr.jl:284 *(x::BigFloat, c::Union{Int16, Int32, Int64, Int8}) in Base.MPFR at mpfr.jl:292 *(x::BigFloat, c::Union{Float16, Float32, Float64}) in Base.MPFR at mpfr.jl:300 *(B::BitArray{2}, J::UniformScaling) in Base.LinAlg at linalg/uniformscaling.jl:149 *(z::Complex, w::Complex) in Base at complex.jl:223 *(z::Complex, x::Bool) in Base at complex.jl:244 *(x::Real, z::Complex{Bool}) in Base at complex.jl:254 *(x::Real, z::Complex) in Base at complex.jl:266 *(z::Complex, x::Real) in Base at complex.jl:267 *(x::Rational, y::Rational) in Base at rational.jl:252 *(a::Integer, index::CartesianIndex{N}) where N in Base.IteratorsMD at multidimensional.jl:83 *(c::Union{UInt16, UInt32, UInt64, UInt8}, x::BigInt) in Base.GMP at gmp.jl:372 *(c::Union{Int16, Int32, Int64, Int8}, x::BigInt) in Base.GMP at gmp.jl:378 *(c::Union{UInt16, UInt32, UInt64, UInt8}, x::BigFloat) in Base.MPFR at mpfr.jl:288 *(c::Union{Int16, Int32, Int64, Int8}, x::BigFloat) in Base.MPFR at mpfr.jl:296 *(c::Union{Float16, Float32, Float64}, x::BigFloat) in Base.MPFR at mpfr.jl:304 *(x::Irrational, y::Irrational) in Base at irrationals.jl:109 *(y::Real, x::Base.Dates.Period) in Base.Dates at dates/periods.jl:88 *(x::Real, r::Base.Use_StepRangeLen_Instead) in Base at deprecated.jl:1226 *(x::Number) in Base at operators.jl:400 *(y::Number, x::Bool) in Base at bool.jl:106 *(x::T, y::T) where T<:Union{Int128, UInt128} in Base at int.jl:567 *(x::T, y::T) where T<:Union{Int128, Int16, Int32, Int64, Int8, UInt128, UInt16, UInt32, UInt64, UInt8} in Base at int.jl:33 *(x::T, y::T) where T<:Number in Base at promotion.jl:336 *(x::Number, y::Number) in Base at promotion.jl:250 *(x::Real, r::StepRangeLen{#s45,#s44,S} where S where #s44<:Base.TwicePrecision where #s45<:Real) in Base at twiceprecision.jl:226 *(x::Number, r::StepRangeLen) in Base at range.jl:743 *(x::Number, r::LinSpace) in Base at range.jl:744 *(x::Number, r::Range) in Base at range.jl:742 *(r::StepRangeLen{#s45,#s44,S} where S where #s44<:Base.TwicePrecision where #s45<:Real, x::Real) in Base at twiceprecision.jl:228 *(r::StepRangeLen, x::Number) in Base at range.jl:747 *(r::LinSpace, x::Number) in Base at range.jl:748 *(r::Base.Use_StepRangeLen_Instead, x::Real) in Base at deprecated.jl:1227 *(r::Range, x::Number) in Base at range.jl:746 *(x::Base.TwicePrecision, v::Integer) in Base at twiceprecision.jl:469 *(x::Base.TwicePrecision{R}, v::S) where {S<:Number, R} in Base at twiceprecision.jl:487 *(v::Number, x::Base.TwicePrecision) in Base at twiceprecision.jl:491 *(rowvec::RowVector{T,CV} where CV<:(ConjArray{T,1,V} where V<:(AbstractArray{T,1} where T) where T), vec::AbstractArray{T,1}) where T<:Real in Base.LinAlg at linalg/rowvector.jl:171 *(rowvec::RowVector{T,V} where V<:(AbstractArray{T,1} where T), vec::AbstractArray{T,1}) where T<:Real in Base.LinAlg at linalg/rowvector.jl:170 *(rowvec::RowVector{T,CV} where CV<:(ConjArray{T,1,V} where V<:(AbstractArray{T,1} where T) where T) where T, vec::AbstractArray{T,1} where T) in Base.LinAlg at linalg/rowvector.jl:172 *(::RowVector, ::RowVector) in Base.LinAlg at linalg/rowvector.jl:182 *(rowvec::RowVector, A::Base.LinAlg.AbstractTriangular) in Base.LinAlg at linalg/triangular.jl:1639 *(rowvec::RowVector, D::Diagonal) in Base.LinAlg at linalg/diagonal.jl:270 *(rowvec::RowVector, vec::AbstractArray{T,1} where T) in Base.LinAlg at linalg/rowvector.jl:176 *(rowvec::RowVector, mat::AbstractArray{T,2} where T) in Base.LinAlg at linalg/rowvector.jl:181 *(A::Union{Base.ReshapedArray{T,2,A,MI} where MI<:Tuple{Vararg{Base.MultiplicativeInverses.SignedMultiplicativeInverse{Int64},N} where N} where A<:DenseArray, DenseArray{T,2}, SubArray{T,2,A,I,L} where L} where I<:Tuple{Vararg{Union{Base.AbstractCartesianIndex, Int64, Range{Int64}},N} where N} where A<:Union{Base.ReshapedArray{T,N,A,MI} where MI<:Tuple{Vararg{Base.MultiplicativeInverses.SignedMultiplicativeInverse{Int64},N} where N} where A<:DenseArray where N where T, DenseArray}, x::Union{Base.ReshapedArray{S,1,A,MI} where MI<:Tuple{Vararg{Base.MultiplicativeInverses.SignedMultiplicativeInverse{Int64},N} where N} where A<:DenseArray, DenseArray{S,1}, SubArray{S,1,A,I,L} where L} where I<:Tuple{Vararg{Union{Base.AbstractCartesianIndex, Int64, Range{Int64}},N} where N} where A<:Union{Base.ReshapedArray{T,N,A,MI} where MI<:Tuple{Vararg{Base.MultiplicativeInverses.SignedMultiplicativeInverse{Int64},N} where N} where A<:DenseArray where N where T, DenseArray}) where {T<:Union{Complex{Float32}, Complex{Float64}, Float32, Float64}, S} in Base.LinAlg at linalg/matmul.jl:74 *(A::SymTridiagonal, B::Number) in Base.LinAlg at linalg/tridiag.jl:130 *(A::Tridiagonal, B::Number) in Base.LinAlg at linalg/tridiag.jl:626 *(A::UpperTriangular, x::Number) in Base.LinAlg at linalg/triangular.jl:531 *(A::Base.LinAlg.UnitUpperTriangular, x::Number) in Base.LinAlg at linalg/triangular.jl:534 *(A::LowerTriangular, x::Number) in Base.LinAlg at linalg/triangular.jl:531 *(A::Base.LinAlg.UnitLowerTriangular, x::Number) in Base.LinAlg at linalg/triangular.jl:534 *(A::Tridiagonal, B::UpperTriangular) in Base.LinAlg at linalg/triangular.jl:1385 *(A::Tridiagonal, B::Base.LinAlg.UnitUpperTriangular) in Base.LinAlg at linalg/triangular.jl:1385 *(A::Tridiagonal, B::LowerTriangular) in Base.LinAlg at linalg/triangular.jl:1385 *(A::Tridiagonal, B::Base.LinAlg.UnitLowerTriangular) in Base.LinAlg at linalg/triangular.jl:1385 *(A::LowerTriangular, B::LowerTriangular) in Base.LinAlg at linalg/triangular.jl:1392 *(A::Base.LinAlg.UnitLowerTriangular, B::LowerTriangular) in Base.LinAlg at linalg/triangular.jl:1400 *(A::UpperTriangular, B::UpperTriangular) in Base.LinAlg at linalg/triangular.jl:1408 *(A::Base.LinAlg.UnitUpperTriangular, B::UpperTriangular) in Base.LinAlg at linalg/triangular.jl:1416 *(A::Base.LinAlg.AbstractTriangular, B::Base.LinAlg.AbstractTriangular) in Base.LinAlg at linalg/triangular.jl:1533 *(A::Base.LinAlg.AbstractTriangular, D::Diagonal) in Base.LinAlg at linalg/diagonal.jl:149 *(A::Base.LinAlg.AbstractTriangular, B::Union{Bidiagonal, SymTridiagonal, Tridiagonal}) in Base.LinAlg at linalg/bidiag.jl:498 *(A::Base.LinAlg.AbstractTriangular, B::AbstractArray{T,1} where T) in Base.LinAlg at linalg/triangular.jl:1557 *(A::Base.LinAlg.AbstractTriangular, B::AbstractArray{T,2} where T) in Base.LinAlg at linalg/triangular.jl:1557 *(A::Union{Base.LinAlg.QRCompactWYQ, Base.LinAlg.QRPackedQ}, b::Union{Base.ReshapedArray{T,1,A,MI} where MI<:Tuple{Vararg{Base.MultiplicativeInverses.SignedMultiplicativeInverse{Int64},N} where N} where A<:DenseArray, DenseArray{T,1}, SubArray{T,1,A,I,L} where L} where I<:Tuple{Vararg{Union{Base.AbstractCartesianIndex, Int64, Range{Int64}},N} where N} where A<:Union{Base.ReshapedArray{T,N,A,MI} where MI<:Tuple{Vararg{Base.MultiplicativeInverses.SignedMultiplicativeInverse{Int64},N} where N} where A<:DenseArray where N where T, DenseArray} where T) in Base.LinAlg at linalg/qr.jl:527 *(A::Union{Base.LinAlg.QRCompactWYQ, Base.LinAlg.QRPackedQ}, B::Union{Base.ReshapedArray{T,2,A,MI} where MI<:Tuple{Vararg{Base.MultiplicativeInverses.SignedMultiplicativeInverse{Int64},N} where N} where A<:DenseArray, DenseArray{T,2}, SubArray{T,2,A,I,L} where L} where I<:Tuple{Vararg{Union{Base.AbstractCartesianIndex, Int64, Range{Int64}},N} where N} where A<:Union{Base.ReshapedArray{T,N,A,MI} where MI<:Tuple{Vararg{Base.MultiplicativeInverses.SignedMultiplicativeInverse{Int64},N} where N} where A<:DenseArray where N where T, DenseArray} where T) in Base.LinAlg at linalg/qr.jl:539 *(A::Union{Base.ReshapedArray{T,2,A,MI} where MI<:Tuple{Vararg{Base.MultiplicativeInverses.SignedMultiplicativeInverse{Int64},N} where N} where A<:DenseArray, DenseArray{T,2}, SubArray{T,2,A,I,L} where L} where I<:Tuple{Vararg{Union{Base.AbstractCartesianIndex, Int64, Range{Int64}},N} where N} where A<:Union{Base.ReshapedArray{T,N,A,MI} where MI<:Tuple{Vararg{Base.MultiplicativeInverses.SignedMultiplicativeInverse{Int64},N} where N} where A<:DenseArray where N where T, DenseArray} where T, Q::Union{Base.LinAlg.QRCompactWYQ, Base.LinAlg.QRPackedQ}) in Base.LinAlg at linalg/qr.jl:627 *(Q::Base.LinAlg.HessenbergQ{T,S} where S<:(AbstractArray{T,2} where T), X::Union{Union{Base.ReshapedArray{S,1,A,MI} where MI<:Tuple{Vararg{Base.MultiplicativeInverses.SignedMultiplicativeInverse{Int64},N} where N} where A<:DenseArray, DenseArray{S,1}, SubArray{S,1,A,I,L} where L} where I<:Tuple{Vararg{Union{Base.AbstractCartesianIndex, Int64, Range{Int64}},N} where N} where A<:Union{Base.ReshapedArray{T,N,A,MI} where MI<:Tuple{Vararg{Base.MultiplicativeInverses.SignedMultiplicativeInverse{Int64},N} where N} where A<:DenseArray where N where T, DenseArray}, Union{Base.ReshapedArray{S,2,A,MI} where MI<:Tuple{Vararg{Base.MultiplicativeInverses.SignedMultiplicativeInverse{Int64},N} where N} where A<:DenseArray, DenseArray{S,2}, SubArray{S,2,A,I,L} where L} where I<:Tuple{Vararg{Union{Base.AbstractCartesianIndex, Int64, Range{Int64}},N} where N} where A<:Union{Base.ReshapedArray{T,N,A,MI} where MI<:Tuple{Vararg{Base.MultiplicativeInverses.SignedMultiplicativeInverse{Int64},N} where N} where A<:DenseArray where N where T, DenseArray}}) where {T, S} in Base.LinAlg at linalg/hessenberg.jl:101 *(X::Union{Union{Base.ReshapedArray{S,1,A,MI} where MI<:Tuple{Vararg{Base.MultiplicativeInverses.SignedMultiplicativeInverse{Int64},N} where N} where A<:DenseArray, DenseArray{S,1}, SubArray{S,1,A,I,L} where L} where I<:Tuple{Vararg{Union{Base.AbstractCartesianIndex, Int64, Range{Int64}},N} where N} where A<:Union{Base.ReshapedArray{T,N,A,MI} where MI<:Tuple{Vararg{Base.MultiplicativeInverses.SignedMultiplicativeInverse{Int64},N} where N} where A<:DenseArray where N where T, DenseArray}, Union{Base.ReshapedArray{S,2,A,MI} where MI<:Tuple{Vararg{Base.MultiplicativeInverses.SignedMultiplicativeInverse{Int64},N} where N} where A<:DenseArray, DenseArray{S,2}, SubArray{S,2,A,I,L} where L} where I<:Tuple{Vararg{Union{Base.AbstractCartesianIndex, Int64, Range{Int64}},N} where N} where A<:Union{Base.ReshapedArray{T,N,A,MI} where MI<:Tuple{Vararg{Base.MultiplicativeInverses.SignedMultiplicativeInverse{Int64},N} where N} where A<:DenseArray where N where T, DenseArray}}, Q::Base.LinAlg.HessenbergQ{T,S} where S<:(AbstractArray{T,2} where T)) where {T, S} in Base.LinAlg at linalg/hessenberg.jl:105 *(A::Base.LinAlg.LQPackedQ, B::Union{Union{Base.ReshapedArray{T,1,A,MI} where MI<:Tuple{Vararg{Base.MultiplicativeInverses.SignedMultiplicativeInverse{Int64},N} where N} where A<:DenseArray, DenseArray{T,1}, SubArray{T,1,A,I,L} where L} where I<:Tuple{Vararg{Union{Base.AbstractCartesianIndex, Int64, Range{Int64}},N} where N} where A<:Union{Base.ReshapedArray{T,N,A,MI} where MI<:Tuple{Vararg{Base.MultiplicativeInverses.SignedMultiplicativeInverse{Int64},N} where N} where A<:DenseArray where N where T, DenseArray}, Union{Base.ReshapedArray{T,2,A,MI} where MI<:Tuple{Vararg{Base.MultiplicativeInverses.SignedMultiplicativeInverse{Int64},N} where N} where A<:DenseArray, DenseArray{T,2}, SubArray{T,2,A,I,L} where L} where I<:Tuple{Vararg{Union{Base.AbstractCartesianIndex, Int64, Range{Int64}},N} where N} where A<:Union{Base.ReshapedArray{T,N,A,MI} where MI<:Tuple{Vararg{Base.MultiplicativeInverses.SignedMultiplicativeInverse{Int64},N} where N} where A<:DenseArray where N where T, DenseArray}} where T) in Base.LinAlg at linalg/lq.jl:142 *(A::Union{Base.ReshapedArray{TA,2,A,MI} where MI<:Tuple{Vararg{Base.MultiplicativeInverses.SignedMultiplicativeInverse{Int64},N} where N} where A<:DenseArray, DenseArray{TA,2}, SubArray{TA,2,A,I,L} where L} where I<:Tuple{Vararg{Union{Base.AbstractCartesianIndex, Int64, Range{Int64}},N} where N} where A<:Union{Base.ReshapedArray{T,N,A,MI} where MI<:Tuple{Vararg{Base.MultiplicativeInverses.SignedMultiplicativeInverse{Int64},N} where N} where A<:DenseArray where N where T, DenseArray}, B::Base.LinAlg.LQPackedQ{TB,S} where S<:(AbstractArray{T,2} where T)) where {TA, TB} in Base.LinAlg at linalg/lq.jl:176 *(A::Union{Hermitian{T,S}, Symmetric{T,S}} where S where T, B::Union{Hermitian{T,S}, Symmetric{T,S}} where S where T) in Base.LinAlg at linalg/symmetric.jl:267 *(A::Union{Base.ReshapedArray{T,2,A,MI} where MI<:Tuple{Vararg{Base.MultiplicativeInverses.SignedMultiplicativeInverse{Int64},N} where N} where A<:DenseArray, DenseArray{T,2}, SubArray{T,2,A,I,L} where L} where I<:Tuple{Vararg{Union{Base.AbstractCartesianIndex, Int64, Range{Int64}},N} where N} where A<:Union{Base.ReshapedArray{T,N,A,MI} where MI<:Tuple{Vararg{Base.MultiplicativeInverses.SignedMultiplicativeInverse{Int64},N} where N} where A<:DenseArray where N where T, DenseArray} where T, B::Union{Hermitian{T,S}, Symmetric{T,S}} where S where T) in Base.LinAlg at linalg/symmetric.jl:268 *(A::Symmetric, x::Bool) in Base.LinAlg at linalg/symmetric.jl:272 *(A::Symmetric, x::Number) in Base.LinAlg at linalg/symmetric.jl:274 *(A::Hermitian, x::Bool) in Base.LinAlg at linalg/symmetric.jl:272 *(A::Hermitian, x::Real) in Base.LinAlg at linalg/symmetric.jl:274 *(D::Diagonal, x::Number) in Base.LinAlg at linalg/diagonal.jl:144 *(Da::Diagonal, Db::Diagonal) in Base.LinAlg at linalg/diagonal.jl:146 *(D::Diagonal, B::Base.LinAlg.AbstractTriangular) in Base.LinAlg at linalg/diagonal.jl:150 *(D::Diagonal, A::SparseMatrixCSC) in Base.SparseArrays at sparse/linalg.jl:114 *(D::Diagonal, V::AbstractArray{T,1} where T) in Base.LinAlg at linalg/diagonal.jl:147 *(D::Diagonal, A::AbstractArray{T,2} where T) in Base.LinAlg at linalg/diagonal.jl:154 *(A::Bidiagonal, B::Number) in Base.LinAlg at linalg/bidiag.jl:346 *(A::Union{Bidiagonal, SymTridiagonal, Tridiagonal}, B::Union{Bidiagonal, SymTridiagonal, Tridiagonal}) in Base.LinAlg at linalg/bidiag.jl:499 *(A::Bidiagonal{T}, B::AbstractArray{T,1}) where T in Base.LinAlg at linalg/bidiag.jl:503 *(X::Union{Base.ReshapedArray{P,N,A,MI} where MI<:Tuple{Vararg{Base.MultiplicativeInverses.SignedMultiplicativeInverse{Int64},N} where N} where A<:DenseArray, DenseArray{P,N}, SubArray{P,N,A,I,L} where L} where I<:Tuple{Vararg{Union{Base.AbstractCartesianIndex, Int64, Range{Int64}},N} where N} where A<:Union{Base.ReshapedArray{T,N,A,MI} where MI<:Tuple{Vararg{Base.MultiplicativeInverses.SignedMultiplicativeInverse{Int64},N} where N} where A<:DenseArray where N where T, DenseArray} where N, y::Real) where P<:Base.Dates.Period in Base.Dates at dates/periods.jl:93 *(x::AbstractSparseArray{Tv,Ti,1} where Ti where Tv, a::Number) in Base.SparseArrays at sparse/sparsevector.jl:1429 *(X::Union{Base.ReshapedArray{TX,2,A,MI} where MI<:Tuple{Vararg{Base.MultiplicativeInverses.SignedMultiplicativeInverse{Int64},N} where N} where A<:DenseArray, DenseArray{TX,2}, SubArray{TX,2,A,I,L} where L} where I<:Tuple{Vararg{Union{Base.AbstractCartesianIndex, Int64, Range{Int64}},N} where N} where A<:Union{Base.ReshapedArray{T,N,A,MI} where MI<:Tuple{Vararg{Base.MultiplicativeInverses.SignedMultiplicativeInverse{Int64},N} where N} where A<:DenseArray where N where T, DenseArray}, A::SparseMatrixCSC{TvA,TiA}) where {TX, TvA, TiA} in Base.SparseArrays at sparse/linalg.jl:102 *(A::Union{Base.ReshapedArray{Ta,2,A,MI} where MI<:Tuple{Vararg{Base.MultiplicativeInverses.SignedMultiplicativeInverse{Int64},N} where N} where A<:DenseArray, DenseArray{Ta,2}, SubArray{Ta,2,A,I,L} where L} where I<:Tuple{Vararg{Union{Base.AbstractCartesianIndex, Int64, Range{Int64}},N} where N} where A<:Union{Base.ReshapedArray{T,N,A,MI} where MI<:Tuple{Vararg{Base.MultiplicativeInverses.SignedMultiplicativeInverse{Int64},N} where N} where A<:DenseArray where N where T, DenseArray}, x::AbstractSparseArray{Tx,Ti,1} where Ti) where {Ta, Tx} in Base.SparseArrays at sparse/sparsevector.jl:1504 *(A::SparseMatrixCSC, x::AbstractSparseArray{Tv,Ti,1} where Ti where Tv) in Base.SparseArrays at sparse/sparsevector.jl:1682 *(A::SparseMatrixCSC{Tv,Ti}, B::SparseMatrixCSC{Tv,Ti}) where {Tv, Ti} in Base.SparseArrays at sparse/linalg.jl:125 *(A::SparseMatrixCSC{TvA,TiA}, B::SparseMatrixCSC{TvB,TiB}) where {TvA, TiA, TvB, TiB} in Base.SparseArrays at sparse/linalg.jl:24 *(A::SparseMatrixCSC, D::Diagonal) in Base.SparseArrays at sparse/linalg.jl:118 *(A::Base.SparseArrays.CHOLMOD.Sparse, B::Base.SparseArrays.CHOLMOD.Sparse) in Base.SparseArrays.CHOLMOD at sparse/cholmod.jl:1286 *(A::Base.SparseArrays.CHOLMOD.Sparse, B::Base.SparseArrays.CHOLMOD.Dense) in Base.SparseArrays.CHOLMOD at sparse/cholmod.jl:1287 *(A::Base.SparseArrays.CHOLMOD.Sparse, B::Union{Array{T,1}, Array{T,2}} where T) in Base.SparseArrays.CHOLMOD at sparse/cholmod.jl:1288 *(A::Symmetric{Float64,SparseMatrixCSC{Float64,Ti}}, B::Union{SparseMatrixCSC{Float64,Ti}, SparseVector{Float64,Ti}}) where Ti in Base.SparseArrays.CHOLMOD at sparse/cholmod.jl:1711 *(A::Hermitian{Complex{Float64},SparseMatrixCSC{Complex{Float64},Ti}}, B::Union{SparseMatrixCSC{Complex{Float64},Ti}, SparseVector{Complex{Float64},Ti}}) where Ti in Base.SparseArrays.CHOLMOD at sparse/cholmod.jl:1713 *(A::Hermitian{Float64,SparseMatrixCSC{Float64,Ti}}, B::Union{SparseMatrixCSC{Float64,Ti}, SparseVector{Float64,Ti}}) where Ti in Base.SparseArrays.CHOLMOD at sparse/cholmod.jl:1715 *(A::Union{SparseMatrixCSC{Float64,Ti}, SparseVector{Float64,Ti}}, B::Symmetric{Float64,SparseMatrixCSC{Float64,Ti}}) where Ti in Base.SparseArrays.CHOLMOD at sparse/cholmod.jl:1718 *(A::Union{SparseMatrixCSC{Complex{Float64},Ti}, SparseVector{Complex{Float64},Ti}}, B::Hermitian{Complex{Float64},SparseMatrixCSC{Complex{Float64},Ti}}) where Ti in Base.SparseArrays.CHOLMOD at sparse/cholmod.jl:1720 *(A::Union{SparseMatrixCSC{Float64,Ti}, SparseVector{Float64,Ti}}, B::Hermitian{Float64,SparseMatrixCSC{Float64,Ti}}) where Ti in Base.SparseArrays.CHOLMOD at sparse/cholmod.jl:1722 *(x::AbstractArray{#s45,2} where #s45<:Number) in Base at abstractarraymath.jl:94 *(B::Number, A::SymTridiagonal) in Base.LinAlg at linalg/tridiag.jl:131 *(B::Number, A::Tridiagonal) in Base.LinAlg at linalg/tridiag.jl:627 *(x::Number, A::UpperTriangular) in Base.LinAlg at linalg/triangular.jl:541 *(x::Number, A::Base.LinAlg.UnitUpperTriangular) in Base.LinAlg at linalg/triangular.jl:544 *(x::Number, A::LowerTriangular) in Base.LinAlg at linalg/triangular.jl:541 *(x::Number, A::Base.LinAlg.UnitLowerTriangular) in Base.LinAlg at linalg/triangular.jl:544 *(x::Number, D::Diagonal) in Base.LinAlg at linalg/diagonal.jl:143 *(B::Number, A::Bidiagonal) in Base.LinAlg at linalg/bidiag.jl:347 *(a::Number, x::AbstractSparseArray{Tv,Ti,1} where Ti where Tv) in Base.SparseArrays at sparse/sparsevector.jl:1430 *(A::Number, B::AbstractArray) in Base at arraymath.jl:44 *(A::AbstractArray, B::Number) in Base at arraymath.jl:47 *(s1::AbstractString, ss::AbstractString...) in Base at strings/basic.jl:70 *(index::CartesianIndex, a::Integer) in Base.IteratorsMD at multidimensional.jl:84 *(vec::AbstractArray{T,1} where T, rowvec::RowVector) in Base.LinAlg at linalg/rowvector.jl:183 *(vec::AbstractArray{T,1} where T, rowvec::AbstractArray{T,1} where T) in Base.LinAlg at linalg/rowvector.jl:184 *(A::AbstractArray{T,2} where T, B::Base.LinAlg.AbstractTriangular) in Base.LinAlg at linalg/triangular.jl:1590 *(A::AbstractArray{T,2} where T, D::Diagonal) in Base.LinAlg at linalg/diagonal.jl:152 *(A::AbstractArray{T,2}, x::AbstractArray{S,1}) where {T, S} in Base.LinAlg at linalg/matmul.jl:78 *(a::AbstractArray{T,1} where T, B::AbstractArray{T,2} where T) in Base.LinAlg at linalg/matmul.jl:87 *(A::AbstractArray{T,2}, B::AbstractArray{S,2}) where {T, S} in Base.LinAlg at linalg/matmul.jl:145 *(A::Base.LinAlg.LQ{TA,S} where S<:(AbstractArray{T,2} where T), B::Union{Union{Base.ReshapedArray{TB,1,A,MI} where MI<:Tuple{Vararg{Base.MultiplicativeInverses.SignedMultiplicativeInverse{Int64},N} where N} where A<:DenseArray, DenseArray{TB,1}, SubArray{TB,1,A,I,L} where L} where I<:Tuple{Vararg{Union{Base.AbstractCartesianIndex, Int64, Range{Int64}},N} where N} where A<:Union{Base.ReshapedArray{T,N,A,MI} where MI<:Tuple{Vararg{Base.MultiplicativeInverses.SignedMultiplicativeInverse{Int64},N} where N} where A<:DenseArray where N where T, DenseArray}, Union{Base.ReshapedArray{TB,2,A,MI} where MI<:Tuple{Vararg{Base.MultiplicativeInverses.SignedMultiplicativeInverse{Int64},N} where N} where A<:DenseArray, DenseArray{TB,2}, SubArray{TB,2,A,I,L} where L} where I<:Tuple{Vararg{Union{Base.AbstractCartesianIndex, Int64, Range{Int where N} where A<:Union{Base.ReshapedArray{T,N,A,MI} where MI<:Tuple{Vararg{Base.MultiplicativeInverses.SignedMultiplicativeInverse{Int64},N} where N} where A<:DenseArray where N where T, DenseArray}}) where {TA, TB} in Base.LinAlg at linalg/lq.jl:126 *(A::Base.LinAlg.LQ{TA,S} where S<:(AbstractArray{T,2} where T), B::Base.LinAlg.QR{TB,S} where S<:(AbstractArray{T,2} where T)) where {TA, TB} in Base.LinAlg at linalg/lq.jl:130 *(A::Base.LinAlg.QR{TA,S} where S<:(AbstractArray{T,2} where T), B::Base.LinAlg.LQ{TB,S} where S<:(AbstractArray{T,2} where T)) where {TA, TB} in Base.LinAlg at linalg/lq.jl:134 *(J1::UniformScaling, J2::UniformScaling) in Base.LinAlg at linalg/uniformscaling.jl:148 *(J::UniformScaling, B::BitArray{2}) in Base.LinAlg at linalg/uniformscaling.jl:150 *(A::AbstractArray{T,2} where T, J::UniformScaling) in Base.LinAlg at linalg/uniformscaling.jl:151 *(J::UniformScaling, A::Union{AbstractArray{T,1}, AbstractArray{T,2}} where T) in Base.LinAlg at linalg/uniformscaling.jl:152 *(x::Number, J::UniformScaling) in Base.LinAlg at linalg/uniformscaling.jl:153 *(J::UniformScaling, x::Number) in Base.LinAlg at linalg/uniformscaling.jl:154 *(R::Base.LinAlg.AbstractRotation{T}, A::Union{AbstractArray{S,1}, AbstractArray{S,2}}) where {T, S} in Base.LinAlg at linalg/givens.jl:9 *(G1::Base.LinAlg.Givens{T}, G2::Base.LinAlg.Givens{T}) where T in Base.LinAlg at linalg/givens.jl:363 *(p::Base.DFT.ScaledPlan, x::AbstractArray) in Base.DFT at dft.jl:255 *(p::Base.DFT.FFTW.cFFTWPlan{T,K,false,N} where N, x::Union{Base.ReshapedArray{T,N,A,MI} where MI<:Tuple{Vararg{Base.MultiplicativeInverses.SignedMultiplicativeInverse{Int64},N} where N} where A<:DenseArray, DenseArray{T,N}, SubArray{T,N,A,I,L} where L} where I<:Tuple{Vararg{Union{Base.AbstractCartesianIndex, Int64, Range{Int64}},N} where N} where A<:Union{Base.ReshapedArray{T,N,A,MI} where MI<:Tuple{Vararg{Base.MultiplicativeInverses.SignedMultiplicativeInverse{Int64},N} where N} where A<:DenseArray where N where T, DenseArray}) where {T, K, N} in Base.DFT.FFTW at fft/FFTW.jl:627 *(p::Base.DFT.FFTW.cFFTWPlan{T,K,true,N} where N, x::Union{Base.ReshapedArray{T,N,A,MI} where MI<:Tuple{Vararg{Base.MultiplicativeInverses.SignedMultiplicativeInverse{Int64},N} where N} where A<:DenseArray, DenseArray{T,N}, SubArray{T,N,A,I,L} where L} where I<:Tuple{Vararg{Union{Base.AbstractCartesianIndex, Int64, Range{Int64}},N} where N} where A<:Union{Base.ReshapedArray{T,N,A,MI} where MI<:Tuple{Vararg{Base.MultiplicativeInverses.SignedMultiplicativeInverse{Int64},N} where N} where A<:DenseArray where N where T, DenseArray} where N) where {T, K} in Base.DFT.FFTW at fft/FFTW.jl:634 *(p::Base.DFT.FFTW.rFFTWPlan{Float32,-1,false,N} where N, x::Union{Base.ReshapedArray{Float32,N,A,MI} where MI<:Tuple{Vararg{Base.MultiplicativeInverses.SignedMultiplicativeInverse{Int64},N} where N} where A<:DenseArray, DenseArray{Float32,N}, SubArray{Float32,N,A,I,L} where L} where I<:Tuple{Vararg{Union{Base.AbstractCartesianIndex, Int64, Range{Int64}},N} where N} where A<:Union{Base.ReshapedArray{T,N,A,MI} where MI<:Tuple{Vararg{Base.MultiplicativeInverses.SignedMultiplicativeInverse{Int64},N} where N} where A<:DenseArray where N where T, DenseArray}) where N in Base.DFT.FFTW at fft/FFTW.jl:704 *(p::Base.DFT.FFTW.rFFTWPlan{Complex{Float32},1,false,N} where N, x::Union{Base.ReshapedArray{Complex{Float32},N,A,MI} where MI<:Tuple{Vararg{Base.MultiplicativeInverses.SignedMultiplicativeInverse{Int64},N} where N} where A<:DenseArray, DenseArray{Complex{Float32},N}, SubArray{Complex{Float32},N,A,I,L} where L} where I<:Tuple{Vararg{Union{Base.AbstractCartesianIndex, Int64, Range{Int64}},N} where N} where A<:Union{Base.ReshapedArray{T,N,A,MI} where MI<:Tuple{Vararg{Base.MultiplicativeInverses.SignedMultiplicativeInverse{Int64},N} where N} where A<:DenseArray where N where T, DenseArray}) where N in Base.DFT.FFTW at fft/FFTW.jl:711 *(p::Base.DFT.FFTW.rFFTWPlan{Float64,-1,false,N} where N, x::Union{Base.ReshapedArray{Float64,N,A,MI} where MI<:Tuple{Vararg{Base.MultiplicativeInverses.SignedMultiplicativeInverse{Int64},N} where N} where A<:DenseArray, DenseArray{Float64,N}, SubArray{Float64,N,A,I,L} where L} where I<:Tuple{Vararg{Union{Base.AbstractCartesianIndex, Int64, Range{Int64}},N} where N} where A<:Union{Base.ReshapedArray{T,N,A,MI} where MI<:Tuple{Vararg{Base.MultiplicativeInverses.SignedMultiplicativeInverse{Int64},N} where N} where A<:DenseArray where N where T, DenseArray}) where N in Base.DFT.FFTW at fft/FFTW.jl:704 *(p::Base.DFT.FFTW.rFFTWPlan{Complex{Float64},1,false,N} where N, x::Union{Base.ReshapedArray{Complex{Float64},N,A,MI} where MI<:Tuple{Vararg{Base.MultiplicativeInverses.SignedMultiplicativeInverse{Int64},N} where N} where A<:DenseArray, DenseArray{Complex{Float64},N}, SubArray{Complex{Float64},N,A,I,L} where L} where I<:Tuple{Vararg{Union{Base.AbstractCartesianIndex, Int64, Range{Int64}},N} where N} where A<:Union{Base.ReshapedArray{T,N,A,MI} where MI<:Tuple{Vararg{Base.MultiplicativeInverses.SignedMultiplicativeInverse{Int64},N} where N} where A<:DenseArray where N where T, DenseArray}) where N in Base.DFT.FFTW at fft/FFTW.jl:711 *(p::Base.DFT.FFTW.r2rFFTWPlan{T,K,false,N} where N, x::Union{Base.ReshapedArray{T,N,A,MI} where MI<:Tuple{Vararg{Base.MultiplicativeInverses.SignedMultiplicativeInverse{Int64},N} where N} where A<:DenseArray, DenseArray{T,N}, SubArray{T,N,A,I,L} where L} where I<:Tuple{Vararg{Union{Base.AbstractCartesianIndex, Int64, Range{Int64}},N} where N} where A<:Union{Base.ReshapedArray{T,N,A,MI} where MI<:Tuple{Vararg{Base.MultiplicativeInverses.SignedMultiplicativeInverse{Int64},N} where N} where A<:DenseArray where N where T, DenseArray}) where {T, K, N} in Base.DFT.FFTW at fft/FFTW.jl:790 *(p::Base.DFT.FFTW.r2rFFTWPlan{T,K,true,N} where N, x::Union{Base.ReshapedArray{T,N,A,MI} where MI<:Tuple{Vararg{Base.MultiplicativeInverses.SignedMultiplicativeInverse{Int64},N} where N} where A<:DenseArray, DenseArray{T,N}, SubArray{T,N,A,I,L} where L} where I<:Tuple{Vararg{Union{Base.AbstractCartesianIndex, Int64, Range{Int64}},N} where N} where A<:Union{Base.ReshapedArray{T,N,A,MI} where MI<:Tuple{Vararg{Base.MultiplicativeInverses.SignedMultiplicativeInverse{Int64},N} where N} where A<:DenseArray where N where T, DenseArray} where N) where {T, K} in Base.DFT.FFTW at fft/FFTW.jl:797 *(p::Base.DFT.FFTW.DCTPlan{T,5,false}, x::Union{Base.ReshapedArray{T,N,A,MI} where MI<:Tuple{Vararg{Base.MultiplicativeInverses.SignedMultiplicativeInverse{Int64},N} where N} where A<:DenseArray, DenseArray{T,N}, SubArray{T,N,A,I,L} where L} where I<:Tuple{Vararg{Union{Base.AbstractCartesianIndex, Int64, Range{Int64}},N} where N} where A<:Union{Base.ReshapedArray{T,N,A,MI} where MI<:Tuple{Vararg{Base.MultiplicativeInverses.SignedMultiplicativeInverse{Int64},N} where N} where A<:DenseArray where N where T, DenseArray} where N) where T in Base.DFT.FFTW at fft/dct.jl:97 *(p::Base.DFT.FFTW.DCTPlan{T,4,false}, x::Union{Base.ReshapedArray{T,N,A,MI} where MI<:Tuple{Vararg{Base.MultiplicativeInverses.SignedMultiplicativeInverse{Int64},N} where N} where A<:DenseArray, DenseArray{T,N}, SubArray{T,N,A,I,L} where L} where I<:Tuple{Vararg{Union{Base.AbstractCartesianIndex, Int64, Range{Int64}},N} where N} where A<:Union{Base.ReshapedArray{T,N,A,MI} where MI<:Tuple{Vararg{Base.MultiplicativeInverses.SignedMultiplicativeInverse{Int64},N} where N} where A<:DenseArray where N where T, DenseArray} where N) where T in Base.DFT.FFTW at fft/dct.jl:100 *(p::Base.DFT.FFTW.DCTPlan{T,K,true}, x::Union{Base.ReshapedArray{T,N,A,MI} where MI<:Tuple{Vararg{Base.MultiplicativeInverses.SignedMultiplicativeInverse{Int64},N} where N} where A<:DenseArray, DenseArray{T,N}, SubArray{T,N,A,I,L} where L} where I<:Tuple{Vararg{Union{Base.AbstractCartesianIndex, Int64, Range{Int64}},N} where N} where A<:Union{Base.ReshapedArray{T,N,A,MI} where MI<:Tuple{Vararg{Base.MultiplicativeInverses.SignedMultiplicativeInverse{Int64},N} where N} where A<:DenseArray where N where T, DenseArray} where N) where {T, K} in Base.DFT.FFTW at fft/dct.jl:103 *(p::Base.DFT.Plan{T}, x::AbstractArray) where T in Base.DFT at dft.jl:214 *(α::Number, p::Base.DFT.Plan) in Base.DFT at dft.jl:257 *(p::Base.DFT.Plan, α::Number) in Base.DFT at dft.jl:258 *(I::UniformScaling, p::Base.DFT.ScaledPlan) in Base.DFT at dft.jl:259 *(p::Base.DFT.ScaledPlan, I::UniformScaling) in Base.DFT at dft.jl:260 *(I::UniformScaling, p::Base.DFT.Plan) in Base.DFT at dft.jl:261 *(p::Base.DFT.Plan, I::UniformScaling) in Base.DFT at dft.jl:262 *(x::P, y::Real) where P<:Base.Dates.Period in Base.Dates at dates/periods.jl:87 *(a, b, c, xs...) in Base at operators.jl:424 * is highly polymorphic
5. 5. Type hierarchy in Julia Any AbstractArray{T,N} Array{T,N} Matrix{T} Symmetric{T,S} Matrix{Float64} NumberString Complex{T} Real FloatingPoint Rational{T} (a tiny part thereof) Float64 Float16
7. 7. Multimethods Any AbstractArray{T,N} Array{T,N} Matrix{T} Symmetric{T,S} Matrix{Float64} NumberString Complex{T} Real FloatingPoint Rational{T} Float64 Float16 double(x) = 2*x double(x::Rational{T}) = Rational{T}(2*x.num, x.den) double(x::Int) = x << 1 Integer Int BigInt
8. 8. Multimethods Any AbstractArray{T,N} Array{T,N} Matrix{T} Symmetric{T,S} Matrix{Float64} NumberString Complex{T} Real FloatingPoint Rational{T} Float64 Float16 double(x) = 2*x double(x::Rational{T}) = Rational{T}(2*x.num, x.den) double(x::Integer) = x << 1 Integer Int BigInt Most generic deﬁnition gets you more features for free
9. 9. Matrix types in Julia • Dense matrix: Matrix{T} = AbstractArray{T, 2} • Sparse matrix: SparseMatrixCSC{T} • Special matrix types: Symmetric{T}, SymTridiagonal{T}, ... • Types with the algebra of matrices but not the indexing behavior of 2D arrays • Matrix factorizations: LU{T,S}, QRCompactWY{T, S} • "Algebraic matrices": QRCompactWYQ{T,S}
10. 10. Multi-methods for linear algebra What can I do with/to a thing? compute spectral factorization compute singular values compute singular values and vectors compute eigenvalues generic function objectsmethods Methods can take advantage of special matrix structures eigvals eigfact svdvals svdfact Matrix SymTridiagonal Bidiagonal
11. 11. The secret to Julia’s speed and composability You can deﬁne new types with new algebra at run time. If the compiler can ﬁgure out exactly which method you need to use when you invoke a function, then it generates optimized code. Composability at run time is strictly more powerful than at compile time, c.f. C++ expression templates, Haskell typeclasses,...
12. 12. easy to call external C functions, e.g. CLAPACK sstev, dstev…
13. 13. textbook algorithm Run time composition enables matrix algebra over user-deﬁned rings
14. 14. Run time composition enables matrix algebra over user-deﬁned rings
15. 15. Native parallelism constructs
16. 16. Claim 1. Julia's generic function system (multimethods/ multiple dispatch) is ergonomically designed to capture mathematical abstraction. Claim 2. We're only just learning how to explain abstractions clearly to a computer. Claim 3. The future of high performance lies in composable abstraction.
17. 17. Implementing Householder notation is serious business • Recently presented at JuliaCon 2017 • Punchline: there are two incompatible abstractions behind vectors that are impossible to implement perfectly in any programming language
18. 18. The surface problem (pre-0.6) I can't compute inner products with Julia! julia> u = [1, 2, 3]; julia> u'*u 1-element Array{Int64,1}: 14 I am an unhappy mathematician! Don't you Julia people know any mathematics?? julia> u*u'/u'*u ERROR: DimensionMismatch("second dimension of A, 1, does not match length of x, 3") in gemv! (::Array{Float64,1}, ::Char, ::Array{Float64,2}, ::Arra y{Float64,1}) at ./linalg/matmul.jl:219 in *(::Array{Float64,2}, ::Array{Int64,1}) at ./ linalg/matmul.jl:80 julia> :'(
19. 19. So I did some research...
20. 20. Deﬁnition 1. A vector is a column of a matrix. Deﬁnition 2. A vector is that which has the algebra of a vector space. naturally isomorphic but not identical AType 2 vector is "ﬂat": no intrinsic orientation as column or row
21. 21. Why these two deﬁnitions?  A brief history of matrices and vectors matrix universe antiquity - 1937 vector universe 1844 - 1930 hypercomplex numbers coordinate geometry simultaneous equations determinants linear maps Householder notation matrix+vector universe  1924 - present linear algebra tuples vector spaces eigenvalues operators
22. 22. A brief history of matrices and vectors matrix universe antiquity - 1937 vector universe 1844 - 1930 hypercomplex numbers scalar/vector products simultaneous equations determinants linear maps Householder notation matrix+vector universe  1924 - present linear algebra tuples vector spaces eigenvalues operators Deﬁnition 1. A vector is a column of a matrix, i.e. a column matrix. Stewart (1966), ... Deﬁnition 2. A vector is that which has the algebra of a vector space. Peano (1888), van der Waerden (1931)Weyl (1928) ?  "column matrix" coined Cullis (1913)  A = [Amn] and b = [bm] notations Weyl (1928), van der Waerden (1931) , Turnbull & Aitken (1932)... column matrices form a vector space Schrier & Sperner (1931) ? "n-tuple" (n-Tupel) coined
23. 23. A brief history of matrices and vectors matrix universe antiquity - 1937 vector universe 1844 - 1930 hypercomplex numbers scalar/vector products simultaneous equations determinants linear maps Householder notation matrix+vector universe  1924 - present linear algebra tuples vector spaces eigenvalues operators 1924 | 1961 Uniﬁcation era: Matrix-vector relationship needed explanation &' Post-uniﬁcation: "It's so obvious" lel 😹 1967 | now Pre-uniﬁcation era 🦊🐺🐳🐉🦀 now told with 100% more emoji!
24. 24. Van der Waerden, ModerneAlgebra (1931) 1931: (Noether, Artin and) van der Waerden, Moderne Algebra 1966: van der Waerden, Algebra 6/e 1970: English translation: https://archive.org/details/algebra01waer The ﬁrst comprehensive treatise on abstract algebra
25. 25. 1931: (Noether, Artin and) van der Waerden, Moderne Algebra 1966: van der Waerden, Algebra 6/e 1970: English translation: https://archive.org/details/algebra01waer The ﬁrst comprehensive treatise on abstract algebra coefﬁcientvector tuple column matrix Van der Waerden, ModerneAlgebra (1931)
26. 26. 1931: (Noether, Artin and) van der Waerden, Moderne Algebra 1966: van der Waerden, Algebra 6/e 1970: English translation: https://archive.org/details/algebra01waer The ﬁrst comprehensive treatise on abstract algebra vector tuple column matrix coefﬁcient (ring element) satisﬁes the algebra of a vector space an indexable collection of coefﬁcients; no algebra satisﬁes the linear algebra of matrices Van der Waerden, ModerneAlgebra (1931)
27. 27. 1931: (Noether, Artin and) van der Waerden, ModerneAlgebra 1966: van der Waerden, Algebra 6/e 1970: English translation: https://archive.org/details/algebra01waer The ﬁrst comprehensive treatise on abstract algebra vector tuple column matrix coefﬁcient (ring element) satisﬁes the algebra of a vector space satisﬁes the linear algebra of matrices most of these words are horribly overloaded an indexable collection of coefﬁcients; no algebra Van der Waerden, ModerneAlgebra (1931)
28. 28. 1935: Thurstone, The Vectors Of Mind: Multiple FactorAnalysis For The Isolation Of Primary Traits https://archive.org/details/vectorsofmindmul010122mbp A quote from the transitional era
29. 29. fast forward to 1953... Alston S. Householder invents the ﬁeld of numerical analysis
30. 30. All numerical analysts today use Householder notation • Three types: scalars α,β,…; vectors, a, b, …; and matrices A,B,… • Inner product u'*v  vector, vector ⟶ scalar • Outer product u*v'  vector, vector ⟶ matrix • Quadratic/bilinear forms u'*A*v  vector, matrix, vector ⟶ scalar only transpose (') and multiplication (*) are needed to write these expressions A. S. Householder, Introduction to NumericalAnalysis (1953)
31. 31. Householder notation only needs matrix operations A. S. Householder, Introduction to NumericalAnalysis (1953) • Inner product u'*v  vector, vector ⟶ scalar u  vector v  vector U  column matrix V  column matrix '  matrix transpose U'  row matrix U'V  1×1 matrix*  matrix multiply u'*v  "true"  scalar + isomorphisms vector to matrix vector to matrix matrix to "true" scalar
32. 32. Householder notation only needs matrix operations A. S. Householder, Introduction to NumericalAnalysis (1953) • Inner product u'*v  vector, vector ⟶ scalar u  vector v  vector U  column matrix V  column matrix '  matrix transpose U'  row matrix U'V  1×1 matrix*  matrix multiply u'*v  "true"  scalar + isomorphisms vector to matrix vector to matrix matrix to "true" scalar Julia's deﬁnition of vector transpose (≤v0.5)
33. 33. Why Julia ≤v0.5 was broken • Inner product u'*v  vector, vector ⟶ scalar u  vector v  vector U  column matrix '  matrix transpose U'  row matrix U'v  1-vector*  matrix-vector product u'*v  "true"  scalar not enough isomorphisms vector to matrix vector to "true" scalar Julia's deﬁnition of vector transpose (≤v0.5) X Result: unhappy Einstein
34. 34. MATLAB ⟵ The Matrix Laboratory An anomaly in modern languages because it still uses Type 1 (column) vectors "Matrices with one dimension equal to one and the other greater than one are called vectors" - MATLAB R2016a manual "For MATLAB, the matrix is the only data type (although scalars, vectors and text are special cases)" - C. Moler, UNM Technical Report (1980) MATLAB® is a trademark of The MathWorks, Inc.
35. 35. MATLAB has no true scalars or vectors • Trailing singleton dimension rule allows on-demand conversion between 1×1 matrix and scalar, vector and column matrix. u  vector v  vector U  column matrix V  column matrix '  matrix transpose U'  row matrix U'V  1×1 matrix*  matrix multiply u'*v  "true"  scalar add [,1] add [,1] drop [1,1] in future  expressions  if needed
36. 36. MATLAB's * must implement (array) shape-based dispatch
37. 37. Language design considerations for Householder notation • Can we rely on just transpose (') and multiplication (*) to write these expressions? • If not, need special parsing rules to make Householder work. Make _'*_ its own special operator    => u'*v != (u')*v    Special parsing rules add complexity to a language.
38. 38. Let A be a m×n matrix and b be a n-vector. Then we observe the following design matrix: Impossible to express Householder notation with only scalar, vector and matrix types!! A*b is a m-vector A*b is a m×1 matrix b' == b u*v' == u'*v outer product wrong u'*A*v is a 1x1 matrix quadratic product wrong b' is a 1×n matrix u'*v is a 1-vector inner product wrong
39. 39. A*b is a m-vector A*b is a m×1 matrix b' == b u*v' == u'*v outer product wrong | | | u'*A*v is a 1x1 matrix quadratic product wrong | | | b' is a 1×n matrix u'*v is a 1-vector inner product wrong b' is a row vector ✅ Householder works Thus was introduced the RowVector... Can express Householder notation with scalar, vector, row vector, and matrix types!
40. 40. A*b is a m-vector A*b is a m×1 matrix b' is an error Lua/Torch b' == b Mathematica  Python Ruby (numo) b' is a 1×n matrix Julia (≤ v0.5) R (MATLAB) b' is a row n-vector Julia (v0.6) PARI/GP What do other languages do?
41. 41. How row vectors work New deﬁnitions of ' and * needed, but syntax tree is simpler u  vector v  vector '  vector transpose u'  row vector u'v  scalar*  row-column product
42. 42. Claim 1. Julia's generic function system (multimethods/ multiple dispatch) is ergonomically designed to capture mathematical abstraction. Claim 2. We're only just learning how to explain abstractions clearly to a computer. Claim 3. The future of high performance lies in composable abstraction.
43. 43. High Performance Julia (2017) • Statistical computations for genomics  BioJulia, OpenMendel, MIT/Intel, ... • Numerical optimization and automatic diﬀerentiation  JuMP (MIT), LowRankModels, ... • Variational Bayesian inference  Celeste (LBNL), ... • Quantitative economics  QuantEcon, FRBNY models, ... • Other industry uses
44. 44. (non-restarted) GKL with partial reorthogonalization ω-recurrence threshold triggered 20 singular values resnorm threshold = 1e-8
45. 45. (non-restarted) GKL with partial reorthogonalization 20 singular values resnorm threshold = 1e-8 ω-recurrence threshold triggered
46. 46. thick-restart GKL with adaptive one-sided full reorthogonalization restart threshold 20 singular values resnorm threshold = 1e-8
47. 47. thick-restart GKL with adaptive one-sided full reorthogonalization restart threshold 20 singular values resnorm threshold = 1e-8
48. 48. βﬁnal ||Δθ||1 ||Δθ||∞ FlashPCA 2280 2616.1 s 6.4 4E+04 4E+04 Thick restart GKL svdl in IterativeSolvers 280 matvecs 397.8 s 6.4 2E-03 2E-03 (non-restarted) GKL with partial reorthogonalization 321 302.4 s 15072 6E-04 6E-04 PROPACK called from Julia (partial reorthogonalization) 298 264.6 s - 4E-08 4E-08 Implicitly restarted Arnoldi (ARPACK called from Julia) 355 858.5 s 7.8 6E-03 1E-03 80000 x 40000 model problem (density = 0.34) 20 singular values, no vectors, thres. = 1E-8, 16-thread OpenBLAS
49. 49. @inline function getindex(M::PLINK1Matrix, i, j) offset = (i-1)*M.n+(j-1) #Assume row major byteoffset = (offset >> 2) + 4 crumboffset = offset & 0b00000011 @inbounds rawbyte = M.data[byteoffset] rawcrumb = (rawbyte >> 6-2crumboffset) & 0b11 ifelse(rawcrumb==0, rawcrumb, rawcrumb-0x01) end function A_mul_B!{T}(y::AbstractVector{T}, M::PLINK1Matrix, b::AbstractVector{T}) y[:] = zero(T) @fastmath @inbounds for i=1:M.m δy = zero(T) @simd for j=1:4:M.n x = M[i,j]; z = b[j] x2 = M[i,j+1]; z2 = b[j+1] x3 = M[i,j+2]; z3 = b[j+2] x4 = M[i,j+3]; z4 = b[j+3] δy += x*z + x2*z2 + x3*z3 + x4*z4 end y[i] += δy end y end the same code developed for ﬂoating point matrices also accepts custom genotype data formats 8000x4000 matvec: Matrix{Float64} (OpenBLAS) 115 ms PLINK1Matrix 115 ms
50. 50. 2 4 8 16 32 64 128 256 512 1024 2048 4096 8192 16384 32768 65536 131072 2621440 1000 2000 3000 4000 5000 6000 7000 8000 9000 10000 11000 12000 13000 14000 speed(mflops) intel-mkl-dfti in-place intel-mkl-dfti out-of-place fftw3 out-of-place fftw3 in-place fftw3-no-simd out-of-place fftw3-no-simd in-place dfftpack emayer julia bloodworth cross cwplib esrfft double-precision complex, 1d transforms powers of two FFT in pure Julia code already comparable to FFTPACK [ actually was even a bit better; some recent inlining regressions in Julia snapshots Steven Johnson, 2014-06 Presented at NAIS Codegen workshop
51. 51. High Performance Julia (2017) • Statistical computations for genomics  BioJulia, OpenMendel, MIT/Intel, ... • Numerical optimization and automatic diﬀerentiation  JuMP (MIT), LowRankModels, ... • Variational Bayesian inference  Celeste (LBNL), ... • Quantitative economics  QuantEcon, FRBNY models, ... • Other industry uses
52. 52. High Performance Julia (2017) • Celeste: high performance astronomical imaging project at LBNL • Pure Julia code for variational Bayesian inference, implemented on a GPAS model written in Julia/MPI-3 • Recent keynote at JuliaCon 2017:  ~1.5 Tﬂops/s on 8,192 Xeon cores  largest computation in Julia to date
53. 53. Summary • High level language design for numerical computations = teaching computers how to do algebra • Generic functions like * and ' are heavily overloaded in Julia and can be extended by users • But ﬁrst we have to be absolutely clear about what data structures and algebra we want • Computers are very poor at reasoning implicitly about isomorphisms we take for granted today, e.g. 1×1 matrix ↔ scalar, vector ↔ column
54. 54. But what about Eigen/C++? • Lazy evaluation with many internal templates    ProductReturnType<MatrixXd, VectorXd>,   Transpose<GeneralProduct<MatrixXd, VectorXd>>,  CwiseUnaryOp<internal::scalar_quotient1_op<typename internal::traits<Matrix<double, -1, -1, 0, -1, -1> >::Scalar>, const Eigen::Matrix<double, -1, -1, 0, -1, -1> >  ... A*b is a m-vector A*b is a m×1 matrix b' is a 1×n matrix Eigen Eigen b' is a row n- vector Eigen Eigen • Result type determined by variable assignment    VectorXd y = x.transpose(); // OK! MatrixXd Y = x.transpose(); // OK!
55. 55. But what about arrays? matrix universe antiquity - 1937 vector universe 1844 - 1930 hypercomplex numbers coordinate geometry simultaneous equations determinants linear maps Householder notation matrix+vector universe  1924 - present linear algebra tuples vector spaces eigenvalues operators array universe  1948 - present 1948: Manchester "Baby" - B line 1967: Strachey: Array{T,N} deﬁnition 1957-61: Arrays in FORTRAN, ALGOL
56. 56. But what about indexing? • Type 1 (column) vectors: Indexing is multiplication and transposition b[1] == e1'*b b[1:2] == *b A[1,2] == e1'*A*e2 • Type 2 ("ﬂat") vectors: No such identities exist 0 B B B B @ 1 0 . . . 0 1 0 . . . ... 0 0 . . . ... 1 C C C C A
57. 57. How to index a row vector? • Choice 1: derive rules from Type 1 (column) vectors by transposition b'[1] == (e1'*b)' = (b'*e1) b'[1:2] == b'* • Choice 2: Consciously break these identities. Treat ', * as separate from indexing 0 B B B B @ 1 0 . . . 0 1 0 . . . ... 0 0 . . . ... 1 C C C C A
58. 58. But what about tensors? • At least 3 generalizations of Type 1 and Type 2 vectors! • Type 1 tensors: to distinguish between columnness and rowness, have indexing apparatus that disambiguates A12, A 1 2, A1 2 , A 12 . • Problem: only one of these (A1 2 ) is a matrix • Problem: matrix transpose != permuting dimensions 1↔2 • Problem: matrix transpose is no longer unique:  A 1 2 ⟶ A 2 1 or A2 1 ? • Type 1' tensors: A 1 2, A1 2 are the same; diﬀerent from A12, A 12 • Type 2 tensors: A12, A 1 2, A1 2 , A 12 are all the same