• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
Introduction to ad-3.4, an automatic differentiation library in Haskell
 

Introduction to ad-3.4, an automatic differentiation library in Haskell

on

  • 257 views

Re-upload of the English version.

Re-upload of the English version.

Statistics

Views

Total Views
257
Views on SlideShare
248
Embed Views
9

Actions

Likes
0
Downloads
0
Comments
0

1 Embed 9

https://twitter.com 9

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

    Introduction to ad-3.4, an automatic differentiation library in Haskell Introduction to ad-3.4, an automatic differentiation library in Haskell Presentation Transcript

    • (An attempt of) Introduction to ad-3.4,an automatic differentiation library in Haskell 3/31/2013 Ekmett study meeting in Shibuya, Tokyo by NebutaAny comments or correction to the material are welcome
    • About myselfNebuta (@nebutalab) https://github.com/nebutaMy interest in softwares: Programming languages (Haskell, Scala, Ruby, etc) Image processing, data visualization, web design Brainstorming and lifehack methods that take advantage of IT, etc.My research areas: A graduate student, studying biophysical chemistry and quantitative biology (2010−) Imaging live cells, analyzing microscopy images by Scala on ImageJWhere my interest in Haskell came from: MATLAB、ImageJで細胞の顕微鏡画像の解析 (2010年) → MATLAB, Javaはいまいち使いづらい → Scalaっていうイケてる言語がある (2011年) → 関数型? → Haskell 面白い!(2011年)
    • ad-3.4, an automatic differentiation libraryWhat you can do Differentiation of arbitrary mathematical functions Taylor expansion Calculation of gradient, Jacobian, and Hessian, etc.Dependencies array (≥0.2 & <0.5), base (4.*), comonad (≥3), containers (≥0.2 & <0.6), data-reify (0.6.*), erf (2.0.*), free (≥3), mtl (≥2), reflection (≥1.1.6), tagged (≥0.4.2.1), template-haskell (≥2.5 & <2.9)Installation$ sudo cabal install ad simple-reflect 記号で微分するのに使う For symbolic differentiation
    • How to use ad-3.4 https://github.com/ekmett/ad/blob/master/README.markdown#examplesDifferentiation of a single-variable scalar function>> :m + Numeric.AD ※Derivative of a>> diff sin 0 trigonometric function1.0>> :m + Debug.SimpleReflect>> diff sin x -- x :: Expr is defined in Debug.SimpleReflectcos x * 1 Derivative with a symbol!>> diff (x -> if x >= 0 then 1 else 0) 00.0 Not delta function nor undefined.Gradient>> grad ([x,y] -> exp (x * y)) [x,y][0 + (0 + y * (0 + exp (x * y) * 1)),0 + (0 + x * (0 + exp (x* y) * 1))]>> grad ([x,y] -> exp (x * y)) [1,1][2.718281828459045,2.718281828459045]
    • How to use (continued)Taylor expansionPrelude Numeric.AD Debug.SimpleReflect> take 3 $ taylor exp 0 d[exp 0 * 1,1 * exp 0 * (1 * d / 1),(0 * exp 0 + 1 * exp 0 * 1) *(1 * d / 1 * d / (1 + 1))]Prelude Numeric.AD Debug.SimpleReflect> take 3 $ taylor exp x d[exp x * 1,1 * exp x * (1 * d / 1),(0 * exp x + 1 * exp x * 1) *(1 * d / 1 * d / (1 + 1))]Prelude Numeric.AD Debug.SimpleReflect> take 3 $ taylor exp x 0[exp x * 1,1 * exp x * (1 * 0 / 1),(0 * exp x + 1 * exp x * 1) *(1 * 0 / 1 * 0 / (1 + 1))]•Taylor expansion is an infinite list! Taylor expansion (general)•No simplification, and slow in higher order terms Exponential function
    • How to use (continued)Equality of functions>> sin x == sin xTrue>> diff sin xcos x * 1>> diff sin x == cos x * 1True>> diff sin x == cos x * 0.5 * 2FalseCool! (no simplification, though...)And so on.
    • Cf. Mechanism of automatic differentiationRead a Wikipedia article http://en.wikipedia.org/wiki/Automatic_differentiationI don’t understand it yet.(What’s the difference from symbolic differentiation?) ?? 要は、合成関数の微分を 機械的に順次適用していく、 (f + g)’ = f’ + g’ という認識で良いかと思われる It seems to be mechanical, successive application of rules of differentiation for composite functions.
    • I’ll try to appreciate the type and class organization注意:Githubに上がっている最新バージョン(https://github.com/ekmett/ad)は4.0で、Hackage(http://hackage.haskell.org/package/ad-3.4)とは違います。ライブラリの構造が若干違うようです。cabal unpack ad-3.4 でver. 3.4のソースをダウンロードするか、Hackageを見て下さい。 I’ll use ad-3.4 on Hackage, not ad-4.0 on Github
    • Package structure http://new-hackage.haskell.org/package/ad-3.4 ここでは複数のモードの実装のうち、デ フォルトのモードがインポート&再エクス ポートされている クラスの定義 いろいろな自動微分の”モード”の実装 型の定義
    • The starting point for exploration: diff functionNumeric.AD.Mode.Forward 1変数スカラー関数の微分{-# LANGUAGE Rank2Types #-}diff :: Num a => (forall s. Mode s => AD s a -> AD s a) -> a -> adiff f a = tangent $ apply f a 微分対象の関数fは(AD s a->AD s a)型として表されるNumeric.AD.Internal.Forward{-# LANGUAGE Rank2Types, TypeFamilies, DeriveDataTypeable,TemplateHaskell, UndecidableInstances, BangPatterns #-}tangent :: Num a => AD Forward a -> atangent (AD (Forward _ da)) = datangent _ = 0bundle :: a -> a -> AD Forward abundle a da = AD (Forward a da)apply :: Num a => (AD Forward a -> b) -> a -> bapply f a = f (bundle a 1) この部分の型: AD Forward a どうやらAD型が になるようだ。
    • AD typeNumeric.AD.Types An instance of Mode class determines the behavior. e.g.) AD (Forward 10 1) :: Num a => AD Forward a {-# LANGUAGE CPP #-} {-# LANGUAGE Rank2Types, GeneralizedNewtypeDeriving, TemplateHaskell, FlexibleContexts, FlexibleInstances, MultiParamTypeClasses, UndecidableInstances #-} {-# ANN module "HLint: ignore Eta reduce" #-} newtype AD f a = AD { runAD :: f a } deriving (Iso (f a), Lifted, Mode, Primal)
    • Classes that have AD type as an instanceNumeric.AD.Types http://hackage.haskell.org/packages/archive/ad/3.4/doc/ html/Numeric-AD-Types.html#t:AD 例:1変数実数スカラー関数 y = f(x) (fがLiftedのインスタンス && aがFloatingのインスタンス)のとき、  AD f a  はFloatingのインスタンス。 diffの第一引数の型 AD s a -> AD s a は、以下の型に適合する* Floating a => a -> a *こういう言い方が正確か分からないが。
    • Let’s look at some examples of values with AD typeadtest.hsimport Numeric.ADimport Numeric.AD.Typesimport Numeric.AD.Internal.Classesimport Numeric.AD.Internal.Forwardf x = x + 3g x | x > 0 = x | otherwise = 0d = diff (g . f)GHCi*Main> :t (g . f)(g . f) :: (Num c, Ord c) => c -> c*Main> :t (g . f) :: (Num a, Ord a, Mode s) => AD s a -> AD s a(g . f) :: (Num a, Ord a, Mode s) => AD s a -> AD s a :: (Num a, Ord a, Mode s) => AD s a -> AD s a*Main> :t ff :: Num a => a -> a*Main> :t f :: (Num a, Lifted s) => AD s a -> AD s af :: (Num a, Lifted s) => AD s a -> AD s a :: (Num a, Lifted s) => AD s a -> AD s a*Main> :t gg :: (Num a, Ord a) => a -> a*Main> :t dd :: Integer -> Integer
    • Let’s play around with apply and tangentGHCi> :l adtest.hs[1 of 1] Compiling Main ( adtest.hs, interpreted )Ok, modules loaded: Main.[*Main]> :t applyapply :: Num a => (AD Forward a -> b) -> a -> b[*Main]> :t apply fapply f :: Num a => a -> AD Forward a Since f :: Num t0 => t0 -> t0,[*Main] b in the type of apply is restricted to> :t apply f 0 AD Forward aapply f 0 :: Num a => AD Forward a[*Main]> apply f 03[*Main]> :t tangenttangent :: Num a => AD Forward a -> a[*Main]> :t tangent $ apply f 0tangent $ apply f 0 :: Num a => a[*Main]> tangent $ apply f 01
    • Lifted class and Mode classChain rule of differentiation http://hackage.haskell.org/packages/archive/ad/3.4/doc/Wrap a (Num a => a) value into t html/Numeric-AD-Internal-Classes.html#t:Mode Definition of different modes. Mode is a subclass of Lifted http://hackage.haskell.org/packages/archive/ad/3.4/doc/ html/Numeric-AD-Internal-Classes.html#t:Lifted e.g.) Forward is an instance of Lifted and Mode
    • Lifted class defines chain rules of differentiation http://hackage.haskell.org/packages/archive/ad/3.4/doc/html/ src/Numeric-AD-Internal-Classes.html#deriveLiftedderiveLifted (in Numeric.AD.Internal.Classes)というTemplateHaskellの関数がLiftedのインスタンスを作ってくれる。A part of deriveLifted exp1 = lift1_ exp const log1 = lift1 log recip1 sin1 = lift1 sin cos1 cos1 = lift1 cos $ negate1 . sin1 tan1 = lift1 tan $ recip1 . square1 . cos1
    • deriveNumeric generates instances of Num, etc. http://hackage.haskell.org/packages/archive/ad/3.4/doc/html/ src/Numeric-AD-Internal-Classes.html#deriveNumeric deriveNumeric (in Numeric.AD.Internal.Classes) -- | @deriveNumeric f g@ provides the following instances: -- -- > instance (Lifted $f, Num a, Enum a) => Enum ($g a) -- > instance (Lifted $f, Num a, Eq a) => Eq ($g a) -- > instance (Lifted $f, Num a, Ord a) => Ord ($g a) -- > instance (Lifted $f, Num a, Bounded a) => Bounded ($g a) -- -- > instance (Lifted $f, Show a) => Show ($g a) -- > instance (Lifted $f, Num a) => Num ($g a) -- > instance (Lifted $f, Fractional a) => Fractional ($g a) -- > instance (Lifted $f, Floating a) => Floating ($g a) -- > instance (Lifted $f, RealFloat a) => RealFloat ($g a) -- > instance (Lifted $f, RealFrac a) => RealFrac ($g a) -- > instance (Lifted $f, Real a) => Real ($g a)
    • Definition of deriveNumericderiveNumeric :: ([Q Pred] -> [Q Pred]) -> Q Type -> Q [Dec]deriveNumeric f t = do members <- liftedMembers let keep n = nameBase n `elem` members xs <- lowerInstance keep ((classP Num [varA]:) . f) t `mapM` [Enum, Eq,Ord, Bounded, Show] ys <- lowerInstance keep f t `mapM` [Num,Fractional, Floating, RealFloat,RealFrac, Real, Erf, InvErf] return (xs ++ ys)lowerInstance :: (Name -> Bool) -> ([Q Pred] -> [Q Pred]) -> Q Type -> Name -> Q DeclowerInstance p f t n = do#ifdef OldClassI ClassI (ClassD _ _ _ _ ds) <- reify n#else ClassI (ClassD _ _ _ _ ds) _ <- reify n#endif instanceD (cxt (f [classP n [varA]])) (conT n `appT` (t `appT` varA)) (concatMap lower1 ds) where lower1 :: Dec -> [Q Dec] lower1 (SigD n _) | p n = [valD (varP n) (normalB (varE n)) []] wheren = primed n lower1 _ = [] primed n = mkName $ base ++ [prime] This looks an important part, where but a bit difficult.... base = nameBase n h = head base prime | isSymbol h || h `elem` "/*-<>" = ! | otherwise = 1
    • A pitfall?>> :t diff ((**2) :: Floating a => a -> a)diff ((**2) :: Floating a => a -> a) :: Floating a => a -> a>> :t diff ((**2) :: Double -> Double)<interactive>:1:7: Couldnt match expected type `ad-3.4:Numeric.AD.Internal.Types.AD s a0 with actual type `Double Expected type: ad-3.4:Numeric.AD.Internal.Types.AD s a0 -> ad-3.4:Numeric.AD.Internal.Types.AD s a0 Actual type: Double -> Double In the first argument of `diff, namely `((** 2) :: Double -> Double) In the expression: diff ((** 2) :: Double -> Double)You need to keep functions polymorphic for differentiation.import Numeric.ADfunc x = x ** 2func2 x = x ** 3fs = [func, func2]test = map (f -> diff f 1.0) fsSo, this does not compile. (there is a GHC extension to accept this, isn’t there..?)
    • SummaryAn interesting library that shows how to use types andtypeclasses effectively and beautifully.Sorry, but I’m still unclear about how Lifted actuallyexecutes differentiation It seems to be done by Num instance of AD f a, via instance (Lifted f, Num a) => Num (AD f a)...Things that might be improved for application • Simplification of terms • Improving the performance of higher order taylor expansion. • Is automatic integration possible, maybe?These might be a good practice for seasoned Haskellers (I’m not...)
    • 感想• Haskellならではのコードの簡潔さのおかげで、ソースのどこを追っていけば良いかはわりと明確。• 他のEkmett氏のライブラリよりは具体的な対象がはっきりしている分、少しは分かりやすい。• とはいっても、私には若干(相当?)手に余る感じで、深いところまでは解説出来ませんでした。知らないGHC拡張も多し。ただ、人に説明しようとすることで少しは理解が進んで、嬉しい。