Ekeko
Applicative logic meta-programming
  using core.logic against Eclipse




              Coen De Roover
         Software Languages Lab
         Vrije Universiteit Brussel
Application: program querying
... commonly implemented by tool builders



                                                          {
                                                               syntactic
                           identify code with                  structural
                           characteristics of interest
                                                              control flow
                                                               data flow

e..g., method returning null
	
  public	
  List<Integer>	
  getChildren()	
  {
	
  	
  	
  	
  	
   return	
  null;
	
  }                                                                              idiom
                                                                   corpus       transformer
e..g., method returning empty collection                           miner
public	
  List<Integer>	
  getChildren()	
  {
	
  	
  	
  	
  	
   return	
  Collections.emptyList();
}


                                                          idiom
                                                          marker
                                                                             tools
Eclipse plugin
                       causally connected


Ekeko
        applications
            program querying

            program transformation

            corpus mining



meta-programming library
for Clojure’s core.logic
 logic meta-programming
   specify code characteristics declaratively, leave search to logic engine

 applicative meta-programming
     script queries over workspace
     manipulate workspace
select
programs to
   query



 start REPL



launch query



  browse
  results
Applicative meta-programming
... against Eclipse JDT using Clojure
                          Object
                                                                          rich in information: ASTs, semantics, structure

                        ASTNode




                                                    BodyDeclaration




                                                                                                  MethodDeclaration




     name            extraDimensions              thrownExceptions             returnType2           parameters       constructor        body   typeParameters    modifiers          javadoc




      SimpleName            int                        Name           Type             SingleVariableDeclaration        boolean         Block   TypeParameter    IExtendedModifier        Javadoc




                                                                                                                         statements




                                                                                                                         Statement




                                                                                                                      ReturnStatement       node
(spit	
  "jdt_docu_graph.dot"	
  
	
  	
  (dot/dot	
  (graph	
  [MethodDeclaration                                                                         expression         property
	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  Block	
  
	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  ReturnStatement])))                          Expression         value kind
Applicative meta-programming
... against Eclipse JDT using Clojure
                                                                      Lisp on JVM
 (defn	
  
 	
  	
  reduce-­‐projects!
 	
  	
  "Clojure	
  reduce	
  over	
  all	
  projects	
  in	
  the	
  workspace.
 	
  	
  	
  Destructive	
  as	
  it	
  opens	
  and	
  closes	
  each	
  project	
  sequentially."
 	
  	
  [f	
  initval	
  projects]
 	
  	
  (reduce	
  
 	
  	
  	
  	
  (fn	
  [sofar	
  p]
 	
  	
  	
  	
  	
  	
  (workspace-­‐project-­‐open!	
  p)
 	
  	
  	
  	
  	
  	
  (workspace-­‐wait-­‐for-­‐builds-­‐to-­‐finish)
 	
  	
  	
  	
  	
  	
  (let	
  [result	
  (f	
  sofar	
  p)]
 	
  	
  	
  	
  	
  	
  	
  	
  (workspace-­‐project-­‐close!	
  p)
 	
  	
  	
  	
  	
  	
  	
  	
  result))
 	
  	
  	
  	
  initval
 	
  	
  	
  	
  projects))

                                         lambda                                        Java interop

 =>	
  (reduce-­‐projects!	
  (fn	
  [sofar	
  p]	
  (conj	
  sofar	
  (.getName	
  p)))
 	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  []
 	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  (workspace-­‐projects))
 ["AmbientTalk20080201"	
  "DesignPatterns"	
  "DissertationExamples"	
  "JHotDraw51"	
  
 "JavaTemplatesUnitTests"	
  "MLIForSootUnitTests"	
  "jEdit"]

                       but what if we want to know more than the name of each project?
Applicative logic meta-programming
... specify code characteristics through Ekeko relations, leave search to core.logic
                               collection of all substitutions for ?s and ?e



        (ekeko [?s ?e]
            (ast :ReturnStatement ?s)
            (has :expression ?s ?e)
            (ast :NullLiteral ?e))


                                   such that the following Ekeko relations hold:
                                          ast/2 holds for :ReturnStatement, ?s
                                                                                                ?e is the value of the
                                          has/3 holds for :expression, ?s, and ?e            property named :expression
                                                                                                    of ASTNode ?s
                                          ast/2 holds for :NullLiteral, ?e
   ([?s1 ?e2] ... [?sn ?en])

                   ([#<ReturnStatement return null;
                     #<NullLiteral null>]
                    ...
                    [#<ReturnStatement return null;
                     #<NullLiteral null>])                                          actual search performed by core.logic
Applicative logic meta-programming
... core.logic in a nutshell
               embedding of logic programming
                   port of Kanren [Friedman, Kiselyov, Bird] to Clojure by David Nolen
                   no operational impurities (e.g., cut), yet high performance
                   features tabling, extensible constraints, extensible unification protocols
  core.logic


                logic goals
                      functions that take a substitution
                        either return a possibly infinite stream of substitutions (success) or nil (failure)
                      constructors:
                         always success: s#, always failure: f#, success if arguments unify: ==

                 composing goals

                          introduces lexically scoped variables
                                 chains goals together                                           abstraction

                                                     (fresh	
  [?x	
  ?y]                        (defn	
  g	
  [?y]
                     (fresh	
  [?x	
  ?y]            	
  	
  	
  (conde	
                        	
  	
  (fresh	
  []
                     	
  	
  	
  (==	
  ?x	
  ?y)    	
  	
  	
  	
  [(==	
  ?x	
  1)	
  ..]     	
  	
  	
  	
  	
  (==	
  ?y	
  5)))	
  
                     	
  	
  	
  (==	
  ?x	
  5))    	
  	
  	
  	
  [(==	
  ?x	
  5)	
  ..]))   	
  
                                                                                                  (fresh	
  [?x]
                                                    interleaves substitutions from goals          	
  	
  (g	
  ?x))	
  
Applicative logic meta-programming
... using relations provided by the Ekeko library


                                                                    {
                                                                             syntactic
                                             concern                         structural
                                          characteristics                    control flow
                                                                             data flow




                                                                    {
                                                                              JDT DOM
                                                derived                       JDT Model
                                                 from                         control flow graph
                                                            SOOT data flow analyses
 e.g. relation of all JDT reference-valued expressions that may alias at run-time
      (defn	
  
      	
  	
  ast-­‐references-­‐may-­‐alias
      	
  	
  [?ast1	
  ?ast2]
      	
  	
  (fresh	
  [?set1	
  ?set2	
  ?model]
      	
  	
  	
  	
  (ast-­‐reference-­‐soot-­‐model-­‐points-­‐to	
  ?ast1	
  ?model	
  ?set1)	
  
      	
  	
  	
  	
  (ast-­‐reference-­‐soot-­‐model-­‐points-­‐to	
  ?ast2	
  ?model	
  ?set2)
      	
  	
  	
  	
  (succeeds	
  (.hasNonEmptyIntersection	
  ^PointsToSet	
  ?set1	
  ?set2))))
Applicative logic meta-programming
... in practice: specifying idiom characteristics incrementally
     (ekeko*	
  [?s	
  ?e]
     	
  	
  (ast	
  :ReturnStatement	
  ?s)	
                                 familiar query
     	
  	
  (has	
  :expression	
  ?s	
  ?e)	
  
     	
  	
  (ast	
  :NullLiteral	
  ?e))



     (defn
     	
  	
  statement-­‐returning-­‐null                                 query in reusable form
     	
  	
  [?s]
     	
  	
  (fresh	
  [?e]	
  
     	
  	
  	
  	
  (ast	
  :ReturnStatement	
  ?s)
     	
  	
  	
  	
  (has	
  :expression	
  ?s	
  ?e)
     	
  	
  	
  	
  (ast	
  :NullLiteral	
  ?e)))


     (defn	
  
     	
  	
  method-­‐returning-­‐null-­‐for-­‐type                                  same, but with the return type of the enclosing method
     	
  	
  [?m	
  ?t]
     	
  	
  (fresh	
  [?s]
     	
  	
  	
  	
  	
  	
  	
  	
  	
  (statement-­‐returning-­‐null	
  ?s)                      supposed to subtype java.util.Collection
     	
  	
  	
  	
  	
  	
  	
  	
  	
  (ast-­‐encompassing-­‐method	
  ?s	
  ?m)
     	
  	
  	
  	
  	
  	
  	
  	
  	
  (has	
  :returnType2	
  ?m	
  ?t)))
Applicative logic meta-programming
... in practice: specifying idiom characteristics incrementally
(defn
	
  	
  method-­‐returning-­‐null-­‐for-­‐ctype                                            same, but only if method returns a Collection type
	
  	
  [?method	
  ?ast-­‐type]
	
  	
  (fresh	
  [?k	
  ?c-­‐type	
  ?type]
	
  	
  	
  	
  	
  	
  	
  	
  	
  (type-­‐qualifiedname	
  ?c-­‐type	
  "java.util.Collection")                    what if there are many method exits?
	
  	
  	
  	
  	
  	
  	
  	
  	
  (method-­‐returning-­‐null-­‐for-­‐type	
  ?method	
  ?ast-­‐type)
	
  	
  	
  	
  	
  	
  	
  	
  	
  (ast-­‐type-­‐type	
  ?k	
  ?ast-­‐type	
  ?type)
	
  	
  	
  	
  	
  	
  	
  	
  	
  (type-­‐super-­‐type	
  ?type	
  ?collection-­‐type))


(defn
	
  	
  method-­‐always-­‐returning-­‐null-­‐for-­‐ctype                                                     same, but only if all paths through the method
	
  	
  [?m	
  ?t]                                                                                             end in a return statement that returns null
	
  	
  (fresh	
  [?cfg]
	
  	
  	
  	
  	
  	
  	
  	
  	
  (method-­‐returning-­‐null-­‐for-­‐ctype	
  ?m	
  ?t)
	
  	
  	
  	
  	
  	
  	
  	
  	
  (method-­‐cfg	
  ?m	
  ?cfg)	
  
	
  	
  	
  	
  	
  	
  	
  	
  	
  (method-­‐cfg-­‐entry	
  ?m	
  ?entry)
	
  	
  	
  	
  	
  	
  	
  	
  	
  (qwal	
  ?cfg	
  ?entry	
  ?end	
  []
               	
  	
  	
  	
  	
  	
  	
  	
  (qall	
  
                                                          (q=>*)                                                     universally quantified regular path expression: 0
	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  (qcurrent	
  [?return]                                  or more transitions before reaching the end of the
	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  (statement-­‐returning-­‐null	
  ?s)))))))       path, which has to be a statement returning null

             Reinout Stevens’
                damp.qwal
The idiom marker tool in action




         continuously adds a marker next to ?m
Status

being ported from SOUL




 but you can already give it a spin!
      https://github.com/cderoove/damp.ekeko

Ekeko Technology Showdown at SoTeSoLa 2012

  • 1.
    Ekeko Applicative logic meta-programming using core.logic against Eclipse Coen De Roover Software Languages Lab Vrije Universiteit Brussel
  • 2.
    Application: program querying ...commonly implemented by tool builders { syntactic identify code with structural characteristics of interest control flow data flow e..g., method returning null  public  List<Integer>  getChildren()  {           return  null;  } idiom corpus transformer e..g., method returning empty collection miner public  List<Integer>  getChildren()  {           return  Collections.emptyList(); } idiom marker tools
  • 3.
    Eclipse plugin causally connected Ekeko applications program querying program transformation corpus mining meta-programming library for Clojure’s core.logic logic meta-programming specify code characteristics declaratively, leave search to logic engine applicative meta-programming script queries over workspace manipulate workspace
  • 4.
    select programs to query start REPL launch query browse results
  • 5.
    Applicative meta-programming ... againstEclipse JDT using Clojure Object rich in information: ASTs, semantics, structure ASTNode BodyDeclaration MethodDeclaration name extraDimensions thrownExceptions returnType2 parameters constructor body typeParameters modifiers javadoc SimpleName int Name Type SingleVariableDeclaration boolean Block TypeParameter IExtendedModifier Javadoc statements Statement ReturnStatement node (spit  "jdt_docu_graph.dot"      (dot/dot  (graph  [MethodDeclaration expression property                                      Block                                        ReturnStatement]))) Expression value kind
  • 6.
    Applicative meta-programming ... againstEclipse JDT using Clojure Lisp on JVM (defn      reduce-­‐projects!    "Clojure  reduce  over  all  projects  in  the  workspace.      Destructive  as  it  opens  and  closes  each  project  sequentially."    [f  initval  projects]    (reduce          (fn  [sofar  p]            (workspace-­‐project-­‐open!  p)            (workspace-­‐wait-­‐for-­‐builds-­‐to-­‐finish)            (let  [result  (f  sofar  p)]                (workspace-­‐project-­‐close!  p)                result))        initval        projects)) lambda Java interop =>  (reduce-­‐projects!  (fn  [sofar  p]  (conj  sofar  (.getName  p)))                                          []                                          (workspace-­‐projects)) ["AmbientTalk20080201"  "DesignPatterns"  "DissertationExamples"  "JHotDraw51"   "JavaTemplatesUnitTests"  "MLIForSootUnitTests"  "jEdit"] but what if we want to know more than the name of each project?
  • 7.
    Applicative logic meta-programming ...specify code characteristics through Ekeko relations, leave search to core.logic collection of all substitutions for ?s and ?e (ekeko [?s ?e] (ast :ReturnStatement ?s) (has :expression ?s ?e) (ast :NullLiteral ?e)) such that the following Ekeko relations hold: ast/2 holds for :ReturnStatement, ?s ?e is the value of the has/3 holds for :expression, ?s, and ?e property named :expression of ASTNode ?s ast/2 holds for :NullLiteral, ?e ([?s1 ?e2] ... [?sn ?en]) ([#<ReturnStatement return null; #<NullLiteral null>] ... [#<ReturnStatement return null; #<NullLiteral null>]) actual search performed by core.logic
  • 8.
    Applicative logic meta-programming ...core.logic in a nutshell embedding of logic programming port of Kanren [Friedman, Kiselyov, Bird] to Clojure by David Nolen no operational impurities (e.g., cut), yet high performance features tabling, extensible constraints, extensible unification protocols core.logic logic goals functions that take a substitution either return a possibly infinite stream of substitutions (success) or nil (failure) constructors: always success: s#, always failure: f#, success if arguments unify: == composing goals introduces lexically scoped variables chains goals together abstraction (fresh  [?x  ?y] (defn  g  [?y] (fresh  [?x  ?y]      (conde      (fresh  []      (==  ?x  ?y)        [(==  ?x  1)  ..]          (==  ?y  5)))        (==  ?x  5))        [(==  ?x  5)  ..]))   (fresh  [?x] interleaves substitutions from goals    (g  ?x))  
  • 9.
    Applicative logic meta-programming ...using relations provided by the Ekeko library { syntactic concern structural characteristics control flow data flow { JDT DOM derived JDT Model from control flow graph SOOT data flow analyses e.g. relation of all JDT reference-valued expressions that may alias at run-time (defn      ast-­‐references-­‐may-­‐alias    [?ast1  ?ast2]    (fresh  [?set1  ?set2  ?model]        (ast-­‐reference-­‐soot-­‐model-­‐points-­‐to  ?ast1  ?model  ?set1)          (ast-­‐reference-­‐soot-­‐model-­‐points-­‐to  ?ast2  ?model  ?set2)        (succeeds  (.hasNonEmptyIntersection  ^PointsToSet  ?set1  ?set2))))
  • 10.
    Applicative logic meta-programming ...in practice: specifying idiom characteristics incrementally (ekeko*  [?s  ?e]    (ast  :ReturnStatement  ?s)   familiar query    (has  :expression  ?s  ?e)      (ast  :NullLiteral  ?e)) (defn    statement-­‐returning-­‐null query in reusable form    [?s]    (fresh  [?e]          (ast  :ReturnStatement  ?s)        (has  :expression  ?s  ?e)        (ast  :NullLiteral  ?e))) (defn      method-­‐returning-­‐null-­‐for-­‐type same, but with the return type of the enclosing method    [?m  ?t]    (fresh  [?s]                  (statement-­‐returning-­‐null  ?s) supposed to subtype java.util.Collection                  (ast-­‐encompassing-­‐method  ?s  ?m)                  (has  :returnType2  ?m  ?t)))
  • 11.
    Applicative logic meta-programming ...in practice: specifying idiom characteristics incrementally (defn    method-­‐returning-­‐null-­‐for-­‐ctype same, but only if method returns a Collection type    [?method  ?ast-­‐type]    (fresh  [?k  ?c-­‐type  ?type]                  (type-­‐qualifiedname  ?c-­‐type  "java.util.Collection") what if there are many method exits?                  (method-­‐returning-­‐null-­‐for-­‐type  ?method  ?ast-­‐type)                  (ast-­‐type-­‐type  ?k  ?ast-­‐type  ?type)                  (type-­‐super-­‐type  ?type  ?collection-­‐type)) (defn    method-­‐always-­‐returning-­‐null-­‐for-­‐ctype same, but only if all paths through the method    [?m  ?t] end in a return statement that returns null    (fresh  [?cfg]                  (method-­‐returning-­‐null-­‐for-­‐ctype  ?m  ?t)                  (method-­‐cfg  ?m  ?cfg)                    (method-­‐cfg-­‐entry  ?m  ?entry)                  (qwal  ?cfg  ?entry  ?end  []                (qall   (q=>*) universally quantified regular path expression: 0                              (qcurrent  [?return] or more transitions before reaching the end of the                                  (statement-­‐returning-­‐null  ?s))))))) path, which has to be a statement returning null Reinout Stevens’ damp.qwal
  • 12.
    The idiom markertool in action continuously adds a marker next to ?m
  • 13.
    Status being ported fromSOUL but you can already give it a spin! https://github.com/cderoove/damp.ekeko

Editor's Notes

  • #2 \n
  • #3 begin by explaining title,\nwhat is program querying\ncommonly implemented by tool builders who need to identify code with chars of interest\nfir instance, imagine you want ..\n
  • #4 \n
  • #5 \n
  • #6 begin by explaining title,\nwhat is program querying\ncommonly implemented by tool builders who need to identify code with chars of interest\nfir instance, imagine you want ..\n
  • #7 begin by explaining title,\nwhat is program querying\ncommonly implemented by tool builders who need to identify code with chars of interest\nfir instance, imagine you want ..\n
  • #8 \n
  • #9 \n
  • #10 \n
  • #11 \n
  • #12 \n
  • #13 \n
  • #14 \n