Creating A Language Editor Using Dltk


Published on

  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Creating A Language Editor Using Dltk

  1. 1. Building a Custom Language Editor leveraging DLTKRupal Patel, Kaniska Mandal.Overview:Eclipse base text editing framework:Eclipse provides rich support for creating programming language editors that operate ontext.The text editing framework follows the same architectural principles as the rest of theEclipse Platform. The four layers are the model (core), the view (SWT- StyledText), the controller (JFace - ISourceViewer), and the presentation context (usuallythe workbench part - ITextEditor).MVC Collaboration in Eclipse Text Editing FrameworkThe goal is to create a language editor based upon dynamic language toolkit.Problem Space:Tool vendors need a free flow editor for the business analysts and tool developers tocode in tool-specific languages. The editor should provide all types of editing,debugging and launching facilities (syntax coloring, auto-completion, content assist,code folding, selection, problem markers, outline view, AST parser, etc.).Solution Approach:Eclipse offers a robust language tooling framework.
  2. 2. The Dynamic Languages Toolkit is a set of extensible frameworks for building IDEs fordynamically-typed languages.Structural Elements of an DLTK tool:ProjectsCode folders and code librariesSource modulesTypes, Functions, VariablesPackage and import declarationsWhat makes DLTK an exemplary tool is Type inference:Without inferring types we can’t build good tools for dynamically-typed languages.This is a must for:Content Assistance (Code Completion)Code Navigation (Go To Declaration)Search (Search For References, Read/Write Access, etc)Code AnalysisRefactoringBeyond editing! DLTK a powerhouse of features:Launch and DebugEnvironment configuration based on different interpreter types and their installationsFull featured Eclipse-based debugger for all scripting languages providedRemote debugging and launchingDebug based on open DBGp protocolInteractive ConsoleCommon console protocolRemote consoleCode completion & assistanceAll standard featuresViewsScript ExplorerOutlineCall HierarchiesNavigation (packages, types, functions)
  3. 3. Type HierarchyQuick Outline / Quick HierarchyMore UIProject propertiesWizardsPreference pagesSearch UILeveraging DLTK:There are three ways, a custom language editor can contribute language-specific thingsto DLTK :Language toolkit – parsers, checkers, selection, completion engines, etc.UI language toolkit – label providers, and so on.Other extension points, like documentation providers, view filters, etc.Creating core plugin:After creating project, we need to setup dependencies for:org.eclipse.dltk.core – core of – Eclipse file – Eclipse Resources, Workspace.We use dltk to develop an Editor for a hypothetical Business Modeling Language(BML).Eclipse Nature : com.bml.core.natureThe nature class BMLNature extends ScriptNature class. ScriptNature class has allrequired stuff for nature management.<extension id="nature" point="org.eclipse.core.resources.natures">
  4. 4. <runtime> <run class="com.bml.language.core.BMLNature"> </run> </runtime> </extension>BMLNature Class package com.bml.language.core; import org.eclipse.dltk.core.ScriptNature; public class BMLNature extends ScriptNature{ public static final String NATURE_ID = BMLPlugin.PLUGIN_ID + ".nature"; } ** We can set up incremental builder for project.Creating a simple Language Toolkit:Most non-UI language-specific things are provided to DLTK by a language toolkit. Itsmost important thing in DLTK and language specific code interaction.Language toolkit requires definition of Eclipse project nature, <extension point="org.eclipse.dltk.core.language"> <language class="com.bml.language.core.BMLLanguageToolKit" nature="com.bml.core.nature">
  5. 5. </language> </extension>Primary thing that language toolkit should provide is a source element parser.This parser builds a source module content, so it could be accessible from DLTK (fromscript explorer for example).Other required thing is a source module validation. Basically it could be just checkingfor a file name extension, in more complex cases it could be checks for file headers andmore. UI stuff could be extended gradually as required.BMLLanguageToolKit implements IDLTKLanguageToolkit, which could extendAbstractLanguageToolkit class for some basic implementation.BML Language Parser:Creating a Source Element Parser:This parser is invoked when the editor is initialized to build the document model. Asource element parserextracts structural and reference information from a piece of source. <extension point="org.eclipse.dltk.core.sourceElementParsers"> <parser class="com.bml.language.internal.parsers.BMLSourceElementParser" nature="com.bml.core.nature" priority="0"> </parser> </extension>BMLSourceElementParser which implements ISourceElementParser interface. Thisclass is used to build model for source modules.
  6. 6. public class BMLSourceElementParser implements ISourceElementParser{private ISourceElementRequestor fRequestor;public ModuleDeclaration parseSourceModule(char[] contents,ISourceModuleInfo astCashe, char[] filename) {try {sourceParser = (ISourceParser) DLTKLanguageManager.getSourceParser(BMLNature.NATURE_ID);} catch (CoreException e1) {if (DLTKCore.DEBUG) {e1.printStackTrace();}return null;}ModuleDeclaration moduleDeclaration = sourceParser.parse(null,contents, null);return moduleDeclaration;}public void setRequestor(ISourceElementRequestor requestor) {this.fRequestor = requestor;}
  7. 7. }Extending Source Element parser to build correct modelFor building model we provide ISourceElementRequestor interface which is passed tothe SourceElementParser when building content of source module. It works as visitor.SourceElementParser should call methods to define model elements. i.e types, methods,fields, package declarations. public ModuleDeclaration parseSourceModule(char[] contents, ISourceModuleInfo astCashe, char[] filename) { ISourceParser sourceParser = null; try { sourceParser = (ISourceParser) DLTKLanguageManager .getSourceParser(BMLNature.NATURE_ID); } catch (CoreException e1) { if (DLTKCore.DEBUG) { e1.printStackTrace(); } return null; } ModuleDeclaration moduleDeclaration = sourceParser.parse(null, contents, null); moduleDeclaration.disableRebuild(); List statements = moduleDeclaration.getStatements(); try { fRequestor.enterModule();
  8. 8. namespacesLevel.push("::"); buildModel(statements, TYPE_MODULE, ""); fRequestor.exitModule(contents.length); } catch (Exception e) { if (DLTKCore.DEBUG_PARSER) { e.printStackTrace(); } } return moduleDeclaration; }Invoking Source Parser :The Source Element Parser retrieves the Source Parser throughDLTKLanguageManager and parses the contents to get the ModeleDeclarations.ModuleDeclaration moduleDeclaration = sourceParser.parse(null, contents, null); <extension point="org.eclipse.dltk.core.sourceParsers"> <parser class="com.bml.language.internal.parsers.BMLSourceParser" nature="com.bml.core.nature" priority="0"> </parser> </extension>
  9. 9. 1. BMLSourceParser passes the contents buffer from the bml file to theSimpleBMLParser.which generates the BMLScript.1.a …. script = SimpleBMLParser.parse(contents); ……… CodeScanner reads the file buffer till EOF.public static BMLScript parse(String content) throws ParseException{ CodeScanner scanner = new CodeScanner(content); BMLScript script = parse(scanner, false); return script; }1.b. The SimpleParser will create a script and keep adding new BMLCommands tillEOF.1.c. Every BMLCommand in turn keeps gathering the BMLWords till an EOL isencountered.2. Next, BMLSourceParser reads the BMLScript.Checks for various Substitutions from every word of the command and respectivelyforms the expressions.If this substitution is Quote then a BMLBlockExpression is created , Otherwise if it isjust a command (add/delete operators) a BMLExecuteExpression is created.Everything else is SimpleReference.3. The newly generated expressions are added to the BML Statement.4. Finally the statements are added to the BML module declaration which extends fromASTNode. import org.eclipse.dltk.ast.declarations.ModuleDeclaration; import com.tibco.cep.ui.rulelanguage.internal.parsers.BMLASTBuilder;
  10. 10. public class BMLModuleDeclaration extends ModuleDeclaration{ public BMLModuleDeclaration(int sourceLength) { super(sourceLength, true); } protected void doRebuild() { BMLASTBuilder.buildAST(this, getTypeList(), getFunctionList(), getVariablesList()); } public void rebuildMethods() { BMLASTBuilder.rebuildMethods(this); }}Sample Code ExamplePackage com.bml.samplesBizModel {Declare { Customer bob=Customer.createCustomer(....); Account savings = Account.createAccount(….);}Get {String name = bob.getName();}Put {
  11. 11. bob.setAccount(savings);}}Completion Engine:BMLCompletionEngine <extension point="org.eclipse.dltk.core.completionEngine"> <completionEngine class="com.bml.language.core.codeassist.BMLCompletionEngine" nature="com.bml.core.nature"> </completionEngine> </extension>BMLCompletionEngine sorts the list of proposals and appends it toCompletionProposals which is displayed on the Popup window.1. BMLCompletionEngine calls parser to parse and compute the list of proposals.public class BMLCompletionEngine extendsScriptCompletionEngine {private BMLCompletionParser parser;public BMLCompletionEngine() {this.parser = new BMLCompletionParser();}}BMLCompletionParser extends from BMLAssistParser which implements IassistParser.public class BMLAssistParser implements IAssistParser {public BMLAssistParser() {
  12. 12. try {this.parser = DLTKLanguageManager.getSourceParser(BMLNature.NATURE_ID);} catch (CoreException e) {if (DLTKCore.DEBUG) {e.printStackTrace();}}}}public class BMLCompletionParser extends BMLAssistParser {/*** Called when element could not be found.*/@Overridepublic void handleNotInElement(ASTNode node, int position){if (node instanceof ModuleDeclaration) {ModuleDeclaration unit = (ModuleDeclaration) node;List exprs = new ArrayList();exprs.add(new SimpleReference(position, position, ""));BMLEmptyCompleteStatement statement = newBMLEmptyCompleteStatement(exprs);unit.addStatement(statement);
  13. 13. this.parseBlockStatements(statement, unit, position);}}private static class BMLEmptyCompleteStatement extendsBMLStatement {public BMLEmptyCompleteStatement(List expressions) {super(expressions);}}public void parseBlockStatements(ASTNode node, ASTNodeinNode, int position) {//when completion proposals are found.ASTNode nde = new CompletionOnKeywordOrFunction(completionToken, completionNode, node, keywords);throw new CompletionNodeFound(nde,((ModuleDeclaration) inNode).scope, isMethod);}}**“node” is the ASTNode.“completionNode” is the Expression from the ASTNode.“completionToken” is the Name of the comletionnode.“keywords” are the list of proposals.
  14. 14. Method handleNotInElement is called when an element is not found, and further theASTNode could be resolved as completionNode and completionToken.For Example:“Pack” + Crtl*Space will be the completionNode and the word “Pack” iscompletionToken. The list of possible proposals would be “Package” for thecompletionToken “Pack”.When the list of proposals are found, BMLCompletionParser forms aCompletionOnKeywordOrFunction, ASTNode and throws CompletionNodeFoundException.This RuntimeException is caught by BMLCompletionEngine such as,public class BMLCompletionEngine extendsScriptCompletionEngine {private BMLCompletionParser parser;public BMLCompletionEngine() {this.parser = new BMLCompletionParser();}public void complete(ISourceModule sourceModule, intcompletionPosition,int pos) {this.requestor.beginReporting();boolean contextAccepted = false;try {this.fileName = sourceModule.getFileName();this.actualCompletionPosition = completionPosition;this.offset = pos;
  15. 15. ModuleDeclaration parsedUnit = (ModuleDeclaration)this.parser.parse(sourceModule);if (parsedUnit != null) {if (DEBUG) {System.out.println("COMPLETION - Diet AST :");System.out.println(parsedUnit.toString());}try {this.lookupEnvironment.buildTypeScope(parsedUnit, null);if ((this.unitScope = parsedUnit.scope) != null) {this.source = sourceModule.getSourceContents().toCharArray();parseBlockStatements(parsedUnit,this.actualCompletionPosition);if (DEBUG) {System.out.println("COMPLETION - AST :"); /System.out.println(parsedUnit.toString());}// parsedUnit.resolve();}} catch (CompletionNodeFound e) {// completionNodeFound = true;if (e.astNode != null) {
  16. 16. if (DEBUG) {System.out.print("COMPLETION - Completion node : ");System.out.println(e.astNode.toString());if (this.parser.getAssistNodeParent() != null) {System.out.print("COMPLETION - Parent Node : ");System.out.println(this.parser.getAssistNodeParent());}}// if null then we found a problem in the completion// nodecontextAccepted = complete(e.astNode, this.parser.getAssistNodeParent(), e.scope,e.insideTypeAnnotation);}}}if (this.noProposal && this.problem != null) {if (!contextAccepted) {contextAccepted = true;CompletionContext context = new CompletionContext();context.setOffset(completionPosition);context.setTokenKind(CompletionContext.TOKEN_KIND_UNKNOWN);this.requestor.acceptContext(context);}
  17. 17. this.requestor.completionFailure(this.problem);if (DEBUG) {this.printDebug(this.problem);}}} finally {if (!contextAccepted) {contextAccepted = true;CompletionContext context = new CompletionContext();context.setTokenKind(CompletionContext.TOKEN_KIND_UNKNOWN);context.setOffset(completionPosition);this.requestor.acceptContext(context);}this.requestor.endReporting();}}private boolean complete(ASTNode astNode, ASTNodeastNodeParent,Scope scope, boolean insideTypeAnnotation) {//sorts the list of proposals and appends it to CompletionProposals which getsdisplayed in a //Popup window.}}