1. Building a Custom Language Editor leveraging DLTK
Rupal Patel, Kaniska Mandal.
Overview:
Eclipse base text editing framework:
Eclipse provides rich support for creating programming language editors that operate on
text.
The text editing framework follows the same architectural principles as the rest of the
Eclipse Platform. The four layers are the model (core), the view (SWT- Styled
Text), the controller (JFace - ISourceViewer), and the presentation context (usually
the workbench part - ITextEditor).
MVC Collaboration in Eclipse Text Editing Framework
The goal is to create a language editor based upon dynamic language toolkit.
Problem Space:
Tool vendors need a free flow editor for the business analysts and tool developers to
code in tool-specific languages. The editor should provide all types of editing,
debugging and launching facilities (syntax coloring, auto-completion, content assist,
code folding, selection, problem markers, outline view, AST parser, etc.).
Solution Approach:
Eclipse offers a robust language tooling framework.
2. The Dynamic Languages Toolkit is a set of extensible frameworks for building IDEs for
dynamically-typed languages.
Structural Elements of an DLTK tool:
Projects
Code folders and code libraries
Source modules
Types, Functions, Variables
Package and import declarations
What makes DLTK an exemplary tool is Type inference:
Without inferring types we can’t build good tools for dynamically-typed languages.
This is a must for:
Content Assistance (Code Completion)
Code Navigation (Go To Declaration)
Search (Search For References, Read/Write Access, etc)
Code Analysis
Refactoring
Beyond editing! DLTK a powerhouse of features:
Launch and Debug
Environment configuration based on different interpreter types and their installations
Full featured Eclipse-based debugger for all scripting languages provided
Remote debugging and launching
Debug based on open DBGp protocol
Interactive Console
Common console protocol
Remote console
Code completion & assistance
All standard features
Views
Script Explorer
Outline
Call Hierarchies
Navigation (packages, types, functions)
3. Type Hierarchy
Quick Outline / Quick Hierarchy
More UI
Project properties
Wizards
Preference pages
Search UI
Leveraging DLTK:
There are three ways, a custom language editor can contribute language-specific things
to DLTK :
Language toolkit – parsers, checkers, selection, completion engines, etc.
UI language toolkit – label providers, and so on.
Other extension points, like documentation providers, view filters, etc.
Creating core plugin:
After creating project, we need to setup dependencies for:
org.eclipse.dltk.core – core of DLTK.
org.eclipse.core.filesystem – Eclipse file system.
org.eclipse.core.resources – Eclipse Resources, Workspace.
We use dltk to develop an Editor for a hypothetical Business Modeling Language
(BML).
Eclipse Nature : com.bml.core.nature
The nature class BMLNature extends ScriptNature class. ScriptNature class has all
required stuff for nature management.
<extension id="nature"
point="org.eclipse.core.resources.natures">
4. <runtime>
<run
class="com.bml.language.core.BMLNature">
</run>
</runtime>
</extension>
BMLNature Class
package com.bml.language.core;
import org.eclipse.dltk.core.ScriptNature;
public class BMLNature extends ScriptNature{
public static final String NATURE_ID = BMLPlugin.PLUGIN_ID +
".nature";
}
** We can set up incremental builder for project.
Creating a simple Language Toolkit:
Most non-UI language-specific things are provided to DLTK by a language toolkit. It's
most important thing in DLTK and language specific code interaction.
Language toolkit requires definition of Eclipse project nature,
<extension
point="org.eclipse.dltk.core.language">
<language
class="com.bml.language.core.BMLLanguageToolKit"
nature="com.bml.core.nature">
5. </language>
</extension>
Primary thing that language toolkit should provide is a source element parser.
This parser builds a source module content, so it could be accessible from DLTK (from
script explorer for example).
Other required thing is a source module validation. Basically it could be just checking
for a file name extension, in more complex cases it could be checks for file headers and
more. UI stuff could be extended gradually as required.
BMLLanguageToolKit implements IDLTKLanguageToolkit, which could extend
AbstractLanguageToolkit class for some basic implementation.
BML Language Parser:
Creating a Source Element Parser:
This parser is invoked when the editor is initialized to build the document model. A
source element parser
extracts structural and reference information from a piece of source.
<extension point="org.eclipse.dltk.core.sourceElementParsers">
<parser
class="com.bml.language.internal.parsers.BMLSourceElementParser"
nature="com.bml.core.nature"
priority="0">
</parser>
</extension>
BMLSourceElementParser which implements ISourceElementParser interface. This
class is used to build model for source modules.
7. }
Extending Source Element parser to build correct model
For building model we provide ISourceElementRequestor interface which is passed to
the SourceElementParser when building content of source module. It works as visitor.
SourceElementParser should call methods to define model elements. i.e types, methods,
fields, package declarations.
public ModuleDeclaration parseSourceModule(char[] contents,
ISourceModuleInfo astCashe, char[] filename) {
ISourceParser sourceParser = null;
try {
sourceParser = (ISourceParser) DLTKLanguageManager
.getSourceParser(BMLNature.NATURE_ID);
} catch (CoreException e1) {
if (DLTKCore.DEBUG) {
e1.printStackTrace();
}
return null;
}
ModuleDeclaration moduleDeclaration = sourceParser.parse(null,
contents, null);
moduleDeclaration.disableRebuild();
List statements = moduleDeclaration.getStatements();
try {
fRequestor.enterModule();
8. namespacesLevel.push("::");
buildModel(statements, TYPE_MODULE, "");
fRequestor.exitModule(contents.length);
} catch (Exception e) {
if (DLTKCore.DEBUG_PARSER) {
e.printStackTrace();
}
}
return moduleDeclaration;
}
Invoking Source Parser :
The Source Element Parser retrieves the Source Parser through
DLTKLanguageManager and parses the contents to get the ModeleDeclarations.
ModuleDeclaration moduleDeclaration = sourceParser.parse(null, contents, null);
<extension
point="org.eclipse.dltk.core.sourceParsers">
<parser
class="com.bml.language.internal.parsers.BMLSourceParser"
nature="com.bml.core.nature"
priority="0">
</parser>
</extension>
9. 1. BMLSourceParser passes the contents buffer from the bml file to the
SimpleBMLParser.
which generates the BMLScript.
1.a …. script = SimpleBMLParser.parse(contents);
……… CodeScanner reads the file buffer till EOF.
public static BMLScript parse(String content) throws ParseException{
CodeScanner scanner = new CodeScanner(content);
BMLScript script = parse(scanner, false);
return script;
}
1.b. The SimpleParser will create a script and keep adding new BMLCommands till
EOF.
1.c. Every BMLCommand in turn keeps gathering the BMLWords till an EOL is
encountered.
2. Next, BMLSourceParser reads the BMLScript.
Checks for various Substitutions from every word of the command and respectively
forms the expressions.
If this substitution is Quote then a BMLBlockExpression is created , Otherwise if it is
just a command (add/delete operators) a BMLExecuteExpression is created.
Everything else is SimpleReference.
3. The newly generated expressions are added to the BML Statement.
4. Finally the statements are added to the BML module declaration which extends from
ASTNode.
import org.eclipse.dltk.ast.declarations.ModuleDeclaration;
import com.tibco.cep.ui.rulelanguage.internal.parsers.BMLASTBuilder;
10. public class BMLModuleDeclaration extends ModuleDeclaration{
public BMLModuleDeclaration(int sourceLength) {
super(sourceLength, true);
}
protected void doRebuild() {
BMLASTBuilder.buildAST(this, getTypeList(), getFunctionList(),
getVariablesList());
}
public void rebuildMethods() {
BMLASTBuilder.rebuildMethods(this);
}
}
Sample Code Example
Package com.bml.samples
BizModel {
Declare {
Customer bob=Customer.createCustomer(....);
Account savings = Account.createAccount(….);
}
Get {
String name = bob.getName();
}
Put {
11. bob.setAccount(savings);
}
}
Completion Engine:
BMLCompletionEngine
<extension point="org.eclipse.dltk.core.completionEngine">
<completionEngine
class="com.bml.language.core.codeassist.BMLCompletionEngine"
nature="com.bml.core.nature">
</completionEngine>
</extension>
BMLCompletionEngine sorts the list of proposals and appends it to
CompletionProposals which is displayed on the Popup window.
1. BMLCompletionEngine calls parser to parse and compute the list of proposals.
public class BMLCompletionEngine extends
ScriptCompletionEngine {
private BMLCompletionParser parser;
public BMLCompletionEngine() {
this.parser = new BMLCompletionParser();
}
}
BMLCompletionParser extends from BMLAssistParser which implements IassistParser.
public class BMLAssistParser implements IAssistParser {
public BMLAssistParser() {
12. try {
this.parser = DLTKLanguageManager
.getSourceParser(BMLNature.NATURE_ID);
} catch (CoreException e) {
if (DLTKCore.DEBUG) {
e.printStackTrace();
}
}
}
}
public class BMLCompletionParser extends BMLAssistParser {
/**
* Called when element could not be found.
*/
@Override
public void handleNotInElement(ASTNode node, int position)
{
if (node instanceof ModuleDeclaration) {
ModuleDeclaration unit = (ModuleDeclaration) node;
List exprs = new ArrayList();
exprs.add(new SimpleReference(position, position, ""));
BMLEmptyCompleteStatement statement = new
BMLEmptyCompleteStatement(exprs);
unit.addStatement(statement);
13. this.parseBlockStatements(statement, unit, position);
}
}
private static class BMLEmptyCompleteStatement extends
BMLStatement {
public BMLEmptyCompleteStatement(List expressions) {
super(expressions);
}
}
public void parseBlockStatements(ASTNode node, ASTNode
inNode, int position) {
//when completion proposals are found.
ASTNode nde = new CompletionOnKeywordOrFunction(
completionToken, completionNode, node, keywords);
throw new CompletionNodeFound(nde,
((ModuleDeclaration) inNode).scope, isMethod);
}
}
**
“node” is the ASTNode.
“completionNode” is the Expression from the ASTNode.
“completionToken” is the Name of the comletionnode.
“keywords” are the list of proposals.
14. Method handleNotInElement is called when an element is not found, and further the
ASTNode could be resolved as completionNode and completionToken.
For Example:
“Pack” + Crtl*Space will be the completionNode and the word “Pack” is
completionToken. The list of possible proposals would be “Package” for the
completionToken “Pack”.
When the list of proposals are found, BMLCompletionParser forms a
CompletionOnKeywordOrFunction, ASTNode and throws CompletionNodeFound
Exception.
This RuntimeException is caught by BMLCompletionEngine such as,
public class BMLCompletionEngine extends
ScriptCompletionEngine {
private BMLCompletionParser parser;
public BMLCompletionEngine() {
this.parser = new BMLCompletionParser();
}
public void complete(ISourceModule sourceModule, int
completionPosition,
int pos) {
this.requestor.beginReporting();
boolean contextAccepted = false;
try {
this.fileName = sourceModule.getFileName();
this.actualCompletionPosition = completionPosition;
this.offset = pos;
16. if (DEBUG) {
System.out.print("COMPLETION - Completion node : ");
System.out.println(e.astNode.toString());
if (this.parser.getAssistNodeParent() != null) {
System.out.print("COMPLETION - Parent Node : ");
System.out.println(this.parser.getAssistNodeParent());
}
}
// if null then we found a problem in the completion
// node
contextAccepted = complete(e.astNode, this.parser
.getAssistNodeParent(), e.scope,
e.insideTypeAnnotation);
}
}
}
if (this.noProposal && this.problem != null) {
if (!contextAccepted) {
contextAccepted = true;
CompletionContext context = new CompletionContext();
context.setOffset(completionPosition);
context.setTokenKind(CompletionContext.TOKEN_KIND_UNKNOWN);
this.requestor.acceptContext(context);
}
17. this.requestor.completionFailure(this.problem);
if (DEBUG) {
this.printDebug(this.problem);
}
}
} finally {
if (!contextAccepted) {
contextAccepted = true;
CompletionContext context = new CompletionContext();
context.setTokenKind(CompletionContext.TOKEN_KIND_UNKNOWN);
context.setOffset(completionPosition);
this.requestor.acceptContext(context);
}
this.requestor.endReporting();
}
}
private boolean complete(ASTNode astNode, ASTNode
astNodeParent,
Scope scope, boolean insideTypeAnnotation) {
//sorts the list of proposals and appends it to CompletionProposals which gets
displayed in a //Popup window.
}
}
