9-roslyn-guidelines

Mohammad Rahimi, MSc. Student
Edit by: Morteza Zakeri
4/20/2020
Reverse Engineering Research Laboratory
(www.parsa.iust.ac.ir/reverse-engineering-lab/)
1

New generation of Compilers
Compilers as services via APIs
APIs for code analysis and refactoring
4/20/2020
2

New generation of Compilers
• CLang for C and C++
• https://clang.llvm.org/
• Roslyn for C# and Visual Basic
• https://github.com/dotnet/roslyn
• This week:
• Roslyn
4/20/2020
3

What is Roslyn?
• .NET Compiler Platform, also known by its nickname Roslyn, is a set
of open-source compilers and code analysis APIs.
• Only for C# and Visual Basic .NET languages
• from Microsoft.
• Traditionally compilers have been a black box for application
developers.
• With increasing complexity and demands for source code analysis in modern
integrated development environments, however, compilers need to expose
application programming interfaces (APIs)
• Will help developers to directly perform phases of compilation such as lexical
and syntactic structure analysis of source code.
4/20/2020
4

There are two different ways to find the .NET Compiler Platform SDK in
the Visual Studio Installer
4/20/2020
5

Install using the Visual Studio Installer - Workloads view
The .NET Compiler Platform SDK is not automatically selected as part of the
Visual Studio extension development workload. You must select it as an
optional component.
1- Run Visual Studio Installer
2- Select Modify
3- Check the Visual Studio extension development workload.
4-Open the Visual Studio extension development node in the summary tree.
5-Check the box for .NET Compiler Platform SDK. You'll find it last under the
optional components.
4/20/2020
6

4/20/2020
7

4/20/2020
8

Optionally, you'll also want the DGML editor to display graphs in the
visualizer:
1- Open the Individual components node in the summary tree.
2-Check the box for DGML editor
4/20/2020
9

Select DGML editor
4/20/2020
10

• Install using the Visual Studio Installer - Individual components tab
1-Run Visual Studio Installer
2-Select Modify
3-Select the Individual components tab
4-Check the box for .NET Compiler Platform SDK. You'll find it at the top
under the Compilers, build tools, and runtimes section.
Optionally, you'll also want the DGML editor to display graphs in the visualizer:
1-Check the box for DGML editor. You'll find it under the Code tools section.
4/20/2020
11

4/20/2020
12

4/20/2020
13

Compiler Pipline
First, the parse phase tokenizes and parses source text into syntax that
follows the language grammar
Second, the declaration phase analyzes source and imported metadata to
form named symbols.
Next, the bind phase matches identifiers in the code to symbols.
Finally, the emit phase emits an assembly with all the information built up
by the compiler.
4/20/2020
14

API layers
The .NET compiler SDK consists of two main layers of APIs: compiler
APIs and workspaces APIs.
4/20/2020
15

Compiler APIs
The compiler layer contains the object models that correspond to
information exposed at each phase of the compiler pipeline, both
syntactic and semantic.
The compiler layer also contains an immutable snapshot of a single
invocation of a compiler, including assembly references, compiler
options, and source code files.
There are two distinct APIs that represent the C# language and the
Visual Basic language.
4/20/2020
16

Workspaces APIs
It assists you in organizing all the information about the projects in a
solution into single object model, offering you direct access to the
compiler layer object models without needing to parse files, configure
options, or manage project-to-project dependencies.
4/20/2020
17

Workspaces APIs
In addition, the Workspaces layer surfaces a set of APIs used when
implementing code analysis and refactoring tools that function within a
host environment like the Visual Studio IDE.
4/20/2020
18

Parz Lordeghan Forest Park
4/20/2020
19

Work with syntax
syntax tree serve two important purposes:
1-To allow tools - such as an IDE, add-ins, code analysis tools, and
refactorings - to see and process the syntactic structure of source
code in a user’s project.
2-To enable tools - such as refactorings and an IDE - to create, modify,
and rearrange source code in a natural manner without having use
direct text edits.
4/20/2020
20

Syntax trees
Syntax trees have three key attributes.
1- The first attribute is that syntax trees hold all the source
information in full fidelity.
2- This enables the second attribute of syntax trees. A syntax tree
obtained from the parser can produce the exact text it was
parsed from.
3- The third attribute of syntax trees is that they are immutable and
thread-safe. This means that after a tree is obtained, it is a snapshot
of the current state of the code, and never changes.
4/20/2020
21

Syntax trees
4/20/2020
22

Syntax trees - nodes
These nodes represent syntactic constructs such as declarations,
statements, clauses, and expressions.
Each category of syntax nodes is represented by a separate class
derived from Microsoft.CodeAnalysis.SyntaxNode
All syntax nodes are non-terminal nodes in the syntax tree. each node
has a parent node that can be accessed through the SyntaxNode.Parent
property.
4/20/2020
23

Syntax trees - nodes
Each node has a SyntaxNode.ChildNodes() method, which returns a list of
child nodes in sequential order based on their position in the source text.
Each node also has methods to examine Descendants, such as
DescendantNodes, DescendantTokens, or DescendantTrivia , …
each syntax node subclass exposes all the same children through strongly
typed properties. For example, a BinaryExpressionSyntax node class has
three additional properties specific to binary operators: Left, OperatorToken,
and Right.
4/20/2020
24

Syntax trees - tokens
Syntax tokens are the terminals of the language grammar, representing the
smallest syntactic fragments of the code.
there is only one structure for all kinds of tokens with a mix of properties
that have meaning depending on the kind of token that is being represented.
the literal token has a Value property that tells you the exact decoded integer
value.
The ValueText property tells you the same information as the Value property;
however this property is always typed as String.
4/20/2020
25

Syntax trees - trivia
Syntax trivia represent the parts of the source text that are largely
insignificant for normal understanding of the code, such as white space,
comments, and preprocessor directives.
The single Microsoft.CodeAnalysis.Syntax Trivia type is used to describe all
kinds of trivia.
You can access trivia by inspecting a token’s SyntaxToken.LeadingTrivia or
SyntaxToken.TrailingTrivia collections.
Unlike syntax nodes and tokens, syntax trivia do not have parents.
4/20/2020
26

Syntax trees - Spans
A TextSpan object is the beginning position and a count of characters, both
represented as integers
Each node has two TextSpan properties: Span and FullSpan.
The Span property is the text span from the start of the first token in the
node’s sub-tree to the end of the last token. This span does not include any
leading or trailing trivia.
The FullSpan property is the text span that includes the node’s normal span,
plus the span of any leading or trailing trivia.
4/20/2020
27

Syntax trees - Kinds
Each node, token, or trivia has a SyntaxNode.RawKind property, of type
System.Int32, that identifies the exact syntax element represented.
Each language, C# or Visual Basic, has a single SyntaxKind enumeration
(Microsoft.CodeAnalysis.CSharp.SyntaxKind and
Microsoft.CodeAnalysis.VisualBasic.SyntaxKind, respectively) that lists all the
possible nodes, tokens, and trivia elements in the grammar.
The RawKind property allows for easy disambiguation of syntax node types
that share the same node class.
4/20/2020
28

Syntax trees - Errors
When the parser encounters code that does not conform to the
defined syntax of the language, it uses one of two techniques to create
a syntax tree.
1- First, if the parser expects a particular kind of token but does not
find it, it may insert a missing token into the syntax tree in the location
that the token was expected.
2- Second, the parser may skip tokens until it finds one where it can
continue parsing.
4/20/2020
29

Work with syntax
4/20/2020
30

Lordegan Miniature Waterfall
4/20/2020
31

Work with semantics
You can use it to discover the following:
• The symbols referenced at a specific location in source.
• The resultant type of any expression.
• All diagnostics, which are errors and warnings.
• How variables flow in and out of regions of source.
• The answers to more speculative questions.
4/20/2020
32

Work with semantics - Compilation
A compilation is a representation of everything needed to compile a C#
or Visual Basic program, which includes all the assembly references,
compiler options, and source files.
The compilation contains a variety of methods that help you find and
relate the symbols that have either been declared in the source code or
imported as metadata from an assembly
Similar to syntax trees, compilations are immutable
4/20/2020
33

Work with semantics - Symbols
A symbol represents a distinct element declared by the source code or
imported from an assembly as metadata.
A variety of methods and properties on the Compilation type help you
find symbols
Symbols also contain additional information that the compiler
determines from the source or metadata, such as other referenced
symbols.
4/20/2020
34

Work with semantics - Symbols
Each kind of symbol is represented by a separate interface derived
from Isymbol.
Symbols present a common representation of namespaces, types, and
members, between source code and metadata.
Symbols are similar in concept to the CLR type system as represented
by the System.Reflection API.
4/20/2020
35

Work with a workspace - Workspace
The Workspaces layer is the starting point for doing code analysis and
refactoring over entire solutions.
A workspace is an active representation of your solution as a collection of
projects, each with a collection of documents.
The Workspace provides access to the current model of the solution.
You can also create stand-alone workspaces that are disconnected from the
host environment or used in an application that has no host environment.
4/20/2020
36

Work with a workspace - Solutions, projects,
documents
A solution is an immutable model of the projects and documents. you
can modify solutions by constructing new instances based on existing
solutions and specific changes.
A project is a part of the overall immutable solution model. It
represents all the source code documents, parse and compilation
options, …
A document is also a part of the overall immutable solution model. A
document represents a single source file.
4/20/2020
37

Work with a workspace - Solutions, projects,
documents
4/20/2020
38

Karun Dam 3
4/20/2020
39

Get started with syntax analysis
Create a new C# Stand-Alone Code Analysis Tool project:
• In Visual Studio, choose File > New > Project to display the New Project dialog.
• Under Visual C# > Extensibility, choose Stand-Alone Code Analysis Tool.
• Name your project "SyntaxTreeManualTraversal" and click OK.
4/20/2020
40

syntax analysis
const string programText =
@"using System;
using System.Text;
namespace HelloWorld
{
class Program
{
static void Main(string[] args)
{
Console.WriteLine(""Hello, World!"");
}
}
}";
4/20/2020
41

syntax analysis - syntax tree
Next, add the following code to build the syntax tree for the code text in the
programText constant. Add the following line to your Main method:
SyntaxTree tree = CSharpSyntaxTree.ParseText(programText);
CompilationUnitSyntax root = tree.GetCompilationUnitRoot();
Those two lines create the tree and retrieve the root node of that tree.
You can now examine the nodes in the tree.
4/20/2020
42

syntax analysis – root property
Add these lines to your Main method to display some of the properties
of the root node in the tree:
WriteLine($"The tree is a {root.Kind()} node.");
WriteLine($"The tree has {root.Members.Count} elements in it.");
WriteLine($"The tree has {root.Usings.Count} using statements. They
are:");
foreach (UsingDirectiveSyntax element in root.Usings)
WriteLine($"t{element.Name}");
4/20/2020
43

syntax analysis - APIs
In this example, you're analyzing code you know to explore the APIs.
Add the following code to examine the first member of the root node:
MemberDeclarationSyntax firstMember = root.Members[0];
WriteLine($"The first member is a {firstMember.Kind()}.");
var helloWorldDeclaration
=(NamespaceDeclarationSyntax)firstMember;
4/20/2020
44

syntax analysis – node types
Add the following code to examine what nodes are declared inside the
HelloWorld namespace:
WriteLine($"There are {helloWorldDeclaration.Members.Count} members
declared in this namespace.");
WriteLine($"The first member is a
{helloWorldDeclaration.Members[0].Kind()}.");
4/20/2020
45

syntax analysis - Main method
Add the following code to find the Main method, and cast it to a
var programDeclaration =
(ClassDeclarationSyntax)helloWorldDeclaration.Members[0];
WriteLine($"There are {programDeclaration.Members.Count} members
declared in the {programDeclaration.Identifier} class.");
WriteLine($"The first member is a
{programDeclaration.Members[0].Kind()}.");
var mainDeclaration =
(MethodDeclarationSyntax)programDeclaration.Members[0];
4/20/2020
46

syntax analysis - method declaration node
WriteLine($"The return type of the {mainDeclaration.Identifier}
method is {mainDeclaration.ReturnType}.");
WriteLine($"The method has
{mainDeclaration.ParameterList.Parameters.Count} parameters.");
foreach (ParameterSyntax item in mainDeclaration.ParameterList.Parameters)
WriteLine($"The type of the {item.Identifier} parameter is
{item.Type}.");
WriteLine($"The body text of the {mainDeclaration.Identifier}
method follows:");
WriteLine(mainDeclaration.Body.ToFullString());
var argsParameter = mainDeclaration.ParameterList.Parameters[0];
4/20/2020
47

syntax analysis - Run the program
The tree is a CompilationUnit node.
The tree has 1 elements in it.
The tree has 4 using statements. They are:
System
System.Collections
System.Linq
System.Text
The first member is a NamespaceDeclaration.
There are 1 members declared in this namespace.
The first member is a ClassDeclaration.
4/20/2020
48

syntax analysis - Run the program
There are 1 members declared in the Program class.
The first member is a MethodDeclaration.
The return type of the Main method is void.
The method has 1 parameters.
The type of the args parameter is string[].
The body text of the Main method follows:
{
Console.WriteLine("Hello, World!");
}
4/20/2020
49

Karun Dam 4
4/20/2020
50

Query methods
you can also explore the syntax tree using the query methods defined
on Microsoft.CodeAnalysis.SyntaxNode.
You can use these methods with LINQ to quickly find things in a tree.
The SyntaxNode has query methods as such DescendantNodes
, AncestorsAndSelf and ChildNodes.
4/20/2020
51

Query methods
You can use these query methods to find the argument to the Main
method as an alternative to navigating the tree.
var firstParameters = from methodDeclaration in
root.DescendantNodes().OfType<MethodDeclarationSyntax>
where methodDeclaration.Identifier.ValueText == "Main"
select methodDeclaration.ParameterList.Parameters.First();
var argsParameter2 = firstParameters.Single();
WriteLine(argsParameter == argsParameter2);
4/20/2020
52

Lake Clare
4/20/2020
53

Querying symbols - semantic analysis
Add the text for the Hello World program as a constant in your Program
class:
next page
4/20/2020
54

const string programText = @“
using System;
using System.Collections.Generic;
using System.Text;
namespace HelloWorld
{class Program
{static void Main(string[] args)
{Console.WriteLine(""Hello, World!"");}
}
}";
4/20/2020
55

Next, add the following code to build the syntax tree for the code text
in the programText constant. Add the following line to your Main
method:
SyntaxTree tree = CSharpSyntaxTree.ParseText(programText);
CompilationUnitSyntax root = tree.GetCompilationUnitRoot();
In the code above, build a CSharpCompilation from the tree you
already created.
4/20/2020
56

Add the following line to your Main method to create a compilation of
your syntax tree, including the reference to the appropriate assembly:
var compilation = CSharpCompilation.Create("HelloWorld")
.AddReferences(MetadataReference.CreateFromFile
typeof(string).Assembly.Location)) .AddSyntaxTrees(tree);
4/20/2020
57

semantic analysis - Querying symbols
The CSharpCompilation.AddReferences method adds references to the
compilation.
The MetadataReference.CreateFromFile method loads an assembly as a
reference.
4/20/2020
58

Querying the semantic model
A SemanticModel can answer questions like "What names are in scope
at this location?", "What members are accessible from this method?",
"What variables are used in this block of text?", and "What does this
name/expression refer to?" Add this statement to create the semantic
model:
SemanticModel model = compilation.GetSemanticModel(tree);
4/20/2020
59

semantic analysis - Binding a name
Add these two lines to your Main method to create the semantic model
and retrieve the symbol for the first using statement:
// Use the syntax tree to find "using System;"
UsingDirectiveSyntax usingSystem = root.Usings[0];
NameSyntax systemName = usingSystem.Name;
// Use the semantic model for symbol information:
SymbolInfo nameInfo = model.GetSymbolInfo(systemName);
4/20/2020
60

Add these two lines to your Main method to create the semantic model
and retrieve the symbol for the first using statement:
// Use the syntax tree to find "using System;"
UsingDirectiveSyntax usingSystem = root.Usings[0];
NameSyntax systemName = usingSystem.Name;
// Use the semantic model for symbol information:
SymbolInfo nameInfo = model.GetSymbolInfo(systemName);
4/20/2020
61

The preceding code also illustrates that
you use the syntax model to find the structure of the code;
you use the semantic model to understand its meaning.
From the SymbolInfo object you can obtain the Microsoft.CodeAnalysis.ISymbol
sing the SymbolInfo.Symbol property.
4/20/2020
62

Add the following code to your Main method. It retrieves the symbol
for the System namespace and then displays all the child namespaces
declared in the System namespace:
var systemSymbol = (INamespaceSymbol)nameInfo.Symbol;
foreach (INamespaceSymbol ns in
systemSymbol.GetNamespaceMembers())
{
Console.WriteLine(ns);
}
4/20/2020
63

Run the program and you should see the following output:
System.Collections
System.Configuration
System.Deployment
System.Diagnostics
System.Globalization
System.IO
System.Numerics
………………………
Press any key to continue . . .
4/20/2020
64

There are other expressions in a C# program that can be bound that
aren't names.
// Use the syntax model to find the literal string:
LiteralExpressionSyntax helloWorldString = root.DescendantNodes()
.OfType<LiteralExpressionSyntax>()
.Single();
// Use the semantic model for type information:
TypeInfo literalInfo = model.GetTypeInfo(helloWorldString);
4/20/2020
65

Add a declaration that assigns this property to a local variable:
var stringTypeSymbol =(INamedTypeSymbol)literalInfo.Type;
4/20/2020
66

To finish this tutorial, let's build a LINQ query that creates a sequence
of all the public methods declared on the string type that return a
string.
The source for this query is the sequence of all members declared on
the string type.
var allMembers = stringTypeSymbol.GetMembers();
4/20/2020
67

var methods = allMembers.OfType<IMethodSymbol>();
Next, add another filter to return only those methods that are public
and return a string:
var publicStringReturningMethods = methods.Where(m =>
m.ReturnType.Equals(stringTypeSymbol) &&
m.DeclaredAccessibility == Accessibility.Public);
4/20/2020
68

Select only the name property, and only distinct names by removing
any overloads:
var distinctMethods = publicStringReturningMethods.Select(m
=>m.Name).Distinct();
4/20/2020
69

You can also build the full query using the LINQ query syntax, and then
display all the method names in the console:
foreach (string name in (from method in stringTypeSymbol
.GetMembers().OfType<IMethodSymbol>() where
method.ReturnType.Equals(stringTypeSymbol)
&& method.DeclaredAccessibility == Accessibility.Public
select method.Name).Distinct())
{
Console.WriteLine(name);
}
4/20/2020
70

Edit Syntax tree
var workspace = new AdhocWorkspace();
var msBuild = MSBuildWorkspace.Create();
var project = workspace.AddProject(ProjectInfo.Create(ProjectId.CreateNewId
(), VersionStamp.Create(), "NewProject", "projName",
LanguageNames.CSharp));
Section1.c1.document1 = workspace.AddDocument(project.Id,
"NewFile.cs", SourceText.From(content));
DocumentEditor editor =
DocumentEditor.CreateAsync(Section1.c1.document1).Result;
int Count =
Section1.c1.document1.GetSyntaxRootAsync().Result.DescendantNodes().OfType
<IfStatementSyntax>().Count();
editor.ReplaceNode(node, newNode);
Section1.c1.document1 = editor.GetChangedDocument();
editor= DocumentEditor.CreateAsync(Section1.c1.document1).Result;
4/20/2020
71

4/20/2020
72

9-roslyn-guidelines

More Related Content

Similar to 9-roslyn-guidelines

More from Morteza Zakeri

Recently uploaded

9-roslyn-guidelines