Cédric Pellerin, Open Source expert at Silicom, presented at OW2con'19 FaSilET², a full end-to-end testing solution and multifunction tool for developers and QA.

1. Confidentiel SILICOM
© 2017
© Silicom 2019
1
Modifiez le style du titre
© Silicom 2019
SILICOM
FaSilET² : End To End Tester
Multifunction Tool for Developers and QA

2. Confidentiel SILICOM
© 2017
© Silicom 2019
2
§ Framework and tool for developers and quality assurance
§ Interact with code in production conditions
§ No need to insert test-only code or stubs
§ Act like a human tester
FaSilET² : Main concepts

3. Confidentiel SILICOM
© 2017
© Silicom 2019
3
§ Actuators :
Scripts launched in a pre-established order defined in a config file.
Simulate human interactions through GUI or CLI and catch their returns.
May be launched in parallel to simulate multiple users at the same time, using
LXC containers.
Examples :
§ Web interface interaction simulation using Selenium
§ Writing into files using ssh or telnet
§ Database modification
§ Physical buttons action simulation using /proc or /sys
§ Etc.
FaSilET² : Vocabulary

4. Confidentiel SILICOM
© 2017
© Silicom 2019
4
§ Validators :
Get information on changes resulting from actuators activities
Zero to many validators for every actuator
Launched in parallel in LXC container prior to the corresponding actuator
Store validation results until needed
Return status PASSED, FAILED, TIMEOUT
FaSilET² : Vocabulary

5. Confidentiel SILICOM
© 2017
© Silicom 2019
5
§ Validators :
Exemples :
§ Reading local or remote logs
§ Database queries
§ Email reception
§ Etc.
FaSilET² : Vocabulary

6. Confidentiel SILICOM
© 2017
© Silicom 2019
6
§ Monitors :
Monitor one or more machine components during actuators execution.
Launched in parallel in LXC containers prior to test session start.
Stopped after the end of the whole process
Timestamped logs to compare with actuator activity
Exemples :
§ CPU usage
§ RAM usage
§ Disk space
§ Etc.
FaSilET² : Vocabulary

7. Confidentiel SILICOM
© 2017
© Silicom 2019
7
FaSilET² is currently a little better than a proof of concept.
Core is ready
Some modules are at production level
Validators :
§ Logs reader using ssh connections
§ Databases reader
Actuators :
§ Web (Selenium)
FaSilET² : Current development status

8. Confidentiel SILICOM
© 2017
© Silicom 2019
8
Current version
FaSilET² : Current architecture
Validator
1
Validator
n
GUI or system
under
tests
Logs
server
L
X
C
Action
GUI returns
Other
(mail, etc.)
Results
Centralization
(JSON file)
Results
analysis
Actuators
Validator
2
Exécute
Database
Monitor
1
Monitor
n
L
X
C
L
X
C

9. Confidentiel SILICOM
© 2017
© Silicom 2019
9
Dynamic operation in list
FaSilET² : Architecture
Actuator
parsing
Containers
creation for
validators
and
monitoring
Validators
and
monitors
started
Actuator
started
Waiting for
validators
Getting and
storing
results
Next
actuator
Other
actuator in
list
?
Et2.pl
JSON
export

10. Confidentiel SILICOM
© 2017
© Silicom 2019
10
§ Scripts based
Operation based on Perl scripts
Use abstraction modules
For exemple :
§ ET2::Web::Selenium
§ ET2::Utils::Report
§ ET2::Validators::SSHLogs
§ Etc.
FaSilET² : Operation

11. Confidentiel SILICOM
© 2017
© Silicom 2019
11
§ Directory layout
Testing scripts must be put into directories like this :
test
├── actuators
│ ├── activate_equipment.pl
│ ├── badlogin.pl
│ ├── finish.pl
│ ├── login.pl
│ ├── ntp.pl
│ └── share.pl
├── et2.conf
├── monitors
│ └── CPU1.pl
└── validators
├── Log1.pl
├── Log2.pl
└── LogNTP.pl
And the main script named et2.pl is called with the top directory as parameter
Execution order of actuators is given in configuration file et2.conf. This file is a simple “ini” format
one.
FaSilET² : Operation

12. Confidentiel SILICOM
© 2017
© Silicom 2019
12
§ Results
Test results are stored in two JSON files :
§ Results from actuators
§ Results from validators
A graphical analysis solution development is planned.
Results are stamped with UIDs
UID format : <actuator_number>-<action_number>-<start_ts>-<stop_ts>
FaSilET² : Running

13. Confidentiel SILICOM
© 2017
© Silicom 2019
13
§ Actuator result example
{
"actuators" : {
"login" : {
"results" : {
"wait_for_text" : {
"main" : "SUCCESS",
"description" : "Waiting for error message",
"UID" : "0001-0001-20190425132501-20190425132512"
}
}
},
"ntp" : {
"results" : {
"change_ntp_server" : {
"main" : "SUCCESS",
"description" : "Checking NTP server is the one entered previously",
"comment" : "NTP server read : 0.debian.pool.ntp.org",
"UID" : "0002-0001-20190425132602-20190425132623"
}
}
},
…
FaSilET² : Running

14. Confidentiel SILICOM
© 2017
© Silicom 2019
14
§ Validator result example{
"maintitle" : "SG4000",
"maindescription" : "SG4000 end to end tests",
"actuators" : {
"ntp" : {
"description" : "Change NTP server address",
"results" : {
"LogNTP" : {
"status" : "PASSED",
"data" : [],
"params_ERROR" : null,
"params_OK" : "server 0.debian.pool.ntp.org",
"UID" : "0002-0001-20190425132400-20190425132730",
"name" : "LogNTP"
}
},
"status" : {
"LogNTP" : "ok"
}
},
"login" : {
"status" : {
"Log1" : "ok"
},
"description" : "Good login tests",
"results" : {
"Log1" : {
"status" : "PASSED",
"params_OK" : "WARN admin_vpn0: connection",
"name" : "Log1",
"UID" : "0001-0001-20190425132412-20190425132842",
"data" : [
"2019/03/25 15:44:15 WARN admin_vpn0: connection "
],
"params_ERROR" : null
}
}
},
…
FaSilET² : Running

15. Confidentiel SILICOM
© 2017
© Silicom 2019
15
FaSilET² : Screenshot

16. Confidentiel SILICOM
© 2017
© Silicom 2019
16
§ Providing a VM image (OVA), LXC or LXD template
§ Add network scanning modules (nmap, wireshark…)
§ Add IA modules
§ GUI to generate actuator and validator scripts
§ Other ways to use this framework :
§ Cyber security (attack and defense)
§ Quality Assurance
§ …
FaSilET² : Near future
Improvements

17. Confidentiel SILICOM
© 2017
© Silicom 2019
17
Modifiez le style du titre
© Silicom 2019
Thank you
Any questions ?
Sources available on gitlab : https://gitlab.com/silicom/fasilet

18. Confidentiel SILICOM
© 2017
© Silicom 2019
18
Actuator example :
FaSilET² : Scripts exemple
#!/usr/bin/perl
use 5.20.0;
use File::Basename;
my $dirname = dirname(__FILE__);
require "$dirname/share.pl";
…
use ET2::Web::Selenium;
use ET2::Utils::Report;
our @VALIDATORS = ( 'Log1' );
…
if(!defined($sel))
{
$sel = ET2::Web::Selenium->new(
host => "https://$host",
driver => "firefox",
encoding => 'latin1',
);
$sel->setname("Sel1");
$sel->start();
}
if(!defined($report))
{
$report = ET2::Utils::Report->new(
title => 'VPN tests',
description => 'End to end tests for VPN project'
);
}

19. Confidentiel SILICOM
© 2017
© Silicom 2019
19
FaSilET² : Scripts example
$sel->goto('/');
$sel->write(name => "username", value =>
"admin_vpn0");
$sel->write(name => "password", value =>
"admin_vpn1");
$sel->clickbtn(class_name => "btn");
$ret = $sel->waitforpage(
timeout => 2,
url =>
'https://10.10.2.4/#/configuration/localnetwork/
ipconfig'
);
if($ret > 0)
{
say “Login : TIMEOUT”;
}
else
{
$rettitle = $sel->gettitle();
$returl = $sel->geturl();
if(($returl =~ /config$/) && ($rettitle =~ /^Param/))
{
say “Login : TEST SUCCESSFUL";
}
else
{
say “Login : TEST FAILED";
}
}

20. Confidentiel SILICOM
© 2017
© Silicom 2019
20
Validator example :
FaSilET² : Scripts example
#!/usr/bin/perl
use 5.20.0;
…
use ET2::Validator::SSHLogs;
sub run
{
my $vlog = new ET2::Validators::SSHLogs(
host => $host,
port => 10022,
key => '/home/cedric/Developpement/keys/ssh.key',
logfile => '/var/log/diags.log',
timeout => '0..20',
autostart => 1,
user => "root"
);

21. Confidentiel SILICOM
© 2017
© Silicom 2019
21
FaSilET² : Scripts example
return undef if(!defined($vlog));
$vlog->setregexok("WARNsw+s: connection");
$vlog->setname("Log1");
$vlog->tail();
return $vlog->getresults();
}
run();

22. Confidentiel SILICOM
© 2017
© Silicom 2019
22
FaSilET² : Why such a name ?
• ET² stands for End To End Testing = ETET = (ET) ²
• Sil stands for Silicom
• Fa(r) is a long, long way to run… because we can do many
things with it, mainly a name one can easily remember J