Automotive Testing vs Test Automation
LoGeek Night Ukraine
September 2020
Speakers: Alexandr Golyak & Nikolay Chertkov

2www.luxoft.com
Agenda
LoGeek Night Ukraine September 2020
1. Career Path
3. Test Environment
1. What is being automated?
2. Test Automation Frameworks
3. Test Automation Solutions examples
Automotive Testing Test Automation
2. A bit of Test Theory
Aim: to share Automotive Test &
Validation Specifics & Challenges,
answering on the following questions:
• What is Automotive Testing in general?
• Is it interesting to work there?
• How to get there?
• What’s the future of this area?
• What about Automotive Testing in Ukraine?
Aim: to share more technical information
regarding specifics of Test Automation in
Automotive, answering the following
questions:
• What is TAF in Automotive?
• Which TA coverage could be achieved?
• Does the “silver bullet” exist?

3www.luxoft.com
About speaker
 Solution Architect leading the Business Office activities for Luxoft’s Automotive System Test &
Validation practice.
 15+ years experience in IT industry 12 of them in Automotive area providing complex business
and technical solutions with a major emphasis on Test & Validation.
 Specializing in pre-sales, solution architecture, project and program management and
enjoy leading a successful and productive team in a fast-paced and challenging environment.
 Passionate about driving the Automotive LOB and expanding customer footprint by using
proven accumulated expertise.
 Prior Luxoft worked in Government Enterprise (Court Information Centre), publishing house,
pharma company.
https://www.linkedin.com/in/oleksandrgoliak/

4www.luxoft.com
DXC History
Created by the merger of CSC and the Enterprise Services business of Hewlett Packard Enterprise,
DXC Technology boasts a long and proud history of innovation, service and value
Roy Nutt and Fletcher Jones
pooled $100 to form CSC, providing
computer manufacturers with
complex programs
Electronic Data Systems
was founded in Dallas,
Texas, by Ross Perot
Hewlett Packard acquired
Electronic Data Systems in
2008, and soon created HP
Enterprise Services
Hewlett Packard Company
split into HP Inc. and Hewlett
Packard Enterprise, and the
Enterprise Services business
became part of HPE
DXC Technology was founded as the
result of the spin-off of Hewlett
Packard Enterprise's Enterprise Service
segmentand its merger with Computer
Sciences Corporation
DXC Technology is world’s leading IT
services company, with around
$20 billion in annual revenues and
nearly 6,000 customers in more than
70 countries!
1959 1962
20172020
2008
2015

5www.luxoft.com
Luxoft Global Presence DXC Global Centers
Luxoft & DXC at a Glance
Global Centers & Top Offerings
$900+M
Revenue FY2019 (USD)
245+
Active Clients
13,000+
Employees Worldwide
8
Industries
14 years
Automotive Experience
40+
Active Clients
3,050+
Employees Worldwide
36
Delivery Offices
Automotive IndustryGlobal Leader in Innovative Tech Consulting,
IT Services & SW Development
$20+B
Revenue FY2019 (USD)
137,000+
Employees Worldwide
6,000+
Active Clients
70+
countries
9
Streamlined Offering Families

6www.luxoft.com
Major Automotive System Test & Validation
Locations in Luxoft
Dearborn, MI, USA
Bucharest, Romania Odesa, Ukraine
Kyiv, Ukraine
Ukraine: Kyiv & Odesa
Test & Validation of IVI system & HMI SW, IOP Testing
Romania: Bucharest
Test & Validation of IVI systems, IOP Testing
Vietnam: Ho Chi Minh City
Test & Validation
USA: Dearborn
Onsite presence, Virtual Testing Platform Solution
Mexico: Guadalajara
Test & Validation
Guadalajara, Mexico
Ho Chi Minh City, Vietnam
Germany: Böblingen, Berlin & Lindau
Test & Validation of ADAS systems, UI testing
Böblingen, Berlin & Lindau, Germany

7www.luxoft.com
1. Career Path in
Automotive Testing

8www.luxoft.com
ISTQB Automotive Software Tester Certification
https://www.istqb.org/certification-path-root/foundation-level-automotive-software-tester.html
https://astqb.org/benefits/road-map/
ISTQB® CTFL ISTQB Foundation Level software testing
ISTQB® CTFL-AuT ISTQB Foundation Level Automotive Software Tester
If you work in the automotive industry, it comes with its own set of testing
challenges, making Automotive Testing Certification important.
English / German Ukrainian / Russian
• Expleo Technology Germany GmbH, Köln, Germany (GTB)
• imbus AG Möhrendorf, Germany (GTB)
• ALTEN Italia S.p.A., Milan, Italy, (ITA-STQB)
Not Available as for
Sep 2020
ISTQB® Accredited Training Providers by language:
https://www.udemy.com/course/istqb-automotive-software-tester-practice-tests/
ISTQB® CTFL-AuT Exam Providers (English):
www.ita-stqb.org, atsqa.org, brightest.org/en/, www.cert-it.com, www.certinstitute.com,
www.gasq.org, hstqb.org, www.isqi.org

9www.luxoft.com
SHADOW Team approach example
as starting point of Automotive Tester Career
Candidate
• Usually has 0 experience in Automotive Testing
• Joins to the project for “Learning by Doing” as intern
• Project Specific Training Program is applied
Project
• Delegates boring but important tasks (often routine)
• Has strategic reserve in case of attrition
• Seniors can train their soft & mentorship skills
Customer
• Usually pays nothing for shadow team but aware
• Gets extra mile & attrition insurance
• Provides access to systems & tools

10www.luxoft.com
Own Story: from Test Engineer to PgM & SA
Career path in Automotive Testing (example)
2006 – Started IT career as Junior Engineer in non-Automotive area
2007 – graduated from the University
2008 – Regular Test Engineer
2009 – Senior Test Engineer
2010 – Test Team Lead
2015 – Project Manager
2017 – Program Manager
2019 – Acting Head of Automotive Testing CoC
2020 – Head of Business Office System Test & Validation
Enablers:
• Technical Background
• No fear of wires
• English skills
• Learning by Doing
• Continuous Education & Self Development
• Soft skills
• Luck & adventurism
• Good vestibular apparatus
Another Career Path examples:
• From Intern to Test Manager
• From Test Engineer to Program Manager
• From Test Engineer to Head of Technology Office

11www.luxoft.com
2. A bit of Automotive
Testing Theory

12www.luxoft.com
Automotive Testing Challenges
Current Automotive Test & Validation business challenges:
 Increasingly intelligent and interconnected products
 Increasing product testing complexity
 Increasing digitization & proliferation
 Hardware-Software separation
 Shorter product life cycles
 Stricter regulatory norms
 Increasing demand of engineering capacity
Automotive testers are to be able to support Automotive manufacturers to
translate these challenges into strategic business opportunities
The Automotive Industry in the state of Digital Transformation. Automotive manufacturers are being challenged by fast-changing
customer needs, advancing digital technologies, comprehensive connectivity, and stricter environmental regulations.
Testing is a critical enabler for the digital transformation journey

13www.luxoft.com
Autonomous Shared Electric
The Future of Automotive Industry (as well as Testing and Validation) is
C.A.S.E.
Connected
• V2V, V2X, 5G
• Cybersecurity
• SOTA & Telematics
• Advanced Diagnostics
• Emergency Services
• Intelligent Traffic
• SAE Levels of driving
automation (1-5)
• Steer by Wire, Brakes by Wire
• Radars and LiDARs
• CV, AR, AI, ML
• Advanced Virtualization like
Digital Twins
• Data Ingesting, Processing,
Analyzing & Annotation
• Virtual Test Drive
• Personalized Smart Mobility
• Personal vehicle sharing (PVS)
• P2P Carsharing
• Ridesharing
• Cloud Based Infrastructure
• App. Security
• Mobility as a Service
* Shifting priorities in the age of COVID-19
• Electric Vehicles Powertrain
technologies: Hybrids, PHEV, BEV,
FCEV(H2), EREV
• Battery Management System
• EV + Charging Stations IOP &
Communication
• EV applications
• Combined Charging System (CCS)

14www.luxoft.com
SAE Levels of driving automation
Level Definitions Driving mode Example ~ Year
0 Zero Automation Everything on
Gauges or system issues
warnings
1885
1 Driver Assistance Everything on ACC, LKA, PA 2000
2
Partial
Automation
Feet Off
full control of the vehicle:
accelerating, braking,
and steering.
2013
3
Conditional
Automation
Hands Off
Traffic Jam Chauffeur or
ALKS
2022
4 High Automation Eyes Off
Supported only in limited
specific areas or under
special circumstances
2026
5 Full Automation Mind Off
No human intervention
is required at all on all
roads all over the world,
all year around, in all
weather conditions
2032
Ways of Testing:
• physically, in a closed environment
• on public roads, where permitted
• in a virtual environment, i.e. using computer simulations
Active AD players: Google (Waymo), Baidu (Apollo), BMW, Nissan, Ford, General Motors (Cruise), Delphi Automotive Systems, Tesla, Mercedes-Benz, Bosch,
Amazon (Zoox), Nuro, Pony.ai, Aurora, Apple, Audi, Uber, Lyft, Yandex, Volvo, Robocar, AutoX, TuSimple and others.
DXC Robotic Drive: accelerates end-to-end engineering and testing in AD development by enabling the autonomous driving platform, toolkit, and expertise.

15www.luxoft.com
Standards for the testing of E/E systems in Automotive
Which Standards influence the work of Test Professional in Automotive?
• 4 Automotive Safety Integrity Levels: ASIL A, ASIL B, ASIL C, ASIL D
• Relevant volumes (parts) for the tester: Volume 4 - 6
ASPICE
• Test Specific Processes: SWE.4; SWE.5; SWE.6; SYS.4; SYS.5
• Assessment levels and capability indicators: e.g. 0 - 3
ISO 26262 FuSa
AUTOSAR
• Objectives of AUTOSAR, Classic & Adaptive Platforms
• Three-layer architecture: BSW, RTE, SW-C
ISO IEEE 29119
• Series of 5 international standards for software testing
• Can be used within any software development lifecycle

16www.luxoft.com
Test Levels ASPICE based & comparison
Test levels are not completely consistent
ASPICE 3.0 ISTQB CTFL ISO 26262
No equivalent Acceptance Test Safety validation (4-9)
System Qualification Test
(SYS.5)
System of Systems Test 1 Item integration and
test (4-8) 2
System Integration Test
(SYS.4)
System Integration Test
Verification of the
Software-safety
requirements (6-11)
Software integration
and test (6-10)
Software Qualification Test
(SWE.6)
System Test
Software Integration Test
(SWE.5)
Component Integration Test
Software Unit Verification
(SWE.4)
Component Test
Software-Unit-Test (6-
9)
1. The testing of several heterogenic distributed systems
2. Item integration and test includes three phases: the integration and the test of
hardware and software of an element, the integration and the test of all elements
belonging to the item, and the integration and the test of the item in connection
with other items in the vehicle

17www.luxoft.com
3. Specifics of the Test
Environment in
Automotive

18www.luxoft.com
Test Environment in general
Open-Loop-System
 Outputs has no relation to the
inputs
 Reactive behavior
 E.g. traffic light, washing
machine, TV remote
Closed-Loop-System
Also known as in-the-Loop
 Outputs collected and
forwarded to the input
 Complex functions could be
tested
 E.g. controllers, ABS, air
conditioner, refrigerator
Start Testing as
early as possible
Use different environment on
different development phases
Often realistic
environment is needed
Motivation for a test
environment:
To be able to perform his activities tester needs a test environment in
which the missing parts are simulated.
1. The test environment is used to stimulate the input interfaces of the
device under test and monitor its outputs through the output interfaces.
2. Afterwards, the behavior at the output interfaces is analyzed.
3. In a successful test, the observed behavior corresponds to the expected
output.
According to ISO/IEC/IEEE 29119, a test environment consists of the
following parts:
• Hardware of the test environment (computer, electronic control unit
(ECU), test bench, development kit, …)
• Software of the test environment (operating system, simulation
software, environment models, …)
• Facilities of communication (access to networks, data loggers, …)
• Tools (oscilloscope, measuring tools)
• Laboratory (protection from electromagnetic radiation and noise)
Testing in XiL test environments
 Model in the Loop (MiL)
 Software in the Loop (SiL)
 Process in the Loop (PiL)
 Hardware in the Loop (HiL)
 Vehicle in the Loop (ViL)

19www.luxoft.com
Test Lab & Test Farm equipment
19
 Test Lab workplace is designed for technology domain specifics
 Test Farm could consist hundreds of Test Benches equipped for Manual
& Automated Testing
 Possibility to run tests 24/7 including overnight and over weekend
 Could be accessible remotely by using an intranet timeslot booking tool
 Vehicle breadboard is also being used by Test Engineers in some cases
Source: https://www.mathworks.com/
Source: https://www.ni.com/

20www.luxoft.com
Test Garage & Test Fleet Management
20
Test Drives
Performed in different countries & regions, e.g. (New Zealand, India,
Saudi Arabia, Oman, South Africa, Mexico, United States, Brazil,
Spain, Portugal, Ireland, Turkey, Greece, Romania, China)
Two types of test drives are being performed:
 Test drives on-demand – where there is a clear requirement to
drive to a specific location or perform a specific test during the
test drive.
 Recurrent test drives – which happen during the working
hours, in the evening or during the week-end. The main purpose
is to drive as many kilometers as possible, using the system as
an end-user would.
Secure Test Fleet Management
 Closed TISAX compliant perimeter in Garage
 The engineers have to pass a defensive driving training and first-aid
training which provided by local companies agreed with the customer.
 “How to garage” training provided internally for new employees
 When not in use, the experimental HW was covered by a textile cover.
 After test drives, the cars had to be parked in closed garages.
Test car setup activities
 Removal of old equipment & installation of new equipment (e.g. Head unit, Touch Pad, Touch Display,
Multimedia Boxes, USB Hub, GNSS antenna, Audio Amplifier)
 Test cars hardware and mechanical modifications according to testing process, installation of data loggers &
sensors.
 Daily maintenance tasks for electronic equipment

21www.luxoft.com
Vehicle Prototyping & Measurement Systems
21
Wiring
LiDAR
Radar
Camera
DGPS
Power
Management HMI
Modular Measurement
Systems & Dataloggers
System conception
 Clarify requirements
 Definition of system and KPIs
 Conception of whole system
System development & setup
 Setup in laboratory
 Check & Validation
 Adaptation of system to
prototype vehicle
System integration
 Integration of various
sensors and components
 Adaptation of HMI
 Space saving wiring in car
Modular Measurement
Solutions

22www.luxoft.com
Scenario Based Testing Approach
Testing Approach example for the most complex projects
 Hundreds of End User Behavior Scenarios created and executed for complex projects.
 Each week certain number of scenarios is re-planned taking into consideration certain test focus.
 Each scenario consist of ~ 10 – 15 basic Domain Test Cases created by Domain test specialist.
 Scenarios Executed in Laboratory or Test Car (using pre-defined routes).
Different End User Profiles:
 Housewife
 Taxi Driver
 Car rental
 Manager
 Salesman
 Senior
 Youngster
Main benefits:
 increase cross domain coverage
 Stress the system
 Check the most common scenarios
 have always a quick overview of the stability of SW
Difficulties:
 using re-mastered or non-prototype car
 all components (ECU) needed to be updated (time-consuming)
 Scenarios to be executed by very experienced testers having deep
knowledge of all domains.
Driver
Co-
Driver
Tester
Rear
Left
Tester
Rear
Right
Tester
Data / Trace logger

23www.luxoft.com
Summary and Conclusions
Automotive Testing is very cool place to
work for both manual & automated testers
What cool things automotive
testers can do?
 Drive prototype camouflaged cars not even
known by Paparazzi yet
 Execute tests in Virtual Environment
 Deal with the top megatrends transforming
businesses
 Set up and operate Test Farms in car
production line
 Drive Special Test Drive Roads e.g. in
Shanghai and California
 Operate Climate Chamber and Sun
Simulator
 Burn and brick Head Units
Pros:
• Exceptional talent pool
• Strong technical background
• Mindset of the people
• Nearshore to OEMs and Tier1s
Cons:
• Driving prototype cars issues
• HW import complexity & price
• Roads condition & safety
Location Specific: Ukraine Although COVID-19 Crisis, Ukraine
locations (Kyiv & Odessa) are growing in
Automotive Line of Business.
What other related benefits?
 To obtain knowledge in technology
domain
 To develop test tools and Smart Test
Bench Solutions
 To develop & Test Automation
Frameworks and Virtual Testing Platforms
 To collect and annotate data
 To learn German, Chinese, etc.
 Games Development & Testing
 Consulting & Advisory

24www.luxoft.com
Part II: Test
Automation in
Automotive

25www.luxoft.com
About speaker
Part II: Test Automation in Automotive
 Lead Expert driving Technology Office activities for Luxoft's System Test & Validation practice.
 Having more than 10 years of knowledge and experience in Automotive Test Automation.
 Responsible for providing and building the relevant expertise and capabilities to support both
customer requests as well as ongoing activities within Automotive LOB across all practices.
 Focusing on establishing best practices across the entire line of business as well as providing his
technical expertise for the team when they engage in pre-sales support
 Prior to Luxoft developed Test Automation solutions for Gyroscope Manufacturer
 Joined Luxoft as Test Engineer
www.linkedin.com/in/nikolay-chertkov-25522585

26www.luxoft.com
Part II. Test Automation in Automotive
What is being
automated?
Test Automation
Frameworks
Automation solutions
examples

27www.luxoft.com
Automotive vs Automation
Employer: now show
me what you can!
Me:

28www.luxoft.com
Automation vs Manual

29www.luxoft.com
Automated vs Manual
ASPICE levels and Test Automation coverage
SYS.5ENG.10 System Qualification Test
Software Integration TestENG.7
SWE.6ENG.8 Software Qualification Test
SWE.5
ENG.9 SYS.4 System Integration Test
ENG.6 SWE.4 Software Unit Verification
Amount of tests
ASPICE group
Manual
Automated

30www.luxoft.com
TA example: domain Diagnostic
Computer Diagnostic

31www.luxoft.com
Example: domain Diagnostic
Purpose
• Self-diagnosis and error reporting
• Additional/hidden system info
• Change of ECU mode
• Sensor calibration
• Feature activation/deactivation
TA
• Potential up to 100%
• Protocol tests
• Functional tests
• Fault injection tests
• Endurance tests Source: dSpace World Conference 2019

32www.luxoft.com
In-car Infotainment trends
• Autonomous driving
• Functional safety
• Cyber security
• ADAS
• Connected mobility
Source: Joyofapple.com
Source: Uslugio.com

33www.luxoft.com
In-car Infotainment trends (McKinsey, 2019)
“In-car infotainment SW
complexity has grown 4
times from 2010 till 2020.
Meanwhile SW-
development productivity
increased just x1.5”
Dr. Ondrej Burkacky,
McKinsey & Company, Inc
dSpace World Conference 2019

34www.luxoft.com
Test Automation. Cruise ship analogy
Source: vk.com

35www.luxoft.com
Test Environment. VTBxx
A picture where you see a biggest
office mess ever with a lot of wires, car
head-units, dozen displays around,
aluminum test rack and a chair in the
middle. You realize it’s not a
warehouse but a workplace and start
laughing… and I start crying
Source: forums.elderscrollsonline.com

36www.luxoft.com
Test Automation Framework
 Customer- and project-specific
 Reusable design
 Automation vs Manual balance
 Independent instruments
 Common Interface (API)
 Scalable approach
 Complex scenarios
 Continuous testing
 "Shift-left" enabler
Unique
Uniting
Level Up
Script Editor
Diagnostic Tool
CAN Gen Tool
Some/IP Tool
D-Bus Comm
Logs & Reports
Tool
Tracing Tools
(DLT, IOC,
CAN, PCAP)
Peripherals Tools
(Frame Grabber, Climate
Chamber, USB Matrix,
Relay Box, ...)
TAF
Hub

37www.luxoft.com
Smart Test Bench Solution
Smart Test Bench
Micro HIL Bench
Head Unit ModemGateway
Virtual Test
Platform
SD CARD
ETH
Audio IN Audio OUT LVDS IN HDMI
USB
UI
Line Out / SPKR MIC Display
 Display virtualization
 Remote Connectivity
 Automation Framework

38www.luxoft.com
Smart Test Bench Farm Solution
Vehicle
Fleet
Smart Bench Farm
Region 1
Smart Bench Farm
Region N
Internal Network Internal Network Internal Network
Scheduler
Internet
System Quality early feedback
• Vast test coverage
• Remote access
• Running tests 24/7
• Region free

Recommended

Recommended

More Related Content

What's hot

What's hot (19)

More from LogeekNightUkraine

More from LogeekNightUkraine (20)

Recently uploaded

Recently uploaded (20)