• Save
Test scenario simulator
Upcoming SlideShare
Loading in...5
×
 

Test scenario simulator

on

  • 735 views

Simple yet powerful simulation framework to simulate

Simple yet powerful simulation framework to simulate

Statistics

Views

Total Views
735
Views on SlideShare
728
Embed Views
7

Actions

Likes
0
Downloads
0
Comments
0

2 Embeds 7

http://www.linkedin.com 6
http://www.lmodules.com 1

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

Test scenario simulator Test scenario simulator Presentation Transcript

  • Test Scenario Simulation Framework TamerA. V.1 August 3, 2009 Acknowledgment: WayneD 1 Nortel Confidential Information
  • Agenda • Importance of Simulation • Modeling Techniques • More on Simulation Models • System-/Hardware- In-The-Loop Techniques • Virtual-Time Based Simulation • Test Scenario Framework • Testimonials • Anti-testimonials • Conclusions • References 2 Nortel Confidential Information
  • Importance of Simulation • Powerful tool to understand complex system interactions • Quick and predictable answers to what-if scenarios • Prototyping of new ideas and assessment of their performance gain • Replacement of yet-to-exist/expensive hardware • Better model of stochastic nature of human/machine interactions, test with realistic scenarios • Not only a decision tool, it is also a design tool! • Expedite research and development, increase product quality 3 Nortel Confidential Information
  • Modeling Techniques • Analytical Models • Best/Worst/Average case analysis • Quick Results • System dynamics may get overseen, no enough details • Statistical Models (Capacity Tool) • Based on actual measurement • Close to real world • No system dynamics, require real-system measurements • Simulation Models (System Level Simulation, Test Scenario) • Detailed view of system dynamics • Evaluate complex system interactions 4 Nortel Confidential Information
  • More on Simulation Models • Require feedback measurements from existing system • Use stochastic processes and discrete-time event-based (OPNET, TS) • Example (SLS) Global Init User Mobility/Call Link Condition PHY Abstraction L2 (MAC/RLC) Pattern 5 Nortel Confidential Information
  • System-/Hardware- In-The-Loop Techniques • Early exposure to real-life like system loading/conditions • Low development cost where virtually every deisgner has a system on hand • Could be used to support decision making process and also as a debug tool • Fast, repeatable, debug-able, and could be time controlled! 6 Nortel Confidential Information
  • Difficulties with SITL/HIL • CLOCK, CLOCK, CLOCK!!! • Type 1: DES matches system clock (wall-clock) • As close as possible to real situations/conditions • Runs as fast as SITL/HIL would but not faster • May need dedicated hardware/cluster of computers to drive system to capacity • External factor may affect test repeatability • Type 2: DES controls system wide clock (virtual-clock/time) • Highly repeatable and debug-able framework • Strips idle time out of the real system runs, could be faster than type 1 • No need for dedicated/fast hardware • Easily drives system to capacity or extraordinary conditions • Distributed/Parallel DES is needed to test e2e aspects 7 Nortel Confidential Information
  • Virtual-Time Based Simulation • Conceals time information from system/hardware under test TTI #i TTI #i+1 TTI #i+2 … TTI #n … Time (ms) Virtual Time TTI #i TTI #i+1 TTI #i+2 … TTI #n±m … Within time Exceeds time Time (ms) budget budget 8 Nortel Confidential Information
  • Test Scenario Framework • Based on C++ and integrates very well with UnitTest++ (MRTestFixture.h/cc and etc.) • Scenario Based framework • Efficient reuse of test scenarios • Scenario could represent • user mobility pattern, • ever changing RF conditions, • and draw various traffic patterns from stochastic distributions • Could have sanity checks and raise error once anomalies are detected • Unlimited number of users! • Associate different users with different test scenarios for even more productivity and test scenarios reuse. 9 Nortel Confidential Information
  • Test Scenario Framework (cont) • TTF4 had a minimal implementation of TS conceptualization • Generation of deterministic pattern of traffic • Only one type of traffic was supported • Fixed air link condition, no deterioration • Used very well to run traffic and NACK some of MAC TB 10 Nortel Confidential Information
  • Test Scenario - Achievements • Main tool to benchmark rel 1 MAC/RLC • Early detection of timing/performance issues • Design tool to pinpoint MAC/RLC anomalies; performance or otherwise • Secured seamless integration of subsequent activities 11 Nortel Confidential Information
  • Test Scenario (infamous BE chart) 12 Nortel Confidential Information
  • Test Scenario (infamous VOIP chart) 13 Nortel Confidential Information
  • Testimonials • “On top of the testcases you took responsibility for (investigating, discussing with technical primes, designing, coding, debugging and execution), the framework enhancement you provided was first-class. With the eNBTester not being available to us, the UnitTest framework was all we had. And if it weren't for your persistent efforts in enhancing the framework, we would not have finished integration last week. We would probably still be scratching our heads wondering how we are going to finish all the testcases. The enhancement work you did with L2 Architect’s help enabled us to carry on and is responsible for some of the most important findings from our testing (timing and performance data). I really appreciate your insights and vision of this enhancement and your persistence to move ahead in spite of schedule pressures. Thank you for that.” [Software Design Team, TTF4 & TTU4 Prime] • “I want to echo the comments TTF4 Prime made regarding the test scenario toolkit - a major addition that helped make TTF4 a success.” [Software Design Team, Senior Software Designer] • “Your changes all look like Greek to me. I looked at the code for about 35- 40 min. I'm sure the changes are good but I just don't understand the code well enough to comment. Thanks very MUCH for doing these changes. I am anxious to try them out.” [Software Design Team, L2 Architect] Nortel Confidential Information
  • Anti-testimonials! • “Coding/Design Skills - you have a lot of knowledge for different design approaches and algorithms. However, I found that sometimes you applied a more complex solution than necessary. Complexity can make things more difficult to understand and debug. It also caused you to spend quite a bit of time in trying to get the framework working.” [Software Design Team, Software Designer] • “The test cases in the l2IntTest suite #109 all do not fail if there is an error in the received counts and this should be looked at eventually, but it's not a scheduler problem, it's just poorly written test cases.” [Software Design Team, Senior Software Designer] 15 Nortel Confidential Information
  • Conclusions • Powerful tool for capacity/performance decision making • Powerful tool for R&D (testing, debug-ability) • Easily model complex mobility, call patterns along with RF variations • It hard to enumerate test vectors that will hit all system corner cases; it is easy to model user mobility/call patterns (city in a box) • Distributed versions could simulate (not test cases) inter/intra eNB interactions • Deterministic soaking does repeat; stochastic soaking does not! • Simulations are repeatable at will • SITL/HIL is now used in many industries; defence, aerospace, car • Bridges gap between SLS and system design 16 Nortel Confidential Information
  • References • Why use Hardware-in-the-Loop Simulation? http://www.adi.com/products_sim_qhilWhy.htm • OPNET System-in-the-Loop (SITL) module. http://www.opnet.com/solutions/network_rd/system_in_the_loop.html • Modeling and performance evaluation of General Packet Radio Service. Communication Networks Laboratory, School of Engineering Science, Simon Fraser University. http://www.ensc.sfu.ca/research/cnl • Distributed HIL Simulation, Applied Dynamics International • Simulation-Based Testing of Software in Space Applications, Dr. Sergio Montenegro 1, Prof. Stefan Jähnichen 1, Dr. Olaf Maibaum. http://sergio.montenegros.de/public/embeddedsys_springer2006-b.pdf • Pratas, A. Rodrigues, “Enhanced Discrete-Timer Scheduler Engine for MBMS E-UMTS System Level Simulator,” Proc Conf. on Telecommunications - ConfTele, May, 2007, http://www.co.it.pt/conftele2007/assets/papers/mobile/paper_80.pdf • Flxcem Measurement. http://livelink-ott.ca.nortel.com/livelink/livelink.exe?func=ll&objid=30760458&objAc 17 Nortel Confidential Information