Video automation testing at Skype         Pierre Gronlier - pierre.gronlier@skype.netVideo Software Development Engineer i...
1   The Video Library2   Continuous Integration      Building      Testing      Feedback3   Unit, Component, System testin...
The Video Library
What is Skype made of ?                                     UI                                  Network                   ...
What is Skype made of ?                                                   UI                                              ...
Continuous Integration
Continuous integration means :    building continuously.    testing continuously.    having an immediate feedback.
Quickbuild  http://www.pmease.com/features/                                Figure: Quickbuild
Quickbuild     There is, only for the Video Library, around 20 dierent build congurations for dierent     platforms and co...
Cross branches builds   Example  Network :             Video :              Codec :      trunk/                trunk/     ...
Cross branches builds   Example  Network :                        Video :                            Codec :      trunk/  ...
Cross branches builds   Example  Network :                      Video :                        Codec :      trunk/        ...
CI as a daily tool   Continuous integration means that :     1 every 10 mins, a script checks for new commits on video tru...
CI as a daily tool                     Figure: Test results
The importance of visual feedback                   Figure: TVs with build/test feedback                           Make it...
Unit, Component, System testing
Who writes and maintains the tests ?       Writing tests is writing code.       When you automate testing, QE are software...
QE and Devs together  1   Dont wait for developers to write your      tests.  2   Dene the tests when you dene the      Ac...
Cross-platform testing
Requirements  We want to have those features :     run our tests on dierent platforms     run our tests with dierent build...
How to conceive a modular testing framework ?           Targets:           - tablets, mobile, notebook w/ and w/o hardware...
How to conceive a modular testing framework ?           Targets:           - tablets, mobile, notebook w/ and w/o hardware...
NFR
What is non-functional ?  Functional vs Non-Functional                           the video works = we see something       ...
Key performance indicators  list of kpis      resolution and frame rate      bitrate      dropped frames and freeze durati...
Pass/Fail vs Score                                    Video Call A        Network Emulation        Video Call B           ...
Pass/Fail vs Score                                     Video Call A        Network Emulation        Video Call B          ...
How to evaluate the best available quality for a call ?  The best quality of a call is given by :                         ...
How to evaluate the best available quality for a call ?  The best quality of a call is given by :                         ...
Compare across revisions / branches
Compare across revisions / branches
Conclusion
Summary   1   Quick feedback between development and testing.   2   Devs and QE in the same team.   3   Collocation helps ...
Questions 1   The Video Library 2   Continuous Integration       Building       Testing       Feedback 3   Unit, Component...
Upcoming SlideShare
Loading in …5
×

Skype testing overview

3,094 views

Published on

Skype testing overview

Published in: Technology
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
3,094
On SlideShare
0
From Embeds
0
Number of Embeds
310
Actions
Shares
0
Downloads
29
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

Skype testing overview

  1. 1. Video automation testing at Skype Pierre Gronlier - pierre.gronlier@skype.netVideo Software Development Engineer in Test - Microsoft Skype division April 2012 - Kiev
  2. 2. 1 The Video Library2 Continuous Integration Building Testing Feedback3 Unit, Component, System testing Some wrappers for testing. Test Driven Development4 Cross-platform testing CI team Plugin mechanisms5 NFR Denition KPIs Increase visibility6 Conclusion
  3. 3. The Video Library
  4. 4. What is Skype made of ? UI Network Video Audio Messaging Figure: Inside Skype
  5. 5. What is Skype made of ? UI Network Video Audio Messaging Figure: Inside Skype Video Codec Streaming ToolBox Platforms Apple, Android, Windows, Linux, Embedded, ... Figure: Inside the Video Library Platforms contains specic code like capturing, rendering methods.
  6. 6. Continuous Integration
  7. 7. Continuous integration means : building continuously. testing continuously. having an immediate feedback.
  8. 8. Quickbuild http://www.pmease.com/features/ Figure: Quickbuild
  9. 9. Quickbuild There is, only for the Video Library, around 20 dierent build congurations for dierent platforms and compilation modes. release/debug internal/external stable/experimental ... We have a farm of building computers. To enable compilation and maintenance across platforms, Makefile is used for compiling and farm agents are in Java Figure: HeatMap
  10. 10. Cross branches builds Example Network : Video : Codec : trunk/ trunk/ trunk/ branches/ branches/ branches/ network-69 video-42 codec-23 network-68 video-41 codec-22 ... ... ...
  11. 11. Cross branches builds Example Network : Video : Codec : trunk/ trunk/ trunk/ branches/ branches/ branches/ network-69 * video-42 codec-23 * network-68 video-41 * codec-22 ... ... ... To enable two dierent dependent teams to develop new features without becoming incompatible, we compile our code with the latest stable release of the dependencies. In addition to trunk source code, we build our latest Long Term Support branch (*) every time there is a backport of a x.
  12. 12. Cross branches builds Example Network : Video : Codec : trunk/ trunk/ trunk/ branches/ branches/ branches/ network-69 * video-42 codec-23 * network-68 video-41 * codec-22 ... ... ... Mode Network Video Codec Video stable ∅ video-41 codec-23 Video release ∅ trunk codec-23 Video experimental ∅ trunk trunk Network release trunk video-41 codec-23 Network experimental trunk trunk trunk
  13. 13. CI as a daily tool Continuous integration means that : 1 every 10 mins, a script checks for new commits on video trunk/ or the branches/. 2 once a build for a platform is done successfully, it triggers a list of short tests. Every test lasts around 30 seconds. 3 at night, a list of longer tests is executed. 4 for every test execution, a report is generated in a database and the results are aggregated on a web page for Devs and QEs
  14. 14. CI as a daily tool Figure: Test results
  15. 15. The importance of visual feedback Figure: TVs with build/test feedback Make it visible ! !
  16. 16. Unit, Component, System testing
  17. 17. Who writes and maintains the tests ? Writing tests is writing code. When you automate testing, QE are software developers in test. The closer and deeper you get into the production source code, more probably it will be a developer test. UI Python Network Video Audio Messaging Figure: Inside Skype Lua, C# Video Codec Streaming ToolBox Platforms Lua Apple, Android, Windows, Linux, Embedded, ... Figure: Inside the Video Library
  18. 18. QE and Devs together 1 Dont wait for developers to write your tests. 2 Dene the tests when you dene the Acceptance Criteria of your PBI. 3 Evaluate the value of your tests (e.g. code coverage). 4 KISS : Keep it Stupid Short and Simple. Figure: Test plan
  19. 19. Cross-platform testing
  20. 20. Requirements We want to have those features : run our tests on dierent platforms run our tests with dierent builds retrieve the results of our tests and analyze it save the result of the analysis output a report, trigger alarms The cross-platform CI team can provide : a pool of devices, platform and capture devices. access to various builds. provide uniform alarming systems (chat, email, sms) a database. a storage space. It is only a matter of contract denition between you and the CI team
  21. 21. How to conceive a modular testing framework ? Targets: - tablets, mobile, notebook w/ and w/o hardware encoding camera, desktop - Windows (desktop + mobile), Linux, Mac (desktop + mobile), Android +/- Full logs Insert/Update entry Frontend DataBase Server Parsing Server Web Rendering Reduced logs Storage Server Figure: Framework
  22. 22. How to conceive a modular testing framework ? Targets: - tablets, mobile, notebook w/ and w/o hardware encoding camera, desktop - Windows (desktop + mobile), Linux, Mac (desktop + mobile), Android +/- Full logs Insert/Update entry Frontend DataBase Server Parsing Server Web Rendering Reduced logs Storage Server Figure: Framework
  23. 23. NFR
  24. 24. What is non-functional ? Functional vs Non-Functional the video works = we see something vs the video has a good quality = we enjoy our video call
  25. 25. Key performance indicators list of kpis resolution and frame rate bitrate dropped frames and freeze durations frame-quality ... list of usecases for every codec for every media protocol version 1-to-1 call and Group Video Calling software encoding vs hardware encoding for dierent network conditions
  26. 26. Pass/Fail vs Score Video Call A Network Emulation Video Call B with controlled inputs with analyzed outputs x x 1. VGA = 640 480, QVGA = 320 240, QQVGA = 160 120 x 2. Image Quality measurement algorithms
  27. 27. Pass/Fail vs Score Video Call A Network Emulation Video Call B with controlled inputs with analyzed outputs Example : KPI Functional Non-functional pass/fail 0% → 100% resolution = 0x0 max = VGA 1 framerate =0 max = 15fps bitrate in the range of [20..5000]kb 350kbps ± 10 % frame-quality frame exist PSNR or SSIM 2 score . . . . . . . . . Everything is automated using stats and feedback values from the Video Library. x x 1. VGA = 640 480, QVGA = 320 240, QQVGA = 160 120 x 2. Image Quality measurement algorithms
  28. 28. How to evaluate the best available quality for a call ? The best quality of a call is given by : optimal settings = gcd(sender , receiver ) with sender = gcd (max (Encoding power ) , max (Network) , max (Camera)) receiver = gcd (max (Decoding power ) , max (Network) , max (Screen)) (gcd = greatest common divisor)
  29. 29. How to evaluate the best available quality for a call ? The best quality of a call is given by : optimal settings = gcd(sender , receiver ) with sender = gcd (max (Encoding power ) , max (Network) , max (Camera)) receiver = gcd (max (Decoding power ) , max (Network) , max (Screen)) where, with some simplications, Encoding power = f1 (CPU power, Power supply mode, Codec perf.) Network = f2 (Bandwidth, RTT, Relay/P2P) Camera = f3 (Resolution, Framerate) Decoding power = f4 (CPU power, Power supply mode, Codec perf.) Screen = f5 (Resolution) (gcd = greatest common divisor)
  30. 30. Compare across revisions / branches
  31. 31. Compare across revisions / branches
  32. 32. Conclusion
  33. 33. Summary 1 Quick feedback between development and testing. 2 Devs and QE in the same team. 3 Collocation helps a lot ! 4 Dont over-complicate your tests/frameworks. 5 Measure the eciency/value of your tests.
  34. 34. Questions 1 The Video Library 2 Continuous Integration Building Testing Feedback 3 Unit, Component, System testing Some wrappers for testing. Test Driven Development Äÿêóþ çà óâàãó ! 4 Cross-platform testing CI team Plugin mechanisms Çàïèòàííÿ ? 5 NFR Denition KPIs Increase visibility 6 Conclusion

×