A talk given at the Microsoft CTO breakfast club on Nov 7th 2016 which described how we designed and built the Applitools server which can be deployed and run on-prem as well as an Azure Cloud Service.

1. Automating visual software testing
BUILDING
UNIVERSAL
SERVERS
ON-PREM MEETS AZURE PAAS
Adam Carmi
Applitools Co-Founder & CTO

2. Automating visual software testing
ABOUT ME

3. Automating visual software testing
AGENDA
• About Applitools
• Why Azure?
• Applitools’ universal server
• Q&A

4. Automating visual software testing
WHAT IS VISUAL TESTING?
A quality assurance activity aimed to verify that a
Graphical User Interface appears correctly to users

5. Automating visual software testing

6. Automating visual software testing

7. Automating visual software testing
THE WORKFLOW
Drive the AUT and take screenshots
Compare screenshots with baseline images
Report differences
Update the baseline

8. Automating visual software testing
DEMO

9. Automating visual software testing
AGENDA
• About Applitools
• Why Azure?
• Applitools’ universal server
• Q&A

10. Automating visual software testing
OPERATIONAL REQUIREMENTS
• A 30K tenant public cloud
• 5M CPU and memory intensive tests / month
• Multiple single tenant dedicated clouds
• On-Prem installations
• Massive image uploads that are rarely fully downloaded
• 1Y data retention
• Permanent availability
• Growing 50% Q/Q

11. Automating visual software testing
SYSTEM ARCHITECTURE
API Server
Tests, Image processing,
Tasks, Auth, Admin,
Externals, App
APP Server
Tests, Image processing,
Tasks, Auth, Admin,
Externals, App
SDK
Data
+
State
Eyes Server (On-Prem / Azure Web-Role) User
APP
Tests

12. Automating visual software testing
AZURE WEB-ROLE
A collection of identical, auto-managed, load-balanced VMs that
run IIS compatible web-applications
• Same web-apps (binaries and resources)
• Same configuration

13. Automating visual software testing
WHY AZURE WEB-ROLE?
• Infinite scale and high availability
• Extremely reliable
• No hardware / OS maintenance concerns
• Rolling deployments
• Immediate emergency rollback to staging
• A thin wrapper around a standard IIS web-site (VS Project)
• .NET

14. Automating visual software testing
AGENDA
• About Applitools
• Why Azure?
• Applitools’ universal server
• Q&A

15. Automating visual software testing
APPLITOOLS’ UNIVERSAL SERVER
• Design
• Deployment
• Runtime
• Storage
• Framework
• Testability

16. Automating visual software testing
DESIGN TOWARDS THE WEB-ROLE
• Multiple stateless instances (scale-out, rolling upgrades)
• Optimistic concurrency and eventual consistency
• Data partitioning
• Automatic instance recycling
• Transient local storage
• Async long running tasks (hard timeouts)

17. Automating visual software testing
BUT DEPEND ON AS FEW PAAS
ARTIFACTS AS POSSIBLE...
In order to keep the On-Prem deployment simple.

18. Automating visual software testing
APPLITOOLS’ UNIVERSAL SERVER
• Design
• Deployment
• Runtime
• Storage
• Framework
• Testability

19. Automating visual software testing
ON-PREM DEPLOYMENT

20. Automating visual software testing
CLOUD-SERVICE DEFINITION
Packaging

21. Automating visual software testing
• Deployed together with a cloud-service package
• Initial instance count
• SSL certificate
• Configuration settings
CLOUD-SERVICE CONFIGURATION

22. Automating visual software testing
PUBLISHING OPTIONS
• Visual Studio
• Azure management portal
• REST APIs
• Programmatically
• Azure PowerShell

23. Automating visual software testing
APPLITOOLS’ UNIVERSAL SERVER
• Design
• Deployment
• Runtime
• Storage
• Framework
• Testability

24. Automating visual software testing
RUNTIME ABSTRACTION
IServiceRuntime
• Access configuration settings
• Access local file storage paths
• Get the instance name (for logging)
• Register runtime event listeners

25. Automating visual software testing
APPLITOOLS’ UNIVERSAL SERVER
• Design
• Deployment
• Runtime
• Storage
• Framework
• Testability

26. Automating visual software testing
NOSQL?
• Infinite storage
• High availability
• Speed
• Pricing
• No schema migration

27. Automating visual software testing
NOSQL STORAGE ABSTRACTIONS
IObjectStore<T>, IByteStore, …
• Key/value based
• Atomic update/create with optimistic
concurrency
• In-memory, File System, Entity Framework,
Azure Table, Azure Blob implementations

28. Automating visual software testing
STORAGE CONFIGURATION
Web-Role
On-Prem

29. Automating visual software testing
APPLITOOLS’ UNIVERSAL SERVER
• Design
• Deployment
• Runtime
• Storage
• Framework
• Testability

30. Automating visual software testing
FRAMEWORK SERVICES
• Framework services are built on top of the storage abstraction
• Access control
• Counters
• Named locks
• Notifications
• Long running tasks

31. Automating visual software testing
APPLITOOLS’ UNIVERSAL SERVER
• Design
• Deployment
• Runtime
• Storage
• Framework
• Testability

32. Automating visual software testing
TESTABILITY
• Run any version of the server on your local machine with a
simple SVN update command.
• Execute thousands of API tests in less than a minute by using
in-memory storage.
• Easily debug local code changes step-by-step running against
production data.

33. Automating visual software testing
SUMMARY
 Universal
 Scale
 Availability
 Deployment
 Testability
 Cost (storage)
× Design complexity (storage, PaaS services)

34. Automating visual software testing
QUESTIONS?
Adam Carmi
Applitools Co-Founder & CTO