5. ALM 4 Azure
Cloud System
Goals: Collaborative
High Quality
Flexible
Automated
Repeatable
Efficient
5
6. Scenario 1: Dev only
Scenario 2: Dev and Test
Scenario 3: Dev, Test and Build, Deploy
Scenario 4: Dev, Auto Tests and Build, Deploy
Scenario 5: Dev, Auto Tests, Build, Deploy, Acc test
and Operations
6
7. Scenario 1: Dev only
Scenario 2: Dev and Test
Scenario 3: Dev, Test and Build, Deploy
Scenario 4: Dev, Auto Tests and Build, Deploy
Scenario 5: Dev, Auto Tests, Build, Deploy, Acc test
and Operations
7
8. Scenario 1:
Developer only
1 Source Production
Repository Staging
C# 3
TEAM (implementatie + unittests) (TFS)
Developers implementeren het systeem
Local unittesting
‘SprintReview’Build
Compile
Run Unit Tests
2
8
17. Scenario 1:
Developer only
Pro: Con:
Easy installation and configuration No collaboration
Single click deployment from VS2010 Easy deployment errors (configuration)
What about test and ops
17
18. Scenario 1: Dev only
Scenario 2: Dev and Test
Scenario 3: Dev, Test and Build, Deploy
Scenario 4: Dev, Auto Tests and Build, Deploy
Scenario 5: Dev, Auto Tests, Build, Deploy, Acc test
and Operations
18
19. Scenario 2:
Developer with
manual tester
Source Production
Repository Staging
C# 3
TEAM (implementatie + unittests) (TFS)
1 Developers implementeren het systeem
Local unittesting
Testers specificeren testcases ‘SprintReview’Build 4
“Dry” run van tests tegen Development Fabric Compile
Run Unit Tests
2
19
22. Source Production
Repository Staging
TEAM (TFS) 3
1
4
2
1 2 4
Functional, system, Build Verification Platform accepatance
manual tests executed Tests (unit tests) and tests executed on
in compute emulator other quality gates, Azure staging - tested
with MTM. executed in the build. with MTM.
39. Capabilities
• Link test cases to requirements
• Fast Forward
• Test data Iterations
• Test case management
• Test configurations
Interesting capabilities 4 Azure
• Shared steps ( 4 VIPS Swap )
– common practice:
new hosted service only change the name in the DNS…
• Diagnostic adapters ( Rich bug reports ) capturing the logs
Not working capabilities in Azure
• Test Impact 4 manual tests
• IntelliTrace 4 manual tests (although possible custom adapter)
39
40. Scenario 2:
Developer with
manual tester
Pro: Con:
Easy installation and configuration Easy deployment errors (configuration)
Single click deployment from VS2010 Time consuming (deploy and test)
Testers connected, same heartbeat as dev Not repeatable (annoyed testers)
Proven quality Testers connected
40
41. Scenario 1: Dev only
Scenario 2: Dev and Test
Scenario 3: Dev, Test and Build, Deploy
Scenario 4: Dev, Auto Tests and Build, Deploy
Scenario 5: Dev, Auto Tests, Build, Deploy, Acc test
and Operations
41
42. Scenario 3:
Developer with manual tester
and deployment build
3
Source Production
Repository Staging
C#
TEAM (implementatie + unittests) (TFS)
1 Developers implementeren het systeem
Local unittesting
Testers specificeren testcases ‘SprintReview’Build 4
Dry run van tests tegen Development Fabric Compile
Run Unit Tests
2
Package (CSRun.exe, CSPack,exe, CSManage.exe)
Deploy 2 Staging (CMdLets, Powershell)
42
47. Deploy during build
1. Change .ccproj project
• New build target which uses the built-in CorePublish target
2. Create PowerShell script
• Called by build target and uses Azure CMDLets
3. Edit Build Config
• Add the necessary parameters
4. Do something with certificates
• Put the Azure certificate on the build server
47
48. Scenario 3:
Developer with manual tester
and deployment build
Pro: Con:
Easy installation and configuration Time consuming testing
No click deployment from build Application can contain ‘annoying’ bugs
Repeatable ‘proven’ deployments* Build workflow knowledge necessary
Testers connected, same heartbeat as dev Powershell, ccproj tweaks, target files,
Proven quality certificates
*demo needs some additional environment tweaks for different csfg files
per environment
48
49. Scenario 1: Dev only
Scenario 2: Dev and Test
Scenario 3: Dev, Test and Build, Deploy
Scenario 4: Dev, Auto Tests and Build, Deploy
Scenario 5: Dev, Auto Tests, Build, Deploy, Acc test
and Operations
49
50. Scenario 4:
Developer with automated regresion
tests, manual tests and deployment build
3
Source Production
Repository Staging
C#
TEAM (implementatie + unittests) (TFS)
Test cases
(automated UI tests)
1 Developers implementeren het systeem
Local unittesting
4
Testers specificeren testcases ‘SprintReview’Build
Dry run van tests tegen Development Fabric Compile
Automation van testcases 5 Run Unit Tests
(moeten gedraaide testcases zijn voor CodedUI) 2
UI automation tests run tegen Development Fabric Package (CSRun.exe, CSPack,exe, CSManage.exe)
en associëren met test plan Deploy 2 Staging (CMdLets, Powershell)
50
58. Where to run the auto test?
a) At the Build server? (no … only unit test)
b) In the Development environment? (only
dry run for autoamtion)
c) In the Test environment? (sure, functional
tests.. You need emulator at the test env)
d) In the Azure environment? Some definitely
not all (money)
58
59. Scenario 4:
Developer with automated regresion
tests, manual tests and deployment build
Pro: Con:
No click deployment from build Build workflow knowledge necessary
Repeatable ‘proven’ deployments* Powershell, ccproj tweaks, target files,
Testers connected, same heartbeat as dev Certificates
Proven quality Test Infrastructure knowledge necessary
Automated BVT on different Environments A balanced thinking of test automation
Comfortable Acc Testing
Done Done
*demo needs some additional environment tweaks for different csfg files
per environment 59
60. Scenario 1: Dev only
Scenario 2: Dev and Test
Scenario 3: Dev, Test and Build, Deploy
Scenario 4: Dev, Auto Tests and Build, Deploy
Scenario 5: Dev, Auto Tests, Build, Deploy, Acc test
and Operations
60
61. Automated ALM for Azure Cloud development App fa
Applic
Efficient, repeatable with proven quality fast Manag
flexible delivering of useful functionality … 4 (June)
Acceptation
Making optimal use of the characteristics of Azure:
- Identical environments (Guest OS versioning).
- Separate in depended staging production environments. ‘Release ’Drop
- Easy up and downgrade of applications. Package (same as test env
- Unambiguous deployment mechanism 3 package)
- rich diagnostic capabilities Acc testing met MTM en
- Simplified Application Deployment & Management Diagnostic Data Collectors
- …
BUILD Production
(TFS) Staging
C#
TEAM (implementation + unittests)
Test cases
(automated UI tests)
Developers implement the system
Local unit testing
Testers specify test cases ‘SprintReview’Build
Dry run tests cases against Development Fabric Compile
Automation of test cases Run Unit Tests
1
2 Package (CSRun.exe, CSPack,exe, CSManage.exe)
UI automation tests execute against Development Deploy 2 Staging (CMdLets, Powershell)
Fabric and associate with a test case Run automated Test Suites (MTC.exe) with different
environments (test controller – agents infrastructure)
Operations determine instrumentation data Collect and publish test results
DFO, SLA “Build Verification Tests” green -> deployment 2 ‘test
Usages Metrics, payment 'production 61
62. Scenario 1: Dev only
Scenario 2: Dev and Test
Scenario 3: Dev, Test and Build, Deploy
Scenario 4: Dev, Auto Tests and Build, Deploy
Scenario 5: Dev, Auto Tests, Build, Deploy, Acc test
and Operations
Scenario 6: TFS on Azure
Scenario 7: Other Infrastructure on Azure
62