Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Exercising and Scaling Up Mobile DevOps in the Enterprise

623 views

Published on

Adopting the mobile devops culture, processes and practices in any organization may not happen overnight. The transformation from agile to true mobile devops requires identification of inefficiencies and understanding of how process, practice and infrastructure can be scaled up.

Stay tuned and join our upcoming webinars at bitbar.com/testing/webinars/

Published in: Technology
  • What Is An Equalizing Charge, Why Is It Needed, And How to Apply One To a Lead Acid Battery.. ★★★ https://tinyurl.com/rtswhls
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • Be the first to like this

Exercising and Scaling Up Mobile DevOps in the Enterprise

  1. 1. The Mobile DevOps Company Ville-Veikko Helppi Head of Demand Generation ville-veikko.helppi@bitbar.com Exercising and Scaling Up Mobile DevOps in the Enterprise WEBINAR
  2. 2. Mobile DevOps •  The ‘Mobile DevOps’, Process and Tools •  Infrastructure & Architecture •  Tips & Tricks for Scaling Up •  Troubleshooting Device Lab •  Demo •  Summary – Q&A Agenda More information about Mobile DevOps can be found at bitbar.com © Bitbar Technologies 2017 – h7p://bitbar.com
  3. 3. The Mobile DevOps
  4. 4. •  On-Demand Mobile Device Infrastructure •  Support Any Development & Testing Framework •  Automate, Automate, Automate – To Save Time, Money and Hassle •  Provide Unprecedented Scale and Throughput •  Allow For Creative Use (Open API, Customizable Integrations) •  Adopt Mobile Development Thought Leadership and Expertise Support all development stages: Prototyping Development Testing Production Monitoring Successful Adoption of Mobile DevOps
  5. 5. •  Modern mobile app development is Agile, but infrastructure does not support Agility well •  DevOps practices are emerging in to the intersection of Development, QA and Operations •  DevOps practice emphasizes the collaboration and communication of both software developers and operations while automating the process of software delivery and infrastructure changes •  DevOps practice establishes a culture and environment where building, testing, and releasing software, can happen rapidly, frequently, and more reliably. Mobile DevOps and Agile Development QA Operations Dev Ops © Bitbar Technologies 2017 – h7p://bitbar.com
  6. 6. SPEED WINS It’s a simple fact. The faster you can deliver new applications, features and upgrades to market, and collect feedback, the better your business performs. For that you need higher development pace. And a lot of automation. THERE IS A DEMAND FOR MOBILE DEVOPS DevOps practices and tools have revolutionized the pace and quality of service delivery – but only for traditional server-based apps. FRAGMENTATION WITH TOOLS In mobile the delivery pipelines are more complex – in terms of tools and used methods. Cohesive and well-integrated development tool flow derives from Mobile DevOps practices. MANUAL DOING IS NOT EFFECTIVE Every stage requires a lot of manual work and silo’ed solutions from various vendors and/or open source tools. Why Mobile DevOps © Bitbar Technologies 2017 – h7p://bitbar.com
  7. 7. Delivery Pipeline for Mobile Apps Development Source Code Management SCM Build Test Deploy Production Dev Tools: •  IDEs •  JIRA •  Slack •  etc. SCM: •  Git/GitHub/ GitLab •  Bitbucket •  MSFT •  etc. Continuous Integration and Build Automated Testing Automation for app store deployment, beta testers & enterprise app stores Production Monitoring and feedback to Dev © Bitbar Technologies 2017 – h7p://bitbar.com
  8. 8. Delivery Pipeline with Mobile DevOps Development Source Code Management SCM Build Test Deploy Production Dev Tools: •  IDEs •  JIRA •  Slack •  etc. SCM: •  Git/GitHub/ GitLab •  Bitbucket •  MSFT •  etc. Continuous Integration and Build Automated Testing Automation for app store deployment, beta testers & enterprise app stores Production Monitoring and feedback to Dev The Scope of Mobile DevOps © Bitbar Technologies 2017 – h7p://bitbar.com
  9. 9. Mobile DevOps - User Experience Development /IDE SCM Build Automated Testing Deploy / Release “Production” Automated builds for every code change Built binaries automatically tested Tested applications, Screenshots for various devices and languages, certificates, metadata ready to push to App stores Deploy To Devices Test apps in real world (Synthetic) Apps Used by Users •  Synthetic Monitoring •  Real User Monitoring Beta Testing (Humans) © Bitbar Technologies 2017 – h7p://bitbar.com
  10. 10. Exercising Mobile DevOps Test Deploy Monitoring SCM Development Build © Bitbar Technologies 2017 – h7p://bitbar.com
  11. 11. Continuous Integration CONTINUOUS INTEGRATION © Bitbar Technologies 2017 – h7p://bitbar.com
  12. 12. The ‘Shift-Left’ – What People Think It Is Shi>-Le> TesAng (happens earlier) TradiAonal TesAng (happens before release) © Bitbar Technologies 2017 – h7p://bitbar.com
  13. 13. The ‘Shift-Horizontal’ Is The Way Forward © Bitbar Technologies 2017 – h7p://bitbar.com
  14. 14. Hardware & Software for Efficient and Scalable Mobile DevOps
  15. 15. •  Parallelism, and Lots of Real Mobile Devices (Plug-and-Play) •  Mobile Test Automation Frameworks and Appropriate Use of Those •  Easy to Adopt, Deploy and Use Admin Configurability •  Use of Virtual Machines & Containers (Cloud, On-Premise) •  Open API and Ability to Integrate with Any Tools •  Efficient Use of Resources Support all development stages: Prototyping Development Testing Production Monitoring What Enables Scalability?
  16. 16. Three Deployment Options for Test Automation with Devices Public Cloud Private Cloud On-Premise Reserved & User-Specific Devices Shared & Publicly Available Devices Hosted by Vendor Hosted Locally © Bitbar Technologies 2017 – h7p://bitbar.com
  17. 17. UNPRECEDENTED SCALE By automating the creation, validation and deployment of the containers we are able to deploy large number of identical device servers in matter of minutes. ALWAYS CLEAN EXECUTION ENVIRONMENT New pristine container is started for each test session so that the environment is always identical from one session to another. FAST AND EASY TO UPDATE AND MAINTAIN Any updates need to be made only to the scripts that create the container images and the changes are orchestrated to all device servers ALLOWS RUN TIME CUSTOMIZATION OF THE ENVIRONMENT Because the tests are executed inside the container we can allow users to customize the environment any way they like during the test session. Why Containers Host Container 1 Container 2 Container 3 Container n … © Bitbar Technologies 2017 – h7p://bitbar.com
  18. 18. Because the tests are executed inside the container we can allow users to customize the environment any way they like during the test session. This is done by using the run.sh script that is used to launch the the container session. Any shell command or tool that is installed on the container can be called from run.sh script. For instance you can build anything with Maven: mvn clean test –DexecutionType=serverside or install some library or tool with Brew brew install python Customizing The Environment Host Container 1 Container 2 Container 3 Container n … © Bitbar Technologies 2017 – h7p://bitbar.com
  19. 19. #!/bin/bash startAppium(){ if [ "$(uname)" == "Darwin" ]; then startAppiumOSX elif [ "$(expr substr $(uname -s) 1 5)" == "Linux" ]; then startAppiumLinux else echo "Unknown OS system, exiting..." exit 1 fi } executeTests(){ echo "Extracting tests.zip..." unzip tests.zip if [ "$(uname)" == "Darwin" ]; then echo "Running iOS Tests..." mvn clean test -Dtest=IosAppiumExampleTest -DexecutionType=serverside elif [ "$(expr substr $(uname -s) 1 5)" == "Linux" ]; then echo "Running Android Tests..." mvn clean test -Dtest=AndroidAppiumExampleTest -DexecutionType=serverside fi echo "Finished Running Tests!" cp target/surefire-reports/junitreports/TEST-*.xml TEST-all.xml } startAppium executeTests Example of run.sh Script Host Container 1 Container 2 Container 3 Container n … © Bitbar Technologies 2017 – h7p://bitbar.com
  20. 20. Example: iOS Virtual Machine Components 1.3.6, 1.4.3, 1.4.13, 1.4.16, 1.6.0, 1.6.3 + Appium_Helper: 1.3.6, 1.4.3, 1.4.13 0.20.4 Test AutomaAon Frameworks Dependencies and ‘Middleware’ iTunes Programming Languages Core Tools © Bitbar Technologies 2017 – h7p://bitbar.com
  21. 21. Example: iOS Virtual Machine Components 1.3.6, 1.4.3, 1.4.13, 1.4.16, 1.6.0, 1.6.3 + Appium_Helper: 1.3.6, 1.4.3, 1.4.13 WGET ideviceinstaller Ios-webkit-debug-proxy usbmuxd/iproxy 0.20.4 AddiAonal frameworks 2.4.9 8.x / 7.x Settings & Config to VM •  Disable software updates, Disable screensaver, Disable sleeping, Disable Spotlight mds agent, Remove dock items, Disable hibernation, Remove sleepimage •  Remove swap files, Zero out free space, Shrink disk •  Set up /etc/hosts, Remove old Keychains, Import keychain •  Create ~/Library/MobileDevice directory, Create ~/Library/ MobileDevice/Provisioning Profiles directory •  Import mobileprovision © Bitbar Technologies 2017 – h7p://bitbar.com
  22. 22. Mobile Test Automation Frameworks XCTEST/ XCUITEST UIAUTOMATOR ESPRESSO APPIUM CALABASH ANDROID No Yes Yes Yes Yes IOS Yes No No Yes Yes MOBILE WEB Yes Limited to x.y clicks No Yes No LANGUAGE Obj-C/Swift Java Java Almost any Ruby TOOL FOR TEST CREATION Xcode UIAutomator viewer Hierarchy viewer Appium.app CLI (Human-Readable syntax) COMMUNITY Apple Google Google Community Community © Bitbar Technologies 2017 – h7p://bitbar.com
  23. 23. © Bitbar Technologies 2017 – h7p://bitbar.com
  24. 24. •  Easy to setup & easily extendable •  Time-independent •  Backward compatibility (supporting various API levels) •  A custom Instrumentation TestRunner with special privileges •  Works on any API levels •  Thin layer on top of Android Instrumentation Framework © Bitbar Technologies 2017 – h7p://bitbar.com
  25. 25. •  Integral part of Xcode, works with both real devices and simulator •  XCTest allows developers to write tests for components at any level •  XCUITest provides rich UI testing capabilities for apps •  Objective-C / Swift •  Nearly all frameworks (Appium, EarlGrey) are working in conjuction with XCTest •  Bitbar Testing products support XCTest + XCUITest with all deployment options XCTest / XCUITest © Bitbar Technologies 2017 – h7p://bitbar.com
  26. 26. •  Different Deployment Options o  Hosted in various locations (data centers) o  Co-location o  On-premise •  Infrastructure Support o  Wireless carrier support o  Rooted devices (optional) o  Support for hardware add-ons (i.e. Bluetooth, Wearables, etc) •  Security (seriously!) o  Isolated network o  Dedicated hardware resources o  VPN Support o  OAuth/SSO What Also Helps Scaling Up? © Bitbar Technologies 2017 – h7p://bitbar.com
  27. 27. Private Cloud / Enterprise Support •  Network Test Support o  Network Performance Optimization o  Charles Proxy o  Fiddler •  Performance o  Built-in CPU / Memory tracking •  Graphics Benchmarking Integration o  FPS, GPU utilization, battery usage, etc •  Value-Added Services o  API Implementations o  On-demand scripting o  Video recording o  Custom reporting / formatting © Bitbar Technologies 2017 – h7p://bitbar.com
  28. 28. iOS Device Cluster Server Mac OS 1.9 or later, i5 or greater CPU, 4GB RAM or 16GM Minimum for Parallel Device Runs, 256GB Hard Disk Minimum Android Device Cluster Server Ubuntu Linux 12.04, i5 or Greater CPU, 4GB RAM Minimum, 500MB Hard Disk, 10 Port USB 3.0 Hub or 1USB 3.0 Port per Device, Motherboard with BIOS that supports turning off USB Power Saving. Up to 10 Android devices per Machine. Web Front End Server Ubuntu Server 12.04 LTS 64-Bit (Virtual or Physical), 4-Core CPU Minimum, 16GB of RAM Minimum, 500GB Hard Disk Space, 1GBit Ethernet Connectivity Database Server Ubuntu Server 12.04 LTS 64-Bit (Virtual or Physical), 4-Core CPU Minimum, 8GB of RAM Minimum, 2TB Minimum of Hard Disk, Storage in RAID Array, 1GBit Ethernet Connectivity Other Equipment To Consider Server Racks, USB Cables, PDUs, WiFi Routers, UPS Hardware Infrastructure n © Bitbar Technologies 2017 – h7p://bitbar.com
  29. 29. Troubleshooting A Device Lab
  30. 30. •  Devices o  So many things can go wrong … •  Infrastructural Hardware o  Device Control Hardware •  ADB and Instruments/Tools •  Hard Disks/Storage •  RAM o  USB •  Hubs •  Cables •  Network o  WiFi o  Cellular •  Software o  Jenkins/CI o  SCM o  Test Automation Frameworks 80% The Most Typical Problems 10% 5% 5% © Bitbar Technologies 2017 – h7p://bitbar.com
  31. 31. The most typical problems •  Device doesn’t show up o  CI/On-Premise Solution/etc •  Device doesn’t react •  Device doesn’t start tests/run •  Device is out of battery/battery level low •  Device shows notifications Troubleshooting Mobile Devices Daily Check-Ups for Mobile Devices In order to maintain device farm, environment and all aspects of SW & HW, the following daily checks are recommended: •  Are devices powered up? •  Are all devices WiFi connected? •  Do all devices have an active USB connection? •  Are battery levels at least 50% (for every device)? •  Are there any pop-ups, system notifications, or other system dialogs open on device screen? •  Are screens unlocked? We recommend checking every device 3 times/day. © Bitbar Technologies 2017 – h7p://bitbar.com
  32. 32. Too Many Devices Connected / Device Server •  Too large device cluster •  Maximum of ... o  3 iOS devices per Mac Mini o  10 Android devices per Linux server •  Enough harddisk/storage for all servers We recommend checking & cleaning harddisks (with a shell script) every day. Typical Mistakes (and How To Tackle) Device Server Check-Ups In addition to checking logs on server machines, it’s highly recommended to check following things: •  Diskspace on all machines (script) •  Available RAM (script) •  Do all devices have an active USB connection? •  Available and consumption of CPU for peaks •  Network conditions We recommend automated scripts for checking above-listed items every day. © Bitbar Technologies 2017 – h7p://bitbar.com
  33. 33. Network Not Working Properly •  Need to check if network is working •  Network for ALL connected devices •  Take WiFi down and up – try connection with your browser We recommend checking network connection 6-9 times every day. Typical Mistakes (and How To Tackle) Other Software Not Working Properly Again, check all logs, that contain important information about the system and health of the environment: •  Syslog (on all server machines) •  Catalina out (on application server) •  Jenkins logs (on device servers) •  MySQL errors (and their logs) We recommend checking these items every day. In addition, it’s recommended to implement alarm script to notify about any occurring problems. © Bitbar Technologies 2017 – h7p://bitbar.com
  34. 34. PROBLEM: If none of the devices connected to one device server are running tests (and you can’t initiate any manual session) SOLUTION: Majority of issues seen with non-running tests are related to Jenkins •  Check Jenkins log (https://hostname:8080/log/all) •  Restart Jenkins – https://hostname:8080/safeRestart •  Restart the device server Troubleshooting - Mobile Devices PROBLEM: Device is not running any tests. SOLUTION: First, try if you can establish a manual test session with any of your devices. If not, follow these steps: •  Check that device is powered and visible on Jenkins o  https://hostname:8080/android-devices or http://hostname:8080/ios-devices •  Check that device is available in Bitbar On-Premise o  https://cloudservername:9080/#service/admin/configuration © Bitbar Technologies 2017 – h7p://bitbar.com
  35. 35. PROBLEM: Tests fail without running the test script successfully. Fail can happen at init or any phase of the script execution. SOLUTION: Find the exact error and follow these steps: •  Retrying test run is a good idea to see if the error is random or reproducable •  Check test run and device run views for logs and errors •  Check console.log for the test run •  Common failures: 1) app installation failed, 2) test zip structure is invalid, 3) Xcode didn’t start (iOS) Troubleshooting - Mobile Devices PROBLEM: None of devices are running any tests. SOLUTION: Start with logs. In all cases, the logs contain information about errors, wrong behavior and root cause of issues can be found in those. Start with these: •  First check application server log (catalina.out) •  Check available diskspace •  Then restart Application server (sudo reboot now) © Bitbar Technologies 2017 – h7p://bitbar.com
  36. 36. Troubleshooting – Mobile Devices © Bitbar Technologies 2017 – h7p://bitbar.com
  37. 37. Test On Real Devices Automation - Quickly and Efficiently Use Existing Assets Now and Future Tools Platform Designed For Agile Development •  Test real world scenarios on real, non-emulated devices with 100% certainty •  See exactly what the user experiences •  Reduce time and headache of device acquisition and management The Key Things to Boost Mobile DevOps •  Use any automation framework or test exercisers (e.g. Testdroid App Crawler) to get instant reports and results •  Hit the ground running and get actionable results on a wide range of devices •  Run more tests across more devices in less time, with fewer resources •  Bring your own hardware, tools and frameworks and the Bitbar team will integrate them into the cloud •  Bitbar solutions will not lock you in to any tool or automation framework •  No vendor lock-in means that you have a long term and future proof platform for all of your mobile development and testing needs •  Bitbar integrates seamlessly to your development tools and CI process •  Instant results give immediate feedback to the development team •  Faster iterations mean more efficient development, faster time to market and more satisfied users © Bitbar Technologies 2017 – h7p://bitbar.com
  38. 38. Demonstration
  39. 39. Summary – Q&A More information about mobile app testing, mobile monitoring and mobile devops at bitbar.com

×