This document summarizes the trials and tribulations of managing a large-scale mobile device test automation lab at Yahoo. It discusses why mobile testing on real devices is important, Yahoo's CI/CD pipeline for mobile apps, and details of Yahoo's mobile device testing lab including supported frameworks and over 300 iOS and Android devices. It also outlines some of the challenges of running such a lab at scale including hardware issues, understanding user needs, operational processes, and keeping teams engaged over time.