Continuous integration (CI) pipelines generate massive amounts of messy log data. At Pure Storage engineering, we run over 65,000 tests per day creating a large triage problem. Spark’s flexible computing platform allows us to write a single application for both streaming and batch jobs to understand the state of our CI pipeline. Spark indexes log data for real-time reporting (Streaming), uses Machine Learning for performance modeling and prediction (Batch job), and re-indexes old data for newly encoded patters (Batch job). Previous work on a mixed streaming and batch environment describes the options for persisting data and their trade-offs: 1) short interval buckets which hurts batch performance 2) long interval buckets which increases micro batch time windows 3) additional software on the background to compact the short interval buckets which adds complexity. This talk will go over how we use the filesystem metadata of our disaggregated compute and storage layers to write over half a million files per day of varied sizes from 52 Billion events and have efficient batch jobs without compaction that allow us to process over 40TB per hour. We will go over the challenges and best practices to achieve efficiency in this mixed environment scenarios.