Scaling CPU Usage in Genomics Workflows

DAG-structured genomics workflows process gigabytes to terabytes of sequencing data in shared clusters, where efficient CPU usage is critical in resource management. Inefficient CPU usage can significantly delay workflow executions and increase computational costs. Two possible reasons for this issue are limited degree of parallelism and suboptimal CPU allocation during execution. This dissertation presents systems which we developed to scale the CPU usage of genomics workflows.

In the first work, we address the issue of limited degree of parallellism by facilitating easier transition between workflow engines. Genomics workflows execution in production uses workflow engines in order to provide reproducibility, data provenance, task scheduling, and other desirable properties. However, the degree of task parallelism that can be achieved during execution vary across various engines. Some engines can only run workflow tasks serially, like cwltool, while others, like Nextflow, support more sophisticated parallel task execution. Transitioning to Nextflow poses a significant challenge due to the differences in programming models, scripting language compatibilities, and the prerequisite for in-depth knowledge between Nextflow language and CWL, the workflow language of cwltool. We develop CNT, a semi-automatic translator which allows easier migration of CWL workflows to Nextflow. This allows us to execute workflows using the Nextflow engine, which provides better task parallelism and consequently results in higher CPU usage. We evaluate the performance improvements achieved by using Nextflow workflows translated by CNT vs. running CWL on cwltool.

In the second work, we address CPU allocation during workflow execution. Existing CPU allocation approaches are often insufficient for genomic workflows due to their unique challenges: input-dependent performance variability, contention-induced interference, and dynamic workflow branching. We first characterize production-level workflows for processing DNA and RNA sequencing data, identifying three critical characteristics: inflatable parallelism (thread surges up to 96x), stall heterogeneity (up to 2x slowdowns) and millisecond-scale idle bursts that make utilization often unreliable. These findings confirm that effective CPU management should satisfy three requirements: 1) responsiveness to thread bursts, 2) robustness under contention, and 3) adaptiveness to workflow dynamics. Motivated by our characterization study, we developed GenoX, an automated CPU quota management system that leverages thread counts as its primary signal for CPU scaling, rather than traditional metrics like utilization. Operating at 5ms intervals through standard Linux cgroups, GenoX responds immediately to changes in CPU demand phases while remaining stable under contention. Evaluation results show that GenoX outperforms state-of-the-art baselines, as it reduces application execution time by up to 50%, eliminates 200% straggler slowdowns, and improves workflow completion by 21%. These evaluation results confirm that thread-based scaling effectively addresses CPU allocation challenges in genomics workflows.