Radiation therapy is a cornerstone of cancer treatment, in which highly conformal dose distributions are designed to eradicate tumor tissue while sparing surrounding normal organs. Achieving this balance requires complex, patient-specific clinical decision making and individualized treatment planning, both of which are subject to variability in physician judgment and planner experience. This dissertation investigates how deep learning can improve clinical decision support and treatment planning across two radiation therapy modalities: High-Dose-Rate Brachy therapy (HDRBT) and Gamma Knife Radiosurgery (GKRS). In HDRBT for Locally Advanced Cervical Cancer (LACC), substantial inter-patient variability in tumor size and shape, as well as organ–tumor and organ–organ spatial relationships, adds significant complexity to treatment planning, where applicator selection between tandem-and-ovoids and interstitial techniques critically shapes achievable dose distributions yet remains guided largely by physician judgment. To address this gap, we developed a supervised Convolutional Neural Network (CNN) which is the first deep learning model that classifies patient cases using anatomical features, which achieves robust performance and supports more consistent applicator selection. In GKRS for patients with multiple Brain Metastases (BM), treatment planning remains a challenging and labor-intensive process that relies on iterative tuning of planning parameters, resulting in variability in plan quality and strong dependence on planner expertise. While inverse planning improves efficiency by formulating the problem as a mathematical optimization, it still requires manual adjustment of objective priorities through a trial-and-error process, motivating the need for automated treatment planning. To address this challenge, we propose a deep reinforcement learning (DRL)–based framework for automatic inverse planning. A key difficulty in applying DRL to this problem is the lack of well-defined reference values for plan quality metrics, as achievable plan quality strongly depends on tumor-specific characteristics such as size, shape, and location of BM. To address this challenge, we propose a two-stage framework. First, we develop a Hierarchically Densely Connected U-Net (HD-U-Net)-based model with a novel Dice similarity coefficient (DSC)–based loss defined on clinically relevant isodose lines to predict patient-specific achievable plan quality metrics. This represents the first use of a DSC-based formulation to directly optimize clinically relevant isodose volumes for plan quality prediction in GKRS. These predicted plan qualities are then used as reference values in the reward function design for DRL, enabling meaningful and patient-specific feedback during the learning process. Second, building upon this predictive framework, we formulate treatment planning as a sequential decision-making problem and develop a DRL-based optimization approach. The proposed method learns to iteratively adjust inverse planning priorities and directly generates clinically acceptable treatment plans without manual parameter tuning, representing the first application of DRL for automated inverse planning in GKRS for patients with multiple brain metastases. Together, these studies highlight the role of deep learning, especially supervised deep learning and deep reinforcement learning, in bridging patient-specific features with dosimetric outcomes to support clinical decision making, which contributes toward consistent and efficient treatment planning in radiation oncology.
