Deep reinforcement learning (RL) based rate adaptation has been popular in the past few years. Unlike the handcrafted rate adaptation which requires manual effort from network domain experts to design and tune, RL-based rate adaptation has shown significant potential to self-adapt to different network conditions. However, it still suffers from two limitations: 1) poor generalizability across diverse network environments; and 2) lack of awareness of the user-perceived quality of experience (QoE). In this thesis, we introduce a universal training framework for RL-based rate adaptation to overcome the three limitations currently faced. Although improving the RL model's generalizability across network environments and customizing an RL-based rate adaptation to inject QoE awareness and improve training efficiency are two separate goals, they can be achieved by the same training framework which makes use of networking domain knowledge to reweight the reward seen by the RL model at the training stage. In this work, we cover the design of the universal training framework and instantiate the frame using use two kinds of network domain knowledge--rule-based baselines and video codecs--to address the limits respectively. Our experiments in simulated environments, emulated environments, and real-world network settings demonstrate that RL-based rate adaptation trained by the proposed training framework does have better generalizability across diverse network environments and can be customized to be aware of application layer QoE. Additionally, the RL training efficiency are largely improved in comparison to traditional RL training methods in network rate adaptation.