Description:
The future of education is human expertise and artificial intelligence working in conjunction, a revolution that will change the education as we know it. The Intelligent Tutoring System is a key component of this future. A quantitative measurement of efficacies of practice to heterogeneous learners is the cornerstone of building an effective intelligent tutoring system that is able to generate practice recommendations adaptive to individual learner’s progress. This thesis proposes a framework for defining and estimating the practice efficacy, which can be applied to a wide variety of learning processes.
When the mastery is assumed to be an ordinal variable, learning is defined as a probabilistic transition from lower level to a higher level. Practice items differ in the probabilities. The practice efficacy is the magnitude of such probability and thus a measure of the instructional value of the practice items. Had the mastery been directly observed, practice efficacy can be directly estimated from data. Unfortunately, the mastery is not observed and practice efficacy needs to be revealed by a statistical inference model based on observed data. The Bayesian Knowledge Tracing model (BKT), a special instance of the Hidden Markov model, formalizes the "practice makes perfect" learning process in a homogeneous learner population based only on the observed response. The Learning Through Practice (LTP) model is an extension of the BKT model by the introduction of learner heterogeneity and the inclusion of learner engagement. The LTP model can be used to describe a variety of learning processes, such as reinforcement learning and zone of proximal development.
Learner heterogeneity in practice efficacy is a major contribution to the literature of dynamic learning process and the foundation of an adaptive practice recommendation system. This thesis introduces two types of learner heterogeneity: the state heterogeneity and the type heterogeneity. The state heterogeneity means learners from different starting mastery level respond to a practice item differently. The type heterogeneity means learners from the same starting level can respond to a practice item differently.
Another important improvement over the classical BKT model is the inclusion of learner engagement. As digital education flourishes, learner behavior data are now widely available. The LTP model capitalizes on these new data to account for learner engagement in the learning process. The two features analyzed in this thesis are stopp decision and effort decision. The former refers to the learner's decision to exit the practice sequence before it ends; the latter refers to the learner's decision to choose a level of effort in solving the problem. The stop decision address the potential bias of differential sample attrition. The effort decision captures the key intuition that "no pain no gain".
Built on the previous learning theory, this thesis develops outlines a Monte Carlo Markov Chain(MCMC) algorithm to estimate parameters of the LTP model. The MCMC algorithm first augments the observed data with latent mastery by Forward Recursion Backward Sampling algorithm given the parameters, then uses Gibbs sampler to update the parameters given the augmented data. In the second step, if the conditional likelihood does not have a conjugate prior distribution, Gibbs sampler draws new parameters with Adaptive Rejection Sampling algorithm. Because a point estimation instead of the posterior distribution of parameters is used in practice, which is only valid if the model is identified from the frequentist view, this thesis also provides the necessary identification condition of the LTP model and the sufficient condition of the BKT model. The latter corrects a long-standing mistake in the literature to base parameter inference on the observed learning curve.
This thesis then provides two applications of the LTP model featuring different aspects of learner engagement. In the first application, the LTP model is applied to a quiz dataset of two-digit multiplication and long division to correct for the dynamic selection bias. The average learning gain from repeated practice decreases by at least 50% for the two-digit multiplication and at least 75% for the long division. In the second application, the LTP model is applied to account for effort choice in a randomized control trial to rank efficacies of a practice question with or without a video instruction set. While the two question forms cannot be ranked by the Difference in Difference regression, the practice question with the video instruction is estimated to have a stronger efficacy by the LTP model after controlling for the differential effort choices.