Joint modeling of multiple closely related quantitative traits with genetic variants is widely applied in genetics to increase power for detecting associations. Linear mixed models (LMMs) are one of the most commonly used approaches. However, when considering a small number of disease-related traits, it is common for one or more of the traits to be binary, not quantitative. Previous work has found that using LMM to analyze binary traits suffers from substantial power loss if covariate effects are important. Generalized linear mixed model methods could, in principle provide a solution to this problem, but in practice the penalized quasi-likelihood estimation methods that make them computationally feasible are too inaccurate to provide reliable type I error control. Furthermore, assessing the significance of multi-trait associations with single or multiple genetic variants is challenging, particularly in samples with population structure and related individuals. There is a lack of methods capable of jointly modeling both binary and quantitative traits in the presence of population structure or relatedness, while also accommodating multiple genetic variants and remaining robust to ascertainment and model misspecification. To address these limitations, we developed BCMAP (Binary and Continuous Multi-trait Association test with Population structure), a novel modeling framework for multi-trait mapping of a combination of binary and quantitative phenotypes, which is based on a mixed-effects quasi-likelihood framework. BCMAP accommodates covariates, population structure, and relatedness, capturing the dichotomous nature of binary traits, and is suitable for testing with both single and multiple genetic variants. Our test employs a retrospective approach, ensuring robustness to both ascertainment and misspecification of the phenotype model. Additionally, we integrate the recently proposed genetic association test method, JASPER (Joint Association analysis in Structured samples based on approximating a PERmutation distribution). JASPER is a fast, powerful, and robust genetic association test that effectively accounts for population structure, enhancing the accuracy and reliability of our analysis. Parameter estimation for the binary trait(s) in this setting presents additional challenges beyond those for the quantitative trait case. As part of estimating the correlation matrix, we explore a recently proposed parametrization which enforces the positive (semi) definiteness and which can be viewed as a multivariate generalization of Fisher’s Z-transformation of a single correlation. In simulations, BCMAP achieves accurate type I error calibration and demonstrates improved power over existing methods. We apply BCMAP to analyze the genetic associations of genetic variants with diabetes and BMI in the Framingham Heart Study.