Path-independent (PI) quantum control has recently been proposed to integrate quantum error correction and quantum control [W.-L. Ma, M. Zhang, Y. Wong, K. Noh, S. Rosenblum, P. Reinhold, R. J. Schoelkopf, and L. Jiang, Phys. Rev. Lett. 125, 110503 (2020)], achieving fault-tolerant quantum gates against ancilla errors. Here we reveal the underlying algebraic structure of PI quantum control. The PI Hamiltonians and propagators turn out to lie in an algebra isomorphic to the ordinary matrix algebra, which we call the PI matrix algebra. The PI matrix algebra, defined on the Hilbert space of a composite system (including an ancilla system and a central system), is isomorphic to the matrix algebra defined on the Hilbert space of the ancilla system. By extending the PI matrix algebra to the Hilbert-Schmidt space of the composite system, we provide an exact and unifying condition for PI quantum control against ancilla noise.