An optical system is provided for laser encoder resolution extension with three-dimensional motion decoupling capability. The optical system includes a first prism mounted on a moving target, and a plurality of prisms, a retroreflector, a laser source, and a detector mounted on a fixed base. The moving target has three-dimensional linear motion freedom. The first prism on the moving target and the plurality of prisms and the retroreflector on the fixed base reflect a laser beam from the laser source to the detector define a three-dimensional optical path. The three-dimensional optical path provides multiple times optical resolution extension power for linear displacement measurement and encoding. This optic system is only sensible to the target motion on X direction and is substantially unaffected by movement in the Y and Z directions.