This paper analyzes the dynamics of an amplitude-modulation atomic force microscopy (AM-AFM) system, and designs a novel output feedback robust adaptive control (OFRAC) law to improve the scanning performance of the AM-AFM system. That is, a control-oriented reduced model is proposed to approximate the mapping from tip-sample separation to oscillation amplitude, whose accuracy is verified by experimental results. Considering the facts that the parameters of an AM-AFM system vary with different combinations of piezo-scanner and cantilever as well as detected samples, and measurement saturation occurs frequently in dynamic AFM systems, an OFRAC strategy for the piezo-scanner is designed to keep the oscillation amplitude of the cantilever staying at the desired setpoint under various complex situations. It is shown theoretically that the proposed control strategy pushes the system away from the saturation state in finite time, and it ensures uniform ultimate boundedness result for the control error. The OFRAC strategy is applied to a virtual AM-AFM system, and the collected results clearly demonstrate that it presents superior imaging performance for high-speed scanning tasks.