This paper studies close formation control problem with prescribed performance and time-varying state constraints for a group of 4-degrees-of-freedom (DOF) underactuated surface vehicles (USVs) subject to actuator faults, input saturation and input delay. A finite-time sliding mode control (SMC) scheme based on reinforcement learning (RL) algorithm is introduced to guarantee prescribed formation performance without violating velocity error constraints. By using actor-critic neural network (NN)-based RL algorithm, the actuator faults and system uncertainties are accurately estimated. Afterwards, an exponential decreasing boundary function is developed to suppress overshoot more reasonably, and a novel mechanism of switching gain is given to alleviate chattering inherent in SMC while the RL-based compensation term is constructed to handle the formation accuracy problem caused by the reduced switching gain. Besides, auxiliary nonlinear continuous function and Pade approximation have been successfully applied to process actuator saturation and input delay, respectively. Numerical simulations and experimental results are exhibited to verify the effectiveness and superior formation performance of the proposed control method.