A new method extended from motion history image called Multi-Layered Mmotion History Images (MLMHI) is proposed to the representation and recognition of human activity using depth images provided by Time-Of-Fligh (TOF) camera. Firstly, the motion-energy image of the depth silhouettes is computed as the global motion information. Then, the forward-MLMHI and backward-MLMHI is computed as the local motion information based on the variable of depth. The global and local motion information constitute the MLMHI lastly. Since the Hu moments are sensitive to disjoint shapes and noise, R transform is employed to extract features from every layered-MHI and concatenated to form a feature vector. The feature vector is used as the input of Support Vector Machine (SVM) for recognition. Experimental results demonstrate the effectiveness of the proposed method.