When the probabilistic packet marking technique for traceback and localization of malicious nodes in Wireless Sensor Networks (WSNs), the determination of marking probability is the key to influence the convergence, the weakest link, and the node burden of the algorithm. First, the disadvantages of the Basic Probabilistic Packet Marking (BPPM) algorithm and the Equal Probabilistic Packet Marking (EPPM) algorithm is analyzed. Then, a Layered Mixed Probabilistic Packet Marking (LMPPM) algorithm is proposed to overcome the defects of the above algorithms. In the proposed algorithm, WSN is clustered, and each cluster is considered as a big cluster nodes, therefore, the whole network consists of some big cluster nodes. Correspondingly, each cluster nodes internal contains a certain number of sensor nodes. The EPPM algorithm is used between the cluster nodes, and the BPPM algorithm is used in the cluster nodes. Experiments show that LMPPM is better than BPPM in convergence and the weakest link, and the node storage burden of the proposed algorithm is lower than that of the EPPM algorithm. The experiments confirm that the proposed algorithm is a kind of whole optimization under the conditions of resource constraint.