Prey often changes their behavior to reduce predation risks, affecting population and community dynamics. Most mathematical population dynamic models incorporating anti-predator behavior assume that prey responds similarly to different predator species. In reality, however, prey can take predator-specific reactions, and there is a possibility that anti-predator behavior against one predator increases predation from other predators.
Using a three-species model of two predators and one prey, we show that predator-specific anti-predator behavior is important for predator coexistence. Two predators used different attack modes. Accordingly, the prey could exhibit two types of costly anti-predator behavior and choose an optimal combination of the two types for given predator densities. Varying the similarity of attack modes between the predators, we evaluated predator coexistence based on invasibility criteria.
Predator-specific anti-predator behavior could facilitate predator coexistence, despite the fact that prey sharing could make predator coexistence difficult. Coexistence was more easily achieved when predator attack modes were more distinct. On the other hand, when their attack modes were similar, priority effects became stronger, making predator coexistence less likely. Our results suggest that attack modes of predators can serve as a niche difference that determines their coexistence in the modern coexistence theory.