There aren't any rules or modulation police (well, this IS TGP so maybe there are ). There are a lot of similarities between those three, but they are different effects. If you used them together it would turn into mud, but aside from that...
If it were me, I'd only have one of those on a board, but that's just me. I use my modulation to thicken the sound a bit and add a touch of texture. I can get what I want with any of the effects you mentioned, so I don't need more than one. I do have several boards, though, with a chorus on one, a phaser on another and an H&K Tube Rotosphere (Leslie simulator) on a third. That's how I cover the bases - I like my modulation too.
I can never have too much modulation. I have two phasers and a ring mod on mine. Plus delay, reverb, and pitchshifting. Play what inspires you. Our opinions don't matter.