Really depends on what sort of tom sound you like.
I have mostly two drumsets. My Ludwigs have 'normal' deep toms/kick and i use those most of the time. I also have a random Lido(some 70s offshore Ludwig-related company IIRC) drumset that has considerably shallower toms and kick. I use those when i want to tune to like actual notes or something, like a jazz drumset sort of sound.
I use the deep ones for when i want more judd than poing, which is like... most of the time.
I will say that the shorter toms take up less space to store and move them around.
But mainly yeah, i feel like the big drums have a certain natural tuning that they like more than anything else, and it tends to be pretty low. The smaller ones seem to have a little more flexibility for tuning to different notes, but they aren't able to get as low and large sounding as the big ones. They do work better with smaller sticks though, just because of the ratio of size to depth, meaning less head movement is needed for a full sound to come out. If you hit a deep tom with a 7A, not much happens; if you use the same stick on a shallow tom it makes a much better sound.