As a former swimmer, I can say that shaving is not only done for drag. Once shaved and in the water, even while casually swimming in warmup, I would feel as if I'm actually floating better. We'd call the "feel for the water" or something similar. It's hard to describe unless you've experienced it, but I suppose it has something to do with a newly exposed layer of skin in contact with the water.

And, of course, there are also the benefits of less drag and psychological placebo effect.