Generally, nothing forces one DNS to resolve to one ip and one ip to correspond to one DNS. So you're losing both a simple way to do load-balancing and virtual hosts.
That being said, both can be worked around by other ways. You definitely lose a way to do geographical load-balancing, though (like in a CDN).
This only moves the logic you need to do in CDN DNS server to somewhere in your web serving stack.
You could still implement crude geographical load-balancing by simply looking at client IP address for example at your load balancer/reverse proxy or the HTML serving server itself.
Nothing forces you to serve same IP address to each client. For media streaming, you can do some IP round robining to balance load. For serving CSS, images, scripts, etc. you'd want to hash some relatively static client derived attribute to pick a server to avoid defeating client side caching.
At its simplest, map first octet of client IP to optimal geographical load-balancing IP(s). I'm sure there'd be some sub-optimal choices, but it'd still be a reasonably good approximation. This works, because first octet determines at least whether a given IP is assigned to RIPE, ARIN, APNIC, specific organization, etc. You get about country level accuracy, at worst about continent.
Or use a full fledged geo IP database to find an optimal server IP for each client IP.
Of course this isn't very practical, because requires more maintenance and is not effortless like using traditional CDN DNS resolution based methods. Maybe some CDN vendor could offer this in a more managed, easily integrable fashion.
I'm sure it could be made work, if the potential latency win is worth the effort for you...
That being said, both can be worked around by other ways. You definitely lose a way to do geographical load-balancing, though (like in a CDN).