You're right. People should be clearer when touting the benefits of SV. It's really about VC-backed startups that require venture capital (VC's are notoriously adverse to funding faraway startups) and that require access to talent that's willing and able to work 70+ hour weeks face to face with other hackers and entrepreneurs.

There are exceptions, but that's the general model and people have made billions with it so arguing against it is an uphill battle.