I disagree. I don't think there's a technology company in the world that can identify talent reliably enough to make the argument that they can find a team of 5 who will outperform 1,000. Even if the 1,000 people are average.
Painting is largely a solo occupation, and you can reliably judge the talent of a painter by looking at their past work. But it's basically impossible to judge the talent of a software engineer by looking at their past code -- there's too much context required. I think the sports analogy is better in this respect, because it's much harder to separate the the record of a sports superstar from the team that they're on. So it is with non-trivial software.
And also: five "brilliant" painters might be so socially dysfunctional as a team that you'd rather take the 1,000 average guys who can take direction well. You wouldn't automatically get a great work of art if you told Van Gogh, Matisse, Monet, Munch and Seurat to collaborate on a mural -- but you see all sorts of great murals that were painted by teams of less-talented artists.
In the early stages of a company, where you need creative thinking, new ideas, rapid iteration, and are building the technical foundations of the company, you need the 5 brilliant engineers, and wouldn't have anything to do with the 1,000.
In the later stages of the company, when you have 500 enterprise customers creating support requests and you have internal payroll systems etc., then you need the 1000 engineers.
And in technology, if you don't have the 5 at all times, you're on borrowed time. Look at what Steve Jobs, Jonny Ive, and pals did to Nokia.
> But it's basically impossible to judge the talent of a software engineer by looking at their past code -- there's too much context required.
I'm not really sure how this is true. I find that I can judge a software engineer very accurately based on his previous work, and intuit from the code a great deal of the context.
In fact, if I can't determine an engineer's talent without having more context, they are likely an average to below average engineer.
> I think the sports analogy is better in this respect, because it's much harder to separate the the record of a sports superstar from the team that they're on. So it is with non-trivial software.
I think what you're actually doing here is elucidating the difference between a 'superstar' and a 'team of average'. A senior high-quality engineer will have a considerable amount of new code that they've architected and implemented, whereas a team of average will have a considerable amount of code they have maintained.
Like most software organizations, our projects are divvied up into components and subcomponents, and individual engineers own components/subcomponents individually, with conformance to a broader set of development standards. The more junior an engineer, the more likely that they'll be assigned maintenance or improvement work rather than writing a component from scratch.
It would be very easy to evaluate the senior engineers by evaluating their components, whereas the junior and/or average engineers can only really be evaluated in context by the senior ones.
"I'm not really sure how this is true. I find that I can judge a software engineer very accurately based on his previous work, and intuit from the code a great deal of the context."
To each their own, I suppose. Applicant-submitted code samples don't correlate strongly with hiring decisions, in my experience. They're too easy to fake, and don't capture the things you want to know about interpersonal communication, thinking style, etc.
Crass Generalization: I think people who strongly emphasize code samples over everything else are sacrificing a chicken to the hiring gods. It smells very cargo-cult.
"It would be very easy to evaluate the senior engineers by evaluating their components, whereas the junior and/or average engineers can only really be evaluated in context by the senior ones."
Nah, it's always really hard. Some senior engineers don't write a lot of code, but are superb at managing and mentoring teams. Others hole-up in their geek caves, churn out code of varying quality, but are dismal leaders.
My point is: once you've got more than a few people, team dynamics matter at least as much as "rock-star" coding ability (probably more), and you can't tell anything about this from code samples.
> To each their own, I suppose. Applicant-submitted code samples don't correlate strongly with hiring decisions, in my experience.
We evaluate their open source code or paid challenge project, not applicant-submitted code samples of unknown origin.
> They're too easy to fake, and don't capture the things you want to know about interpersonal communication, thinking style, etc.
All of those skills are important, but they're moot if an engineering applicant can't actually code.
> Nah, it's always really hard. Some senior engineers don't write a lot of code, but are superb at managing and mentoring teams. Others hole-up in their geek caves, churn out code of varying quality, but are dismal leaders.
You're conflating quite a few areas of responsibility here. Some people are good leaders, but leadership is also where poor engineers will run to if they can't actually code -- this is not something you want to occur in your organization unless they actually belong in management, and even then, you risk those individuals pushing for poor engineering decisions from a position of authority.
If a senior engineer isn't actually designing and writing software, they're not an engineer anymore, and should be evaluated by a distinct criteria. If you require non-coding engineers to provide a small engineering team with direction, then you likely either have an overly junior team, or too many directionless/mediocre engineers.
> My point is: once you've got more than a few people, team dynamics matter at least as much as "rock-star" coding ability (probably more), and you can't tell anything about this from code samples.
Nobody (least of all me) ever used the term "rockstar". There is simply an enormous difference between the efficiency and code quality of great engineers as compared to mediocre ones.
Most of what you're saying sounds like the standard bandaid approaches to big-enterprise engineering management with mediocre teams.
Applicant-submitted code samples don't correlate strongly with hiring decisions, in my experience. They're too easy to fake, and don't capture the things you want to know about interpersonal communication, thinking style, etc.
Yes, by themselves they're not much good, but if you explicitly ask for a recent piece of code that they're proud of, you can ask them questions about it: "Why did you use data structure X?", or "How does function Y work?" Much, much harder to fake.
The key untested question then, I think, is whether you can successfully identify the 5 vs 1000 when recruiting. I think it would be pretty easy to show that there exist teams of 5 that outperform teams of 1,000 (certainly teams of 20 outperforming teams of 4,000). (be grateful you haven't had to work for any of the latter)
A secondary open question would be whether an existing team of 5 who currently outperform some larger number tend to continually outperform them after being acquired. I would bet that the performance gap would be close to the same. At the very least I would suggest that the cases from other industries give us next to no insight for software development.
"I think, is whether you can successfully identify the 5 vs 1000 when recruiting."
Hm...I think the question is, can you do it reliably? Anyone can get lucky once, but we're talking about acquihires for tens of millions of dollars. You've gotta have a pretty high degree of confidence in your decision to shell out that kind of cash for the promise of the future productivity of a team.
"A secondary open question would be whether an existing team of 5 who currently outperform some larger number tend to continually outperform them after being acquired."
If the research cited in the article is to be believed, most super-stars cease being super-stars after acquisition. Maybe the odds get better for teams, but it sure seems like a lot of brilliant product teams get acquired by big companies (even "good" companies, like Google), never to be heard from again.
This line of reasoning has always sounded like Valley Conventional Wisdom to me. Take a small, high-performing team and stuff it into a big bureaucracy, and the culture of the bureaucracy is going to win.
Painting is largely a solo occupation, and you can reliably judge the talent of a painter by looking at their past work. But it's basically impossible to judge the talent of a software engineer by looking at their past code -- there's too much context required. I think the sports analogy is better in this respect, because it's much harder to separate the the record of a sports superstar from the team that they're on. So it is with non-trivial software.
And also: five "brilliant" painters might be so socially dysfunctional as a team that you'd rather take the 1,000 average guys who can take direction well. You wouldn't automatically get a great work of art if you told Van Gogh, Matisse, Monet, Munch and Seurat to collaborate on a mural -- but you see all sorts of great murals that were painted by teams of less-talented artists.