There are many components to cognitive ability. For example, there's the Cattell-Horn-Carroll theory that has several different abilities:

  Crystallized Intelligence 
  Fluid Intelligence
  Quantitative Reasoning
  Reading & Writing Ability
  Short-Term Memory
  Long-Term Storage and Retrieval
  Visual Processing
  Auditory Processing
  Processing Speed

This was cut and pasted from wikipedia (http://en.wikipedia.org/wiki/Cattell-Horn-Carroll_theory). If you've taken an IQ test, you'll see that they test various types of cognitive abilities, not just math or logic, but also the ability to rotate objects in your head and visualize them.

I know plenty of people who are "smart" who have a very hard time with that. In addition, I know plenty of people who aren't academically smart, but who are great visualizers. There are many different components to cognitive ability. I have a friend and both he and his father are exceedingly intelligent. My friend's son also is also a gifted child. However, when the son took some cognitive tests, his spacial/visualization scores were average, whereas the verbal and math were off the charts. I don't think that affects whether or not the kid is smart, it's just interesting how cognitive ability is broken up into different measurable areas.

Is it so hard to imagine that computer programming takes a certain subset of cognitive ability that some people, who are considered smart, just don't have? I know a lot of very intelligent people who wouldn't be able to program at all. It's not because they're "dumb", but as I've said, but the way they process information is completely different. You most definitely could develop the skills, however, if their brains are wired a certain way that causes them to be predisposed AGAINST programming, then they won't enjoy it very much at all.

So there is a logical jump in your reasoning.

- Some brain visualisation skills are innate

- Programming (probably?) requires some of those traits.

- JUMP

- "either you are born with the ability to be a programmer or you're not."

What you haven't demonstrated (beyond some weak anecdotes), is that those innate skills can't be taught.

Even if you proved that, you'd then have to prove that developing software, will always involve those skills. E.G. programming in the 60s required a different set of skills to programming now (with some overlap).

Abstract thought. Many people have real problems going from "A is X" to "A is a reference pointing to X" or "A is of class X".

Whether that is genetic is unknown to me, but it strikes me as something that coders have to be good at.