The fact the author can think of one particular way to design an AI that would result it in falling over and dying once it figures out how to modify itself does not imply that all possible intelligences would behave similarly. Somehow, humans are able to "want to be happy" while still refusing to take heroin or pain killers, and we have no reason yet to believe that AI is fundamentally unable to grasp this distinction between "stuff I care about" and "the different stuff I would care about if I changed my own design".
The fact the author can think of one particular way to design an AI that would result it in falling over and dying once it figures out how to modify itself does not imply that all possible intelligences would behave similarly.
I was actually surprised a bit to see that the author was somewhat familiar with Eliezer Yudkowsky's writings on the topic (he cited http://lesswrong.com/lw/wp/what_i_think_if_not_why/), because the line of thought doesn't seem to incorporate a real understanding of what he's said on the topic (which, to be fair, is a huge body of work...).
Most of EY's "Friendly AI" worries are rooted in this idea that when considering the entire universe of algorithms that could be described as intelligent or self improving, we need to be exceptionally careful not to assume that more than a negligible percentage of them share anything in common with human intelligence, because for the most part, they won't, unless they're carefully and explicitly designed to do so.
Here, the author assumes that the AI is simply trying to optimize some internal measure of happiness, with complete disregard for the meaning of that measure. This is an incredibly naive view of how deeply important and carefully constructed any optimization target would have to be in any self-improving intelligent machine; it's literally the core of the entire problem of friendly AI, and to trivialize it by assuming that such an AI would ever even consider rewriting its "happiness button" to be always-on is to miss the entire difficulty of the problem.
Hell, it's even the core of the problem of non-friendly AI, because it doesn't even require human level intelligence to realize that if you rewrote your own code so that you were always thrilled with the result, that's the easiest way to increase "utility". Any self-rewriting algorithm that's capable of real self improvement has to, by design, be able to consider the likelihood that changes to its objective function will end up with negative expected value.
None of which is to say this isn't a valid concern, by any stretch. But it's not a proof of universality; in fact, getting around this type of problem is exactly what any real AI designer must contend with. It's an issue that's very well known, and it's certainly not well-accepted as an insurmountable hurdle.
A transhuman AI worthy of the name would be very concerned with how well it was able to model the real world -- if for no other reason than the sake of its own survival. Think of that as a 2nd order "survival imperative" of sentient beings.
The overlooked point here is, We do not have any control over how we build next generation. We have a little control by teaching them. The example author gave assumes each generation having lots of control over the next generation, and hence figured out what bugs them and solved it. I do not think we know what bugs us. For some people other people having different religions bugs them, so they kill them and be happy. For some people unknown unified theory bugs them, they will not be happy till it found. So It is not possible to find universal solution to different problem. But It could happen that people with similar problem join a group and find a shortcut way of being happy, without the problem being solved.
>We do not have any control over how we build next generation.
Says who?
The author assumes uncertainty and exploits uncertainty to make a certain conclusion, when the uncertainty should just prevent him from forming a conclusion at all.
In fact it's fairly regular, I think, for singulitarians to let their imaginations run amok this way.
The point is that there are humans who will refuse to have that machine hooked up, even though they understand completely that it'll make them feel so good they won't care if they use it.
