Human civilization is currently limited to planet Earth with a few minor exceptions, so it makes sense to consider it one big system.
Actually it makes sense to consider all life as one big system. It was created by the planet itself, so, who knows we might even have to ascribe motivations to the planet. It's as if the planet (kind of like Lem's Solaris) has been brewing organisms for millions of years in order to do something with them . We might not be able to conceive these purposes with our antrhropomorphic thinking.
So, the human race is now coming to the point where it can modify and advance itself by tinkering with its own circuits. What we don't know is 1) what the planet plans to do with us and b) what are its movitivation and reward signals. It's not explained in the article how the AI knows what are the reward signals of its creator or why it would ever want to change them.
Actually it makes sense to consider all life as one big system. It was created by the planet itself, so, who knows we might even have to ascribe motivations to the planet. It's as if the planet (kind of like Lem's Solaris) has been brewing organisms for millions of years in order to do something with them . We might not be able to conceive these purposes with our antrhropomorphic thinking.
So, the human race is now coming to the point where it can modify and advance itself by tinkering with its own circuits. What we don't know is 1) what the planet plans to do with us and b) what are its movitivation and reward signals. It's not explained in the article how the AI knows what are the reward signals of its creator or why it would ever want to change them.