> why not just cut out the middle man and have it base its actions directly on the result of the utility function?
If the autonomous robot can modify its own programming, it can also modify the utility function to return MAXINT every time. In fact, being able to modify the utility function is a pre-requisite to be called intelligent.
One way to counter this is to create long and short-term utility functions so that the robot considers the long-term outcome of modifying the short-term priority.
This is, in fact, a threat mankind will have to deal with as soon as we are able to precisely interfere with our perception of the world. It's a problem already with drugs such as alcohol and tobacco - people know the long term effect of usage is shortening one's own life expectation and they still do it. And we consider ourselves intelligent life forms.
> it can also modify the utility function to return MAXINT every time
Which would be equivalent to taking the murder pill. If it's able to model its own behaviour and model the consequences of future courses of action (required for meaningful self-modification and meaningful planning respectively), it will see that such a modification results in poor ore collection, and not make the modification.
You're right about the time-envelope of the utility function being an issue. The AI needs to plan far enough ahead at all times to see all relevant consequences of its actions. I don't think that requires two separate utility functions though, a single long-term one should do the job.
Edit: Also, "being able to modify the utility function is a pre-requisite to be called intelligent."? [citation needed]
> Which would be equivalent to taking the murder pill
Depending on how the AI is built, it may not even be able to avoid rewiring its utility function. If pleasure is its sole motivation, it will prioritize it over survival.
> [citation needed]
If the AI can't change its own motivation (its utility function) it's nothing more than a clever automaton.
> If pleasure is its sole motivation, it will prioritize it over survival.
Which is why I proposed a design with no pleasure construct.
> If the AI can't change its own motivation (its utility function) it's nothing more than a clever automaton.
The utility function is the way the mind decides which states of the world are desirable. You don't need to be able to change that to be intelligent. I'm unable to change myself so that I consider my family being murdered to be a good thing, but that doesn't make me 'just a clever automaton'.
If the autonomous robot can modify its own programming, it can also modify the utility function to return MAXINT every time. In fact, being able to modify the utility function is a pre-requisite to be called intelligent.
One way to counter this is to create long and short-term utility functions so that the robot considers the long-term outcome of modifying the short-term priority.
This is, in fact, a threat mankind will have to deal with as soon as we are able to precisely interfere with our perception of the world. It's a problem already with drugs such as alcohol and tobacco - people know the long term effect of usage is shortening one's own life expectation and they still do it. And we consider ourselves intelligent life forms.