The comment under “Future Directions” at the end of that paper would seem to apply to the use of AI in software development, business management, and other areas as well:
“For AI[-assisted mathematics] research, improving research judgement and research-state representation may require training and evaluation on records of mathematics as a process, including failed approaches, strategic decisions, and evolving assessments of evidence. For mathematical practice, while these limitations persist, human expertise is likely to remain most valuable in these global functionalities: deciding when to persist or reframe a direction, and maintaining an accurate representation of accumulated progress.”
Yeah, the end goal is that you can run the economy autonomously, at higher levels of complexity than people could understand or reasonably make decisions about.
This makes no sense. It would be smart enough to understand but not enough to explain it, or are you assuming the human brain can’t comprehend the world by default?
“For AI[-assisted mathematics] research, improving research judgement and research-state representation may require training and evaluation on records of mathematics as a process, including failed approaches, strategic decisions, and evolving assessments of evidence. For mathematical practice, while these limitations persist, human expertise is likely to remain most valuable in these global functionalities: deciding when to persist or reframe a direction, and maintaining an accurate representation of accumulated progress.”