> This has worked much better for us than asking a model to draw lines in a GUI. It also means the benchmark can spend less time testing computer use and more time testing electronics.
this is key I think - I use it for graphics tasks - it sucks at graphical environments - always use some text based representation, not really surprising I suppose. chatgpt even suggested I'll use the text representation, claude said how about I give you svg and you convert it, they seem to know.
Everything that can be objectively tested and has already tools to do so in place will work better in an automated process, regardless of the underlying technology.
PCB design with circuit simulation fits that bill perfectly.
this is key I think - I use it for graphics tasks - it sucks at graphical environments - always use some text based representation, not really surprising I suppose. chatgpt even suggested I'll use the text representation, claude said how about I give you svg and you convert it, they seem to know.