512,000 cycles per poll is probably plenty but I think the SRAM might be a bit small for that code. Could something be made to fit in that and still do the same things? Probably but it likely wouldn't be as easy on either the software or hardware side. Sometimes a lunch is worth more than the extra hassle of optimizing.
The Teensy 3.2 was fast enough; we're running matrix math for remapping the stick gates and a modified Kalman filter at 1000 Hz, and bitbanging the GameCube protocol meanwhile.
Teensy 4 is overkill and we've been clocking them at 150 MHz to reduce power draw.
I looked in to the ADC issues and decided it was fine. It has slightly lower effective resolution because certain ADC values are more likely to occur. It looks like in the end not a huge deal.
We already redesigned to switch away from Teensy 3.2, which had production suspended indefinitely due to parts shortages.