In my opinion, is slow. And I bet you -or others- have room for increase performance by 2-4x in next years. Even without SIMD intrinsics (autovectorization-friendly code is OK, though), but with proper data structure usage, better code L1 cache usage, increasing data cache hit ratio, reducing function/pointer indirection, reduce data cache pollution, etc.
Executive summary: yes, I could probably get it twice as fast with identical accuracy (in the absolute best case -- 50% faster is more realistic); but I would sacrifice the portability, maintainability and readability of the code. This is a big problem because I have very limited time and resources as a single person. It's because the code is so straight-forward that I'm able to rapidly fix bugs when they are found.
First of all, your proyect is OK as is, I respect your criteria, of course.
My point in that regard was about the 3GHz requirement was because of the implementation, mainly, not because accuracy requirement. I bet 2-4x faster is possible, without reducing portability. If for "maintainability and readability" you mean keep it in C++ (L1 code cache impact, indirections, data structure penalty, etc.), I agree, that will be a penalty as fix cost in time and space.
