Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

Added some additional logging for an edge case, rolled it out to production and then went camping in the remote wilderness for a week. Two days in, the edge case got hit and the logging wasn't sufficiently tested. It logged as intended... and kept logging and logging... until out of disk space :(

Oh yea, I run a proprietary trading firm (still at the same spot), as a result of that bug we went down and lost about $250k over the next few hours. Testing is important in automated trading :)



Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: