I find that this is exactly true. Caffeine makes me feel worse in the medium term and exacerbates stressful situations. Having said that, it sometimes feels like a necessary evil to keep operating when sleep deprived :(