It's clear that secondhand smoke does harm others, but the extent to which it does is a matter of debate. There's non-tobacco funded research that indicates it's not a significant risk unless you live with an in-home smoker for 30 years[1] (lung cancer only, not sure about heart disease). There are studies that paint a far more extreme situation, I'm sure, but I think this puts occasional outdoor exposure into perspective.

[1] http://annonc.oxfordjournals.org/content/26/1/221.short