I have a yubikey myself. The smart card method is much better though. When using a smart card, it is impossible even for the owner (who knows the access pin) to read the actual key. The crypto operations involved in the authentication take place on the card it's self. The smart card is protected by a pin, and will wipe it's self if the pin is entered incorrectly too many times. You can even use a smart card reader with a hardware pin pad built in so neither soft nor hard keyloggers can intercept it.
The weak point is you still don't know what you're signing, who or what you're authenticating, etc. if you don't have a trusted display and do the entire transaction on trusted hardware. (your host may be compromised, and could show arbitrary data, proxy connections, etc.) Protecting the key does go a long way, but even with a secure passphrase input and smartcard key operations, there is risk.
The ideal, IMO, would be some secondary hardware with a limited, secure OS which lets you put specific limits on how the key is used, showing a hash or identifying details of each transaction on the secure display, prompting you to verify each. E.g. to send a payment, the amount and payee (and date/sequence number) are shown on the secure display to verify before authorizing.
