The problem still persists with 2.7.3 - if you use ask_password, you still get ERROR: unexpected error - Passphrase must be specified to import encrypted private keys if your key is encrypted. It also does not work if you ssh-added your key.
Downgrading to 2.5.4 solves the issue with the ssh agent for me.