After my wordpress was updated to 4.0, I had the same problem. (alert handshake failure)
My BackWPup version was 3.1.2
I updated this version to 3.1.4 and it looked like the problem was solved.
It was not, there was an error “access token not found”
What i did: First, login to https://www.dropbox.com
and also
login to your wordpress admin area.
BackupWP, Jobs, Edit
Goto tab “To: Dropbox”
Delete Dropbox Authentication
then
press button “Get Dropbox App auth code” or press the “full dropbox auth code” whatever your pre is. (The first is good enough)
This opens the browser which asks you “BackWPupApp would like access to its own folder, enz. Click “Allow”
The accesscode is generated.
Copy the code and paste it in your wordpress “App Access to Dropbox”-box. (or the other, whatever your first choice was)
Save it. Test it via “Run now”. It worked for me;-)
Regards,
rjthebest