Make sure the background threads actually finish before
deleting the object.

Signed-off-by: Luca Fulchir <luker@fenrirproject.org>