There's no particular health risk of any kind, but they are pests and create problems for native (and preferable) birds. If you want to get rid of them, by all means go ahead.
I have a house sparrow nest under my window ac unit on the 3rd floor of my house. I don't want to hurt them but I think I have to get rid of them. I heard that they carry microscopic bugs on them that can get into my house. Plus they've been poking holes in the duct tape I have to seal the window unit.
Do I have a legitimate concern for my heath here of should I wait for them to leave before I trash their nest.