Should I feel bad...it's just business?!?!?

You absolutely did the right thing. It's business. You listened to your gut and saved yourself a lot of grief. "Sorry, I won't be taking this place. I'm sure you'll be able to find someone else very soon." Why, if he asks? "Those are my personal reasons. Have a nice day." Click!