Do you think that it really bothers anyone if someone calls them a bigot ?

I had a guy call me prejudice because he was begging for money and I said I had none (and I truly didn't but had I, I wouldn't have handed it to him but to the guy whom owned the hotel he claimed he was staying at). It bothered me to no end because I am not a bigot.