Sit and watch the Handicapped spaces for a while. After you see all the people with no sticker, and no apparent disability using them, you won't feel bad at all about using them.
OTH. It's always polite to leave the spaces for others. My wife has a disabled tag. I frequently drop her at the door and go park in a regular space.