I frequently bump into beggars on the street who ask me for small change. Is it okay to refuse to give them money?
There is no obligation to give money to panhandlers, especially those who do not appear to need it or who appear as though they would waste it. If your personal safety is not at risk, you could politely offer them information on the nearest homeless shelter. Since you regularly meet panhandlers, you might ask a nearby shelter for business cards with contact information that you could distribute instead of money.
In any event, if you meet your obligation to support the Church and you give to reputable charities, you need have no qualms about gently refusing to give spare change to panhandlers.