Next time you need to decline something small, don’t over explain. Don’t apologize 20 times. Say it clearly and watch what happens in the next hour.
If they can stay decent while disappointed, that’s information.
If the room gets colder until you fold, that’s also information and it’s usually more honest than their words.