Everyone is allowed to do what they’d like to with their free time. When you’re off the clock, you’re off the clock and you don’t have to answer to anyone else. That is, unless you have toddlers, then you definitely have to answer to them. Either way, nobody can force you to do anything you don’t want to during your time off, even your coworkers. It’s nice to join in after-work activities when you can since it helps build rapport and you might get a free drink or two out of the deal. But if you already have a personal plan, then those tend to come first. After all, work already takes up so much of your time.
You’re not required to give them a couple more hours. But some folks don’t feel this way and revolve their social lives around work. Their coworkers are their best friends, and the after-work outings are their main social outlet. This is fine, too, but it doesn’t have to be everyone.