Should I invite him to my wedding?

If you want no regrets, invite them all. That way you cannot end up feeling badly and wondering "what if." Most likely they won't come anyway since you aren't close with them.