A man has a mistress for 30 odd years. About 12 years ago his wife dies. For some reason, his family know about the mistress.
One day after the wife dies, the man asks his son if he can come and stay the night en route elsewhere.
And bring his friend.
The son ( I suspect egged on by the wife ) says no.
So the man and his new live-in-lover ( ex mistress) never go to any of his family together, although they live perfectly happily and have a circle of their own friends, many from before the man's wife dies.
The man continues to see his family on his own. The couple visit her family together without any difficulty.
Now the lover effectively looks after the man until he dies ( she would be about 25/30 years his junior), and brings him great happiness in his declining years.
My view would be that the man's children were being petty.