Apparently that happens only if you start getting away from the group -- until you magically teleport back to where the rest are at. Hope you enjoy being with a bunch of strangers because you don't have a choice.

[Except if you get transported into the Burgh, but yeah, apparently it's not a constant feeling!]