You don’t need to know well all those dozens of people; it’s just more about not being total strangers. We would ideally have 1-3 very close friends and then a slightly wider circle of the next ring, and so on.
I would say your level of exclusivity is very rare. Few people can tolerate that little contact for that long. There is certainly a middle ground for everyone’s satisfaction and it’s great that you found yours but the majority of friendless society is lonely. Maybe you jive with your remote coworkers more than other people do with their regular company.
This is the way.
This is also how Mormon churches work, last I checked; they’re called “wards,” and when a single ward grows to >500 members, they split off into two wards, to make sure everyone knows each other decently enough (probably to make sure they’re all tithing regularly and crap, but without such a sinister agenda, that can legitimately be a beautiful thing).