I think
Malazan Book of the Fallen by Steven Erikson falls into that sweet spot you mentioned. It is somewhat popular but it is not so big as Game of Thrones. This is what I consider to be the main set of 10, but there are many more books in the universe. There are several substantial entries by Ian Esslemont, there are novellas, and it looks like Erikson has started on a new Malazan trilogy. I'm not fully caught up on books outside of that core 10.
It's high fantasy -- large cast, large well-developed world, rich mythology, world-changing events. The main reason I love the series is that the mythology is strong and it feels fresh, and it gets revealed to the reader in a very organic fashion. I am somewhat ashamed to admit that I have not yet read LotR. I tried it once years back but I couldn't get through the early chapters where it just felt like reading dry history. The Malazan books never did that -- they just drop you into conflict and the mythology gets filled in as you go. The way that magic works and how all the gods are related (and also, how the pantheon changes over time) is quite engrossing.
The first book is entitled Gardens of the Moon.
But that's actually not even my favourite series right now. My current favourite is
Shadows of the Apt by Adrian Tchaikovsky. It will also be a 10 novel series; 8 have been published and the 9th should be coming out this summer.
The reason I LOVE this series is that it doesn't use the usual fantasy tropes. There aren't any elves or orcs or dragons. Instead, Tchaikovsky creates something that feels completely new (though he inevitably does draw on traditional stuff). Instead of the usual fantasy races, he creates "kinden". These are races of people that take their name from bugs. They also typically develop certain abilities or characteristics based on their totem animal (and note: this is not considered magical at all). These abilities are typically called "Art". People of the same kinden don't even all share the same Art, just like how different people in real life may be more athletic or have different talents.
As one particularly interesting example,
Ant kinden people have one very notable trait. At a young age, Ants develop Art that grants them a mind-link with others of their city-state. That means they are in constant telepathic communication with their fellows. This gives them an interesting advantage in battle, and they are indeed a people of war... often fighting against foreign Ant cities.It is extremely delightful when a new kinden is introduced in the story and we as readers learn about their unique Arts and how they fit into the history of this world.
But there is more! Kinden in the world are generally divided between the Apt and the Inapt. The apt have no magic (and in fact, they do not believe in magic, regarding such things as superstition and myth) but are technologically capable. At the start of the series they use things like crossbows. The inapt, on the other hand, have the potential for magic but are completely incapable of using or even understanding mechanical devices. They once ruled the world, but the Apt have since overthrown them; as a result, magic is rare.
While world building like this is really interesting, the author is also skilled enough to use this as a solid foundation to tell compelling stories. I would go on, but it's tough without giving too much away.
As a bonus, the author has a
blog that includes numerous short stories set in the world. I've been going through them recently.