I've rebuilt fandoms how I wanted, I've built fantasy from scratch and I've agonized over every scientific detail of hify adventures. In the end, the most annoying thing for all of them for me was a common sentiment here: the geography. Geography, to some degree, is meant to have a sense of permanence, although yes it can change. The world life lives in, evolves in, thrives in and dies in all ultimately have the largest role of all elements to worldbuilding. What has been around the longest? The land. With the simple alteration of a flow, what can wipe out a species and give birth to a new one? The sea. The more science and realism applied to this, the deeper you get into cause and effect - then ultimately there is a huge grey zone at the stuff we actually don't even know, including about Earth. Worldbuilding uses models from of course great acts of literature, but they all come from the same and only true model we have: our own planet. The reason I dislike geography in general is because for it to truly have the impact it should, an implied sense of permanence comes with it, and that permanence will somehow permeate nearly every part of the subsequent roleplay in the most minuscule ways. I just dislike knowing that I'm building what could be the riskiest part of the world in terms of opportunity trade-offs and realistic expectations.
Currently, I'm working with fandoms; my normal roleplays are more akin to Modern Fantasy. Either way, my favorite part of world building is the social engineering aspect. After creating this whole world, I want to know how an abundance of player characters are going to interact in it. I prefer large scale. I prefer tons of people influencing the world while the world influences them right back. I love maximizing interaction without minimizing quality or thematic elements, and finding ways to have people interact with each other in productive ways that create an interesting narrative is my favorite part of world building. It is then that you see every system, whether its magical, sociopolitical, geographic, biological, etc., come together and influence the real life of the world you just created: its players. I even focus HIGHLY on my setting only so that this feeling is more satisfying when it comes to fruition. While the world I build should be it's own living, breathing entity, I want to figure out how the entities of others are going to coexist in and on this complex dynamic I - and often others - created. It is also for this reason I tend to be strict with a ton of rejections, although honestly none more than a high fantasy roleplay. Everything needs to fit perfectly before it goes on to influence everything and become part of the system.
Admittedly, a close second is actually on spot with Jorick: I love to build and understand magical systems. I have this issue with almost always wanting to convert things into science magic or magic realism; true high fantasy escapes me. I guess I never escape to that place. In all my years roleplaying, I never have. Still, building magical systems is probably the most fun I have almost solely because it is the only system you really get to create 100% on your own. Magic does not exist; it's up to interpretation and inspiration. It's a true child of creativity.