:/ I think the biggest thing I can suggest is to just not have such a specific endgoal in mind. RP's, by their nature, are open-ended in where they can go. It should be expected that the ending of an RP is at least somewhat unpredictable, and almost impossible to predict when you're only at the starting point of an RP. If you have a story you want to tell and you already know how you want it to end, maybe you should just write a book.
In my RP's, I focus less on following a specific sequence of events and more on just giving the players something to do. Have the players gone off on their own little side-quest that's engaging and giving everyone something to do? That's good. Let them go do that. Is everyone just kind of derping around without having anything to do? Then it's time to give them something to do. If you have an RP with a plot in mind, then that means moving the plot along in some way -- such as introducing the next threat that the players are up against.
And even with side-plots, it's not like you can't still bring things back around to the big bad eventually. Just don't overthink the details or create a specific list of bullet points to hit. Instead, just... go with what the players want to do, and adjust the amount of control that you have on the RP depending on what is needed. Players generally don't like being told exactly what to do or having their own plot ideas stifled, but players will also get bored and leave if nothing interesting is happening. You have to find the balance between the two. If your players are engaged and imaginative and have lots of awesome side-plots going on, then that's something you definitely want to encourage. But when you hit a lull, it's important to get the main plot going again so that things don't remain slow and boring.
As for how to nudge players in the direction you want without being "aggressive" about it -- just try to set up the IC in a way that encourages the players to go down the route that you want. For example, say you want your players to head in the direction of the next boss fight. But then the characters say "um yeah no, how about we not go that way and do something different instead". Instead of trying to force anyone's hand and making the players go that way through OOC means alone, you can do something about it in the IC. For example, maybe you could place an even bigger threat on whatever alternate route the players wanted to go down, or just come up with any other reason why they can't go that way. This encourages things to go the way you want. And if the players do go a different direction and still find a way to make it work -- well, that at least gives you time to re-structure your plans so that the big bad remains relevant.
The big thing is always flexibility. Having a main goal for the ending is good, but don't make it a specific sequence of events that need to be hit. When your players change things up, roll with the punches, and figure out how to gently steer things in your direction based on what happens. Your haunted house monster is hiding behind a specific door, and you're frustrated that your players aren't opening the right door? Guess what: there's no reason why it had to be that specific door to begin with. You can put that monster behind any door that the players open. So long as you haven't yet revealed all the plot twists and turns that are yet to come, there's no reason why you can't re-write all those things to accommodate for changes made by the players.
And, even then, if you're too specific about what you want, then some things just aren't feasible after the story has deviated too much. So, again, I just recommend only having vague plans about how you want the story to reach a certain point. Have backup plans for specific events. If the players don't defeat the monster the way you wanted, think about what would happen if the players tried something else. Be flexible.
And if there's something that you really want to happen, then make sure you have multiple plans on how exactly to reach that point, so that your plans aren't completely thrown off by some small deviation in the story.
As a general rule of thumb, the farther ahead something is in the future, the more vague your plans about it should be. Because the further you go into the future, the less certain you can be about where the story will be at that point. If you're just planning the immediate future for the RP, then it's easy to plan for specific actions taken by the players and to guide them down specific paths. But determining the exact series of events that should lead up to the final boss fight just doesn't make any sense, because you have no idea what the RP will look like by that point in time. When you get closer to these far-off events, then you can start planning things more concretely -- but only because there's less uncertainty standing between the present situation and the desired ending. Not to mention, if your RP has made it that far, then you've probably already changed quite a few details about the ending in order to match where the RP has gone -- or at least changed how you planned for the players to get there.
Hopefully some of this helps. And, like I said, if you really can't handle the story deviating at all from what you had in mind, then... maybe solo-writing would be a better choice for you. :/ RPing is a collaborative effort, and so, uncertainty in its story's future is just the nature of the beast.
