Let's say we have, like, 200 guards.
We've got an ABSOLUTE MINIMUM of 5 or 6 in each section of the city, so that's what, 30 guards to cover the city?
We've got four by the city gates, multiplied by four gates, that's another 16 guards busy.
Two guards stand by the gates that lead onto the royal estate, two guards stand by the doors of the castle, and two guards stand by the throne room doors. Two guards with the magical ability to turn invisible remain beside the throne at all times. That's another eight. So now we have 54 in specific positions.
The castle has three floors, not including the guards' quarters and servants' quarters. They're not huge, so if we even dispatch a minimum of ten guards for each floor, we'll have a total of 84 on patrol at any given time.
That leaves us with 116 guards unassigned. So, obviously, these guards will be added to the various sections of the city, added to the hallway patrol, and sent on specific tasks. Some of them will be off-duty, eating, visiting family, training, derping around in the castle, sleeping, waiting to take over the shift of another guard who's currently working...some will be doing specific tasks, like Diodora's group being sent to arrest Loran.
I think that 200 is a good number. Given that about half of them will be off-duty at any given time, and the captain wouldn't have daily interaction with every single one of them every single day.
Also, I want to question why a highly trained assassin can take on multiple guards, but a guard can't take on multiple opponents? Who's to say that a highly-trained royal guard can't take on 25 highly-trained assassins? Does the pure power of evil make assassins more deadly than guards? It's like the opposite of the situation in movies where villains can't aim but heroes have perfect aim all the time.