Yes, of course you could. The point I made was that most people have pointed out seeing the wall while going straight from the ship, which would technically be impossible with the inn blocking sight.
So people would have to walk to the Inn to actually see it...
Sneaky...I think most people missed that though xD
I was imagining something like a giant wall that covered half the island or something.