While spectating a live game or observing a recorded game you are still stuck with things like control group banners (which are useless).
This is because AoT never had an observer UI programmed into it so it just takes your default UI. I have adjusted the UI to remove all the unnecessary buttons and moved things around to give a clean view of the game you're watching.
This mod is best used in conjunction with my Enhanced Tooltips mod since it has a very big tooltip window. Using a bigger font will now not result in losing info from the tooltip.

When watching recorded games you must press the button named "REC" in the top right corner. This will transform your UI to be suited for observing recorded games.
The progress bar is hidden by default. But you now have a button to make it visible again.
In the top right you can see the idle Villagers / Fishing Ships / Caravans of the selected player.

Recent Voobly Balance Patch only:

The UI has built in buttons to say !t/!r/!1/!2~~ in allied chat to activate the map scripts. These can be pulled up using the button right of the minimap.
They will be displayed above the score and will have the same color as the corresponding player.
They are hidden by default and each time you press the button 2 come up. Thus you can swap between 1v1 2v2 3v3 and 4v4 mode. Pressing the button
when all 9 buttons are visible results into them becoming all hidden again, factually resetting the UI.
These buttons are hidden while watching recorded games since they do not work there. You can still type !1 etc in the chat to activate the command but somehow these buttons don't function anymore ;(.


You can also use commands in your user.con to hotkey these chat commands.
map("1", "game", "sendChatToAllies(\"!1\")")
map("2", "game", "sendChatToAllies(\"!2\")")
map("3", "game", "sendChatToAllies(\"!3\")")
map("4", "game", "sendChatToAllies(\"!4\")")
map("5", "game", "sendChatToAllies(\"!5\")")
map("6", "game", "sendChatToAllies(\"!6\")")
map("7", "game", "sendChatToAllies(\"!7\")")
map("8", "game", "sendChatToAllies(\"!8\")")
map("t", "game", "sendChatToAllies(\"!t\")")
map("r", "game", "sendChatToAllies(\"!r\")")

You can use this command to toggle the fog of war while casting recorded games.
map("k", "world", "blackmap")