When the viewer has bugs we can only suspect that the GE does too.
There are different kind of bugs. You can see what's happening on the scoreboard and the player statistics. For example, commentary saying a player achieves a double-double when he missed a shot and lost the rebound and his stat line shows all zeros is pretty obviously a presentational bug, not a bug in the GE that calculates what happens in the game. Or when the shot clock goes to zero and the possession does not end, 15 seconds later there is a steal. We are pretty safe to assume it is presentational. We should suspect a GE bug if the team went and scored to finish a 40 second possession.
While purely presentational bugs are typically easy to spot, some stuff we cannot see from the score and other stats. To present suspicioin on such bugs with any credibility, a lot of information is needed. Many users who claim there is a GE bug base their claim on a single game, or two or three games they chose after seeing something suspicious. Now there could be a bug, of course, but it's pretty much impossible to say that based on such limited observation.
So, to answer your opening line:
Just for the sake of argument, what do we, as players, have to go on to know that the GE is working correctly or not...
We need to trust the BB's. And we need to observe more and more and more.
I don't agree that a squeeky clean match viewer gives us any assurance that the GE works correctly. But I agree it would be very nice to get a squeeky clean match viewer. An old presentational bug is annoying, a new presentational bug is confusing.