I could imagine something like that after 2-3 game lost, not only for 1 game in a seanson.
Just noticed you said that, if you look at the 3 games before this drop in attendance you will see the following:
1. Lost your cup game
2. Won against 7th place team
3. Lost against 1st place team
You lost 2 out of your last 3 games before that match where your attendance dropped :)