You think this is a psychological thing?
Have you ever ridden something you felt was huge gone back to it the next day and it feels so small?
After riding so well it's easy to think you're better than you are, meaning when you do ride to your real ability you feel like you're capable of more, which in turn does affect your confidence which kills your balance + confidence.
There are two sides of the brain, the remembering side and the doing side, it is easy to let the remembering side take over, for example, have you ever driven home then not been able to remember how you got there? That's because you're no longer thinking how to get home, and it's your remembering side which takes over.
It's the same with this, Having a few days rest you're more likely to actually think what you're doing, instead of remembering and doing the wrong thing and not knowing why.