I know exactly what you're getting at. Anyone who's come from another diciplin of biking picks it up fairly easily 'cause as you said, they normally have the basics covered. But anyone who starts trials/biking from scratch will find it much harder.
But yes, further progression will become much harder the bigger you go! Just keep at it and enjoy it!