If you want to ride trials just for fun/hobby, it's never too late. You've already noticed that there are many people similar age riding. Various, if you want to go pro, i'd say it's a bit too late (just like in any sport, i guess).
Edit: so no don't sweat, keep riding, be confident.