There are estimated to be 1.57 billion Muslims in the world. Let's go over the top and say that ten thousand of them have been terrorists and/or extremists in the past decade. With those figures you get, wait for it, 0.0006369426752% of the entire Muslim population.
Yes, it is a minority.
Another point of view?
Remember that Troyston guy who admitted he'd shot and killed a sheep. He's one of many many trials riders. Would it be fair or accurate for, say, a farmer to assume that, based on the actions of a very tiny minority of a select group, all trials riders are likely to kill his sheep?