Normally, the booster doesn't actually fit and work properly without using the spacers. For example, on a 4bolt magura one, if it has no spacers the curvature of the clamp means, well, you need spacers. So use them, it's designed for them, unless you really can fit and fully use it without the spacers, in which case, whatever you like.