Sometimes I don't wear it because it makes my head itch and is uncomfortable, but recently I've been wearing it most rides. In fact, the other day D-L-B forced me to wear it, and he (like some of the comments here) had a good point.
It's a small price to pay to insure your head. You can't get a new brain, and damage to the skull takes a long time to heal and isn't fun. When I first started riding trials I went over the bars near the top of a big flight of slippery steps. Totally idiotic thing to do, riding down them, and I was knocked out with a helmet.
It is important, and you'd regret it if something went wrong. Think about the things that can happen...
1) Hubs skips, over the bars
2) Chain staps, over the bars super-quick
3) Gears slip, over the bars
4) Back brake fails, off the back (head to rail wouldn't be fun...)
5) Snapped forks, forehead smashed
I could go on...
And Simon, I think it's good that people get pestered to wear helmets. A lot of the time it's for silly reasons like trying to look cool, or negative peer pressure from people who don't wear lids.