Yes it will rub off but with smooth rims the brakes tend to slip alot more often,My rim had a old grind it slipped alot now its got a fresh grind it makes the brakes alot sharper,if you grind your rim use a angle grinder not a bench grinder,
thats just a link if you are going to grind and if you dont know how.