An alternate possibility would be to wrap some sand paper around a small file and open the hole up to get the bolt to clear. Diamond files are also useful in this regard. A set of budget import diamond files was like $5 file. The diamond doesn't last long on those budget ones but enough to do the job a few times to make them worthwhile.
Sometimes after a heat treat I've wanted to do other handle options, picked up some carbide drills for the fasteners I use. From $20-$40/bit and they are savers when that realization of "Oh Sheep!" hits.
