'06 Scion Xb automatic overdrive?

I like to "downshift" out of overdrive a lot by pressing the on on the shift handle. I do that mainly to "ist" in breaking, but also for other reasons. Will doing that cause early wear or damage to anything ... the , the overdrive, the switch?