You are right, without enough power the pellet does bounce straight back 20 feet and hit me.
Dave, you need to think outside of the box, literally. Glue the rubber mud flaps to all sides of the outside of those traps with clear silicone construction adhesive. The two densities of rubber will deaden the sound and last forever. A couple tubes of clear silicone from Home Depot should make a good 1/4" thick layer on the outside of them, with the mud