Hatsan Hatsan Jet POI shift.

I have a 22 cal jet 2 set up in the carbine configuration with a scope. I currently polished the trigger and barrel, lightened the springs on the mags, polished the hammer and removed the 2 hs preload washers. It now gets 3 mags with great accuracy out to 35 yards on a fill. According to the pressure gauge I could shoot 4 mags but poi starts dropping a little. I also put a .25 cal buckrail moderator on it. It shoots way better with the moderator than without. Interestingly of all the pellets I tested the hatsan vortex 17grain shoot the best. My problem is with how the barrel is fastened in the polymer housing, if it gets bumped even a little my poi shifts drastically. I tried to fix this by drilling small divots where the set screws contact the barrel but it didn’t seem to help. Did anyone else with a jet have this problem? Any suggestions to fix it?