Hmm I think I get what you're saying. I don't think the caliper mount would move as well, that is constrained to the swingarm essentially.
All I am certain of is that the wheel bearing install process isn't symmetrical, and relies upon one bearing being seated fully in the wheel, and the other not being fully seated into the wheel by design. I know for a fact if you installed the wheel bearings in the wrong order or incorrect seating procedure, the whole wheel (along with brake rotor, and sprocket) would be offset along the axle. The wheel axle spacers and caliper mount are pressed up against the swingarm inside on both sides, and they will always be seated there, even if you did the bearing install wrong.
All to say, the more I think about the way these parts are assembled, the more likely I think it is that you have incorrectly installed wheel bearings or a reversed mounted brake rotor.
If you installed the bearings wrong in the rear wheel, you would reassemble the whole wheel and axle and spacers and everything, and you wouldn't be able to notice that you did the bearing seating incorrectly until you got to the caliper install, like you have. The offset error is probably small enough that you can't obviously see the wheel shifted right or left, but enough that it could cause brake caliper clearance issues.
I'd recommend just tearing it all down and replacing the bearings. If it doesn't solve the problem, then at least you got fresh bearings, and there is a bigger problem at play here. If you want nice bearings, OEM, if you're cheap, then AllBalls wheel bearings aren't bad and are available. I'm using them and they seem good.