Having participated in my first reenactment today, and firing multiple times, I can confirm there is absolutely no such thing as your rifle swinging around when you first aim. Even with my arms extremely tired, I could hold it as stable as a rock on the ground. My arms shook slightly, however this made no effect on the rifle itself.
Personally I think the mechanic should be replaced by a different one: your gun drooping if aiming it for too long, and there being a regeneration on this when not aiming.