The connector is quite stiff to get in but also prone to wiggle itself loose over time if you have some pressure on the cable from the side of the headset.
Make sure its seated correctly and make sure the cable has enough play when guiding it to the side and you should be fine.
@Jarhead, havent come across that suggestion yet. Will try.