For me the game started getting laggy after Christmas (before the patch). My guess is there many more people now (and more with poor connections). I assume most people that could afford the xbox one on day one probably also has a decent connection.... Hopefully it is just the patch though.
Also for rage quitters, even if they figure out a way to take the points away, they will still drop... which to me is very annoying if you get several drops in a row. They should take twice the points away if you quit early, that might change a few things.