The problem is that the video itself is sourced through Content Distribution Networks, which means you're getting video from a server installed in a city close to you, and in many cases from a server that's physically installed in one of your ISP's data centers.
In a lot of ways when you say "I watched the game on Vue" it's like saying "I buy Wonder bread at Wal-mart." Let's say you try to go to a Chicago Wal-mart every Sunday afternoon and by the time you get there they have completely sold out of Wonder bread. It doesn't help if I go to a Minneapolis store and then say "that's strange, I just went there and had no problem buying it." We both went to "Wal-mart" at the same time, but every local store is going to be slightly different in stock and experience.
It's the same thing with CDNs. Your experience is going to vary based on a number of factors, starting with the CDN provider. SlingTV primarily uses Fastly, DirecTV NOW uses Akamai, PS Vue uses Level(3) with some Akamai, Amazon uses their Cloudfront CDN, Netflix uses their own "open connect" CDN.
Even when you break down a single CDN, it's still complicated. For example, if I do packet captures of DirecTV NOW at home I can see that I connect to an Akamai server directly connected to Comcast's network in Roseville, MN. My folks live just a couple miles away and if I connect using their Centurylink ISP I connect to an Akamai server on CL's network in Minneapolis. If I connect over my ATT LTE hotspot, I land on yet a different appliance somewhere close by (based on latency).
Each of those appliances can sustain 2000-3000 simultaneous streams (depending on specs), and once they start to hit that limit, the only path to add capacity is someone needs to negotiate with each individual ISP for space/power/cooling, server hardware needs to be ordered, shipped, installed, commissioned, etc. This is a long lead-time process, and why some folks are going to continue to have bad performance for long stretches of time.
