So it seems this sort of thing is kinda normal for most LCD TVs. I sent my panel to Hisense to check, and they reported no issues. The funny thing is that since I noticed it on my TV I started to check for it on other displays as well, and I am noticing it everywhere now, it's just much easier to spot on a TV this size. Compared to others I still think my TV is worse than some but it doesn't bother me anymore in most content, and my wife doesn't care at all.
A word of advice to new buyers, don't look at test patterns on your TV because if you see that the uniformity of your panel isn't 100% (and no TV is), then you will start looking for the faults in regular content. Ignorance is bliss.