I might need confirmation on this but my assumption is that the instant a person watches a video on youtube it is loaded into a local RSA server cache "somewhere" so that the next person who views it can download it much more easily. So if your the first person viewing it, it might take a while.
I notice that Philip deFranco vids load up faster because he has a bigger fan base here.
My other assumption is that certain countries are given priority over others. For instance watching a vid from USA might be faster because most South Africans watch content there. Whereas if it was from Japan or China it might be slower because there is less priority given to those countries based on viewership.
But this might just apply to iBurst because of the 1mbps(average user) speed limit.