Depends if the background is a colour or an image.
^ This
Image vs No Image yes. If they are both images, then there will be no difference if the whole image is black vs the whole image white (unless compression is used (png or jpeg) and the colours are not pure black or pure white).
So OP... if a solid colour is the only differentiator, then no, absolutely no difference (Agree with ghoti, you are being a bit silly). There are a ton of other things to look at, so as was mentioned in an earlier post, include a link to the page if you want the real reason for the slow down.