I have a feeling it is FB. Try turning its contact sync off?
The FB app has to be one of the crappest apps in the store. Friendcaster is better.
Shot for the tip, I just did another factory reset. I uninstalled each app individually and nothing changed, but I will start with the essential apps and see how it goes.