So now to answer where the latency comes from.
(1) Distance contributes about 3 µsec per km (Free space) and 4 µsec per km through optic fibre. So while distance is the dominant factor internationally, distance is not the factor locally all that much.
(2) Properly configured hardware is a few µsec per device, so the higher the number of routers etc in the chain the more added latency one gets. So again, mostly this is not the dominant factor either.
(3) Next is the enscapsulation techniques used has quite a large impact on latency. So those of you on full GE system are really at the forefront here. Even with GE, if not implemented correctly, will show up as increased latency.
(4) Next is compression and encryption.
(5) Routers add latency if links and nodes are congested. Hence the quick way out when users complain. If all else fails blame exchange congestion.
(6) Closer to home, one needs to look at PCs and their operating systems as well. Those with incorrect window sizes will also show up as increased latency, affecting the bandwidth-delay product, ideally the window size needs to match the bandwidth-delay product.
(7) the "contention" present a network loading is probably the biggest factor in our environment. Hence, at the moment FTTH is working very well as those networks should be effectively lightly loaded, but ahs more and more clients come on board, and the ISPs do not increase capacity, we can expect performance to decrease and latency to increase.
(8) Routing has a major effect, especially under fault conditions, as well as how traffic is restored after faults are fixed . (example is the Blizzard issue) at the moment.
(9) Servers contribute quite a bit to latency, proportional to the demand for connections.
Any more anyone wants to add?