You definitely shouldn't be seeing any latency on your 1st hop... How many devices do you have on your WiFi connection at the moment?
2 phones and this pc.
Pinging default gateway (other side of my AP). C:\Users\Nicki>ping 192.168.10.1
Pinging 192.168.10.1 with 32 bytes of data:
Reply from 192.168.10.1: bytes=32 time=3ms TTL=64
Reply from 192.168.10.1: bytes=32 time=3ms TTL=64
Reply from 192.168.10.1: bytes=32 time=5ms TTL=64
Reply from 192.168.10.1: bytes=32 time=6ms TTL=64
Ping statistics for 192.168.10.1:
Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
Minimum = 3ms, Maximum = 6ms, Average = 4ms
C:\Users\Nicki>ping 192.168.10.1
Pinging 192.168.10.1 with 32 bytes of data:
Reply from 192.168.10.1: bytes=32 time=4ms TTL=64
Reply from 192.168.10.1: bytes=32 time=6ms TTL=64
Reply from 192.168.10.1: bytes=32 time=2ms TTL=64
Reply from 192.168.10.1: bytes=32 time=2ms TTL=64
Ping statistics for 192.168.10.1:
Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
Minimum = 2ms, Maximum = 6ms, Average = 3ms
C:\Users\Nicki>