Testing Google Drive performance
Preface
Recently I've been getting into storing my backups offsite since I now have the bandwidth to do so. I chose Google Drive as I could get unlimited storage via their GSuite offering, which I use for personal mail. For this reason, it should be expected that I require as much bandwidth as possible to complete these backups in a timely manner.
Initial testing
I'll be conducting tests using a linux based server directly connected to the router via cable, the gdrive-cli client (here:
https://github.com/prasmussen/gdrive) and my 200Mbit Cool Ideas line.
I started by first creating a 1Gb sparse file which I would use to benchmark the speeds, this will eliminate factors such as file system speed and disk fragmentation.
View attachment 491247
I then ran the client using the pure stock configuration, and logged the results via my firewall (to compare to the results reported via gdrive-cli)
View attachment 491249
Hmmm, not quite as I expected. I fired up Wireshark and looked for the server I was uploading to and found this: 216.58.223.42
View attachment 491251
Well then, the server is local which rules out any TCP window problems. I spun up DSTAT for an overview of the system's performance during the test just to see that I was not maxing out any other piece of hardware and thats when I noticed something interesting. My box wasn't sending data continuously but rather send out chunks at a time, and the gdrive-cli client simply gave me an average speed over a period of time. The speed it reported was not real time.
View attachment 491253
Troubleshooting
Odd, but not uncommon. I've seen the same with a service called BackBlaze which I had also used in the past. The service sends data in chunks, waiting for the server to verify the integrity of the chunk before requesting the rest of the file. I decided to look into the gdrive-cli manpages and found that I was able to set the chunk size myself, keeping in mind this would mean the chunk be loaded into memory first which meant my system's memory usage would increase (slightly, on modern systems you probably wont be able to tell the difference)
I went for it, doubling the chunk size from it's default 8388608 to 16777216 bytes.
View attachment 491255
Pretty neat huh?
Comparison testing
Running with 8388608byte chunks (x1):
- Average speed over entire upload: 4.7MB/s
- Average memory usage change was negligible.
Running with 16777216byte chunks (x2):
- Average speed over entire upload: 7.3MB/s
- Average memory usage change was 0.01GB
Running with Running with 33554432byte chunks (x4):
- Average speed over entire upload: 9.9MB/s
- Average memory usage change was 0.01GB
Looks like we're hitting diminishing returns. With a x4 increase in chunk size I managed to more than double the speed of uploads. More optimisation is possible, but beyond the scope of this post.