redarrow
Expert Member
Heya,
Thought I would share this in case someone is looking for a similar solution. Basically, I was looking for a means of downloading Steam games during my ISP's zero-rated data hours (Vox Telecom), to save on data. During steam sales I sometimes end up with erm, a rather lot of games which can consume a rather lot of data. HumbleBundles also contribute to this.
I had two problems though, firstly the Steam client offers no built in scheduling and secondly I would really rather not do this on my main PC as it's kinda near where I sleep and flickering lights/fan noises are not conducive to sleep.
Ideally I wanted to run it on my headless Linux server box (basically my glorified NAS device).
Amazingly I discovered Valve actually has a CLI Steam client, it's just not very well advertised. It basically does what I needed: Download Steam games. Best part is that it can download games for any OS. I.e., it can download a Windows game even though it's running on Linux.
Here's the official Wiki page for it: https://developer.valvesoftware.com/wiki/SteamCMD
The page contains all you need to download and get steamcmd running.
The steamcmd client itself is pretty dumb though and doesn't have any scheduling or anything like that so you need to set something up yourself. My solution was cron and a python script. I've shared my script below, it's pretty simple to use.
As I only use Steam on Windows and Linux I only wrote it to handle Windows/Linux games but it wouldn't be hard to add Mac in if someone wants that.
To start the downloading process:
To abort:
You just pretty much need to use cron to start the process at the start of zero rated time and then again to kill it when it ends.
When run it scans the Linux and/or Windows directory (at least one must be specified) looking for subdirectories with digit only names, it assumes those to be Steam AppID's and pretty much passes them onto steamcmd for download (games will be downloaded into those dirs). It does some simple sorting to ensure partially downloaded games get first priority over others, thereafter it sorts by the creation date of the subdirectories (oldest first). Once it determines a game is completely downloaded the directory is renamed to the actual game name (pulled out of the "appmanifest" file).
Passing the steam password is not strictly necessary (in case you're paranoid) as once you've setup steamcmd it usually remembers it, I add it in though cos I've had it randomly lose it once or twice so less hassle this way.
To actually install the downloaded games in Steam you need to copy the game directory to your steam library's "common" dir. On Windows that defaults to "Program Files (x86)\Steam\SteamApps\common". You also need to move the "appmanifest" file which will be in "<Gamedir>\steamapps" (named "appmanifest_<STEAMAPPID>.acf") to the base Steam library dir (so "Program Files (x86)\Steam\SteamApps"). After this restart Steam and it will find the game. Once Steam has restarted I always get it to verify the game files as I have had one or two cases where steamcmd for some reason missed a few files (was just a few MB each time).
Basically now that I have this setup on my NAS box, anytime I want a Steam game downloaded I just create a new directory with the Steam AppID under my main Windows or Linux directories depending on which OS I want it for. In the last week or so I've downloaded over 80GB's of Steam games with this setup.
My python script (needs psutil module):
Thought I would share this in case someone is looking for a similar solution. Basically, I was looking for a means of downloading Steam games during my ISP's zero-rated data hours (Vox Telecom), to save on data. During steam sales I sometimes end up with erm, a rather lot of games which can consume a rather lot of data. HumbleBundles also contribute to this.
I had two problems though, firstly the Steam client offers no built in scheduling and secondly I would really rather not do this on my main PC as it's kinda near where I sleep and flickering lights/fan noises are not conducive to sleep.
Ideally I wanted to run it on my headless Linux server box (basically my glorified NAS device).
Amazingly I discovered Valve actually has a CLI Steam client, it's just not very well advertised. It basically does what I needed: Download Steam games. Best part is that it can download games for any OS. I.e., it can download a Windows game even though it's running on Linux.
Here's the official Wiki page for it: https://developer.valvesoftware.com/wiki/SteamCMD
The page contains all you need to download and get steamcmd running.
The steamcmd client itself is pretty dumb though and doesn't have any scheduling or anything like that so you need to set something up yourself. My solution was cron and a python script. I've shared my script below, it's pretty simple to use.
As I only use Steam on Windows and Linux I only wrote it to handle Windows/Linux games but it wouldn't be hard to add Mac in if someone wants that.
To start the downloading process:
Code:
steamdown.py -l <linux dir> -w <windows dir> -u <steam username> -p <steam password> -c <full path to steamcmd.sh>
Code:
steamdown.py -xWhen run it scans the Linux and/or Windows directory (at least one must be specified) looking for subdirectories with digit only names, it assumes those to be Steam AppID's and pretty much passes them onto steamcmd for download (games will be downloaded into those dirs). It does some simple sorting to ensure partially downloaded games get first priority over others, thereafter it sorts by the creation date of the subdirectories (oldest first). Once it determines a game is completely downloaded the directory is renamed to the actual game name (pulled out of the "appmanifest" file).
Passing the steam password is not strictly necessary (in case you're paranoid) as once you've setup steamcmd it usually remembers it, I add it in though cos I've had it randomly lose it once or twice so less hassle this way.
To actually install the downloaded games in Steam you need to copy the game directory to your steam library's "common" dir. On Windows that defaults to "Program Files (x86)\Steam\SteamApps\common". You also need to move the "appmanifest" file which will be in "<Gamedir>\steamapps" (named "appmanifest_<STEAMAPPID>.acf") to the base Steam library dir (so "Program Files (x86)\Steam\SteamApps"). After this restart Steam and it will find the game. Once Steam has restarted I always get it to verify the game files as I have had one or two cases where steamcmd for some reason missed a few files (was just a few MB each time).
Basically now that I have this setup on my NAS box, anytime I want a Steam game downloaded I just create a new directory with the Steam AppID under my main Windows or Linux directories depending on which OS I want it for. In the last week or so I've downloaded over 80GB's of Steam games with this setup.
My python script (needs psutil module):
Code:
#!/usr/bin/python
import os, sys, getopt, operator, shlex, psutil
scriptName = os.path.basename(__file__)
# Defaults
linuxDir = ""
windowsDir = ""
steamUsername = ""
steamPassword = ""
steamCmd = "steamcmd.sh"
# Kill any instances of this script and steamcmd
def killScript():
killScriptCnt = 0
killSteamCnt = 0
# First kill any instances of the script (except this one)
# so they cannot start another steamcmd when that is killed
for proc in psutil.process_iter():
if proc.name == scriptName and proc.pid != os.getpid():
killScriptCnt += 1
proc.kill()
# Now kill any instances of steamcmd
for proc in psutil.process_iter():
if proc.name == "steamcmd":
killSteamCnt += 1
proc.kill()
if killScriptCnt > 0 or killSteamCnt > 0:
print "Steam downloader script terminated."
return
# Print help message
def printHelp():
print "Usage: "+ scriptName +" <options>"
print " -h Display this help"
print " -l <dir> Base directory for Linux games"
print " -w <dir> Base directory for Windows games"
print " -u <username> Steam username"
print " -p <password> Steam password"
print " -c <cmd> Full path to steamcmd.sh"
print " -x Kill any running instances of this script and steamcmd"
return
# Options
try:
opts, args = getopt.getopt(sys.argv[1:], "hl:w:u:p:c:x")
except getopt.GetoptError:
printHelp()
sys.exit(2)
for opt, arg in opts:
if opt == "-h":
printHelp()
sys.exit()
elif opt == "-x":
killScript()
sys.exit()
elif opt == "-l":
linuxDir = arg
elif opt == "-w":
windowsDir = arg
elif opt == "-u":
steamUsername = arg
elif opt == "-p":
steamPassword = arg
elif opt == "-c":
steamCmd = arg
# Need a steam username
if steamUsername == "":
print "Please supply a steam username."
sys.exit(2)
# Confirm steamcmd exists
if not os.path.isfile(steamCmd):
print "Please specifiy a valid path to the steamcmd.sh executable."
sys.exit(2)
# Was a linux dir specified?
if linuxDir <> "":
linuxGames = True
else:
linuxGames = False
# Was a windows dir specified?
if windowsDir <> "":
windowsGames = True
else:
windowsGames = False
# Must have at least one directory
if not linuxGames and not windowsGames:
print "Please specify at least a linux or windows games directory"
sys.exit(2)
# Make sure directories are valid
if (linuxGames):
if not os.path.isdir(linuxDir):
print "Invalid linux games directory specified!"
sys.exit(2)
if (windowsGames):
if not os.path.isdir(windowsDir):
print "Invalid windows games directory specified!"
sys.exit(2)
# List to contain games to be downloaded
gamesList = []
for i in range(0, 2):
if i == 0:
if linuxGames: cpath = linuxDir
else: continue
if i == 1:
if windowsGames: cpath = windowsDir
else: continue
for file in os.listdir(cpath):
# Ignore files
if not os.path.isdir(os.path.join(cpath, file)): continue
# Ignore directories with names that contain anything other than numbers
if not str.isdigit(file): continue
# Check if dir is empty, non empty dirs are assumed to have partial downloads
# already and thus are prioritised
if not os.listdir(os.path.join(cpath, file)): empty = 1
else: empty = 0
# Dir age, older directories are prioritised over newer (after non empty)
age = os.path.getmtime(os.path.join(cpath, file))
gamesList.append([i, file, empty, age])
# Sort the list
gamesList.sort(key=operator.itemgetter(2,3))
for game in gamesList:
if game[0] == 0:
gOs = "linux"
curOsDir = linuxDir
gpath = os.path.join(linuxDir, game[1])
else:
gOs = "windows"
curOsDir = windowsDir
gpath = os.path.join(windowsDir, game[1])
#print "%s +@sSteamCmdForcePlatformType %s +login %s %s +force_install_dir "%s" +app_update %s +quit" % (steamCmd, gOs, steamUsername, steamPassword, gpath, game[1])
print "Passing download information for %s game id# %s to steamcmd." % (gOs, game[1])
if os.system(steamCmd +" +@sSteamCmdForcePlatformType "+ gOs +" +login "+ steamUsername +" "+ steamPassword + " +force_install_dir "+ gpath +" +app_update "+ game[1] +" +quit") == 0:
sadir = os.path.join(gpath, "steamapps")
ddir = os.path.join(sadir, "downloading")
appManifest = os.path.join(sadir, "appmanifest_"+ game[1] +".acf")
# Download dir must exist and be empty
if not os.path.isdir(ddir): continue
if os.listdir(ddir): continue
if os.path.isfile(appManifest):
bytesToDownload = 0
bytesDownloaded = 0
installDir = ""
mf = open(appManifest, 'r')
for line in mf:
sl = shlex.split(line)
if sl[0].lower() == "bytestodownload": bytesToDownload = int(sl[1])
if sl[0].lower() == "bytesdownloaded": bytesDownloaded = int(sl[1])
if sl[0].lower() == "installdir": installDir = sl[1]
# If bytesToDownload and bytesDownloaded are equal and greater than zero we can assume
# the download is complete and successful - we can rename the dir if we also have a valid name
if bytesToDownload == bytesDownloaded and bytesToDownload > 0 and installDir <> "":
print "Download appears complete and successful, renaming installation directory.."
os.rename(gpath, os.path.join(curOsDir, installDir))
