Improved workshop download reliability #2
Conversation
There was a problem hiding this comment.
Sorry it took so long for me to look at this, must have gotten lost in the email.
I like the idea behind both params but think it needs a bit more work / cleanup. If your able to do it I can take another look or at some point I'll be able to do the work. Thanks for the PR!
| @@ -14,6 +14,7 @@ RUN apt-get update \ | |||
| python3 \ | |||
| rename \ | |||
| wget \ | |||
| git \ | |||
There was a problem hiding this comment.
It's a remnant for the solution we adopted.
It can be removed, I simply forgot to remove it.
Basically I taught missions makers how to upload missions to a git repo. When the server starts it pulls the repo in the mpmissions folder. This way if the server is off (and in my community case will be most of the time) they can still upload it somewhere without having to turn it on and ftp the file to it.
| @@ -75,12 +77,29 @@ def log(msg: str): | |||
| print(msg) | |||
| print("{{0:=<{}}}".format(len(msg)).format("")) | |||
|
|
|||
| def count_sub_directories(rootPath: str): | |||
|
|
||
| def call_steamcmd(params: str): | ||
| debug('steamcmd {}'.format(params)) | ||
| os.system("{} {}".format(STEAM_CMD, params)) | ||
| print("") | ||
|
|
||
| def call_steamcmd_script(file: str): |
There was a problem hiding this comment.
Not really seeing the benefit of using this over the call_steamcmd fn.
There was a problem hiding this comment.
I have lost the forum thread I read that gave me the idea.
What I remember reading was that when the mod download times out, it's actually continuing in the background.
When you give steam_cmd a new command, it stops the workshop_item_download and it goes on to the next one.
So I thought, maybe if I go down the steamcmd script solution, that won't happen?
I don't have any statistical data regarding what's more reliable between the 2 approaches, it was a blind attempt to fix the problem. For all I know, both might work the same.
| path = "{}/{}".format(A3_STEAM_WORKSHOP_DIR, mod_id) | ||
| WORKSHOP_MODS[mod_name] = mod_id | ||
|
|
||
| if os.path.isdir(path) and mod_last_updated: |
There was a problem hiding this comment.
You've removed the directory datetime check and made it so every mod is redownloaded every startup, that's no bueno. This actually may be the more valid place to download the mod with the new DOWNLOAD_ATTEMPTS so each mod gets the same download attempt individually and we can copy the mod keys to the right folder after it's downloaded.
The current implementation bundles all the workshop mod downloads. I believe that's because steam would have to keep signing in and would cause timeouts for large number of mods trying to download too fast, but that could be fixed with the retries too? maybe...
There was a problem hiding this comment.
The reason why it was removed I believe was because after the folder is created and the download times out, it would skip the partially downloaded mod since it was "up to date". Steamcmd is an unreliable pile of crap, to be extra sure I simply removed the check, forcing a validation on every mod every time.
It is impossible (at least with larger modsets) to do individual attempts on the mods because as you said, there's a very strict login timeout on Steam's end. The only way it can be done is to do one or two steamcmd calls and hope to do everything in a single batch.
I would like to know though, why do you consider it a big no no?
| for _, match in enumerate(matches, start=1): | ||
| mod_id = match.group(1) | ||
| check_workshop_mod(mod_id) | ||
| download_updated_workshop_mods_script() |
There was a problem hiding this comment.
Because mods are still bundled (and it's changed to all mods in this pr), if any download fails it will retry downloading every mod again from the start.
There was a problem hiding this comment.
That doesn't happen, it validates the currently downloaded mods. I've used it for a couple of months in my arma 3 community and we don't have to redownload everything every time.
|
Alright, thanks for the input, I'll do some cleanup. Just let me know your stance on "directory check time" removal or anything you don't approve after the explanations I gave. |
I've slightly modified the a3update.py to be more reliable. The problem is that steamcmd times out when downloading big mods.
So the main changes are that instead of launching steamcmd with the "+argument" I made it so that it launches using a .txt file. The txt file is generated by a3update.py it self and then when the workshop download has to be called, it launches steam with the specified script file. From what I've read online this system makes it more reliable.
2 more environment variables have been added: