diff --git a/docs/PLATFORMS.md b/docs/PLATFORMS.md index 71b1252f..fd5f2f3b 100644 --- a/docs/PLATFORMS.md +++ b/docs/PLATFORMS.md @@ -101,13 +101,23 @@ Outline: There is also a an verbatim example, you may skip the next section initially and can continue with the [hands-on example](PLATFORMS.md#sample-usecase-adding-platform-satellaview). -### Updating `peas.json` (or `peas_local.json`) and `platforms_idmap.csv` +### Updating `peas_local.json` and `platforms_idmap_local.csv` These two files are ment to be locally edited and extended for additional -platforms. Whenever you add a new platform block to the `peas.json` do also -lookup the corresponding platform ids and add them to `platforms_idmap.csv` for +platforms. Whenever you add a new platform block to the `peas_local.json` do also +lookup the corresponding platform ids and add them to `platforms_idmap_local.csv` for the scraping sites with an API. +From Skyscraper 3.15.0 onwards creating/editing the `peas_local.json` and +`platforms_idmap_local.csv` is the preferred way. In any case both files (e.g. +`peas.json` and `peas_local.json`) will be evalutated but the `_local` +configuration has precedence over the distributed `peas.json`. The same rule +applies to `platforms_idmap.csv` and `platforms_idmap_local.csv`. +If you have made local changes before Skyscraper 3.15 to either `peas.json` or +`platforms_idmap.csv` read the section [Transferring Local Platform +Changes](PLATFORMS.md#transferring-local-platform-changes) on how to transfer +your changes to the corresponding `*_local.*` files. + To find the platform ids for Screenscraper, Mobygames and The Games DB, please consult the files `screenscraper_platforms.json`, `mobygames_platforms.json` and `tgdb_platforms.json` which are located sibling to your `config.ini` of the @@ -126,7 +136,7 @@ list in the `peas.json` for the respective platform/system at ``. The platforms ScummVM or Steam do not have an exact match on Mobygames, however you may scrape successfully for ScummVM and Steam games if you use 'PC', 'DOS', 'Windows', 'Linux' or similar as `"aliases": ...` in the `"scummvm": ...` or `"steam": ...` section of `peas.json`. Usually you find the platform information if you lookup the game manually on the scraping website. -### Sample Usecase: Adding Platform _Satellaview_ +### Sample Usecase: Adding Platform _Satellaview_ Let the platform/systemname be `satellaview`. You may read about this SNES enhancing peripheral [here](https://en.wikipedia.org/wiki/Satellaview). @@ -187,6 +197,7 @@ at the end may be a less cumbersome manual merge with your local `platforms_idmap.csv`. Add this information: + ```csv satellaview,107,-1,-1 ``` @@ -234,7 +245,7 @@ additional information on this. 1. Scrape and generate the `satellaview/gamelist.xml` as in the [introductive use case](USECASE.md) using `Skyscraper -p - satellaview -s screenscraper` and `Skyscraper -p satellaview` +satellaview -s screenscraper` and `Skyscraper -p satellaview` 2. Restart EmulationStation, respective trigger reload of the gamelist in your frontend. 3. Smile :) @@ -243,10 +254,112 @@ additional information on this. Thanks to retrobit @ GitHub for contributing this usecase. +### Transferring Local Platform Changes + +This section describes how to transfer your changes from `peas.json` and +`platforms_idmap.csv` to `*_local.*` files with the same format. If you never +changed these files, you can safely ignore this section. +Whenever there is an update and maybe changes to `peas.json` and +`platforms_idmap.csv` Skyscraper will place the distribution files as +`peas.json.rp-dist` and `platforms_idmap.csv.rp-dist`. +Before Skyscraper 3.15.0 you had to manually transfer updates from +`peas.json.rp-dist` and `platforms_idmap.csv.rp-dist`. With Skyscraper 3.15.0 +onwards there is a semi-automated approach. + +!!! note Non-RetroPie Users + + If you are using Skyscraper without RetroPie + context these files will have the suffix `.dist`. + The manual will use `.rp-dist` as synonym for both. + + +#### Step 1: Transfer Platform Information (`peas`) to Local File + +Install Python Deepdiff: `sudo apt install python3-deepdiff`. Then navigate to +`/opt/retropie/supplementary/skyscraper/` and find the script +`deepdiff_peas_jsonfiles.py`. + +!!! tip Non-RetroPie Users + + If you are using Skyscraper without RetroPie + context the you can find the Python script in source in the + `supplementary/scraperdata` folder. + +The script expects at least two parameters: + +1. ``: The pristiine/baseline file with all platform + information +2. ``: The file with your local changes +3. Optionally ``: Once you have reviewed the changes provide + this file to store the platform "diff" between the pristine file and your + changes + +**Example(s)** + +Create a diff on the console: +```bash +python3 deepdiff_peas_jsonfiles.py peas.json.rp-dist peas.json +``` + +Review the diff, then run: +```bash +python3 deepdiff_peas_jsonfiles.py peas.json.rp-dist peas.json peas_local.json +``` + +Backup your `peas.json` if needed and when satisfied move +`peas.json.rp-dist` to `peas.json`. + +#### Step 2: Transfer Platform Scraper IDs (`platforms_idmap`) to Local File + +The logic is the same as before and provides an output of the lines you have +changed in `platforms_idmap.csv` in relation to the baseline +`platforms_idmap.csv.rp-dist`. +Navigate to the folder with the `platforms_idmap.csv`. Then run: + +```bash +diff \ + --new-line-format="%L" \ + --old-line-format="" \ + --unchanged-line-format="" \ + platforms_idmap.csv.rp-dist platforms_idmap.csv > platforms_idmap_local.csv +``` + +Also add the column header (folder, screenscraper_id, mobygames_id, tgdb_id) to +the `platforms_idmap_local.csv` file, for example with: + +```bash +echo folder,screenscraper_id,mobygames_id,tgdb_id | \ +cat - platforms_idmap_local.csv > tmp_piggy.csv && \ +mv tmp_piggy.csv platforms_idmap_local.csv +``` + +Backup your `platforms_idmap.csv` if needed and when satisfied move +`platforms_idmap.csv.rp-dist` to `platforms_idmap.csv`. + +#### Step 3: Automatically Apply Distribution Updates for Platform Configuration + +Navigate to the folder with the `_local.*` files mentioned before. Create an +empty file names `.platformcfg_overwrite_ok` (note the heading dot). If this +file is present Skyscraper will overwrite the existing `peas.json` and +`platforms_idmap.csv` with the files from the distribution when updating to a +newer version. +If you remove the file `.platformcfg_overwrite_ok`, Skyscraper will install the +`*.rp-dist` files again on the next install. + +### One More Thing... + +If you have changes which would be beneficial for the community, feel free to +file an issue with the proposed additions/changes or table it in the [RetroPie +Forum/Skyscraper Thread](https://retropie.org.uk/forum/topic/34588). Thank you! + ### Migrating `platforms.json` and `screenscraper.json` !!! info + This section is only applicable if you update from Skyscraper 3.7.7-2. + +!!! tip + If you neither edited `platforms.json` nor `screenscraper.json` or do not have these files in the Skyscraper config folder (sibling to the `config.ini`) you can safely ignore this section. @@ -259,26 +372,29 @@ the platform ids of two more web API sites. Use the script `convert_platforms_json.py` (sibling to the Skyscraper executable) to convert the `platforms.json` to a `peas_mine.json` file which you -can then diff to the `peas.json` and transfer your changes to `peas.json` or +can then diff to the `peas.json` and transfer your changes to `peas_local.json`. Use the script `check_screenscraper_json_to_idmap.py` to identify differences from your `screenscraper.json`. Then use the three `_platforms.json` files to identify the matching platform ids to be entered in -`platform_idmap.csv`. Use `-1` in this file, if there is no matching platform id. +`platforms_idmap_local.csv`. Use `-1` in this file, if there is no matching platform id. ### Summary of Changes in the Config Files Filenames shown ~~strikethrough~~ are superseded. Filenames shown _italic_ are user editable. -|
File
| Introduced with Version | Notes (version) | -| ----------------------------------- | ----------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------- | -| ~~`mobygames.json`~~ | 3.8.0 | superseded by `mobygames_platforms.json` (v3.9.0); not to be edited; IDs are used in `platforms_idmap.csv` | -| ~~`platforms.json`~~ | 3.7.7-2 (@detain) | superseded by _`peas.json`_ (3.9.0) and _`peas_local.json`_ (3.13.0); do edit the latter to add/change platforms; these files use a leaner format than `platforms.json` | -| _`platforms_idmap.csv`_ | 3.9.0 | maps the platform names (handles) from _`peas.json`_ / _`peas_local.json`_ to exact platform IDs used in scrapers MobyGames, Screenscraper or TGDB; do edit to add new platforms | -| ~~`screenscraper.json`~~ | 3.7.7-2 (@detain) | IDs formerly used in here are part of `platforms_idmap.csv` (3.9.0); superseded by `screenscraper_platforms.json` (3.9.0) which is not to be edited | -| `tgdb_developers.json` | 2.5.3 (@muldjord) | API mapping of 'Developers'; Uses leaner format as before (3.9.0); not to be edited | -| `tgdb_genres.json` | 3.9.0 | API mapping of 'Genres' (3.9.0); not to be edited | -| `tgdb_platforms.json` | 3.9.0 | API mapping of 'Platforms' (3.9.0); not to be edited; IDs are used in `platforms_idmap.csv` | -| `tgdb_publishers.json` | 2.5.3 (@muldjord) | API mapping of 'Publishers'; Uses leaner format as before (3.9.0); not to be edited | +|
File
| Introduced with Version | Notes (version) | +| ----------------------------------- | ----------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| ~~`mobygames.json`~~ | 3.8.0 | superseded by `mobygames_platforms.json` (v3.9.0); not to be edited; IDs are used in `platforms_idmap.csv` | +| ~~`platforms.json`~~ | 3.7.7-2 (@detain) | superseded by _`peas.json`_ (3.9.0) and _`peas_local.json`_ (3.13.0); do edit the latter to add/change platforms; these files use a leaner format than the initially used `platforms.json` | +| _`platforms_idmap.csv`_ | 3.9.0 | maps the platform names (handles) from _`peas.json`_ / _`peas_local.json`_ to exact platform IDs used in scrapers MobyGames, Screenscraper or TGDB; do edit to add new platforms | +| _`peas.json`_ | 3.9.0 | maps platform names (read: ROM folder names) to extensions and aliases for that platform | +| _`peas_local.json`_ | 3.13.0 | same as usage as `peas.json`, the `_local.json` file will not be altered by Skyscraper updates. Entries in this file have higher precedence than the distribution file `peas.json` | +| _`platforms_idmap_local.csv`_ | 3.15.0 | same as usage as `platforms_idmap.csv`, the `_local.csv` file will not be altered by Skyscraper updates. Entries in this file have higher precedence than the distribution file `platforms_idmap.csv` | +| ~~`screenscraper.json`~~ | 3.7.7-2 (@detain) | IDs formerly used in here are part of `platforms_idmap.csv` (3.9.0); superseded by `screenscraper_platforms.json` (3.9.0) which is not to be edited | +| `tgdb_developers.json` | 2.5.3 (@muldjord) | API mapping of 'Developers'; Uses leaner format as before (3.9.0); not to be edited | +| `tgdb_genres.json` | 3.9.0 | API mapping of 'Genres' (3.9.0); not to be edited | +| `tgdb_platforms.json` | 3.9.0 | API mapping of 'Platforms' (3.9.0); not to be edited; IDs are used in `platforms_idmap.csv` | +| `tgdb_publishers.json` | 2.5.3 (@muldjord) | API mapping of 'Publishers'; Uses leaner format as before (3.9.0); not to be edited | diff --git a/docs/stylesheets/mods.css b/docs/stylesheets/mods.css index e5b525a6..5aef3699 100644 --- a/docs/stylesheets/mods.css +++ b/docs/stylesheets/mods.css @@ -95,3 +95,7 @@ article a:hover { height: 50px; width: 50px; } + +.md-typeset h4 { + margin-bottom: -0.7em; +} \ No newline at end of file diff --git a/src/config.cpp b/src/config.cpp index 109119f0..918c7970 100644 --- a/src/config.cpp +++ b/src/config.cpp @@ -102,6 +102,13 @@ void Config::copyFile(const QString &src, const QString &dest, FileOp fileOp) { if (QFileInfo::exists(src)) { if (QFileInfo::exists(dest)) { if (fileOp == FileOp::OVERWRITE) { + if (QFileInfo(QFileInfo(dest).dir().path() % + "/.platformcfg_overwrite_ok") + .exists() && + (src.endsWith("peas.json") || + src.endsWith("platforms_idmap.csv"))) { + QFile::remove(dest % ".dist"); + } QFile::remove(dest); QFile::copy(src, dest); qDebug() << "Overwritten file" << dest; @@ -163,54 +170,43 @@ void Config::setupUserConfig() { } QMap> configFiles = { - {"ARTWORK.md", QPair("", FileOp::OVERWRITE)}, - {"artwork.xml.example1", QPair("", FileOp::OVERWRITE)}, - {"artwork.xml.example2", QPair("", FileOp::OVERWRITE)}, - {"artwork.xml.example3", QPair("", FileOp::OVERWRITE)}, - {"artwork.xml.example4", QPair("", FileOp::OVERWRITE)}, - {"cache/priorities.xml.example", - QPair("", FileOp::OVERWRITE)}, - {"config.ini.example", - QPair("config.ini.example", FileOp::OVERWRITE)}, - {"CACHE.md", - QPair("cache/README.md", FileOp::OVERWRITE)}, - {"hints.xml", QPair("", FileOp::OVERWRITE)}, - {"import/definitions.dat.example1", - QPair("", FileOp::OVERWRITE)}, - {"import/definitions.dat.example2", - QPair("", FileOp::OVERWRITE)}, - {"import/IMPORT.md", - QPair("import/README.md", FileOp::OVERWRITE)}, - {"mameMap.csv", QPair("", FileOp::OVERWRITE)}, - {"mobygames_platforms.json", - QPair("", FileOp::OVERWRITE)}, - {"README.md", QPair("", FileOp::OVERWRITE)}, - {"resources/boxfront.png", - QPair("", FileOp::OVERWRITE)}, - {"resources/boxside.png", - QPair("", FileOp::OVERWRITE)}, - {"screenscraper_platforms.json", - QPair("", FileOp::OVERWRITE)}, - {"tgdb_developers.json", QPair("", FileOp::OVERWRITE)}, - {"tgdb_genres.json", QPair("", FileOp::OVERWRITE)}, - {"tgdb_platforms.json", QPair("", FileOp::OVERWRITE)}, - {"tgdb_publishers.json", QPair("", FileOp::OVERWRITE)}, + // clang-format off + {"ARTWORK.md", QPair("", FileOp::OVERWRITE)}, + {"artwork.xml.example1", QPair("", FileOp::OVERWRITE)}, + {"artwork.xml.example2", QPair("", FileOp::OVERWRITE)}, + {"artwork.xml.example3", QPair("", FileOp::OVERWRITE)}, + {"artwork.xml.example4", QPair("", FileOp::OVERWRITE)}, + {"cache/priorities.xml.example", QPair("", FileOp::OVERWRITE)}, + {"config.ini.example", QPair("config.ini.example", FileOp::OVERWRITE)}, + {"CACHE.md", QPair("cache/README.md", FileOp::OVERWRITE)}, + {"hints.xml", QPair("", FileOp::OVERWRITE)}, + {"import/definitions.dat.example1", QPair("", FileOp::OVERWRITE)}, + {"import/definitions.dat.example2", QPair("", FileOp::OVERWRITE)}, + {"import/IMPORT.md", QPair("import/README.md", FileOp::OVERWRITE)}, + {"mameMap.csv", QPair("", FileOp::OVERWRITE)}, + {"mobygames_platforms.json", QPair("", FileOp::OVERWRITE)}, + {"README.md", QPair("", FileOp::OVERWRITE)}, + {"resources/boxfront.png", QPair("", FileOp::OVERWRITE)}, + {"resources/boxside.png", QPair("", FileOp::OVERWRITE)}, + {"screenscraper_platforms.json", QPair("", FileOp::OVERWRITE)}, + {"tgdb_developers.json", QPair("", FileOp::OVERWRITE)}, + {"tgdb_genres.json", QPair("", FileOp::OVERWRITE)}, + {"tgdb_platforms.json", QPair("", FileOp::OVERWRITE)}, + {"tgdb_publishers.json", QPair("", FileOp::OVERWRITE)}, // do not overwrite - {"config.ini.example", - QPair("config.ini", FileOp::KEEP)}, - {"import/definitions.dat.example2", - QPair("import/definitions.dat", FileOp::KEEP)}, - {"resources/frameexample.png", - QPair("", FileOp::KEEP)}, - {"resources/maskexample.png", QPair("", FileOp::KEEP)}, - {"resources/scanlines1.png", QPair("", FileOp::KEEP)}, - {"resources/scanlines2.png", QPair("", FileOp::KEEP)}, - // create .dist if exists - {"aliasMap.csv", QPair("", FileOp::CREATE_DIST)}, - {"artwork.xml", QPair("", FileOp::CREATE_DIST)}, - {"peas.json", QPair("", FileOp::CREATE_DIST)}, - {"platforms_idmap.csv", - QPair("", FileOp::CREATE_DIST)}}; + {"config.ini.example", QPair("config.ini", FileOp::KEEP)}, + {"import/definitions.dat.example2", QPair("import/definitions.dat", FileOp::KEEP)}, + {"resources/frameexample.png", QPair("", FileOp::KEEP)}, + {"resources/maskexample.png", QPair("", FileOp::KEEP)}, + {"resources/scanlines1.png", QPair("", FileOp::KEEP)}, + {"resources/scanlines2.png", QPair("", FileOp::KEEP)}, + // create .dist by default if exists + {"aliasMap.csv", QPair("", FileOp::CREATE_DIST)}, + {"artwork.xml", QPair("", FileOp::CREATE_DIST)}, + {"peas.json", QPair("", FileOp::CREATE_DIST)}, + {"platforms_idmap.csv", QPair("", FileOp::CREATE_DIST)} + // clang-format off + }; for (auto src : configFiles.keys()) { QString dest = configFiles.value(src).first; @@ -227,6 +223,9 @@ void Config::setupUserConfig() { } else if (src.startsWith("resources/")) { tgtDir = getSkyFolder(SkyFolderType::RESOURCE); dest = dest.replace("resources/", ""); + } else if (QFileInfo(tgtDir % "/.platformcfg_overwrite_ok").exists() && + (src == "peas.json" || src == "platforms_idmap.csv")) { + configFiles[src] = QPair("", FileOp::OVERWRITE); } QString tgt = tgtDir % "/" % dest; copyFile(localEtcPath % src, tgt, configFiles.value(src).second); diff --git a/supplementary/scraperdata/deepdiff_peas_jsonfiles.py b/supplementary/scraperdata/deepdiff_peas_jsonfiles.py new file mode 100644 index 00000000..ebed66d9 --- /dev/null +++ b/supplementary/scraperdata/deepdiff_peas_jsonfiles.py @@ -0,0 +1,113 @@ +#! /usr/bin/env python3 + +# Compares platform definition files of Skyscraper and prints differences as new +# JSON object: Fo example, it identifies changes of your local platforms +# peas.json in comparison to the maintainer's peas.json.rp-dist. +# +# See docs/PLATFORMS.md for details. +# +# Your local changes will be printed and can also be stored in a local platform +# peas_local.json, this will not be altered by any upstream changes to +# peas.json. +# +# Once you have saved your local changes move the peas.json.rp-dist to +# peas.json. +# +# For the platform_idmap.csv changes to the maintainer's file run: + +# You will need Deepdiff to run this script: sudo apt install python3-deepdiff + +# (c) 2025 Gemba @ GitHub +# SPDX-License-Identifier: GPL-3.0-or-later + +from deepdiff import DeepDiff +from pathlib import Path +import json +import sys + + +def load_peas_json(fn): + if not fn.exists(): + print(f"[-] File {fn} not found. Please fix.") + sys.exit(1) + with open(fn, encoding="utf8") as fh: + d = json.load(fh) + + for k, v in d.items(): + d[k].pop("scrapers", None) + return d + + +def check_outfile(): + fn_localonly_peas = None + if len(sys.argv) == 4: + fn_localonly_peas = Path(sys.argv[3]) + if fn_localonly_peas.exists(): + print( + f"[!] File {fn_localonly_peas} already exists and will not be overwritten." + ) + fn_localonly_peas = None + return fn_localonly_peas + + +if __name__ == "__main__": + if len(sys.argv) < 3 or len(sys.argv) > 4: + print( + f"[*] Usage: {sys.argv[0]} []" + ) + print( + f" e.g: {sys.argv[0]} peas.json.rp-dist peas.json peas_local.json\n" + " identifies changes to your local peas.json compared to peas.json.rp-dist\n" + " and saves them in peas_local.json" + ) + sys.exit(0) + + fn_upstream_peas = Path(sys.argv[1]) + d1 = load_peas_json(fn_upstream_peas) + + fn_changed_peas = Path(sys.argv[2]) + d2 = load_peas_json(fn_changed_peas) + + diff = DeepDiff(d1, d2, ignore_order=True, report_repetition=True) + + dout = {} + for what, diffpath in diff.items(): + platform = None + if what == "dictionary_item_added": + platform = diffpath[0].split("'")[1] + elif what == "iterable_item_added": + platform = next(iter(diffpath)).split("'")[1] + + if platform: + dout[platform] = d2[platform] + + platforms = dict(sorted(dout.items())) + + # sort subkeys + for platform, sub_dict in platforms.items(): + for k, v in sub_dict.items(): + if type(v) == list: + sub_dict[k] = sorted(v) + platforms[platform] = dict(sorted(sub_dict.items())) + + fn_localonly_peas = check_outfile() + + if not platforms: + print(f"[*] No additions to '{fn_upstream_peas}' detected. Quitting.") + sys.exit(0) + + if fn_localonly_peas: + with open(fn_localonly_peas, "w", encoding="utf8") as fh: + json.dump(platforms, fh, indent=4, ensure_ascii=False) + print(f"[*] Additions saved to '{fn_localonly_peas}'") + else: + print(f"[*] Detected additions to baseline file '{fn_upstream_peas}':") + print( + json.dumps(platforms, indent=4, ensure_ascii=False).encode("utf8").decode() + ) + if len(sys.argv) == 3: + print( + f"[*] Re-run and provide a filename as third parameter to persist these." + ) + else: + check_outfile()