Open .bsps inside archives with a single path, using load_bsp

[snake-biscuits]

Building on #21, I want free up my harddrives by only extracting maps when I load them
Rather than add even more complex infrastructure to tests/megatest.py, I'd rather just use paths into archives

e.g. D:/SteamLibrary/steamapps/common/Quake/Id1/PAK0.PAK/maps/e1m1.bsp

Challenges

• identifying which part of the filepath string is the archive
• identifying if the archive contains another archive
  • D:/Emulators/Sega/Dreamcast/quakeiii.cdi/quake.16.iso/QUAKE3/BASEQ3/MAPS/DC_MAP01/BSP.ZIP/maps/dc_map01.bsp
.cdi -> .iso -> .zip -> .bsp
• getting associated files inside an archive (os.listdir will be useless)
• opening associated files (RespawnBsp .ent & .bsp_lump)
  • some maps are split across multiple .rpak, extra complexity yay

Related

• Create bsp class from bytes / stream #21

[snake-biscuits]

The Dreamcast release of Quake III is going to be a pretty essential test case

• D:/Emulators/Sega/Dreamcast/quakeiii.cdi
• quake.16.iso
• QUAKE3/BASEQ3/MAPS/DC_MAP01/BSP.ZIP
  • AAS.ZIP/botfiles/botfiles.zip/botfiles/game.h (QuakeC paired with the map?)
• maps/dc_map01.bsp

[snake-biscuits]

We'll also want to roll extensions.archives into the core bsp_tool
And to read from archives like Vpk & RPak we'll need other libraries for decompressing Oodle & LZHAM
This means bsp_tool will no longer be builtins only

However, this means we can roll lightmaps into core as well
And we can use Pillow for MipTextures (I think, can we do texture palletes?)

[snake-biscuits]

archives should move away from the zipfile.ZipFile __init__ form
.from_stream etc. would be better

since we will be opening archives inside other archives in memory

source.valve.PakFile could make a good base for this

[snake-biscuits]

.from_stream etc. would be better

archives.respawn.Vpk already kind of has a ._from_stream method, but it's not a @classmethod
a proper @classmethod would be much nicer

[snake-biscuits]

Here's what our deepest load could end up looking like:

from bsp_tool.archives.padus import Cdi
cdi = Cdi.from_file("D:/Emulators/Sega/Dreamcast/quakeiii.cdi")
from bsp_tool.archives.iso import Iso
iso = Iso.from_bytes(cdi.read("1.0.iso"))
from bsp_tool.branches.valve.source import PakFile  # or archives.id_software.Pk3
zip_ = PakFile.from_bytes(iso.read("QUAKE3/BASEQ3/MAPS/DC_MAP01/BSP.ZIP"))
from bsp_tool.id_software import IdTechBsp
from bsp_tool.branches.id_software import quake3
bsp = IdTechBsp.from_bytes(
    quake3,
    "D:/Emulators/Sega/Dreamcast/quakeiii.cdi/1.0.iso/QUAKE3/BASEQ3/MAPS/DC_MAP01/BSP.ZIP/maps/dc_map01.bsp",
    zip_.read("maps/dc_map01.bsp"))

As a one-liner:

from bsp_tool.archives.iso import Iso
from bsp_tool.archives.padus import Cdi
from bsp_tool.branches.id_software import quake3
from bsp_tool.branches.valve.source import PakFile  # or archives.id_software.Pk3
from bsp_tool.id_software import IdTechBsp

bsp = IdTechBsp.from_bytes(
    quake3,
    "D:/Emulators/Sega/Dreamcast/quakeiii.cdi/1.0.iso/QUAKE3/BASEQ3/MAPS/DC_MAP01/BSP.ZIP/maps/dc_map01.bsp",
    PakFile.from_bytes(
        Iso.from_bytes(
            Cdi.from_file("D:/Emulators/Sega/Dreamcast/quakeiii.cdi").read(
                "1.0.iso")).read(
                    "QUAKE3/BASEQ3/MAPS/DC_MAP01/BSP.ZIP")).read(
                        "maps/dc_map01.bsp"))

Yes, a properly indented one-liner ends up being longer

bsp = IdTechBsp.from_bytes(quake3, "D:/Emulators/Sega/Dreamcast/quakeiii.cdi/1.0.iso/QUAKE3/BASEQ3/MAPS/DC_MAP01/BSP.ZIP/maps/dc_map01.bsp", PakFile.from_bytes(Iso.from_bytes(Cdi.from_file("D:/Emulators/Sega/Dreamcast/quakeiii.cdi").read("1.0.iso")).read("QUAKE3/BASEQ3/MAPS/DC_MAP01/BSP.ZIP")).read("maps/dc_map01.bsp"))

bsp_tool.load_bsp("D:/Emulators/Sega/Dreamcast/quakeiii.cdi/1.0.iso/QUAKE3/BASEQ3/MAPS/DC_MAP01/BSP.ZIP/maps/dc_map01.bsp")

[snake-biscuits]

I'm currently focused on quakeiii.cdi
Looking though my Dreamcast folder, I found a few other .cdi:

• Half Life (US).cdi
• PARANOIADC1.1.CDi (Half-Life: Paranoia Dreamcast Port)
• Quake (US).cdi

NOTE: Other Dreamcast games w/ .bsps exist (see #117)

All 3 contain .pak pakfiles
For #21 the proof of concept that showed we could open .bsp inside archives was Quake's id1/PAK0.PAK
Kinda funny that my other .cdi all contain .pak

[snake-biscuits]

archives.sega.GDRom maps directly to the GD-ROM Data Area files
This means we can go skip indexing the .iso inside the .cdi when we want to load from a full path
Making our "one-liner" much simpler

from bsp_tool.archives import sega
gdrom = sega.GDRom.from_file("D:/Emulators/Sega/Dreamcast/quakeiii.cdi")
from bsp_tool.branches.valve.source import PakFile
zip_ = PakFile.from_bytes(gdrom.read("QUAKE3/BASEQ3/MAPS/DC_MAP01/BSP.ZIP"))
...

Should be nice for other images too:

• .cue + .bin & .raw
• .gdi + .bin & .raw

IDK if any full .iso GD-ROM images are out there
You could make one, in theory (maybe with ISOBuster)
All the empty sectors would make it pretty wasteful though

[snake-biscuits]

The new GDRom interface is quite nice
@classmethod initialisers, .tree(), .listdir() & other filesystem utilities
It'd be nice to add things like .is_file(path: str) -> bool to all archives

Working out a new standard for archives will probably be part of #191

Also, valve.source.PakFile might work better living in archives
Seeing as it already has .from_bytes, and filesystem utils could be handy additions

Just have to decide on where to keep it
archives.cdrom already breaks the naming convention of developer.FileFormat
The .zip format has a developer: PKWARE Inc.¹²
Not exactly common knowledge, but pkware.Zip fits archives & is clearly distinct from zipfile.ZipFile

Footnotes

1. https://en.wikipedia.org/wiki/ZIP_(file_format) ↩

2. https://en.wikipedia.org/wiki/PKWare ↩

[snake-biscuits]

Looking though my Dreamcast folder, I found a few other .cdi:

All of the .cdi files in my collection have 1 Mode2 Track in their 2nd session
In the code I was assuming the GD Area would always be there, worked out nicely
Other .cdi GD-ROMs might not fit this assumption tho, so it's good to keep note of it being an assumption

Not going to seek out other .cdi GD-ROMs, I've got all we need for bsp_tool rn

[snake-biscuits]

WE HAVE IGNITION

>>> from bsp_tool.archives import sega
>>> gdrom = sega.GDRom.from_file("D:/Emulators/Sega/Dreamcast/quakeiii.cdi")
>>> from bsp_tool.branches.valve.source import PakFile
>>> zip_ = PakFile.from_bytes(gdrom.read("QUAKE3/BASEQ3/MAPS/DC_MAP01/BSP.ZIP"))
>>> from bsp_tool.id_software import IdTechBsp
>>> from bsp_tool.branches.id_software import quake3
>>> bsp = IdTechBsp.from_bytes(
...     quake3,
...     "D:/Emulators/Sega/Dreamcast/quakeiii.cdi/QUAKE3/BASEQ3/MAPS/DC_MAP01/BSP.ZIP/maps/dc_map01.bsp",
...     zip_.read("maps/dc_map01.bsp"))
<IdTechBsp 'dc_map01.bsp' id_software.quake3 (IBSP version 46)>
>>> _.VERTICES
<BspLump(19827 Vertex) at 0x0000021809B10750>

[snake-biscuits]

what should be possible with .from_archive:

IdTechBsp.from_archive(quake3, "maps/dc_map01.bsp",
    pkware.Zip.from_archive("QUAKE3/BASEQ3/MAPS/DC_MAP01/BSP.ZIP", 
        sega.GDRom.from_file("D:/Emulators/Sega/Dreamcast/quakeiii.cdi")))