gdal_merge fails when using vsizip//vsicurl raster files as input

[Lawrence-Revere]

What is the bug or the crash?

gdal_merge fails when input layer source is HTTPS raster file accessed by vsicurl.

Receive processing error: RuntimeError:`vsicurl/https://exampleurl.com/raster.zip' does not exist in the file system, and is not recognized as a supported dataset name.

Steps to reproduce the issue

1. Go to Data Source Manager
2. Got to Raster
3. Set Source Type to Protocol: HTTP(S)... and set Protocol to HTTP/HTTPS/FTP
4. Enter Source URI e.g. https://exampleurl.com/raster.zip and Add layer
5. Repeat for next Raster Layer
6. Go to gdal_merge from Processing Toolbox
7. Select raster layers from previous steps as Input Layers
8. Do not change default Parameters
9. Run and observe error in Log Messages

Versions

QGIS version
3.38.3-Grenoble
QGIS code revision
37f9e6e
Qt version
5.15.13
Python version
3.12.6
GDAL/OGR version
3.9.2
PROJ version
9.4.0
EPSG Registry database version
v11.004 (2024-02-24)
GEOS version
3.12.2-CAPI-1.18.2
SQLite version
3.45.1
PDAL version
2.6.3
PostgreSQL client version
16.2
SpatiaLite version
5.1.0
QWT version
6.2.0
QScintilla2 version
2.14.1
OS version
Windows 11 Version 2009

Active Python plugins
processing_saga_nextgen
1.0.0
qfieldsync
v4.9.1
db_manager
0.1.20
grassprovider
2.12.99
MetaSearch
0.3.6
processing
2.12.99

Supported QGIS version

• I'm running a supported QGIS version according to the roadmap.

New profile

• I tried with a new QGIS profile

Additional context

Same command is successful with same inputs in v3.36.3.

[NyakudyaA]

Can you share the gdal command line generated, Can you also run these commands outside of QGIS context i.e shell command

[Lawrence-Revere]

Have not yet run from shell cmd, will attempt next. Data source anonymised

Python command:
processing.run("gdal:merge", {'INPUT':['/vsizip/vsicurl/https://tccspatialbulkassets.blob.core.windows.net/lidar/2019/DEM/Tiles/451000_7882000.zip/451000_7882000_DTM.asc','/vsizip/vsicurl/https://tccspatialbulkassets.blob.core.windows.net/lidar/2019/DEM/Tiles/451000_7883000.zip/451000_7883000_DTM.asc'],'PCT':False,'SEPARATE':False,'NODATA_INPUT':None,'NODATA_OUTPUT':None,'OPTIONS':'','EXTRA':'','DATA_TYPE':5,'OUTPUT':'TEMPORARY_OUTPUT'})
qgis_process command:
qgis_process run gdal:merge --distance_units=meters --area_units=m2 --ellipsoid=EPSG:7019 --INPUT='/vsizip/vsicurl/https://tccspatialbulkassets.blob.core.windows.net/lidar/2019/DEM/Tiles/451000_7882000.zip/451000_7882000_DTM.asc' --INPUT='/vsizip/vsicurl/https://tccspatialbulkassets.blob.core.windows.net/lidar/2019/DEM/Tiles/451000_7883000.zip/451000_7883000_DTM.asc' --PCT=false --SEPARATE=false --OPTIONS= --EXTRA= --DATA_TYPE=5 --OUTPUT=TEMPORARY_OUTPUT

GDAL / OGR console call / command:
gdal_merge.bat -ot Float32 -of GTiff -o C:/Users/user/AppData/Local/Temp/processing_NRPaxT/c94e2933aafb49cd84b679c38b1ad750/OUTPUT.tif --optfile C:/Users/user/AppData/Local/Temp/processing_NRPaxT/938151ca2c54463fb136927dc4fe0b10/mergeInputFiles.txt

[NyakudyaA]

I have tested this and I confirm there is a bug .
To produce the bug do the following

• Using raster menu, add a raster data using i.e /vsizip//vsicurl/http://localhost:9000/cog/S29E027.zip
• Navigate to the processing menu and try to run gdal_merge

QGIS version: 3.38.3-Grenoble
QGIS code revision: 37f9e6efeec
Qt version: 5.15.2
Python version: 3.9.5
GDAL version: 3.3.2
GEOS version: 3.9.1-CAPI-1.14.2
PROJ version: Rel. 8.1.1, September 1st, 2021
PDAL version: 2.3.0 (git-version: Release)
Algorithm started at: 2024-10-10T21:28:32
Algorithm 'Merge' starting…
Input parameters:
{ 'DATA_TYPE' : 1, 'EXTRA' : '', 'INPUT' : ['/vsizip//vsicurl/http://localhost:9000/cog/S29E027.zip','/vsizip//vsicurl/http://localhost:9000/cog/S29E028.zip'], 'NODATA_INPUT' : None, 'NODATA_OUTPUT' : None, 'OPTIONS' : '', 'OUTPUT' : '/private/tmp/deas.tif', 'PCT' : False, 'SEPARATE' : False }

GDAL command:
gdal_merge.py -ot Int16 -of GTiff -o /private/tmp/deas.tif --optfile /private/var/folders/c8/bdpg_b590qz9pfw7jqvs4t740000gn/T/processing_kuVVUc/05eeb5cacdc0457ba793f9bd47e8c490/mergeInputFiles.txt
GDAL command output:
ERROR 4: `/vsicurl/http://localhost:9000/cog/S29E027.zip' not recognized as a supported file format.
ERROR 4: `/vsicurl/http://localhost:9000/cog/S29E028.zip' not recognized as a supported file format.
Traceback (most recent call last):
File "/Applications/QGIS.app/Contents/MacOS/bin/gdal_merge.py", line 4, in <module>
__import__('pkg_resources').run_script('GDAL==3.3.2', 'gdal_merge.py')
File "/Applications/QGIS.app/Contents/MacOS/lib/python3.9/site-packages/pkg_resources/__init__.py", line 651, in run_script
self.require(requires)[0].run_script(script_name, ns)
File "/Applications/QGIS.app/Contents/MacOS/lib/python3.9/site-packages/pkg_resources/__init__.py", line 1448, in run_script
exec(code, namespace, namespace)
File "/Applications/QGIS.app/Contents/Resources/python/site-packages/GDAL-3.3.2-py3.9-macosx-10.13.0-x86_64.egg/EGG-INFO/scripts/gdal_merge.py", line 11, in <module>
sys.exit(main(sys.argv))
File "/Applications/QGIS.app/Contents/MacOS/lib/python3.9/site-packages/GDAL-3.3.2-py3.9-macosx-10.13.0-x86_64.egg/osgeo_utils/gdal_merge.py", line 428, in main
ulx = file_infos[0].ulx
IndexError: list index out of range
Process returned error code 1
Execution completed in 1.74 seconds
Results:
  OUTPUT: /private/tmp/deas.tif

Loading resulting layers
The following layers were not correctly generated.
• /private/tmp/deas.tif
You can check the 'Log Messages Panel' in QGIS main window to find more information about the execution of the algorithm.

• Notice that the output files generated contain the following
  

[NyakudyaA]

I tested it with Master and it seems to work. https://github.com/qgis/QGIS/blob/master/python/plugins/processing/algs/gdal/GdalUtils.py#L465-L468 fixes the issue

[Lawrence-Revere]

Tested with changes to Gdalutils.py, still receiving same error.

Command output:

QGIS version: 3.38.3-Grenoble
QGIS code revision: 37f9e6efee
Qt version: 5.15.13
Python version: 3.12.6
GDAL version: 3.9.2
GEOS version: 3.12.2-CAPI-1.18.2
PROJ version: Rel. 9.4.0, March 1st, 2024
PDAL version: 2.6.3 (git-version: b5523a)
Algorithm started at: 2024-10-14T08:19:59
Algorithm 'Merge' starting…
Input parameters:
{ 'DATA_TYPE' : 5, 'EXTRA' : '', 'INPUT' : ['/vsizip//vsicurl/https://tccspatialbulkassets.blob.core.windows.net/lidar/2019/DEM/Tiles/472000_7866000.zip/472000_7866000_DTM.asc','/vsizip//vsicurl/https://tccspatialbulkassets.blob.core.windows.net/lidar/2019/DEM/Tiles/472000_7867000.zip/472000_7867000_DTM.asc'], 'NODATA_INPUT' : None, 'NODATA_OUTPUT' : None, 'OPTIONS' : '', 'OUTPUT' : 'TEMPORARY_OUTPUT', 'PCT' : False, 'SEPARATE' : False }

GDAL command:
gdal_merge.bat -ot Float32 -of GTiff -o C:/Users/user/AppData/Local/Temp/processing_TfHJRF/488c5ae6c73f493786c43e9b9fd0deba/OUTPUT.tif --optfile C:/Users/user/AppData/Local/Temp/processing_TfHJRF/48433f0c30b6450885e91f945dd05996/mergeInputFiles.txt
GDAL command output:
Traceback (most recent call last): 
File "C:\PROGRA~1\QGIS33~1.3\apps\Python312\Scripts\gdal_merge.py", line 12, in <module>
sys.exit(main(sys.argv)) 
^^^^^^^^^^^^^^ 
File "C:\PROGRA~1\QGIS33~1.3\apps\Python312\Lib\site-packages\osgeo_utils\gdal_merge.py", line 657, in main 
return gdal_merge(argv) 
^^^^^^^^^^^^^^^^ 
File "C:\PROGRA~1\QGIS33~1.3\apps\Python312\Lib\site-packages\osgeo_utils\gdal_merge.py", line 397, in gdal_merge
return _gdal_merge(argv=argv) 
^^^^^^^^^^^^^^^^^^^^^^ 
File "C:\PROGRA~1\QGIS33~1.3\apps\Python312\Lib\site-packages\osgeo_utils\gdal_merge.py", line 530, in _gdal_merge
file_infos = names_to_fileinfos(names) 
^^^^^^^^^^^^^^^^^^^^^^^^^ 
File "C:\PROGRA~1\QGIS33~1.3\apps\Python312\Lib\site-packages\osgeo_utils\gdal_merge.py", line 213, in names_to_fileinfos
if fi.init_from_name(name) == 1: 
^^^^^^^^^^^^^^^^^^^^^^^ 
File "C:\PROGRA~1\QGIS33~1.3\apps\Python312\Lib\site-packages\osgeo_utils\gdal_merge.py", line 247, in init_from_name
fh = gdal.Open(filename) 
^^^^^^^^^^^^^^^^^^^ 
File "C:\PROGRA~1\QGIS33~1.3\apps\Python312\Lib\site-packages\osgeo\gdal.py", line 8817, in Open
return _gdal.Open(*args) 
^^^^^^^^^^^^^^^^^ 
RuntimeError: `/vsicurl/https://tccspatialbulkassets.blob.core.windows.net/lidar/2019/DEM/Tiles/472000_7866000.zip' not recognized as being in a supported file format.
Process returned error code 1

[NyakudyaA]

I am not sure how Windows is calling the processing scripts, I would grab the latest QGIS from the builds or just grab a more recent build of QGIS from a PR and test i.e. #59061