[Feature]: Support more storage backend

[gromain]

Feature detail

Right now, only a local folder is supported natively. Even though more can be supported through fuse mounted filesystems, but this method has drawbacks (performance hit mainly), so it would be nice to have support for more back-ends eventually.

S3 is an obvious choice that comes to mind and that would support a lot of providers, but there could be others (like WebDAV perhaps).

Platform

Server

[Nonobis]

Minio would be great !

[gromain]

Minio is compatible with S3!

[Nonobis]

@gromain : i know :)

[palitu]

Would this be local storage for thumbnails, and remote storage for files? That would maintain speed for scrolling.

Then you would need to pass a Uri to the app to retrieve the full file for viewing?

[akashkj]

If it allows to have a backup also for the media, that would be awesome.
Like 2 storage backends - s3 and G-drive. If one of them is not reachable, switch to other. This could be defined in a priority list.

[gromain]

In my mind it could have been for all objects, but @palitu you're right that thumbnails could still be local.
And yes, @akashkj, that's how I felt about it. Having two or more storage back-ends possible, that can either work as RAID0 or RAID1 (not exactly RAID, but either mirror or concatenate the available storage).

[tarasis]

I would definitely love to see this (I guess I could use a symbolic link for the upload folder). I have a Hetzner Storage Box with 5TB of space, and would love to use it to store the photos, and leave the thumbnails local to the host running immich. (free Oracle Cloud instance).

That way I have some additionally safety as they take snapshots of the data.

[zackpollard]

In my mind, supporting a well defined platform such as S3-like APIs is a good idea and is something we can look into after the first major release is completed. As for the other idea of having multiple backends that we switch between, in my mind that is the job of the storage layer, not something that immich should natively support. We'll leave this here and probably get around to it at some point in the future!

[palitu]

Really great to hear your desire for an S3 type backend. FYI, i use the local filesystem for most of immich, then a mounted NFS share for the main photo storage. My server doesnt have the local storage for the bulk data (especially with all the video's), but thumbs is more than appropriate for local.

I do think that there are some delays with video playing as a result of this, but i assume in the future it will be improved with this S3 feature.

ta

[Cellivar]

I just started writing this up as a new feature request before discovering this one. I would love to be able to use S3 storage for all blob objects and make a raw disk volume optional. Big +1 to this feature request!

At home I run a Nomad cluster, like a Kubernetes cluster but with less yaml involved. We use CSI-managed iSCSI storage volumes that are mounted directly into running docker images (or other tasks). The preference here is a single writer job and maybe multiple read-only mounts for other volumes. This means that sharing a directory is hard while communicating over the network is easy. Right now the docker-compose files seem to suggest mounting one volume to multiple hosts that all have write access which is a little scary (and hard to do with iSCSI). The performance hit of switching to NFS is very visible on our network.

There's an official TS SDK for all the bells and whistles MinIO provides, but I'd strongly suggest looking at just using the S3 SDK which would allow folks to use any S3-compatible storage backend.

This is of course asking a lot, adding a new storage backend like S3 is completely different than writing to disk 😄 I'm perfectly happy to be patient for this sort of feature.

[athornfam2]

Would like to add that I am interested in S3 storage access as well to offload.

[Zen3515]

I try mounting S3 with rclone and found that immich uses softlink which is not allowed in S3.

I guess this is one of the things to consider when implement this in the future.

Error: ENOSYS: function not implemented, link 'upload/04f8f839-9ee4-47fc-89b1-fb3883cf3008/original/WEB/33809e6c-1f1f-4bec-a8cc-7a0367493e8d.png' -> 'upload/04f8f839-9ee4-47fc-89b1-fb3883cf3008/2022/2022-12-11/hahaha.png'

[nperez0111]

I ran into this same issue while trying to use a webdav mount via davfs2

[NiklasRosenstein]

I've run into the same issue, sadly. If it didn't try to use soft links, I think Immich could already run pretty well using S3 fuse mounted storage!

[jcollie]

Had similar issues trying to run Immich in Azure/Azure Kubernetes. Azure does not really have a good ReadWriteMany solution for sharing volumes between pods since it's using CIFS under the hood.

[carballude]

Same thing using SMB mounts:
ERROR [StorageTemplateService] Error: ENOTSUP: operation not supported on socket, link '<<source>>' -> '<<destination>>'

[NiklasRosenstein]

It works well with the yandex csi-s3 driver using geesefs.

[nicolopozzato]

Hi, i setup immich on a arm vps on oracle, with ocamel fuse google drive, but I have the same error:

immich_server | [Nest] 1 - 03/01/2023, 12:47:57 AM ERROR [StorageService] Error: ENOSYS: function not implemented, link 'upload/85ff3fb7/original/WEB/408963f5-45f7-40e8-960c-79f36116e086.jpeg' -> 'upload/85ff3fb7/2023/2023-02-25/originalName.jpeg'

So in the folder 2023-02-25 the image is not present but it's only in the original/WEB/....

[cycneuramus]

If I'm reading the source code correctly, the reason Immich currently doesn't support rclone mounted backends is simply because:

• its way of moving files is by relying on the mv library, which:
  • works by calling fs.link (at least on clobber: false, which Immich is currently using), which:
    • creates a hard link of the source file at the destination path, at which:
      • mv will unlink the source file, thus having "moved" the file from source to destination.

Now, the problem is, rclone doesn't support hard links. Hence the move operation fails halfway with the ENOSYS: function not implemented, link ... error.

I don't have a thorough understanding of the way Immich is supposed to work here, but if the point is to move the files and nothing more, it seems to me we could achieve full rclone compatibility—and with it support for countless storage backends—simply by reworking the move command (say, by using fs.rename(sourcePath, desPath); or something).

I've created an issue about this here.

EDIT: PR here

EDIT2: PR is now merged, so all rclone supported remotes should work with Immich probably in the next release.

[kajdo]

great!!!

[pinpox]

Is rclone the way to go for using immich with s3 currently? Are there plans to support s3 natively?

[Koveras-hmm]

webdav plz!

[cycneuramus]

Already supported, at least indirectly, via rclone.

[alkmim]

FYI: I did some tests using rclone and I reported my findings (without a lot of data though) here: #3814.

In a nutshell, it did not work reliably enough for me.

Maybe we should not consider this feature supported (indirectly via rclone)

[david-polak]

I just installed Immich on k8s and connected it to an s3 compatible storage, all hosted on Vultr. It's bit of a kludge setup and took me four hours to debug.

Immich pods - mounted nfs dir -> NFS server pod - mounted s3 dir using s3fs -> s3

The results, well it works. Kinda, if parallelism is reduced to 1 it almost handles uploads of regular photos. There are missed thumbnails here and there and corresponding filesystem errors. Videos are mostly broken, haven't had time to see what exactly is the issue with those apart from ffmpeg errors.

Needs more investigation, these are my initial impressions so far.

[NiklasRosenstein]

I'm running things on a CIFS mount as well and it's pretty smooth sailing. Haven't reached the level of brave required to nest an S3 mount inside it, though.

[floonym]

I am currently using rclone to mount google drive with seemingly no errors using vfs-cache-mode write. I only mounted the library and upload folder of the docker containers from the rclone folder, having the thumbnails mounted from the server locally.

rclone mount drive: /mnt/drive --vfs-cache-mode writes -vv

[StanfordMATT]

Can you share a Docker Compose? Interested in using Google Drive as a persistence layer

[floonym]

I'm using ansible instead of docker-compose, but its not really anything special. Anywhere where normally the "upload" folder is mounted into the container, I instead mounted the individual folders inside.

For the microservices container, it looks like this (again not docker-compose syntax)

    mounts:
      - source: "{{ immich.upload_path }}"
        target: /usr/src/app/upload/upload
        type: bind
      - source: "{{ immich.library_path }}"
        target: /usr/src/app/upload/library
        type: bind
      - source: "{{ immich.thumbs_path }}"
        target: /usr/src/app/upload/thumbs
        type: bind
      - source: "{{ immich.media_path }}"
        target: /mnt/media
        type: bind
        read_only: true

To make sure that rclone mounts before the containers mount, this is the systems service that was set up:

[Unit]
Description=mount rclone drive
After=network.target
# run before docker
Before=docker.service

[Service]
Type=simple
Restart=on-failure
ExecStart=/usr/bin/rclone mount google-drive: /mnt/google-drive --vfs-cache-mode writes --vfs-write-back 5m --vfs-cache-min-free-space 5G -vv
TimeoutStartSec=20s

[Install]
WantedBy=multi-user.target

In my example, the folder /mnt/google-drive with the folders you want to save on the rclone mount must be created first. I now only save the "library" folder on the rclone mount, and have the rest on the server itself. be aware, the "thumbs" folder will grow quite a bit, as it has all the images you normally see when using immich, the "library" folder only stores the original images.

[stv0g]

RClone provides a Docker Volume Plug-in which can make the integration a bit nicer: https://rclone.org/docker/

But still this is just a workaround in my opinion. If supported by the storage backend, up and downloads should target the storage backend directly rather than beeing proxied by Immich or some Fuse file system as it will never scale as good as a direct access.

[janbuchar]

I was thinking about contributing s3 support to immich. I found that

1. the uploads are stored directly to the library by multer - I could probably use multer-s3 to take care of this
2. exiftool needs a local copy of the file, but it shouldn't be a problem to download a temporary copy just for metadata extraction
3. most other filesystem-related functionality is handled by IStorageRepository, it should be enough to make an alternative implementation of that
4. files are served using sendFile from express, and it may be tricky to adjust serveFile to support direct streaming from s3

Am I missing some other roadblocks? Would this level of support for s3 even be welcome?

[stv0g]

Does this mean if I share a library, I'm sharing the s3 api keys?

No, S3 has a feature called Presigned URLs, which can be used to generate short lived URLs to S3 objects without exposing the S3 credentials. That would even allow for direct client to S3 uploads.

[pinpox]

No, S3 has a feature called Presigned URLs, which can be used to generate short lived URLs to S3 objects without exposing the S3 credentials. That would even allow for direct client to S3 uploads.

This is how most products using S3 do it, e.g. ente photos works like this.

[PatrickHuetter]

Presigned URLs is the way to go.

[uhthomas]

Presigned URLs have quite a few caveats, especially at home:

• Clients may have a much harder time caching content.
• The S3 host may not be publicly accessible.
• URLs are not friendly, and if visited directly in a browser then it won't be possible to refresh when the URL expires.
• More?...

Redirecting to a different location is not always desirable and often actually harmful. The server has much less control over the how the content is served and it can get quite messy. It also means the behaviour of how assets are served will differ hugely between local assets vs s3 vs any other storage medium, which adds complexity. IMO I would just avoid presigned URLs entirely.

I personally also quite liked the idea, until I implemented it and used it. I then removed it entirely because it has a lot of drawbacks.

[PatrickHuetter]

Clients may have a much harder time caching content.

Could you explain why? I don't get this point since in my opinion it much better and easier to cache. Even you could cluster your pictures and have direct I/O via different IP routes over the internet to different storage servers. Also in home scenarios if someone decides to use s3 as backend he could decide wether to route traffic to a different device (e.g. NAS with s3 service while Immich is running on a different homeserver).

The S3 host may not be publicly accessible.

If this is the case, your immich instance wouldn't be accessible too? Otherwise you have reverse tunnels like tailscale, cloudflared etc.? If this is the case you could just don't use the s3 backend or am i wrong? Since it's just a decision which storage layer you choose or isn't it?

URLs are not friendly, and if visited directly in a browser then it won't be possible to refresh when the URL expires.

What do you mean with "urls are not friendly"? And what's bad about not beeing able to refresh after an presigned url expires? It just the way presigned URLs work. Knowing it from my own software development and also a bunch of other open source projects where this works absolutely fine.

I can only speak for our spring boot environments where s3 integration with presigned urls works perfect without any issues. We will never look back until we find something better for storing blobs/files :)

[systemmonkey42]

+! for this to support structured storage for original media while keeping thumbnails and cache data on a local filesystem.

This would also benefit from some method to take a database dump and upload it to the same structured storage for future DR scenarios.

Currently I use snapshots for the database, but that not going to help in the event of a fire, even though all the original media is currently backed up to an S3 bucket.

[sorbanbela]

rClone support is great, but I think that native S3 support would be really nice. Serving thumbnails and images with presigned URLs would improve overall performance and scalability.

[mrvanes]

I just deployed a test instance of Immich with an S3 backed storage via https://github.com/s3ql/s3ql
Seems to fare well at first sight. I know it's not native S3, but at least I now have an elastic storage backend that will only cost what I use.

[dubrowin]

I worked this week to bring up an Immich server backed by S3. I'm using an AWS built and supported tool. Documentation on my setup is here: https://github.com/dubrowin/Immich-backed-by-S3/

[Cohinem]

I currently mount WebDAV with rclone. It works fine most of the time. but there are some problems. native support would surely be great.

[dubrowin]

In general, I found the costs of using S3 as the back-end too expensive. It was much cheaper to expand the EBS volume I'm using to a larger volume size (in GP3).

[dillfrescott]

Storj is not just $4 a TB but also $7 a TB of ingress

[archerallstars]

@dillfrescott Did you mean egress (download)? Since ingress (upload) has always been free on Storj. Moreover, every S3 storage costs you for egress, one way or another. For example, if you store 1 TB on Backblaze, you'll have free egress until 3 TB, and another $10 a month (total of $16) for 4 TB egress.

[dillfrescott]

Sorry yeah, i couldnt think of the right word!

[leonidasv]

Be aware that EBS GP3 has 99.8%-99.9% durability, while S3 has 99.999999999%.

[mnpenner]

Backblaze has an alliance with Cloudflare, so you can get free egress if you put B2 behind a Cloudflare domain.

[surapuramakhil]

@gromain is this done / activity working on this?

[gromain]

I'm not the developer, just a user requesting a feature.

And I don't have the time to work on this unfortunately.

[justin13888]

Just adding my two cents. I've been playing around building an image/video (assets in general really) management software similar in nature to Immich but currently, specifically for temporary sharing. It purely uses S3 for storing both original copies and generated assets (e.g. thumbnails, transcoded versions).

I'll spend more time working on it again later but one thing I'd like to point out is the architecture optimized for S3 versus some filesystem-level or block-level storage looks quite different. From a developer's point of view, for example, uploading images for S3 over others could look different although sure you could just implement a way that is the least common denominator.

I'd suggest before any actual implementation work, we should gauge what type of hardware and scale people are actually using Immich for. Personally I love many things that its done for me so far but there's a lot of "cool" or high impact features will still require significant rewrites. (I'm also not a fan of nestjs backend but I digress.)

[luanjot]

I would be happy to help on this effort as I was looking for this kind of solution. Has this been discussed / decided by the community that native S3 support is something they want / don't want to do?

I cannot see any kind of actual discussion on here on whether the feature is needed / wanted or not.

[CrushedAsian255]

Thinking of how this could actually be implemented, should only the originals be uploaded to S3 or should the thumbnails also be stored on S3?

[pinpox]

Best would be both, as one of the use-cases for this is to be able to host the application on a low-cost server which would have little storage itself

[luanjot]

The use case I have in mind is of a person that wants to have minimal infrastructure for their photos optimising the cost. In this scenario, the need to have the thumbnails locally would be counterproductive as you would require either a large disk (not cost effective) or a disk you can expand often (not minimal infrastructure). I guess what I am asking is whether this case should be supported or not. I think that there are lots of people that would be happy with this if set up efficiently because it removes the need to pay for a fixed amount of storage and the operations with thumbnails would be very few. But maybe doing something like this moves people away from self hosting and this is not the point of the project. El domingo, 17 de noviembre de 2024, CrushedAsian255 < ***@***.***> escribió:

…

Thinking of how this could actually be implemented, should only the originals be uploaded to S3 or should the thumbnails also be stored on S3? — Reply to this email directly, view it on GitHub <#1683 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAOIJVSB6CEH4MDGPOCFFH32BCONZAVCNFSM6AAAAAAVLNS2VWVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTCMRYGM2DMMA> . You are receiving this because you commented.Message ID: ***@***.***>

[zatricky]

The definition of "self hosting" is not so narrow. Hosting on a Raspberry Pi at home, hosting on a VPS with a cloud provider, and hosting on a physical rented server in a datacentre all qualify as self-hosted.

Additionally, block storage systems are often available for on-prem systems like Unraid or TrueNAS, typically using Minio, which is S3-compatible.

[sgofferj]

One can self-host S3-compatible storage, e.g. Minio. In fact, object storage has many advantages over block storage because many things can be handled transparently by the storage system, such as versioning, object life-cycle management, redundancy, ...
I'm self-hosting Minio and some of my use cases include:

• Storage of surveillance video with life-cycle management (automatic moving of objects from SSD to HDD after a while and automatic expiry/deletion after another while)
• Storage backend for Nextcloud (object versioning handled by Minio vs. Nextcloud native)
• Automatic replication/backups

[AngelaDMerkel]

@sgofferj This is somewhat off topic, but how can you use Minio as a storage backend for Nextcloud? Secondly, wouldn't it be somewhat redundant if using a filesystem with versioning?

[sgofferj]

https://docs.nextcloud.com/server/latest/admin_manual/configuration_files/primary_storage.html

[sgofferj]

As far as I understand, when using S3 as backend for NC, the versioning is done through S3 natively.