Replace localstack with minio

[javiermtorres]

What's changing

The localstack service is replaced with minio for local deployments.

Closes #487

How to test it

Start lumigator (e.g. using make local-up) and check that minio starts instead of localstack.

All automated tests should work as usual.

NOTE: .env file will need to be recreated based on the .env.example (or see .env.example for examples of environment variables that must be configured). This is because some defaults have changed.

Additional notes for reviewers

N/A

I already...

• Tested the changes in a working environment to ensure they work as expected
• Added some tests for any new functionality
• Updated the documentation (both comments in code and product documentation under /docs)
• Checked if a (backend) DB migration step was required and included it if required
  • No DB update is needed.

[javiermtorres]

One issue I found is that, if the .env.example is modified, then the integration tests should also be run, just in case the env vars are no longer valid. I can make a follow up PR to amend that.

Also, I didn't notice a clear indication that the .env would be generated from the .env.example and used within the docker-compose. If such indication is indeed present, could anyone please point me to it? @agpituk maybe? Thanks in advance.

[aittalam]

Also, I didn't notice a clear indication that the .env would be generated from the .env.example and used within the docker-compose. If such indication is indeed present, could anyone please point me to it? @agpituk maybe? Thanks in advance.

Not sure if it is in the documentation, but you can find it in the Makefile:

.env:
    @if [ ! -f .env ]; then cp .env.example .env; echo ".env created from .env.example"; fi

[aittalam]

Thanks a lot for the PR @javiermtorres, I think this is great! I tested this and it runs for me (I had a few issues at first, mostly due to old env vars being present and the .env/.env.example thing going on, but afterwards things just ran fine). Accessing all persisted data via Web UI is very cool too :-)

Can I ask you to please go through our sources and look for 4566 and localstack too? I think there are few other spots we might need to update contextually to this change such as:

• .env
• documentation (README.md, installation.md, dev.md, etc)

re: conftest.py, I see we are still using localstack. Do you see the need to move tests to minio too? IMHO it is not something urgent, but testing on the same stack we are running makes a bit more sense (and reduces the amount of dependencies + lumigator's fingerprint)

[javiermtorres]

@aittalam thanks for trying out the PR and checking that it works :) I'll address the concerns first thing on Monday.
Re: the dependence on localstack for tests.... It's being addressed in #453 , but unfortunately there's one single test case that is inexplicably failing; I cannot even get meaningful logs from Ray. I know that it somehow has to do with inference, but I can't see how. I'll try rebasing to main, but it should be up to date. If something there rings a bell, please please let me know :D

Once that other PR is in place, we'll remove any dependency on localstack in source and tests.

[aittalam]

@aittalam thanks for trying out the PR and checking that it works :) I'll address the concerns first thing on Monday. Re: the dependence on localstack for tests.... It's being addressed in #453 , but unfortunately there's one single test case that is inexplicably failing; I cannot even get meaningful logs from Ray. I know that it somehow has to do with inference, but I can't see how. I'll try rebasing to main, but it should be up to date. If something there rings a bell, please please let me know :D

Once that other PR is in place, we'll remove any dependency on localstack in source and tests.

That's great @javiermtorres , many thanks!
I'll check #453 today and see if I can be of any help there.

[javiermtorres]

A follow up question: is there any way to persist the minio configuration so we can pick it up if available and run the following only if that is not already set?

Hm. Good question. If the commands are idempotent it doesn't matter (so buckets are not recreated anew if they already exist, etc etc). Was there something specific that failed in your setup?

Leaving the notes on minio persistence here for a future issue, since the original comment is resolved.

[aittalam]

LGTM, thank you!
I saw the branch got out-to-date wrt main, I pre-approve this so you'll be able to land this after merging from main.

[javiermtorres]

Thanks a lot @aittalam :)