[Bug]: LakeFS fails to delete branches due to block storage not found

[isaac-jordan]

What happened?

Slack thread: https://lakefs.slack.com/archives/C016726JLJW/p1736357024429429

I'm trying to delete several branches, but they refuse to be deleted due to a failing precondition: read from block storage: not found.

lakectl branch delete -y lakefs://survey-data-level0/dev-raw_versioned_survey_files-20241122170517
Branch: lakefs://survey-data-level0/dev-raw_versioned_survey_files-20241122170517
list actions from commit: listing actions: manage metarange 96e356c38f6b348f98645061033cfd7be8f93aac8a647da729be17060e8153dc: open sstable file s3://survey-data-level0 96e356c38f6b348f98645061033cfd7be8f93aac8a647da729be17060e8153dc: read from block storage: not found
412 Precondition Failed

I have a hunch these branches may have been created before a reset of my storage layer (minio), so it's failing to find the items it expects to be on S3.

They still show on up the LakeFS UI, but fail to load data:

Expected behavior

I'd expect it to allow me to delete the branches, even if LakeFS can't find the data for them. Perhaps with a --force/-f flag?

lakeFS version

lakeFS 1.47.0

How lakeFS is installed

Local Docker container

Affected clients

lakectl version: 1.48.0

Relevant log output

2025-01-08 17:59:21 time="2025-01-08T17:59:21Z" level=warning msg="aborted by hooks" func="pkg/api.(*Controller).handleAPIErrorCallback" file="build/pkg/api/controller.go:2691" error="pre-delete-branch hook aborted, run id '56qtu1rv8q7c72qji1bg': list actions from commit: listing actions: manage metarange 96e356c38f6b348f98645061033cfd7be8f93aac8a647da729be17060e8153dc: open sstable file s3://survey-data-level0 96e356c38f6b348f98645061033cfd7be8f93aac8a647da729be17060e8153dc: read from block storage: not found" host="127.0.0.1:28000" method=DELETE operation_id=DeleteBranch path=/api/v1/repositories/survey-data-level0/branches/dev-raw_versioned_survey_files-20241122170517 run_id=56qtu1rv8q7c72qji1bg service=api_gateway user=everything-bagel

Contact details

GitHub Issue or LakeFS Slack