Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Consistency for File State #374

Merged
merged 64 commits into from
Jan 15, 2025
Merged
Show file tree
Hide file tree
Changes from 62 commits
Commits
Show all changes
64 commits
Select commit Hold shift + click to select a range
1350081
Delete file immediately (not async) in CachePurge
foodprocessor Nov 25, 2024
e58a516
Set test timeout to 1s.
foodprocessor Nov 25, 2024
e37f609
Remove unused deleteEvent channel
foodprocessor Nov 25, 2024
63f412d
Use cacheTimeoutMonitor even when timeout is zero.
foodprocessor Nov 26, 2024
3e71a0f
Do not use CachePurge for eviction, even when timeout is zero.
foodprocessor Nov 26, 2024
5d2e6af
fixup
foodprocessor Nov 26, 2024
8a6f792
Merge branch 'zero-timeout-evicts-all' into delete-files-synchronously
foodprocessor Nov 26, 2024
2b62986
Use fileLocks to keep file info coherent
foodprocessor Nov 26, 2024
1704b57
Merge remote-tracking branch 'origin/main' into delete-files-synchron…
foodprocessor Nov 26, 2024
77206e8
Remove locks in Chmod & FlushFile to avoid deadlock
foodprocessor Nov 27, 2024
383e8bf
Use object paths when locking files in RenameDir.
foodprocessor Nov 27, 2024
965676c
Close loopback file handles to avoid issues (especially on Windows).
foodprocessor Nov 27, 2024
4c77941
Prevent attribute data corruption
foodprocessor Nov 27, 2024
60321b5
Fix remaining race conditions detected in file cache
foodprocessor Nov 27, 2024
ff15366
Lock file to protect calls to stat.
foodprocessor Nov 27, 2024
4f3f95a
Merge branch 'fix-race-conditions' into delete-files-synchronously
foodprocessor Nov 27, 2024
7b45c80
Make timeout minimum 1s.
foodprocessor Nov 27, 2024
f810edf
Merge branch 'timeout-minimum-1s' into delete-files-synchronously
foodprocessor Nov 27, 2024
bcb288d
Reinstate test which was flaky due to a race condition
foodprocessor Nov 27, 2024
2ea9b61
Remove protection from stat call, which is not within the scope of th…
foodprocessor Nov 27, 2024
74e3769
Merge branch 'fix-race-conditions' into delete-files-synchronously
foodprocessor Nov 27, 2024
3fe1ede
Change test config default to default timeout (120s)
foodprocessor Nov 27, 2024
5b3a6a1
Slow down test check loops to 100ms
foodprocessor Nov 27, 2024
95f6228
Add custom configs for eviction tests.
foodprocessor Nov 27, 2024
f5707fb
Don't open to prevent eviction in tests (unless testing that mechanic).
foodprocessor Nov 27, 2024
0caeb2f
Open from cache when no object exists
foodprocessor Dec 21, 2024
16c8290
Rename downloadFile to openFileInternal
foodprocessor Dec 21, 2024
6eb909b
openFileInternal runs with flock already locked
foodprocessor Dec 21, 2024
787c623
Remove redundant code
foodprocessor Dec 21, 2024
a5b2e51
Replace lost flock.Inc
foodprocessor Dec 21, 2024
2d0e64a
Add test to verify that OpenFile changes what GetAttr returns when op…
foodprocessor Dec 21, 2024
730c271
Do not use &internal.ObjAttr when you mean nil!
foodprocessor Dec 21, 2024
b0c87be
Merge remote-tracking branch 'main' into delete-files-synchronously
foodprocessor Dec 31, 2024
b3136ef
Merge branch 'local-files-open-right-away' into delete-files-synchron…
foodprocessor Dec 31, 2024
1c83708
Add lazy open state in flock, since it affects the whole file's state…
foodprocessor Jan 1, 2025
d59c324
Refactor isDownloadRequired, and use LazyOpen state variable to allow…
foodprocessor Jan 1, 2025
d9d2845
Refactor openFile
foodprocessor Jan 1, 2025
e21dcc1
Update file state in CreateFile
foodprocessor Jan 1, 2025
80fee42
Use O_CREAT flag when testing creation with OpenFile.
foodprocessor Jan 1, 2025
dc7fc99
Merge branch 'local-files-open-right-away' into refactor-openFile
foodprocessor Jan 1, 2025
78be394
Merge branch 'refactor-openFile' into delete-files-synchronously
foodprocessor Jan 1, 2025
b0cbc15
Update state in DeleteFile
foodprocessor Jan 1, 2025
ffd5aea
Update file state in CachePurge (and not speculatively outside of it)
foodprocessor Jan 1, 2025
2ffa0cb
Improve openFile file state logic
foodprocessor Jan 1, 2025
9f47003
Protect file state in Close and Flush (and Sync by consequence)
foodprocessor Jan 1, 2025
094417d
make flushFileInternal update the file state itself
foodprocessor Jan 1, 2025
5e1a000
make renameCachedFile protect and update file states
foodprocessor Jan 1, 2025
80d659c
When locking more that one file, always go in lexical order to preven…
foodprocessor Jan 2, 2025
143e7c4
Add TODOs for directory operations
foodprocessor Jan 2, 2025
4841a19
Lock all files before renaming or deleting a whole directory in one o…
foodprocessor Jan 2, 2025
56a509c
Merge branch 'delete-files-synchronously' into maintain-file-state
foodprocessor Jan 2, 2025
d6a3c72
Make CachePurge update the file state
foodprocessor Jan 2, 2025
f712e7f
Remove file state flags which are never read, and the complexity they…
foodprocessor Jan 2, 2025
ccaca08
Protect file state in GetAttr and Chmod
foodprocessor Jan 2, 2025
b519f5b
Remove unused flags in platform-specific file cache files
foodprocessor Jan 2, 2025
1defe41
Avoid deadlock when flushFileInternal calls Chmod
foodprocessor Jan 2, 2025
74ed602
Don't use base name to find the object name when WalkDir is recursive
foodprocessor Jan 2, 2025
681b140
Don't search and lock files in DeleteDir, since it is only called on …
foodprocessor Jan 7, 2025
c23d36b
Simplify DeleteDir, since the directory being deleted is empty
foodprocessor Jan 7, 2025
6655269
Revert accidental code comment change
foodprocessor Jan 2, 2025
a45a761
Merge branch 'main' into delete-files-synchronously
foodprocessor Jan 11, 2025
077d979
Move helper functions to where they are used
foodprocessor Jan 11, 2025
0cefa32
Remove deadlock bug from incorrect branch merge
foodprocessor Jan 14, 2025
64997fd
Fix open handle count issue with Rename
jfantinhardesty Jan 15, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions common/lock_map.go
Original file line number Diff line number Diff line change
Expand Up @@ -44,6 +44,8 @@ type LockMapItem struct {
exLocked bool
mtx sync.Mutex
downloadTime time.Time
// track if file is in lazy open state
LazyOpen bool
}

// Map holding locks for all the files
Expand Down
2 changes: 1 addition & 1 deletion component/azstorage/block_blob.go
Original file line number Diff line number Diff line change
Expand Up @@ -569,7 +569,7 @@ func (bb *BlockBlob) List(prefix string, marker *string, count int32) ([]*intern
var dirList = make(map[string]bool)
for _, blobInfo := range listBlob.Segment.BlobItems {
blobInfo.Name = bb.getFileName(*blobInfo.Name)
attr := &internal.ObjAttr{}
var attr *internal.ObjAttr
if blobInfo.Properties.CustomerProvidedKeySHA256 != nil && *blobInfo.Properties.CustomerProvidedKeySHA256 != "" {
log.Trace("BlockBlob::List : blob is encrypted with customer provided key so fetching metadata explicitly using REST")
attr, err = bb.getAttrUsingRest(*blobInfo.Name)
Expand Down
4 changes: 2 additions & 2 deletions component/file_cache/cache_policy.go
Original file line number Diff line number Diff line change
Expand Up @@ -56,8 +56,8 @@ type cachePolicy interface {

UpdateConfig(cachePolicyConfig) error

CacheValid(name string) // Mark the file as hit
CachePurge(name string) // Schedule the file for deletion
CacheValid(name string) // Mark the file as hit
CachePurge(name string, flock *common.LockMapItem) // Delete the file from cache

IsCached(name string) bool // Whether or not the cache policy considers this file cached

Expand Down
Loading
Loading