Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

adjust sqlite settings #380

Open
lyuboxa opened this issue Dec 18, 2024 · 0 comments
Open

adjust sqlite settings #380

lyuboxa opened this issue Dec 18, 2024 · 0 comments

Comments

@lyuboxa
Copy link

lyuboxa commented Dec 18, 2024

Since the addition of WAL, sqlite can handle multiple connections however this can often spur database locked errors.
As part of the https://github.com/ConduitIO/conduit project we ran some tests and found that tx locking is more tolerable when transactions are not deferred, which is the default for sqlite thus setting busy_timeout=N[1] and _txlock=immediate[2] allows for concurrent access.

By allowing the tx lock to take place immediately, this allows for a form of serialization of writes in the span of the busy timeout, until when the error will return. In comparison a deferred tx will not trigger a lock until it reaches a statement which requires it, but by that time there may be a non-reconcilable tx in progress or data change and no retry is possible.

  1. https://www.sqlite.org/pragma.html#pragma_busy_timeout
  2. See https://pkg.go.dev/modernc.org/sqlite#Driver.Open, https://www.sqlite.org/lang_transaction.html
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant