diff --git a/notes/adrs/009-poatgresql.md b/notes/adrs/009-postgresql.md similarity index 100% rename from notes/adrs/009-poatgresql.md rename to notes/adrs/009-postgresql.md diff --git a/notes/adrs/011-sqlite.md b/notes/adrs/011-sqlite.md new file mode 100644 index 0000000..13cdedd --- /dev/null +++ b/notes/adrs/011-sqlite.md @@ -0,0 +1,157 @@ +# ADR: Continued Support for SQLite for Testing Environments in Abilian SBE + +**Status**: Accepted + +## Introduction +This ADR explains the rationale for maintaining **SQLite** support exclusively for **testing and development** purposes in Abilian SBE. While PostgreSQL and MariaDB are adopted as the tier-one databases for production deployments, SQLite offers simplicity, ease of setup, and minimal overhead, making it an ideal choice for local testing and automated test suites. + +## Summary +SQLite will remain a supported database backend for **testing** and **local development** but will **not** be recommended or supported for production environments. Its lightweight nature and lack of configuration requirements make it an excellent tool for automated tests and developer workflows. + +## Context and Goals + +### Context +Abilian SBE currently supports multiple relational database systems, including SQLite, MySQL, and PostgreSQL. With the adoption of **PostgreSQL** and **MariaDB** as tier-one databases for production (see related ADRs), there is a need to clarify SQLite’s role within the system. + +SQLite is widely used in development and testing workflows due to its simplicity, file-based storage, and zero-configuration nature. It eliminates the need for managing database servers in test environments, speeding up test execution and simplifying developer onboarding. + +While SQLite lacks the scalability and advanced features required for production, its advantages for testing make it a valuable tool that should continue to be supported. + +### Goals +1. **Testing Efficiency**: Use SQLite for lightweight and fast testing without requiring a database server. +2. **Developer Productivity**: Allow developers to set up local testing environments quickly and easily. +3. **Isolation**: Use a self-contained, file-based database to avoid external dependencies in CI/CD pipelines and local tests. +4. **Cost Efficiency**: Avoid the overhead of configuring and managing PostgreSQL or MariaDB for automated tests. + +## Tenets +- **Simplicity**: SQLite requires no configuration or external dependencies, making it ideal for tests. +- **Speed**: SQLite offers fast setup and teardown times for automated test suites. +- **Isolation**: SQLite’s file-based nature ensures tests remain isolated and reproducible. +- **Consistency**: Developers and CI/CD systems can run tests reliably without external infrastructure. + +## Decision +We will retain **SQLite** as the default database backend for **testing** and **local development** in Abilian SBE. SQLite will not be supported or recommended for production environments. + +### Why SQLite for Testing? +1. **Zero Configuration**: + - SQLite runs as a single library with file-based storage, requiring no database server, user management, or configuration. + +2. **Speed**: + - Tests execute faster with SQLite compared to server-based databases. + - Database creation and teardown are near-instantaneous, which is ideal for unit tests and CI/CD pipelines. + +3. **Self-Contained**: + - SQLite operates as an embedded database, ensuring test environments remain isolated and reproducible. + +4. **Ease of Use for Developers**: + - Developers can run tests locally without installing or managing PostgreSQL or MariaDB. + +5. **CI/CD Integration**: + - SQLite simplifies test infrastructure, reducing overhead in CI/CD pipelines. + +## Detailed Design + +### Use Cases for SQLite +1. **Unit Tests**: + - SQLite will serve as the default backend for running fast, isolated unit tests during development and CI/CD workflows. + +2. **Local Development**: + - Developers can use SQLite for quick local testing without needing a database server. + +3. **Integration Tests**: + - For isolated integration tests that do not rely on advanced database features, SQLite can provide lightweight, consistent behavior. + +### Production Considerations +- SQLite will be **explicitly marked as unsupported for production** due to its limitations: + - Lack of scalability for concurrent writes and large datasets. + - Limited support for advanced features like JSONB, full-text search, and complex queries. + - Potential issues with data integrity in multi-process environments. + +### Configuration +The default testing configuration will use SQLite: +```yaml +database: + backend: sqlite + sqlite: + filename: ":memory:" +``` + +For development, a local SQLite file can be used: +```yaml +database: + backend: sqlite + sqlite: + filename: "./dev.db" +``` + +## Examples and Interactions + +1. **In-Memory SQLite for Tests** + Run unit tests with an ephemeral in-memory database: + ```python + from sqlalchemy import create_engine + + engine = create_engine("sqlite:///:memory:") + # Initialize schema, run tests, and tear down automatically. + ``` + +2. **Local Development Database** + Use a file-based SQLite database for quick development: + ```python + engine = create_engine("sqlite:///local_dev.db") + ``` + +3. **CI/CD Pipeline Example** + Integrate SQLite into a CI pipeline for testing: + ```yaml + steps: + - name: Run Unit Tests + run: pytest --db-backend=sqlite + ``` + +## Consequences + +### Benefits +- **Simplified Testing**: No need for database servers in test environments, reducing setup time and resource consumption. +- **Faster Tests**: SQLite’s in-memory mode ensures rapid execution of test cases. +- **Developer Onboarding**: Developers can start testing without additional configuration or installations. +- **CI/CD Efficiency**: Reduces infrastructure complexity in CI/CD pipelines. + +### Drawbacks +- **Feature Limitations**: SQLite does not support advanced features like JSONB, full-text search, or window functions used in PostgreSQL/MariaDB. +- **Behavior Differences**: Minor inconsistencies may arise between SQLite and production databases (e.g., stricter SQL compliance in PostgreSQL). +- **Not Suitable for Integration Tests with Complex Queries**: Some database-specific behaviors may require PostgreSQL or MariaDB for more robust integration testing. + +## Lessons Learned +SQLite has consistently provided value as a lightweight, low-maintenance database for testing. However, its limitations make it unsuitable for production workloads. Retaining SQLite for testing ensures efficiency while avoiding unnecessary overhead. + +## Action Items +1. Update Abilian SBE’s documentation to clarify SQLite’s role as a **testing-only** database. +2. Update default test configurations to use SQLite in in-memory mode. +3. Add warnings in documentation and logs when SQLite is used in production-like environments. +4. Ensure PostgreSQL and MariaDB remain the focus for production deployments. + +## Alternatives +- **PostgreSQL for Testing**: + - Running PostgreSQL in test environments ensures feature parity with production but increases setup complexity and slows down tests. + +- **MariaDB for Testing**: + - Similar to PostgreSQL, MariaDB introduces resource overhead for local tests and CI pipelines. + +- **SQLite for Production**: + - Not a viable option due to scalability and concurrency limitations. + +## Unresolved Questions +- Should we include optional support for PostgreSQL or MariaDB-based testing in CI/CD pipelines for specific integration tests? + +## Future Work +- Explore using PostgreSQL or MariaDB in automated integration tests where feature parity is critical. +- Provide tools for easy switching between SQLite and production databases for advanced testing workflows. + +## Related +- ADR 009: Adoption of PostgreSQL as a Tier-One Supported Database +- ADR 010: Adoption of MariaDB as a Tier-One Supported Database + +## References +- **SQLite Documentation**: https://sqlite.org/docs.html +- **Testing with SQLite**: https://docs.sqlalchemy.org/en/14/dialects/sqlite.html