diff --git a/README.md b/README.md index b2a647e..cffe1cd 100644 --- a/README.md +++ b/README.md @@ -50,11 +50,96 @@ if storing copies. archived is built with microservice architecture containing the following components: -* publisher - HTTP server to allow data listing and fetching -* manager - gRPC API to manage containers, versions and objects +* archived-publisher - HTTP server to allow data listing and fetching +* archived-manager - gRPC API to manage containers, versions and objects +* archived-exporter - Prometheus metrics exporter for metadata entities * CLI - CLI application to interact with manage component * migrator - metadata migration tool +## Deploy + +archived is distributed as a number of prebuilt binaries which allows to choose +any particular way to deploy it from systemd services to Kubernetes. + +The main things are required to know before deployment: + +* archived-publisher can use RO replica of PostgreSQL for operation + and can scale +* archived-manager requires RW PostgreSQL instance since it performs + writes, can also scale +* archived-exporter is sufficient to run in the only copy since it just + provides metrics for the database stuff, RO replica access is also enough +* archived-migrator must be ran each time archived is upgrading right before + other components +* archived-cli could run anywhere and will require network access to + archived-manager +* there's no authentication on any stage at the moment (yes, even for + cli/manager) + +An example for Kubernetes deployment specs is available in +[docs/examples/deploy/k8s](docs/examples/deploy/k8s) directory. + +## CLI + +archived-cli provides an CLI interface to operate archived including creating +containers, versions and objects. It works with archived-manager to handle +requests. + +```shell +usage: archived-cli --endpoint=ENDPOINT [] [ ...] + +CLI interface for archived + + +Flags: + --[no-]help Show context-sensitive help (also try --help-long and --help-man). + -d, --[no-]debug Enable debug mode ($ARCHIVED_CLI_DEBUG) + -t, --[no-]trace Enable trace mode (debug mode on steroids) ($ARCHIVED_CLI_TRACE) + -s, --endpoint=ENDPOINT Manage API endpoint address ($ARCHIVED_CLI_ENDPOINT) + --[no-]insecure Do not use TLS for gRPC connection + --[no-]insecure-skip-verify + Do not perform TLS certificate verification for gRPC connection + --cache-dir="~/.cache/archived/cli/objects" + cache directory for objects + +Commands: +help [...] + Show help. + +container create + create new container + +container delete + delete the given container + +container list + list containers + +version create [] + create new version for given container + +version delete + delete the given version + +version list + list versions for the given container + +version publish + publish the given version + +object list + list objects in the given container and version + +object create + create object(s) from location + +object url + get URL for the object + +object delete + delete object +``` + ## How build the project manually archived requires the following dependencies to build: diff --git a/docs/examples/deploy/k8s/000-namespace.yaml b/docs/examples/deploy/k8s/000-namespace.yaml new file mode 100644 index 0000000..0719225 --- /dev/null +++ b/docs/examples/deploy/k8s/000-namespace.yaml @@ -0,0 +1,7 @@ +--- +apiVersion: v1 +kind: Namespace +metadata: + name: archived + labels: + app: archived diff --git a/docs/examples/deploy/k8s/001-postgresql-cluster.yaml b/docs/examples/deploy/k8s/001-postgresql-cluster.yaml new file mode 100644 index 0000000..9abd250 --- /dev/null +++ b/docs/examples/deploy/k8s/001-postgresql-cluster.yaml @@ -0,0 +1,69 @@ +--- +apiVersion: postgresql.cnpg.io/v1 +kind: Cluster +metadata: + name: metadatadb + namespace: archived +spec: + instances: 3 + imageName: ghcr.io/cloudnative-pg/postgresql:16.3-7 + + minSyncReplicas: 2 + maxSyncReplicas: 2 + + postgresql: + syncReplicaElectionConstraint: + enabled: true + nodeLabelsAntiAffinity: + - topology.kubernetes.io/node + + replicationSlots: + highAvailability: + enabled: true + updateInterval: 10 + + primaryUpdateStrategy: unsupervised + switchoverDelay: 60 + storage: + pvcTemplate: + resources: + requests: + storage: 30Gi + storageClassName: openebs-hostpath + volumeMode: Filesystem + resizeInUseVolumes: false + + resources: + requests: + memory: "1Gi" + cpu: 1 + limits: + memory: "1Gi" + cpu: 1 + + backup: + barmanObjectStore: + destinationPath: "s3://" + endpointURL: https://s3.example.com # (CHANGEME: s3 endpoint) + s3Credentials: + accessKeyId: + name: cnpg-backup-creds + key: ACCESS_KEY_ID + secretAccessKey: + name: cnpg-backup-creds + key: ACCESS_SECRET_KEY + retentionPolicy: "30d" + + monitoring: + enablePodMonitor: true +--- +apiVersion: postgresql.cnpg.io/v1 +kind: ScheduledBackup +metadata: + name: metadatadb-backups + namespace: archived +spec: + schedule: "0 58 */9 * * *" + backupOwnerReference: self + cluster: + name: metadatadb diff --git a/docs/examples/deploy/k8s/002-configmap.yaml b/docs/examples/deploy/k8s/002-configmap.yaml new file mode 100644 index 0000000..e6becb1 --- /dev/null +++ b/docs/examples/deploy/k8s/002-configmap.yaml @@ -0,0 +1,10 @@ +--- +apiVersion: v1 +kind: ConfigMap +metadata: + name: s3-blob-repository + namespace: archived +data: + BLOB_S3_ENDPOINT: https://s3.example.com # (CHANGEME: s3 endpoint) + BLOB_S3_BUCKET: "" + BLOB_S3_CREATE_BUCKET: "true" diff --git a/docs/examples/deploy/k8s/003-migrator-job.yaml b/docs/examples/deploy/k8s/003-migrator-job.yaml new file mode 100644 index 0000000..c063e0b --- /dev/null +++ b/docs/examples/deploy/k8s/003-migrator-job.yaml @@ -0,0 +1,30 @@ +--- +apiVersion: batch/v1 +kind: Job +metadata: + name: archived-migrator + namespace: archived + labels: + app.kubernetes.io/name: archived-migrator + app.kubernetes.io/app: archived-migrator +spec: + template: + metadata: + name: archived-migrator + labels: + app.kubernetes.io/name: archived-migrator + app.kubernetes.io/app: archived-migrator + spec: + containers: + - name: migrator + image: ghcr.io/teran/archived/migrator:latest + imagePullPolicy: Always + env: + - name: METADATA_DSN + valueFrom: + secretKeyRef: + name: metadatadb-app + key: uri + - name: LOG_LEVEL + value: "trace" + restartPolicy: OnFailure diff --git a/docs/examples/deploy/k8s/README.md b/docs/examples/deploy/k8s/README.md new file mode 100644 index 0000000..402fdb9 --- /dev/null +++ b/docs/examples/deploy/k8s/README.md @@ -0,0 +1,14 @@ +# Kubernetes deployment example + +## Prerequisites + +* ingress-nginx +* openebs with hostpath enabled for PVC +* VictoriaMetrics or Prometheus operator for PodMonitors +* CloudNativePG for PostgreSQL +* External S3 service to store blobs + +## Deploy + +* Change all the fields marked as "CHANGEME" to appropriate values +* `kubectl apply -f .` diff --git a/docs/examples/deploy/k8s/archived-exporter.yaml b/docs/examples/deploy/k8s/archived-exporter.yaml new file mode 100644 index 0000000..6332b4f --- /dev/null +++ b/docs/examples/deploy/k8s/archived-exporter.yaml @@ -0,0 +1,69 @@ +--- +apiVersion: apps/v1 +kind: Deployment +metadata: + name: archived-exporter + namespace: archived + labels: + app.kubernetes.io/name: archived-exporter + app.kubernetes.io/app: archived-exporter +spec: + replicas: 1 + strategy: + type: RollingUpdate + revisionHistoryLimit: 10 + selector: + matchLabels: + app.kubernetes.io/name: archived-exporter + app.kubernetes.io/app: archived-exporter + template: + metadata: + labels: + app.kubernetes.io/name: archived-exporter + app.kubernetes.io/app: archived-exporter + spec: + affinity: + podAntiAffinity: + requiredDuringSchedulingIgnoredDuringExecution: + - labelSelector: + matchExpressions: + - key: app.kubernetes.io/app + operator: In + values: + - archived-exporter + topologyKey: "kubernetes.io/node" + terminationGracePeriodSeconds: 30 + containers: + - name: exporter + image: ghcr.io/teran/archived/exporter:latest + imagePullPolicy: Always + env: + - name: METADATA_DSN + valueFrom: + secretKeyRef: + name: metadata-database-ro + key: METADATA_DSN_RO + - name: LOG_LEVEL + value: "trace" + ports: + - name: metrics + containerPort: 8081 + protocol: TCP + resources: + requests: + cpu: 10m + memory: 128Mi + limits: + memory: 1Gi + readinessProbe: + httpGet: + path: /metrics + port: metrics + timeoutSeconds: 1 + livenessProbe: + httpGet: + path: /metrics + port: metrics + securityContext: + allowPrivilegeEscalation: false + readOnlyRootFilesystem: true diff --git a/docs/examples/deploy/k8s/archived-gc.yaml b/docs/examples/deploy/k8s/archived-gc.yaml new file mode 100644 index 0000000..7716f95 --- /dev/null +++ b/docs/examples/deploy/k8s/archived-gc.yaml @@ -0,0 +1,35 @@ +--- +apiVersion: batch/v1 +kind: CronJob +metadata: + name: archived-gc + namespace: archived + labels: + app.kubernetes.io/name: archived-gc + app.kubernetes.io/app: archived-gc +spec: + schedule: "48 * * * *" + jobTemplate: + spec: + template: + metadata: + name: archived-gc + labels: + app.kubernetes.io/name: archived-gc + app.kubernetes.io/app: archived-gc + spec: + containers: + - name: gc + image: ghcr.io/teran/archived/gc:latest + imagePullPolicy: Always + env: + - name: METADATA_DSN + valueFrom: + secretKeyRef: + name: metadatadb-app + key: uri + - name: LOG_LEVEL + value: "trace" + - name: DRY_RUN + value: "false" + restartPolicy: OnFailure diff --git a/docs/examples/deploy/k8s/archived-manager.yaml b/docs/examples/deploy/k8s/archived-manager.yaml new file mode 100644 index 0000000..e8cf7a9 --- /dev/null +++ b/docs/examples/deploy/k8s/archived-manager.yaml @@ -0,0 +1,77 @@ +--- +apiVersion: apps/v1 +kind: Deployment +metadata: + name: archived-manager + namespace: archived + labels: + app.kubernetes.io/name: archived-manager + app.kubernetes.io/app: archived-manager +spec: + replicas: 3 + strategy: + type: RollingUpdate + revisionHistoryLimit: 10 + selector: + matchLabels: + app.kubernetes.io/name: archived-manager + app.kubernetes.io/app: archived-manager + template: + metadata: + labels: + app.kubernetes.io/name: archived-manager + app.kubernetes.io/app: archived-manager + spec: + affinity: + podAntiAffinity: + requiredDuringSchedulingIgnoredDuringExecution: + - labelSelector: + matchExpressions: + - key: app.kubernetes.io/app + operator: In + values: + - archived-manager + topologyKey: "kubernetes.io/node" + terminationGracePeriodSeconds: 30 + containers: + - name: manager + image: ghcr.io/teran/archived/manager:latest + imagePullPolicy: Always + envFrom: + - configMapRef: + name: s3-blob-repository + - secretRef: + name: s3-blob-repository + env: + - name: METADATA_DSN + valueFrom: + secretKeyRef: + name: metadatadb-app + key: uri + - name: LOG_LEVEL + value: "trace" + ports: + - name: grpc + containerPort: 5555 + protocol: TCP + - name: metrics + containerPort: 8081 + protocol: TCP + resources: + requests: + cpu: 10m + memory: 1Gi + limits: + memory: 1Gi + readinessProbe: + httpGet: + path: /metrics + port: metrics + timeoutSeconds: 1 + livenessProbe: + httpGet: + path: /metrics + port: metrics + securityContext: + allowPrivilegeEscalation: false + readOnlyRootFilesystem: true diff --git a/docs/examples/deploy/k8s/archived-publisher.yaml b/docs/examples/deploy/k8s/archived-publisher.yaml new file mode 100644 index 0000000..0653367 --- /dev/null +++ b/docs/examples/deploy/k8s/archived-publisher.yaml @@ -0,0 +1,77 @@ +--- +apiVersion: apps/v1 +kind: Deployment +metadata: + name: archived-publisher + namespace: archived + labels: + app.kubernetes.io/name: archived-publisher + app.kubernetes.io/app: archived-publisher +spec: + replicas: 3 + strategy: + type: RollingUpdate + revisionHistoryLimit: 10 + selector: + matchLabels: + app.kubernetes.io/name: archived-publisher + app.kubernetes.io/app: archived-publisher + template: + metadata: + labels: + app.kubernetes.io/name: archived-publisher + app.kubernetes.io/app: archived-publisher + spec: + affinity: + podAntiAffinity: + requiredDuringSchedulingIgnoredDuringExecution: + - labelSelector: + matchExpressions: + - key: app.kubernetes.io/app + operator: In + values: + - archived-publisher + topologyKey: "kubernetes.io/node" + terminationGracePeriodSeconds: 30 + containers: + - name: publisher + image: ghcr.io/teran/archived/publisher:latest + imagePullPolicy: Always + envFrom: + - configMapRef: + name: s3-blob-repository + - secretRef: + name: s3-blob-repository + env: + - name: METADATA_DSN + valueFrom: + secretKeyRef: + name: metadata-database-ro + key: METADATA_DSN_RO + - name: LOG_LEVEL + value: "trace" + ports: + - name: http + containerPort: 8080 + protocol: TCP + - name: metrics + containerPort: 8081 + protocol: TCP + resources: + requests: + cpu: 10m + memory: 128Mi + limits: + memory: 1Gi + readinessProbe: + httpGet: + path: /metrics + port: metrics + timeoutSeconds: 1 + livenessProbe: + httpGet: + path: /metrics + port: metrics + securityContext: + allowPrivilegeEscalation: false + readOnlyRootFilesystem: true diff --git a/docs/examples/deploy/k8s/ingress.yaml b/docs/examples/deploy/k8s/ingress.yaml new file mode 100644 index 0000000..9f91210 --- /dev/null +++ b/docs/examples/deploy/k8s/ingress.yaml @@ -0,0 +1,69 @@ +--- +apiVersion: networking.k8s.io/v1 +kind: Ingress +metadata: + annotations: + ingress.kubernetes.io/ssl-redirect: "true" + nginx.ingress.kubernetes.io/ssl-redirect: "true" + kubernetes.io/ingress.class: nginx + kubernetes.io/tls-acme: "true" + cert-manager.io/cluster-issuer: homelab-ca-issuer # CHANGEME: issuer name + nginx.ingress.kubernetes.io/proxy-read-timeout: "3600" + nginx.ingress.kubernetes.io/proxy-send-timeout: "3600" + nginx.ingress.kubernetes.io/client-body-buffer-size: 256m + nginx.ingress.kubernetes.io/proxy-body-size: 256m + labels: + app.kubernetes.io/name: archived-access + app.kubernetes.io/app: archived-access + name: archived + namespace: archived +spec: + ingressClassName: nginx + rules: + - host: archived.example.com # CHANGEME: publisher domain + http: + paths: + - backend: + service: + name: archived-publisher + port: + number: 8080 + path: / + pathType: Prefix + tls: + - hosts: + - archived.archived.example.com # CHANGEME: publisher domain + secretName: archived.archived.example.com # CHANGEME: publisher domain +--- +apiVersion: networking.k8s.io/v1 +kind: Ingress +metadata: + annotations: + ingress.kubernetes.io/ssl-redirect: "true" + nginx.ingress.kubernetes.io/ssl-redirect: "true" + kubernetes.io/ingress.class: nginx + kubernetes.io/tls-acme: "true" + nginx.ingress.kubernetes.io/proxy-read-timeout: "3600" + nginx.ingress.kubernetes.io/proxy-send-timeout: "3600" + nginx.ingress.kubernetes.io/client-body-buffer-size: 256m + nginx.ingress.kubernetes.io/proxy-body-size: 256m + nginx.ingress.kubernetes.io/backend-protocol: "GRPC" + name: archived-manage + namespace: archived +spec: + ingressClassName: nginx + rules: + - host: archived-manage.example.com # CHANGEME: manager domain + http: + paths: + - path: / + pathType: Prefix + backend: + service: + name: archived-manager + port: + number: 5555 + tls: + - hosts: + - archived-manage.example.com # CHANGEME: manager domain + secretName: archived-manage.example.com # CHANGEME: manager domain diff --git a/docs/examples/deploy/k8s/podmonitor.yaml b/docs/examples/deploy/k8s/podmonitor.yaml new file mode 100644 index 0000000..80967dc --- /dev/null +++ b/docs/examples/deploy/k8s/podmonitor.yaml @@ -0,0 +1,48 @@ +--- +apiVersion: monitoring.coreos.com/v1 +kind: PodMonitor +metadata: + name: archived-publisher + namespace: archived + labels: + app.kubernetes.io/name: archived-publisher + app.kubernetes.io/app: archived-publisher +spec: + selector: + matchLabels: + app.kubernetes.io/name: archived-publisher + app.kubernetes.io/app: archived-publisher + podMetricsEndpoints: + - port: metrics +--- +apiVersion: monitoring.coreos.com/v1 +kind: PodMonitor +metadata: + name: archived-manager + namespace: archived + labels: + app.kubernetes.io/name: archived-manager + app.kubernetes.io/app: archived-manager +spec: + selector: + matchLabels: + app.kubernetes.io/name: archived-manager + app.kubernetes.io/app: archived-manager + podMetricsEndpoints: + - port: metrics +--- +apiVersion: monitoring.coreos.com/v1 +kind: PodMonitor +metadata: + name: archived-exporter + namespace: archived + labels: + app.kubernetes.io/name: archived-exporter + app.kubernetes.io/app: archived-exporter +spec: + selector: + matchLabels: + app.kubernetes.io/name: archived-exporter + app.kubernetes.io/app: archived-exporter + podMetricsEndpoints: + - port: metrics diff --git a/docs/examples/deploy/k8s/secrets.yaml b/docs/examples/deploy/k8s/secrets.yaml new file mode 100644 index 0000000..7f5fd62 --- /dev/null +++ b/docs/examples/deploy/k8s/secrets.yaml @@ -0,0 +1,36 @@ +--- +apiVersion: v1 +stringData: + ACCESS_KEY_ID: "" + ACCESS_SECRET_KEY: "" +kind: Secret +metadata: + name: cnpg-backup-creds + namespace: archived +type: Opaque +--- +apiVersion: v1 +stringData: + BLOB_S3_ACCESS_KEY_ID: "" + BLOB_S3_SECRET_KEY: "" +kind: Secret +metadata: + name: s3-blob-repository + namespace: archived +type: Opaque +--- +apiVersion: v1 +stringData: + # Password could be obtained from metadatadb-app secret after CloudNativePG + # will create a new database + # + # Unfortunately it does not provide a complete URL for RO database instances + # so this secret covers that case. If you have the only database instance you + # can easily use RW instance just in manager or migrator do. + # + METADATA_DSN_RO: "postgresql://app:@metadatadb-ro.archived:5432/app" +kind: Secret +metadata: + name: metadata-database-ro + namespace: archived +type: Opaque diff --git a/docs/examples/deploy/k8s/service.yaml b/docs/examples/deploy/k8s/service.yaml new file mode 100644 index 0000000..a5a5c1f --- /dev/null +++ b/docs/examples/deploy/k8s/service.yaml @@ -0,0 +1,38 @@ +--- +apiVersion: v1 +kind: Service +metadata: + name: archived-publisher + namespace: archived + labels: + app.kubernetes.io/name: archived-publisher + app.kubernetes.io/app: archived-publisher +spec: + ports: + - name: http + port: 8080 + protocol: TCP + targetPort: 8080 + selector: + app.kubernetes.io/name: archived-publisher + app.kubernetes.io/app: archived-publisher + type: ClusterIP +--- +apiVersion: v1 +kind: Service +metadata: + name: archived-manager + namespace: archived + labels: + app.kubernetes.io/name: archived-manager + app.kubernetes.io/app: archived-manager +spec: + ports: + - name: grpc + port: 5555 + protocol: TCP + targetPort: 5555 + selector: + app.kubernetes.io/name: archived-manager + app.kubernetes.io/app: archived-manager + type: ClusterIP