minikube helm chart

modification of quarry to allow for deployment to minikube.
Additional changes will be needed when k8s is available in
production to allow for production to be deployed to k8s.

** Do not deploy this **

Bug: T301469

README.md

@@ -4,6 +4,7 @@ queries against Wikipedia and sister projects databases.
 
 ## Setting up a local dev environment ##
 
+# docker-compose
 Quarry uses [Docker](https://docs.docker.com/engine/install/) to set up a local
 environment. You can set it up by:
 
@@ -18,19 +19,43 @@ will imediatelly be taken into account.
 A worker node is also created to execute your queries in the background (uses the
 same image). Finally, redis and two database instances are also started.
 
-One database is your quarry database the other is a wikireplica-like database
-named `mywiki`. This (or `mywiki_p`) is the correct thing to enter in the
-database field on all local test queries.
-
-In your local environment, you can query Quarry internal db itself. Use then
-"quarry" as database name.
-
 To stop, run `docker-compose stop` or hit CTRL-C on the terminal your docker-compose
 is running in. After that, to start with code changes, you'll want to `docker-compose down`
 to clean up. Also, this creates a docker volume where sqlite versions of query
 results are found. That will not be cleaned up unless you run `docker-compose down -v`
 
 
+
+# minikube
+It is possible to run a quarry system inside [minikube](https://minikube.sigs.k8s.io/docs/)!
+At this time, you need to set it up with a cluster version before 1.22, most likely.
+
+First build the containers:
+```
+eval $(minikube docker-env)
+docker build . -t quarry:01
+cd docker-replica/
+docker build . -t mywiki:01
+```
+
+You will need to install minikube (tested on minikube 1.23) and [helm](https://helm.sh) and kubectl on your system. When you are confident those are working, start minikube with:
+ - `minikube start --kubernetes-version=v1.23.15`
+ - `minikube addons enable ingress`
+ - `kubectl create namespace quarry`
+ - `helm -n quarry install quarry helm-quarry -f helm-quarry/dev-env.yaml`
+
+The rest of the setup instructions will display on screen as long as the install is successful.
+
+# local databases
+Both local setups will create two databases.
+
+One database is your quarry database the other is a wikireplica-like database
+named `mywiki`. This (or `mywiki_p`) is the correct thing to enter in the
+database field on all local test queries.
+
+The other database is the Quarry internal db. In your local environment, you can query Quarry internal db itself. Use then
+"quarry" as database name.
+
 ### Updating existing containers ###
 
 If you had already run a dev environment (that is, ran `docker-compose up`) you might want to update

docker-replica/Dockerfile

@@ -0,0 +1,3 @@
+FROM mariadb:10.4-focal
+
+COPY replica.sql /docker-entrypoint-initdb.d/replica.sql

helm-quarry/.helmignore

@@ -0,0 +1,23 @@
+# Patterns to ignore when building packages.
+# This supports shell glob matching, relative path matching, and
+# negation (prefixed with !). Only one pattern per line.
+.DS_Store
+# Common VCS dirs
+.git/
+.gitignore
+.bzr/
+.bzrignore
+.hg/
+.hgignore
+.svn/
+# Common backup files
+*.swp
+*.bak
+*.tmp
+*.orig
+*~
+# Various IDEs
+.project
+.idea/
+*.tmproj
+.vscode/

helm-quarry/Chart.yaml

@@ -0,0 +1,4 @@
+apiVersion: v2
+description: A Helm chart for quarry
+name: helm-quarry
+version: "1.0"

helm-quarry/dev-env.yaml

@@ -0,0 +1,2 @@
+localdev:
+  enabled: true

helm-quarry/dev_config.yaml

@@ -0,0 +1,31 @@
+# The OAUTH tokens are callbacks to a quarry.local domain
+# it is ok that they are public.
+OAUTH_CONSUMER_TOKEN: "deeb8dbabbf157cb0ed120b89971d800"
+OAUTH_SECRET_TOKEN: "38fdce038bbf848453a856aa329f8a06c456c791"
+SECRET_KEY: 'aaarggghhpleaserescuemeiamstuckinarandomnumbergeneratorfactory'
+DEBUG: True
+DB_HOST: 'db'
+DB_NAME: 'quarry'
+DB_USER: 'quarry'
+DB_PASSWORD: 'quarry'
+broker_url: 'redis://redis'
+result_backend: 'redis://redis'
+worker_concurrency: 24  # Since all tasks are IO bound
+task_acks_late: True  # Tasks are idempotent!
+task_track_started: True
+worker_prefetch_multiplier: 1 # Tasks can run for a long time
+# Just query the quarry database itself.
+REPLICA_DOMAIN: ''
+REPLICA_HOST: 'mywiki'
+REPLICA_DB: 'mywiki_p'
+REPLICA_USER: 'repl'
+REPLICA_PASSWORD: 'repl'
+REPLICA_PORT: 3306
+OUTPUT_PATH_TEMPLATE: '/results/%s/%s/%s.sqlite'
+REDIS_HOST: 'redis'
+REDIS_PORT: 6379
+REDIS_DB: 0
+QUERY_TIME_LIMIT: 60  # 1 minute
+QUERY_RESULTS_PER_PAGE: 50
+KILLER_LOG_PATH: 'killer.log'
+MAINTENANCE_MSG: 'This is your local development environment.'

helm-quarry/prod-env.yaml

@@ -0,0 +1,2 @@
+localdev:
+  enabled: false

helm-quarry/prod_config.yaml

@@ -0,0 +1,33 @@
+## Replace these with prod values and encrypt
+
+# The OAUTH tokens are callbacks to a quarry.local domain
+# it is ok that they are public.
+OAUTH_CONSUMER_TOKEN: "deeb8dbabbf157cb0ed120b89971d800"
+OAUTH_SECRET_TOKEN: "38fdce038bbf848453a856aa329f8a06c456c791"
+SECRET_KEY: 'aaarggghhpleaserescuemeiamstuckinarandomnumbergeneratorfactory'
+DEBUG: True
+DB_HOST: 'db'
+DB_NAME: 'quarry'
+DB_USER: 'quarry'
+DB_PASSWORD: 'quarry'
+broker_url: 'redis://redis'
+result_backend: 'redis://redis'
+worker_concurrency: 24  # Since all tasks are IO bound
+task_acks_late: True  # Tasks are idempotent!
+task_track_started: True
+worker_prefetch_multiplier: 1 # Tasks can run for a long time
+# Just query the quarry database itself.
+REPLICA_DOMAIN: ''
+REPLICA_HOST: 'mywiki'
+REPLICA_DB: 'mywiki_p'
+REPLICA_USER: 'repl'
+REPLICA_PASSWORD: 'repl'
+REPLICA_PORT: 3306
+OUTPUT_PATH_TEMPLATE: '/results/%s/%s/%s.sqlite'
+REDIS_HOST: 'redis'
+REDIS_PORT: 6379
+REDIS_DB: 0
+QUERY_TIME_LIMIT: 60  # 1 minute
+QUERY_RESULTS_PER_PAGE: 50
+KILLER_LOG_PATH: 'killer.log'
+MAINTENANCE_MSG: 'This is your local development environment.'

helm-quarry/templates/NOTES.txt

@@ -0,0 +1,14 @@
+Thank you for installing {{ .Chart.Name }}!
+
+An essential part of setting up your dev environment is adding a hosts file entry.
+Get the IP of minikube with:
+minikube ip
+Add this line to your hosts file:
+<ip address> quarry.local
+
+You can override any values you need in a file called dev-values.yaml.
+
+If you create one, run:
+helm -n {{ .Release.Namespace }} upgrade {{ .Release.Name }} helm-quarry/ -f dev-values.yaml
+Happy hacking.
+

helm-quarry/templates/config_yaml.yaml

@@ -0,0 +1,10 @@
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: config
+data:
+{{ if .Values.localdev.enabled }}
+  config.yaml: {{ .Files.Get "dev_config.yaml" | quote }}
+{{ else }}
+  config.yaml: {{ .Files.Get "prod_config.yaml" | quote }}
+{{ end }}

helm-quarry/templates/db_deployment.yaml

@@ -0,0 +1,36 @@
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: db
+  labels:
+    app: db
+spec:
+  replicas: 1
+  selector:
+    matchLabels:
+      app: db
+  template:
+    metadata:
+      labels:
+        app: db
+    spec:
+      containers:
+        - name: db
+          image: mariadb:10.1.48-bionic
+          imagePullPolicy: Always
+          env:
+          - name: MYSQL_USER
+            value: "quarry"
+          - name: MYSQL_PASSWORD
+            value: "quarry"
+          - name: MYSQL_DATABASE
+            value: "quarry"
+          - name: MYSQL_RANDOM_ROOT_PASSWORD
+            value: "1"
+          volumeMounts:
+          - name: schema
+            mountPath: /docker-entrypoint-initdb.d/
+      volumes:
+        - name: schema
+          configMap:
+            name: db-schema

helm-quarry/templates/db_schema_cm.yaml

@@ -0,0 +1,68 @@
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: db-schema
+data:
+  schema.sql: |
+    CREATE DATABASE IF NOT EXISTS quarry CHARACTER SET utf8;
+    USE quarry;
+    CREATE TABLE IF NOT EXISTS user(
+        id INT UNSIGNED NOT NULL PRIMARY KEY AUTO_INCREMENT,
+        username VARCHAR(255) BINARY NOT NULL UNIQUE,
+        wiki_uid INT UNSIGNED NOT NULL UNIQUE
+    );
+    CREATE UNIQUE INDEX IF NOT EXISTS user_username_index ON user( username);
+    CREATE UNIQUE INDEX IF NOT EXISTS user_wiki_uid ON user(wiki_uid);
+    
+    CREATE TABLE IF NOT EXISTS user_group(
+        id INT UNSIGNED NOT NULL PRIMARY KEY AUTO_INCREMENT,
+        user_id INT UNSIGNED NOT NULL,
+        group_name VARCHAR(255) BINARY NOT NULL
+    );
+    CREATE INDEX IF NOT EXISTS user_group_user_group_index ON user_group(user_id, group_name);
+    CREATE INDEX IF NOT EXISTS user_group_user_id_index ON user_group(user_id);
+    CREATE INDEX IF NOT EXISTS user_group_group_name_index ON user_group(group_name);
+    
+    CREATE TABLE IF NOT EXISTS query(
+        id INT UNSIGNED NOT NULL PRIMARY KEY AUTO_INCREMENT,
+        user_id INT UNSIGNED NOT NULL,
+        title VARCHAR(1024) BINARY,
+        latest_rev_id INT UNSIGNED,
+        last_touched TIMESTAMP DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
+        published SMALLINT DEFAULT 0 NOT NULL,
+        description TEXT BINARY,
+        parent_id INT UNSIGNED
+    );
+    CREATE INDEX IF NOT EXISTS query_parent_id_index ON query(parent_id);
+    
+    CREATE TABLE IF NOT EXISTS query_revision(
+        id INT UNSIGNED NOT NULL PRIMARY KEY AUTO_INCREMENT,
+        text TEXT BINARY NOT NULL,
+        query_database VARCHAR(1024) BINARY,
+        query_id INT UNSIGNED NOT NULL,
+        latest_run_id INT UNSIGNED,
+        timestamp TIMESTAMP DEFAULT CURRENT_TIMESTAMP
+    );
+    CREATE INDEX IF NOT EXISTS query_rev_query_id_index ON query_revision(query_id);
+    CREATE INDEX IF NOT EXISTS query_rev_query_database_index ON query_revision(query_database);
+    
+    CREATE TABLE IF NOT EXISTS query_run(
+        id INT UNSIGNED NOT NULL PRIMARY KEY AUTO_INCREMENT,
+        query_rev_id INT UNSIGNED NOT NULL,
+        status TINYINT UNSIGNED NOT NULL DEFAULT 0,
+        timestamp TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
+        task_id VARCHAR(36) BINARY,
+        extra_info TEXT BINARY
+    );
+    CREATE INDEX IF NOT EXISTS query_run_status_index ON query_run(status);
+    
+    CREATE TABLE IF NOT EXISTS star(
+        id INT UNSIGNED NOT NULL PRIMARY KEY AUTO_INCREMENT,
+        user_id INT UNSIGNED NOT NULL,
+        query_id INT UNSIGNED NOT NULL,
+        timestamp TIMESTAMP DEFAULT CURRENT_TIMESTAMP
+    );
+    CREATE INDEX IF NOT EXISTS star_user_id_index ON star(user_id);
+    CREATE INDEX IF NOT EXISTS star_query_id_index ON star(query_id);
+    CREATE UNIQUE INDEX IF NOT EXISTS star_user_query_index ON star(user_id, query_id);
+

helm-quarry/templates/db_service.yaml

@@ -0,0 +1,11 @@
+apiVersion: v1
+kind: Service
+metadata:
+  name: db
+spec:
+  selector:
+    app: db
+  ports:
+  - name: db
+    port: 3306
+    targetPort: 3306

helm-quarry/templates/mywiki_deployment.yaml

@@ -0,0 +1,29 @@
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: mywiki
+  labels:
+    app: mywiki
+spec:
+  replicas: 1
+  selector:
+    matchLabels:
+      app: mywiki
+  template:
+    metadata:
+      labels:
+        app: mywiki
+    spec:
+      containers:
+        - name: mywiki
+          image: mywiki:01 # replace with quay.io
+          imagePullPolicy: Never
+          env:
+          - name: MYSQL_USER
+            value: "repl"
+          - name: MYSQL_PASSWORD
+            value: "repl"
+          - name: MYSQL_DATABASE
+            value: "repl"
+          - name: MYSQL_RANDOM_ROOT_PASSWORD
+            value: "1"

helm-quarry/templates/mywiki_service.yaml

@@ -0,0 +1,11 @@
+apiVersion: v1
+kind: Service
+metadata:
+  name: mywiki
+spec:
+  selector:
+    app: mywiki
+  ports:
+  - name: mywiki
+    port: 3306
+    targetPort: 3306

helm-quarry/templates/redis_deployment.yaml

@@ -0,0 +1,20 @@
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: redis
+  labels:
+    app: redis
+spec:
+  replicas: 1
+  selector:
+    matchLabels:
+      app: redis
+  template:
+    metadata:
+      labels:
+        app: redis
+    spec:
+      containers:
+        - name: redis
+          image: redis:alpine
+          imagePullPolicy: Always

helm-quarry/templates/redis_service.yaml

@@ -0,0 +1,11 @@
+apiVersion: v1
+kind: Service
+metadata:
+  name: redis
+spec:
+  selector:
+    app: redis
+  ports:
+  - name: redis
+    port: 6379
+    targetPort: 6379

helm-quarry/templates/results_pv.yaml

@@ -0,0 +1,12 @@
+apiVersion: v1
+kind: PersistentVolume
+metadata:
+  name: results
+spec:
+  storageClassName: manual
+  capacity:
+    storage: 1Gi
+  accessModes:
+    - ReadWriteMany
+  hostPath:
+    path: "/results"