Proof of Concept TDP Based Kibana Auth (#2775)

* Added formating for header and autofit columns * Formatted the headers * added year/month to the columns * Added contants - translation column * added friendly names to T1 and T2 * added friendly name to m1 and m2 * added friendly name to m3 * added friendly_name to t3 * added friendly_name to t4 and t5 * added friendly_name to t7 * correct missing friendly_name * correction on failing tests * addedfriendly name to excel report * linting * linting * linting * delete contants.py * added test for json field in error model * linting * linting * linting * 2599-added friendly name to postparsing validators * refining the validator tests * added returning fields names to validators * added friendly_name to error field * linting * corrections on views/tests * corrections for fields * failing test corrected * failing test corrected * correcting test failures * linting * corrected the excel fiel generator * removed excessive space in validator * linting * listing * added m6 * lint * corrected new line break * refactored validator logic * linting and correction on t1 * friendly_name correction from comments * friendly_name correction * corrected failing test for m5 * refactor the field_json creation DRY * - Added Kibana config * friendly_name corrections * linting and cleaning errors * linting * correction on friendly_names * corrected friendly_name for test_util * correction child care - number of months * fixed a few more typos and some spacing. (#2767) * fixed a few more typos and some spacing. * fixed linting issues * missed a spot. --------- Co-authored-by: George Hudson <ghudson@teamraft.com> * - Added basic security to Kibana/Elastic - Added setup container to init elastic users, roles, and passwords * - Remove debug code * - change provider name * - Updating settings to reference environment variables * - Add elastic dependency * - Fix network issue * - Added bulk creation of elastic indices * - Updated schemas to reference model based off of elastic document * - Remove password auth from elastic/kibana * - Remove password auth * - Fix tests * - Fix lint * - remove debug print * Changes for fully local development - Enables direct frontend/backend communication sans Login.gov/Cloud.gov - Drives off new DEVELOPMENT env var - Pre-configures and disables frontend auth functionality - Testing based on new dev user - Install via web: ./manage.py generate_dev_user * Reorganized front end logic on REACT_APP_DEVAUTH env var * Reorganized backend logic on REACT_APP_DEVAUTH env var * - Added proof on concept for tdp based kibana auth * - Fixing type issue * added is_superuser and is_staff attrs to dev user * - Add group check * - Add frontend group check for kibana * - fix lint * - Fix lint errors * - Fix doc strings * - Adding authenticated permission * - Renaming variables to clarify things * - fix lint * Revert "- Remove password auth from elastic/kibana" This reverts commit 522ca38. * - Setting up anonymous users with kibana_admin privileges * - Adding password to settings in cloud.gov * - remove incorrect auth - use admin only in frontend and backend * - Add elastic profile * DevAuth feature redesign inspired by Cypress - Initializing frontend w/POST /login/cypress: {devEmail, local-cypress-token} - Changed REACT_APP_DEVAUTH to provide the email of the desired dev user - Modified CustomAuthentication.authenticate to handle both known use cases - Added stt_id=31 to the initial dev user - Disabled ES disk threshold checking for local dev which blocked ES startup - Removed DevAuthentication and other now unnecessary code * Fixed CustomAuthentication.authenticate return val for login.py use case * Fixed CustomAuthentication.authenticate logging for login.py use case * Removed unneeded permissions import * Updates to REACT_APP_DEVAUTH env var settings - Enabled with an email address value - Disabled by default * - debugging env vars * - Testing what settings are used * Revert "- debugging env vars" This reverts commit 900efa8. * Revert "- Testing what settings are used" This reverts commit 784530e. * - debugging env vars again * - Switching to container networking * Restored support for CustomAuthentication.authenticate username keyword * Modified CustomAuthentication.authenticate comment to satisfy flake8 * commit * asdfgvasd * Revert "Modified CustomAuthentication.authenticate comment to satisfy flake8" This reverts commit 761e4eb. * Revert "Restored support for CustomAuthentication.authenticate username keyword" This reverts commit 4bf8957. * Revert "Updates to REACT_APP_DEVAUTH env var settings" This reverts commit 7fc2a09. * Revert "Removed unneeded permissions import" This reverts commit c18383f. * Revert "Fixed CustomAuthentication.authenticate logging for login.py use case" This reverts commit 2b9b46f. * Revert "Fixed CustomAuthentication.authenticate return val for login.py use case" This reverts commit 97a0cf6. * Revert "DevAuth feature redesign inspired by Cypress" This reverts commit 1497d4a. * Revert "commit" This reverts commit a284856. * Revert "added is_superuser and is_staff attrs to dev user" This reverts commit 6ffbee8. * Revert "Reorganized backend logic on REACT_APP_DEVAUTH env var" This reverts commit 7fd7b4d. * Revert "Reorganized front end logic on REACT_APP_DEVAUTH env var" This reverts commit 32a4671. * Revert "Changes for fully local development" This reverts commit 556221b. * asdf * - Adding integration tests for elastic bulk doc creation * Revert "asdf" This reverts commit 26455b4. * - fix lint * fasdf * - Added usage of document to tribal * - Updated based on feedback * - Fixing error * - Updating frontend to only allow access to kibana sitemap if the user is Dev or Sys Admin * - fix lint --------- Co-authored-by: Mo Sohani <msohani@goraft.tech> Co-authored-by: raftmsohani <97037188+raftmsohani@users.noreply.github.com> Co-authored-by: George Hudson <georgehudson78@gmail.com> Co-authored-by: George Hudson <ghudson@teamraft.com> Co-authored-by: Thomas Tignor <thomas.tignor@QP9VN4FgnorRaft.fios-router.home> Co-authored-by: Thomas Tignor <thomas.tignor@QP9VN4F4RH-thomastignor-Raft.local> Co-authored-by: Andrew <84722778+andrew-jameson@users.noreply.github.com>
raft-tech · Feb 12, 2024 · c9c0c74 · c9c0c74
1 parent eecad70
commit c9c0c74
Show file tree

Hide file tree

Showing 15 changed files with 467 additions and 9 deletions.
diff --git a/.circleci/build-and-test/jobs.yml b/.circleci/build-and-test/jobs.yml
@@ -4,7 +4,7 @@
     steps:
       - checkout
       - docker-compose-check
-      - docker-compose-up-backend
+      - docker-compose-up-with-elastic-backend
       - run:
           name: Run Unit Tests And Create Code Coverage Report
           command: |
@@ -47,7 +47,7 @@
     steps:
       - checkout
       - docker-compose-check
-      - docker-compose-up-backend
+      - docker-compose-up-with-elastic-backend
       - docker-compose-up-frontend
       - install-nodejs-machine
       - disable-npm-audit
@@ -61,7 +61,7 @@
                wait-for-it --service http://web:8080 --timeout 180 -- echo \"Django is ready\""
       - run:
           name: apply the migrations
-          command: cd tdrs-backend; docker-compose exec web bash -c "python manage.py makemigrations; python manage.py migrate" 
+          command: cd tdrs-backend; docker-compose exec web bash -c "python manage.py makemigrations; python manage.py migrate"
       - run:
           name: Remove existing cypress test users
           command: cd tdrs-backend; docker-compose exec web python manage.py delete_cypress_users -usernames new-cypress@teamraft.com cypress-admin@teamraft.com

diff --git a/.circleci/util/commands.yml b/.circleci/util/commands.yml
@@ -11,6 +11,12 @@
           name: Build and spin-up Django API service
           command: cd tdrs-backend; docker network create external-net; docker-compose up -d --build
 
+  docker-compose-up-with-elastic-backend:
+    steps:
+      - run:
+          name: Build and spin-up Django API service
+          command: cd tdrs-backend; docker network create external-net; docker-compose --profile elastic_setup up -d --build
+
   cf-check:
     steps:
       - run:

diff --git a/tdrs-backend/docker-compose.yml b/tdrs-backend/docker-compose.yml
@@ -50,20 +50,50 @@ services:
     ports:
       - 5601:5601
     environment:
-      - xpack.security.encryptionKey="something_at_least_32_characters"
+      - xpack.security.encryptionKey=${KIBANA_ENCRYPTION_KEY:-something_at_least_32_characters}
       - xpack.security.session.idleTimeout="1h"
       - xpack.security.session.lifespan="30d"
     volumes:
       - ./kibana.yml:/usr/share/kibana/config/kibana.yml
     depends_on:
       - elastic
 
+  # This task only needs to be performed once, during the *initial* startup of
+  # the stack. Any subsequent run will reset the passwords of existing users to
+  # the values defined inside the '.env' file, and the built-in roles to their
+  # default permissions.
+  #
+  # By default, it is excluded from the services started by 'docker compose up'
+  # due to the non-default profile it belongs to. To run it, either provide the
+  # '--profile=elastic_setup' CLI flag to Compose commands, or "up" the service by name
+  # such as 'docker compose up elastic_setup'.
+  elastic_setup:
+    profiles:
+      - elastic_setup
+    build:
+      context: elastic_setup/
+      args:
+        ELASTIC_VERSION: "7.17.6"
+    init: true
+    environment:
+      ELASTIC_PASSWORD: ${ELASTIC_PASSWORD:-changeme}
+      KIBANA_SYSTEM_PASSWORD: ${KIBANA_SYSTEM_PASSWORD:-changeme}
+      OFA_ADMIN_PASSWORD: ${OFA_ADMIN_PASSWORD:-changeme}
+      ELASTICSEARCH_HOST: ${ELASTICSEARCH_HOST:-elastic}
+    depends_on:
+      - elastic
+
   elastic:
     image: elasticsearch:7.17.6
     environment:
       - discovery.type=single-node
       - logger.discovery.level=debug
-      - xpack.security.enabled=false
+      - xpack.security.enabled=true
+      - xpack.security.authc.anonymous.username="ofa_admin"
+      - xpack.security.authc.anonymous.roles="ofa_admin"
+      - xpack.security.authc.anonymous.authz_exception=true
+      - ELASTIC_PASSWORD=${ELASTIC_PASSWORD:-changeme}
+      - KIBANA_SYSTEM_PASSWORD=${KIBANA_SYSTEM_PASSWORD:-changeme}
     ports:
       - 9200:9200
       - 9300:9300
@@ -101,6 +131,7 @@ services:
       - CYPRESS_TOKEN
       - DJANGO_DEBUG
       - SENDGRID_API_KEY
+      - BYPASS_KIBANA_AUTH
     volumes:
       - .:/tdpapp
     image: tdp

diff --git a/tdrs-backend/elastic_setup/Dockerfile b/tdrs-backend/elastic_setup/Dockerfile
@@ -0,0 +1,10 @@
+ARG ELASTIC_VERSION
+
+FROM docker.elastic.co/elasticsearch/elasticsearch:${ELASTIC_VERSION}
+
+COPY . /
+
+RUN ["chmod", "+x", "/entrypoint.sh"]
+RUN ["chmod", "+x", "/util.sh"]
+
+ENTRYPOINT ["/entrypoint.sh"]
diff --git a/tdrs-backend/elastic_setup/entrypoint.sh b/tdrs-backend/elastic_setup/entrypoint.sh
@@ -0,0 +1,110 @@
+#!/usr/bin/env bash
+
+set -eu
+set -o pipefail
+
+source "${BASH_SOURCE[0]%/*}"/util.sh
+
+
+# --------------------------------------------------------
+# Users declarations
+
+declare -A users_passwords
+users_passwords=(
+	[kibana_system]="${KIBANA_SYSTEM_PASSWORD:-}"
+	[ofa_admin]="${OFA_ADMIN_PASSWORD:-}"
+)
+
+declare -A users_roles
+users_roles=(
+	[kibana_system]='kibana_system'
+    [ofa_admin]='kibana_admin'
+)
+
+# --------------------------------------------------------
+# Roles declarations for custom roles
+
+declare -A roles_files
+roles_files=(
+
+)
+
+# --------------------------------------------------------
+
+
+log 'Waiting for availability of Elasticsearch. This can take several minutes.'
+
+declare -i exit_code=0
+wait_for_elasticsearch || exit_code=$?
+
+if ((exit_code)); then
+	case $exit_code in
+		6)
+			suberr 'Could not resolve host. Is Elasticsearch running?'
+			;;
+		7)
+			suberr 'Failed to connect to host. Is Elasticsearch healthy?'
+			;;
+		28)
+			suberr 'Timeout connecting to host. Is Elasticsearch healthy?'
+			;;
+		*)
+			suberr "Connection to Elasticsearch failed. Exit code: ${exit_code}"
+			;;
+	esac
+
+	exit $exit_code
+fi
+
+sublog 'Elasticsearch is running'
+
+log 'Waiting for initialization of built-in users'
+
+wait_for_builtin_users || exit_code=$?
+
+if ((exit_code)); then
+	suberr 'Timed out waiting for condition'
+	exit $exit_code
+fi
+
+sublog 'Built-in users were initialized'
+
+for role in "${!roles_files[@]}"; do
+	log "Role '$role'"
+
+	declare body_file
+	body_file="${BASH_SOURCE[0]%/*}/roles/${roles_files[$role]:-}"
+	if [[ ! -f "${body_file:-}" ]]; then
+		sublog "No role body found at '${body_file}', skipping"
+		continue
+	fi
+
+	sublog 'Creating/updating'
+	ensure_role "$role" "$(<"${body_file}")"
+done
+
+for user in "${!users_passwords[@]}"; do
+	log "User '$user'"
+	if [[ -z "${users_passwords[$user]:-}" ]]; then
+		sublog 'No password defined, skipping'
+		continue
+	fi
+
+	declare -i user_exists=0
+	user_exists="$(check_user_exists "$user")"
+
+	if ((user_exists)); then
+		sublog 'User exists, setting password'
+		set_user_password "$user" "${users_passwords[$user]}"
+	else
+		if [[ -z "${users_roles[$user]:-}" ]]; then
+			suberr '  No role defined, skipping creation'
+			continue
+		fi
+
+		sublog 'User does not exist, creating'
+		create_user "$user" "${users_passwords[$user]}" "${users_roles[$user]}"
+	fi
+done
+
+log "Elastic setup completed. Exiting with code: $?"