Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

2592/separate celery #2773

Open
wants to merge 46 commits into
base: develop
Choose a base branch
from
Open

2592/separate celery #2773

Show file tree
Hide file tree
Changes from 37 commits
Commits
Show all changes
46 commits
Select commit Hold shift + click to select a range
9c36091
WIP updated terraform and started celery setup.
Nov 30, 2023
fc082b5
WIP
Nov 30, 2023
f14c946
creating redis services, separate celery instance, and new connection…
Dec 7, 2023
0f6bc5c
Merge branch 'develop' into 2592/separate-celery
Dec 7, 2023
b0d581e
update pipeline and deploy script logic to deploy celery and make mor…
Dec 8, 2023
5a99192
removed deploy strategy command from pipeline since it's now in scripts
Dec 8, 2023
9e4e926
fixed spacing in variables.tf
Dec 8, 2023
a8910f6
fixed spacing in variables.tf
Dec 8, 2023
332525c
fixed spacing in main.tf
Dec 8, 2023
24a4ba6
fixed syntax errors
Dec 11, 2023
6b336ec
updated logic for deploy backend
Dec 12, 2023
535bc30
cleaned up spaces, etc.
Dec 12, 2023
59e0c8b
cleaned up more spaces, format of if statements, and indentation etc.
Dec 13, 2023
56f9eeb
made celery_start.sh executable
Dec 18, 2023
38990b0
set env vars to celery as well
Dec 18, 2023
aca5d8f
added cf bind-services for staticfiles (fixed typo).
Dec 18, 2023
4a2e4d9
removed REDIS_URI from ENV list because we should be grabbing from VC…
Dec 18, 2023
6495c78
added debug info
Dec 19, 2023
fa97c70
trying out unsetting REDIS_URI
Dec 20, 2023
849f42e
trying to hardcode REDIS_URI in start script
Dec 20, 2023
c672ef1
removing unset REDIS_URI commands and hardcoded redis
Dec 20, 2023
55aa406
setting last celery task to not be backgrounded.
Dec 22, 2023
606e60c
no route needed or port health checking for celery worker.
Dec 26, 2023
90fb827
Merge branch 'develop' into 2592/separate-celery
Dec 29, 2023
387f91b
Fixed misnamed variables
Jan 2, 2024
57e91b3
added backend appname to env for celery and backend to share during d…
Jan 3, 2024
2e8fa9f
put quotes around env vars
Jan 3, 2024
71552f4
pass env backend name before and after.
Jan 3, 2024
663f761
don't need to serialize a basic string for appname
Jan 3, 2024
c21a04e
added some documentation for Celery and Redis.
Jan 4, 2024
f10e2f5
updated docker compose files to create celery to mimic new celery ins…
Jan 4, 2024
673a338
remove .env requirement for circleci celery build
Jan 4, 2024
ae02550
fix typo"
Jan 4, 2024
7c1015f
import settings
Jan 4, 2024
1d08d18
import settings
Jan 5, 2024
c091a2c
linting
Jan 5, 2024
83f49f8
Merge branch 'develop' into 2592/separate-celery
Jan 5, 2024
4e13d1a
fixed typo, removed unneeded comments
Jan 9, 2024
6f32657
use 3 concurrent celery workers on the app
Jan 10, 2024
94c59be
Merge branch 'develop' into 2592/separate-celery
Jan 11, 2024
ed66a88
allowing routing to port 8080 and configuring flower to run on 8080 i…
Jan 11, 2024
e88d422
map route for celery
Jan 11, 2024
7f8833a
wrapped map routing for celery to non-prod space
Jan 11, 2024
ff069d2
updated tf for staging and prod spaces
Jan 12, 2024
388411a
Merge branch 'develop' into 2592/separate-celery
jtimpe Jan 29, 2024
10cadcd
one redis per space
jtimpe Jan 30, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
73 changes: 11 additions & 62 deletions .circleci/deployment/commands.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,12 +2,7 @@
deploy-cloud-dot-gov:
parameters:
environment:
description: The environment to deploy to.
type: enum
enum: [ "development", "production" ]
default: development
backend-appname:
default: tdp-backend
default: raft
type: string
cf-password:
default: CF_PASSWORD_DEV
Expand All @@ -21,9 +16,6 @@
cf-username:
default: CF_USERNAME_DEV
type: env_var_name
frontend-appname:
default: tdp-frontend
type: string
steps:
- checkout
- sudo-check
Expand All @@ -34,14 +26,11 @@
cf-space: <<parameters.cf-space>>
cf-username: <<parameters.cf-username>>
- deploy-backend:
backend-appname: <<parameters.backend-appname>>
frontend-appname: <<parameters.frontend-appname>>
cf-space: <<parameters.cf-space>>
- deploy-frontend:
environment: <<parameters.environment>>
backend-appname: <<parameters.backend-appname>>
frontend-appname: <<parameters.frontend-appname>>
- deploy-frontend:
cf-space: <<parameters.cf-space>>
environment: <<parameters.environment>>

clamav-cloud-dot-gov:
parameters:
Expand Down Expand Up @@ -73,26 +62,19 @@

deploy-backend:
parameters:
backend-appname:
default: tdp-backend
type: string
frontend-appname:
default: tdp-frontend
type: string
cf-space:
default: tanf-dev
type: string
environment:
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

does this mean it only deploys for raft? or it's only the default

default: raft
type: string
steps:
- get-app-deploy-strategy:
appname: <<parameters.backend-appname>>
- run:
name: Deploy backend application
command: |
bash ./scripts/deploy-backend.sh \
$DEPLOY_STRATEGY \
<<parameters.frontend-appname>> \
<<parameters.backend-appname>> \
<<parameters.cf-space>>
<<parameters.cf-space>> \
<<parameters.environment>>

deploy-clamav:
steps:
Expand All @@ -104,58 +86,25 @@

deploy-frontend:
parameters:
environment:
description: The environment to deploy to.
type: enum
enum: [ "development", "production" ]
default: development
backend-appname:
default: tdp-backend
type: string
frontend-appname:
default: tdp-frontend
type: string
# So the frontend knows what space its in for the banner.
# I am unclear if the domain is a reliable metric to make this function
# It seems like it might not be working
cf-space:
type: string
default: dev
environment:
type: string
default: raft
steps:
- install-nodejs:
node-version: "16.13"
- disable-npm-audit
- install-nodejs-packages:
app-dir: tdrs-frontend
- get-app-deploy-strategy:
appname: <<parameters.frontend-appname>>
- run:
name: Deploy frontend application
command: |
bash ./scripts/deploy-frontend.sh \
$DEPLOY_STRATEGY \
<<parameters.frontend-appname>> \
<<parameters.backend-appname>> \
<<parameters.cf-space>> \
<<parameters.environment>>

get-app-deploy-strategy:
parameters:
appname:
type: string
steps:
- run:
name: Determine deploy strategy
command: |
# NOTE: The || true is a no-op included to suppress exit codes which
# would cause the step to exit early due to use of pipefail
APP_GUID=$(cf app <<parameters.appname>> --guid || true)
if [ "$APP_GUID" == "FAILED" ]; then
echo "export DEPLOY_STRATEGY=initial" >> $BASH_ENV
else
echo "export DEPLOY_STRATEGY=rolling" >> $BASH_ENV
fi

deploy-infrastructure:
parameters:
tf-path:
Expand Down
13 changes: 4 additions & 9 deletions .circleci/deployment/jobs.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,16 +7,14 @@
working_directory: ~/tdp-deploy
steps:
- deploy-cloud-dot-gov:
backend-appname: tdp-backend-<< parameters.target_env >>
frontend-appname: tdp-frontend-<< parameters.target_env >>
environment: << parameters.target_env >>

deploy-staging:
executor: docker-executor
working_directory: ~/tdp-deploy
steps:
- deploy-cloud-dot-gov:
backend-appname: tdp-backend-staging
frontend-appname: tdp-frontend-staging
environment: staging
cf-password: CF_PASSWORD_STAGING
cf-space: tanf-staging
cf-username: CF_USERNAME_STAGING
Expand All @@ -26,8 +24,7 @@
working_directory: ~/tdp-deploy
steps:
- deploy-cloud-dot-gov:
backend-appname: tdp-backend-develop
frontend-appname: tdp-frontend-develop
environment: develop
cf-password: CF_PASSWORD_STAGING
cf-space: tanf-staging
cf-username: CF_USERNAME_STAGING
Expand Down Expand Up @@ -145,9 +142,7 @@
working_directory: ~/tdp-deploy
steps:
- deploy-cloud-dot-gov:
environment: production
backend-appname: tdp-backend-prod
frontend-appname: tdp-frontend-prod
environment: prod
cf-password: CF_PASSWORD_PROD
cf-space: tanf-prod
cf-username: CF_USERNAME_PROD
Expand Down
11 changes: 11 additions & 0 deletions docs/Technical-Documentation/redis-and-celery.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
# Redis Service and Celery Instance

We use a CloudFoundry Redis service and a separate instance to run celery to run background processes like parsing data-file documents that have been submitted and have passed the [ClamAV scan](./clamav.md).

## Redis Deployment

As part of the move towards each environment being self-contained, one redis service is created per environment, deployed through the [CircleCI pipeline](./circle-ci.md), defined using [terraform](../../terraform/README.md).

## Celery Deployment

Celery is deployed at the same time as the backend through the [CircleCI pipeline](./circle-ci.md), with the details configured in the [celery manifest](../../tdrs-backend/manifest.celery.yml) and the [deploy-backend script](../../scripts/deploy-backend.sh)
Loading
Loading