Deploy celery as a separate cloud.gov application

[jtimpe]

Per #2347 (comment)

• Deploy celery as a separate cloud.gov application (separate manifest)
  • independently scalable
  • requires separate redis service - draft pr [DRAFT] Connect to managed redis service #2590

Description:

Deploying celery as a separate application alleviates the application server and allows each to be independently scalable. This (roughly) requires:

1. Deploy redis as a separate service

   1. See draft pr allowing django/celery to communicate with an external redis service as a broker.
   2. Remove redis-related lines from gunicorn_start.sh

      set -e
      
      echo "REDIS_SERVER"
      echo "redis local: $REDIS_SERVER_LOCAL"
      if [[ "$REDIS_SERVER_LOCAL" = "TRUE" || "$CIRCLE_JOB" = "backend-owasp-scan" ]]; then
          echo "Run redis server on docker"
      else
          echo "Run redis server locally"
          export LD_LIBRARY_PATH=/home/vcap/deps/0/lib/:/home/vcap/deps/1/lib:$LD_LIBRARY_PATH
          ( cd  /home/vcap/deps/0/bin/; ./redis-server /home/vcap/app/redis.conf &)
      fi

   3. Remove redis from deployed application dependencies apt.yml
   4. Remove REDIS_URI env manifest.buildpack.yml
   5. Deploy redis in cloud.gov and bind to application deployment using circleci
      • Add redis service binding in deploy-backend.sh

2. Deploy celery as separate cloud.gov application, with its own manifest

   1. Make a copy manifest.buildpack.yml (named manifest.celery.yml or similar)
   2. Update the command in the new manifest to run a script starting celery, removing these lines from gunicorn_start.sh

      celery -A tdpservice.settings worker -c 1 &
      sleep 5
      # TODO: Uncomment the following line to add flower service when memory limitation is resolved
      celery -A tdpservice.settings --broker=$REDIS_URI flower &
      celery -A tdpservice.settings beat -l info --scheduler django_celery_beat.schedulers:DatabaseScheduler &

   3. Deploy the new celery application in cloud.gov using circleci
      • must be bound to all same services as main application
      • copy/abstract logic in deploy-backend.sh - env var logic needs to exist for both celery/app, cf push new manifest

Acceptance Criteria:
Create a list of functional outcomes that must be achieved to complete this issue

• Celery is deployed as its own cloud.gov application, changes are deployed via circle ci
• Redis is deployed as a standalone cloud.gov service, bound to both celery and the main application
• Testing Checklist has been run and all tests pass
• README is updated, if necessary

Tasks:
Create a list of granular, specific work items that must be completed to deliver the desired outcomes of this issue

• Task 1
• Task 2
• Task 3
• Run Testing Checklist and confirm all tests pass

Notes:
Add additional useful information, such as related issues and functionality that isn't covered by this specific issue, and other considerations that will be helpful for anyone reading this

• Note 1
• Note 2
• Note 3

Supporting Documentation:
Please include any relevant log snippets/files/screen shots

• Doc 1
• Doc 2

Open Questions:
Please include any questions or decisions that must be made before beginning work or to confidently call this issue complete

• Open Question 1
• Open Question 2

[George-Hudson]

PR here: #2773

[lhuxraft]

Jan to create a follow-on ticket to capture work that is left to do