Get correct service tags for nginx logs in EC2

[timmc-edx]

nginx logs for our remaining EC2 services (edxapp, insights, etc.) are getting tagged in Datadog with service:edx-nginx rather than the relevant service. This is making it harder to find logs and to sort out which logs go with which service.

Notes:

• Logs are actually getting double tagged, e.g. service:edx-edxapp-lms, service:edx-nginx -- the latter appears to override the former.
• The unwanted tag is coming from a common datadog config block for the entire environment that is explicitly tagging all nginx logs as service:edx-nginx: https://github.com/edx/edx-internal/blob/ec0fc544b3eca361c7d2e25fa0d4be9f4ba3b60d/ansible/vars/prod-edx.yml#L187
  • Can we just remove that service attribute?
  • If not, file a Datadog support ticket asking how this is supposed to work. (It doesn't seem to make sense that you'd have to explicitly tag log sources when there's also a host-level tag available.)
• If this turns out to be too much work, we can just document it and let people know about the problem, and hope it doesn't cause too much trouble in the time before everything is containerized.

[timmc-edx]

@nadeemshahzad Do you know if it's possible for us to remove the service: edx-nginx from this config? It's overriding and conflicting with the host service tags. I vaguely recall that there was some issue with removing the log-level service tag but I don't recall what it was.