Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

More Fluentbit config parsing logic for isolated regions with different domains #133

Merged
merged 1 commit into from
Dec 4, 2024

Conversation

whoix
Copy link
Contributor

@whoix whoix commented Dec 3, 2024

Issue #, if available:

Related to #94 there are separate endpoints for other isolated partitions that are currently in build. This PR adds the correct configs for those partitions.

Description of changes:
Images/addon components are already onboarded to an internal ImageReplicationService. This automatically syncs CW images lowside and transfers them up to all supported EKS regions. We have worked with the EKS team to ensure you guys are already onboarded.

I was able to confirm metrics, log groups, and that my metrics for Application Signals and GPU Container insights show up in Container Insights. I am not able to attach/illustrate specific screenshots or logs due to security implications.

Changes only affect ADC regions to incorporate the endpoint line to the configmap. Commercial is unaffected and i did test there too.

In commercial the configmap remains the same

[INPUT]
  Name                tail
  Tag                 application.*
  Exclude_Path        /var/log/containers/cloudwatch-agent*, /var/log/containers/fluent-bit*, /var/log/containers/aws-node*, /var/log/containers/kube-proxy*
  Path                /var/log/containers/*.log
  multiline.parser    docker, cri
  DB                  /var/fluent-bit/state/flb_container.db
  Mem_Buf_Limit       50MB
  Skip_Long_Lines     On
  Refresh_Interval    10
  Rotate_Wait         30
  storage.type        filesystem
  Read_from_Head      ${READ_FROM_HEAD}

[INPUT]
  Name                tail
  Tag                 application.*
  Path                /var/log/containers/fluent-bit*
  multiline.parser    docker, cri
  DB                  /var/fluent-bit/state/flb_log.db
  Mem_Buf_Limit       5MB
  Skip_Long_Lines     On
  Refresh_Interval    10
  Read_from_Head      ${READ_FROM_HEAD}

[INPUT]
  Name                tail
  Tag                 application.*
  Path                /var/log/containers/cloudwatch-agent*
  multiline.parser    docker, cri
  DB                  /var/fluent-bit/state/flb_cwagent.db
  Mem_Buf_Limit       5MB
  Skip_Long_Lines     On
  Refresh_Interval    10
  Read_from_Head      ${READ_FROM_HEAD}

[FILTER]
  Name                kubernetes
  Match               application.*
  Kube_URL            https://kubernetes.default.svc:443
  Kube_Tag_Prefix     application.var.log.containers.
  Merge_Log           On
  Merge_Log_Key       log_processed
  K8S-Logging.Parser  On
  K8S-Logging.Exclude Off
  Labels              Off
  Annotations         Off
  Use_Kubelet         On
  Kubelet_Port        10250
  Buffer_Size         0

[OUTPUT]
  Name                cloudwatch_logs
  Match               application.*
  region              ${AWS_REGION}
  log_group_name      /aws/containerinsights/${CLUSTER_NAME}/application
  log_stream_prefix   ${HOST_NAME}-
  auto_create_group   true
  extra_user_agent    container-insights

while Isolated region now have the new added endpoint param

[INPUT]
  Name                tail
  Tag                 application.*
  Exclude_Path        /var/log/containers/cloudwatch-agent*, /var/log/containers/fluent-bit*, /var/log/containers/aws-node*, /var/log/containers/kube-proxy*
  Path                /var/log/containers/*.log
  multiline.parser    docker, cri
  DB                  /var/fluent-bit/state/flb_container.db
  Mem_Buf_Limit       50MB
  Skip_Long_Lines     On
  Refresh_Interval    10
  Rotate_Wait         30
  storage.type        filesystem
  Read_from_Head      ${READ_FROM_HEAD}

[INPUT]
  Name                tail
  Tag                 application.*
  Path                /var/log/containers/fluent-bit*
  multiline.parser    docker, cri
  DB                  /var/fluent-bit/state/flb_log.db
  Mem_Buf_Limit       5MB
  Skip_Long_Lines     On
  Refresh_Interval    10
  Read_from_Head      ${READ_FROM_HEAD}

[INPUT]
  Name                tail
  Tag                 application.*
  Path                /var/log/containers/cloudwatch-agent*
  multiline.parser    docker, cri
  DB                  /var/fluent-bit/state/flb_cwagent.db
  Mem_Buf_Limit       5MB
  Skip_Long_Lines     On
  Refresh_Interval    10
  Read_from_Head      ${READ_FROM_HEAD}

[FILTER]
  Name                kubernetes
  Match               application.*
  Kube_URL            https://kubernetes.default.svc:443
  Kube_Tag_Prefix     application.var.log.containers.
  Merge_Log           On
  Merge_Log_Key       log_processed
  K8S-Logging.Parser  On
  K8S-Logging.Exclude Off
  Labels              Off
  Annotations         Off
  Use_Kubelet         On
  Kubelet_Port        10250
  Buffer_Size         0

[OUTPUT]
  Name                cloudwatch_logs
  Match               application.*
  region              ${AWS_REGION}
  log_group_name      /aws/containerinsights/${CLUSTER_NAME}/application
  log_stream_prefix   ${HOST_NAME}-
  auto_create_group   true
  endpoint             logs.${AWS_REGION}.c2s.ic.gov
  extra_user_agent    container-insights

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

@jefchien jefchien merged commit a5ade85 into aws-observability:main Dec 4, 2024
0 of 3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants