Agenda for the Flatcar community call on Tuesday, 8th of February 5:30pm CET / 10pm IST / 4:30pm GMT / 11:30am EST / 8:30am PST

[miao0miao]

Agenda for the Flatcar community call on Tuesday, 8th of February 5:30pm CET / 10pm IST / 4:30pm GMT / 11:30am EST / 8:30am PST

• Zoom link: https://us06web.zoom.us/j/84336116610
  • Meeting ID: 843 3611 6610
  • Passcode: 444888
• Youtube live stream / recording: YouTube
• Community Calls calendar (all future calls): link
  • iCal version: link
• other ways to keep in touch here

Attendees

Mark Peek - VMware
Tony Gosselin - Adobe
Joseph Sandoval -Adobe
Anuruddha Basak
Kai Lüke
Mathieu Tortuyaux
William Light
Sayan Chowdhury
Dongsu Park
Danielle Tal

Welcome

• Brief intro / check-in of all participants in the Zoom call. Please introduce yourself and share what brings you here today.

News

• first bug-smashing day of 2022!
  • ~40 bugs closed / fixed, see below for details
  • We use "good first issue" label for entry-level issues
  • There will be a new bug smashing day end of February, stay tuned!
• Community / issues stats for January/February
  • See below
• Cluster API v1.1.0 released with Ignition / Flatcar support: https://github.com/kubernetes-sigs/cluster-api/releases/tag/v1.1.0
  • Ignition PR took ~10 months of work & review
  • Individual providers require separate PRs
  • Work in progress / ready to merge are Azure and AWS; VMware PR is almost feature-complete (we also take care of broader concerns like bump to v1.1.0)

Spotlight

• Switch to CGroupsV2: Feedback, impact, and path forward

  • November flatcar stable release made Cgroups V2 the default for new provisionings.
  • We felt it was the right time to switch to V2, following mainstream V2 support in Docker and Kubernetes, as well as several other distributions having already switched to using V2 .
  • Since we did not want to break existing workloads, all instances upgrading to that new major release would remain on Cgroups V1
  • Something we did not account for is use cases and scenarios where users upgrade to new versions through re-provisioning, i.e. recycling the instance.
  • Also, while we know many users do pull Beta updates, we over-estimated the use of the Beta channel releases across all scenarios.
  • We received reports from a number of users who upgrade through instance recycling and are currently researching the best path forward.
  • It’s already possible to configure Flatcar to use V1, but this currently requires interacting with the instance after provisioning, and also rebooting the instance once, or using a custom image.
  • We are considering supporting an early boot time flag which can be set through provisioning, and would make the OS use Cgroups V1 from the start. Since systemd is already running (and using cgroups) at the point where ignition data is being evaluated this requires thorough inspection and testing.
  • We are also considering alternative approaches like shipping images which use CGroups v1 by default.
  • In any case, we will provide a solution that will enable users to provision V1 instances.
  • We expect to have it included in the late February release (the one after the ongoing release).
  • Learnings
  • For the Flatcar team, the lesson is that we need to consider all the scenarios in which a user will get a new version.
  • For users, we continue to encourage everyone to run some instances with beta and to report any issues that you encounter, as we cannot test in all environments.
    We will halt promotion of beta to stable if we hear from users that something doesn’t work for them – but we can only do so if we get that feedback.

• Discussion

  • Joseph shares his experience with the upstream kubernetes community re: cgroups v2 support. Many things aren't fully functioning with v2 - even 1.24 needs more validation - and upstream acknowledges this. SIG Node does not have a clear roadmap for full v2 support. This should be important for any kubernetes workload that uses Flatcar.
  • Kai shares features included in the upcoming support for Ignition v3 in Flatcar, which will among other things add options to set kernel parameters. Even if sometimes a reboot is required to make a kernel option apply, this reboot is transparent to provisioning.
  • Mathieu elaborates on his ignition v3 work and underlines that Flatcar will continue supporting v2 alongside v3.

• FOSDEM wrap-up: Flatcar talks, other cool presentations

  • How to make OS auto-updates boring: https://fosdem.org/2022/schedule/event/container_flatcar_autoupdate/
  • Extending the OS image (from sources): https://fosdem.org/2022/schedule/event/modding_the_immutable_how_to_extend_flatcar/
  • All's well that tests well: https://fosdem.org/2022/schedule/event/overview_of_flatcar_container_linux_test_framework/
  • Immutable Infrastructure with Flatcar Container Linux: https://fosdem.org/2022/schedule/event/flatcar_container_linux/
  • Securing and Hardening Container Host: https://fosdem.org/2022/schedule/event/securing_and_hardening_a_container_host/

• Question from youtube chat: When will talks be published?

  • Publishing has already started, the FOSDEM team work through all videos; all recordings should be available soon. Check the above links regularly, talks will pop up there.

Release planning

• Ongoing and upcoming releases (https://github.com/orgs/flatcar-linux/projects/5)

Q&A

• Questions from community participants, answered by the Flatcar maintainers

[miao0miao]

CAPI v1.1 - Add support for generating bootstrap data in Ignition format
kubernetes-sigs/cluster-api#4172
https://github.com/kubernetes-sigs/cluster-api/releases/tag/v1.1.0

[miao0miao]

Last month Issues opened/Closed (only Flatcar repo) up to February 7th:

Issues Opened

• #619: alpha:3139.0.0
• #618: Release Flatcar Container Linux Alpha 3139.0.0, Beta 3066.1.2, Stable 3033.2.2, LTS 2605.26.1
• #617: [RFE] bpftool on Flatcar
• #614: Improve text on Packer
• #613: Community Call 2022-02-08
• #611: After successful upgrade to 3033.2.0 the second reboot goes back to 2905.2.2
• #609: Has the GCP image for 2765.2.6 been deleted or the required permissions changed?
• #608: LSILogicParallel disk is still listed by lsblk after hot adding\removing
• #607: [RFE] support for toolbox without docker.service running
• #606: CVE-2021-4034
• #605: Slow ssh on gcp with oslogin
• #604: Release Flatcar Container Linux Alpha 3127.0.0, Beta 3066.1.1, Stable 3033.2.1, LTS 2605.25.1
• #603: Bug smashing Friday - 2022-01-28
• #602: Handling CVE-2022-0185
• #600: Release Team Call 2021-01-25
• #599: motd: race in reboot coordinator report (probably only with flatcar.autologin)
• #598: Can't install Kubernetes >=1.22 with RKE due to missing SELinux custom policies
• #597: Performance degradation in 3033.2.0?
• #596: Leaked selinux policy store files to the image root directory
• #595: Release Flatcar Container Linux Alpha 3115.0.0

Issues Closed

• #619: alpha:3139.0.0
• #611: After successful upgrade to 3033.2.0 the second reboot goes back to 2905.2.2
• #604: Release Flatcar Container Linux Alpha 3127.0.0, Beta 3066.1.1, Stable 3033.2.1, LTS 2605.25.1
• #603: Bug smashing Friday - 2022-01-28
• #602: Handling CVE-2022-0185
• #600: Release Team Call 2021-01-25
• #596: Leaked selinux policy store files to the image root directory
• #595: Release Flatcar Container Linux Alpha 3115.0.0
• #589: Community Call 2022-01-11
• #581: There is "metric 1024" for ip command
• #564: Spliting Container Linux Configuration in multiple files
• #561: mantle: rename packet to equinix-metal
• #538: Unable to update from old version due to SSL certificate problem
• #506: [cork] Failed to run create with UTF-8 user.name in git config
• #500: [RFE] publish containerd versions via flatcar_production_image_packages.txt
• #477: How to manualy set DNS during baremetal install (No DHCP)
• #456: AMI for 2765.2.6 on AWS lost
• #441: Simplify SDK format and distribution
• #440: Simplify the SDK sources
• #422: [RFE] Stricter dm-verity error handling
• #397: DHCPv6 broken since 2765.2.0/Systemd 247
• #394: Update Iptables from 1.6.2 to 1.8.x in flatcar OS version 2605.8.0
• #384: IPv6 address auto-configuration doesn't work on AWS EC2 instances
• #365: Include Amazon Root CAs in Flatcar's trust store
• #340: flatcar image with open-vm-tools + cloud-init metadata from openstacks' metadata_urls
• #331: [RFC] GPIO support
• #328: flatcar 2605.9.0+ with systemd 246 journal logs produced is not readable by older versions
• #291: [Tracking ] Update dev-vcs/repo to 2.8 in stable
• #268: Remove unused repositories from manifest
• #218: Stable 2605.6.0 systemd log spam Failed to set up mount unit: Invalid argument
• #209: Add favicon.ico to Flatcar website
• #204: Relabeling of content in /usr
• #198: Make BPF tracing tools (bcc, bpftrace) easier to use in Flatcar
• #187: UEFI Boot Support for Bare Metal Installs
• #259: Docs does not mention what UPDATE_STATUS_UPDATED_NEED_REBOOT means
• #178: [RFC] additional Reboot strategies needed?
• #166: AWS C5A support
• #270: Update ansible-bootstrap to Python 3
• #243: [ignition] Add luks support
• #135: Ignition ignores non-transpiled config without warnings or errors
• #133: Swap file instructions don't work
• #257: were these "Flatcar" images in GovCloud actually pushed by Flatcar?
• #97: Ship Beta, Stable images for ARM
• #95: Public feature / patch tracking system, automated Release Notes generation from PRs
• #89: Run automatic tests for BPF before each Flatcar release
• #88: Document dependencies of flatcar-install script
• #43: ARM 32-bit builds on the download page
• #38: Document how to build a custom image
• #37: Can't upload 2345.3.0 as a custom image on Digital Ocean

[pothos]

Want to share it here, we have a script to generate cgroupv1 AWS EC2 images for custom AMIs now:
https://github.com/flatcar-linux/flatcar-docs/blob/main/create_cgroupv1_ami.sh