diff --git a/.gitignore b/.gitignore index 4b0bd10406..50a85df8e6 100644 --- a/.gitignore +++ b/.gitignore @@ -49,3 +49,17 @@ yarn-error.log* # provider config /provider/ +/cmd/tke-installer/app/installer/provider/ +/cmd/tke-installer/app/installer/data/ + +# data log +/data +/data/ + +# hooks +/hooks +/hooks/ + +# manifests +/manifests +/manifests/ diff --git a/docs/design-proposals/expansion-framework.md b/docs/design-proposals/expansion-framework.md new file mode 100644 index 0000000000..be2359289f --- /dev/null +++ b/docs/design-proposals/expansion-framework.md @@ -0,0 +1,156 @@ +# Expansion Framework For TKEStack + +**Author**: madongdong([@MaDongdong99](https://github.com/MaDongdong99)) + +**Status** (20210413): Designing + +## Background + +[TKEStack Hook Framework](/docs/user/cluster/hooks.md) helps users to deploy the customized hooks to TKEStack, in which way user can inject customized logic during TKEStack life cycle. + +There are a group of hook points for users to choose and configure. Deep users will generate their own **tke.json** with hook points specified. + +Unfortunately, to make things work automatically, users usually need to generate the hook points by themselves so that it can work together with TKE-installer running. +At the same time, users also need to make logic to deal with customized materials such as Images,Charts,Static-files etc, which TKEStack is able to handle. + +For automation, a **TKEStack Wrapper** is always needed to handle the scenes above, which deep users need to think about and cannot get help from TKEStack. + +## Motivation + +To make better use of TKEStack Hook Framework, in other words: **Do it A Native Way**, we consider **generating hook points** and **dealing with deploy materials** by a new mechanism: **Expansion Framework**. + +Expansion Framework will empower users to plan hook points while they are building their own packages, let TKEStack deal with customized materials, and then get rid of things such as **TKEStack Wrapper**. + +## Goals + +- Users put an **Expansion Package** at a specified place, start TKE-installer, and expansion framework will do all the rest. +- Hook Points and Hook Files are auto setup. +- Customized materials are auto uploaded. +- When users disable expansion, TKEStack works as it did before. (And disable expansion means users do not put any **Expansion Package** in that place, which is also the same as what they did before) + +## Non Goals + +- Support multiple expansions. It's complicated to deal with conflicts between expansions, it's better to merge them together before packaging. + +## Design + +- Users make package layout as blow, and expansion framework does the following work: + - Scan expansion directory to find expansion.yaml which specifies the layout + - Verify files, hooks, images, charts materials + - Merge hook config into tke.json + - Copy hooks files into hook point + - Put images, charts where they should be + +![](/docs/images/expansion-framework/expansion-framework-design.png) + +### How it works + +- We define expansion.yaml as blow: which describes how users customize the hook framework. + +``` +charts: [] +images: [] +files: [] +hooks: [] +provider: [] +installer_skip_steps: [] +create_cluster_skip_conditions: [] +create_cluster_delegate_conditions: [] +``` + +- charts: Users put their customized charts (maybe helm charts) into the ${EXPANSION_DIR}/charts directory, and expansion framework will scan & check them, then upload to a chart repo. +- images: Users package their own images into ${EXPANSION_DIR}/images.tar.gz, and expansion framework will load/tag them, then push them to an image registry. +- files: Users put cluster hook files(for cluster.CopyFile hook) into the ${EXPANSION_DIR}/files directory, and expansion framework will scan & check them, then merge to cluster.feature.files in tke.json. +- hooks: User put install hook files(pre-install,post-cluster-ready,post-install) into the ${EXPANSION_DIR}/hooks directory, and expansion framework will scan & check them, then override origin hook files by them. +- provider: User put provider files into ${EXPANSION_DIR}/provider directory, and expansion framework will override the origin provider files +- installer_skip_steps: Install step list that users want to skip, expansion framework will merge them into config.Skipsteps in tke.json +- create_cluster_skip_conditions: Create cluster conditions list that users want to skip, expansion framework will merge them into cluster.feature.skipConditions in tke.json +- create_cluster_delegate_conditions: Create cluster conditions list that users want to delegate to an external operator, expansion framework will merge them into cluster.feature.delegateConditions(which is a new configuration for TKEStack) in tke.json +- **note**: files may need to be rendered before they are copied. + +### Mechanism of **delegate_conditions** + +We introduce a new mechanism called delegate_conditions when creating cluster. + +This means TKEStack can delegate any condition(can be controlled by a whitelist) to an **external operator**, which will give users more abilities to customized their own TKEStack. +For delegate_conditions, TKEStack do nothing but just waiting for the condition to be **READY**, an **external operator** will take responsibility. + +So an **external operator**(a beside container with tke-installer, or a sidecar with tke-platform), should complete conditions delegated to it and update cluster.Status.Conditions. + +What TKEStack need to do is providing an **expansion-operator-SDK** for users to develop their own operator. + +### Interactive Design + +![](/docs/images/expansion-framework/expansion-framework-interactive.png) + +## Detailed design + +### Expansion Framework Modules + +- Expansion Framework Lib - Base functions to support the framework. +- Expansion Operator SDK - If users want to develop expansion operator, give them a hand. +- Expansion Build Tools - A tool collections for users to build/check/test their expansion package easily. + +![](/docs/images/expansion-framework/expansion-framework-modules.png) + +### Design of Expansion Framework module + +- expansion Driver module + +``` +type expansionDriver struct { + log log.Logger + Operator string `json:"operator" yaml:"operator"` + Values map[string]string + Charts []string `json:"charts" yaml:"charts"` + Files []string `json:"files" yaml:"files"` + Hooks []string `json:"hooks" yaml:"hooks"` + Provider []string `json:"provider" yaml:"provider"` + Images []string `json:"images" yaml:"images"` + globalKubeconfig []byte + InstallerSkipSteps []string `json:"installer_skip_steps" yaml:"installer_skip_steps"` + CreateClusterSkipConditions []string `json:"create_cluster_skip_conditions" yaml:"create_cluster_skip_conditions"` + CreateClusterDelegateConditions []string `json:"create_cluster_delegate_conditions" yaml:"create_cluster_delegate_conditions"` + // TODO: save image lists in case of installer restart to avoid load images again + ImagesLoaded bool `json:"images_loaded" yaml:"images_loaded"` + // if true, will prevent to pass the same expansion to cluster C + DisablePassThroughToPlatform bool `json:"disable_pass_through_to_platform" yaml:"disable_pass_through_to_platform"` +} +``` + +- main method for TKEStack calling + - scan() error + - merge(t *TKE) + - loadOperatorImage(ctx context.Context) error + - patchHookFiles(ctx context.Context) error + - startOperator(ctx context.Context) error + - loadExpansionImages(ctx context.Context) error + - tagExpansionImages(ctx context.Context) error + - pushExpansionImages(ctx context.Context) error + - uploadExpansionCharts(ctx context.Context) error + +### Changes in TKEStack + +- Global Changes + - All modules with a cluster-installer (tke-installer, tke-platform-controller) add a member *expansionDriver, which supports calling of expansion functions. + - All modules with a cluster.OnCreate method (tke-installer, tke-platform-controller) , when looping for createCluster, has break points for delegate steps, and do reload-cluster when the step fails. +- Modules + - apiv1.ClusterFeature add member "DelegateConditions []string" + - cluster.Interface add const "ReasonDelegated = \"Delegated\"" +- TKE-installer + - tke.do() add "mergeFeatures()" + - initSteps add "Load Expansion Operator Image", "Patch Hook Files", "Start Expansion Operator", + - Step::loadImages add "loadExpansionImages" + - Step::tagImages add "tagExpansionImages" + - Step::pushImages add "pushExpansionImages" + - Step::writeKubeconfig add "write kubeconfig to share dir" +- tke-platform + - TODO: +- Cluster.Create + - Split EnsureDocker into EnsureDocker and EnsureStartDocker + - Split EnsureKubelet into EnsureKubelet and EnsureStartKubelet + + +## Further Discussion + +- Addon expansion: giving a **Native Way** for user to install their own apps on TKEStack. diff --git a/docs/images/expansion-framework/expansion-framework-design.png b/docs/images/expansion-framework/expansion-framework-design.png new file mode 100644 index 0000000000..9365618a95 Binary files /dev/null and b/docs/images/expansion-framework/expansion-framework-design.png differ diff --git a/docs/images/expansion-framework/expansion-framework-interactive.png b/docs/images/expansion-framework/expansion-framework-interactive.png new file mode 100644 index 0000000000..3c6d397ada Binary files /dev/null and b/docs/images/expansion-framework/expansion-framework-interactive.png differ diff --git a/docs/images/expansion-framework/expansion-framework-modules.png b/docs/images/expansion-framework/expansion-framework-modules.png new file mode 100644 index 0000000000..230af8d605 Binary files /dev/null and b/docs/images/expansion-framework/expansion-framework-modules.png differ