Add Kubernetes backend

[ashb]

I know Kubernetes isn't a key-value store so this might not make immediate sense but confd's mode of operation (of watching for changes, then rendering templates and running reload commands) has been a massive help in moving some of our enterprise applications to kubernetes.

I have written some docs which you can view in rendered form https://godoc.org/github.com/ashb/confd/backends/kubernetes (will work while my fork lasts) I have included a snippet below that shows the keys and their values that I "synthesised" from the API objects.

I have been using this for a week and it's been working for me. I have also written an integration test script (note - it will create a confd-test namespace in what ever kube cluster you run it in) but I couldn't find any easy/obvious way to run these tests from Travis.

Mapping API Object to Variables

Since confd expects a key-value store and the kubernetes API doesn't expose this directly we have to define our own pattern of variables from the API objects.

The only API objects (and thus initial key path) supported are endpoints.

For a service called "mysvc" it will create the following variables under "/endpoints/mysvc":

• A "ports/$port_name" variable for each named port with the port number as the value. Ports with numbers only names are not present.

• A set of keys under "ips" for each ready pod

  /endpoints/mysvc/ips/0: 172.17.0.6
  /endpoints/mysvc/ips/1: 172.17.07
  

• A set of keys under "allips" that combines ready and notready pods

  /endpoints/mysvc/allips/0: 172.17.0.6
  /endpoints/mysvc/allips/1: 172.17.0.7
  /endpoints/mysvc/allips/2: 172.17.0.5
  

A complete listing of all the variables created in this example service are:

/endpoints/mysvc/ports/http: 8080
/endpoints/mysvc/ips/0: 172.17.0.6
/endpoints/mysvc/ips/1: 172.17.07
/endpoints/mysvc/allips/0: 172.17.0.6
/endpoints/mysvc/allips/1: 172.17.0.7
/endpoints/mysvc/allips/2: 172.17.0.5

[ashb]

Oh, and I'm not sure I drove godeps right - i was struggling a bit with it. Should it be updating files under vendor/ and should those be commited? If so I couldn't work out how to make that happen.

[ahmetb]

@ashb download godep to your $PATH, and run godep restore then godep update k8s.io/client-go/kubernetes and then just git add+git commit should be bringing that package to the latest version.

[ashb]

Something's still not adding up -- To start with even to k8s.io/client-go/tools/cache is listed in the Godeps.json (now) it's not writing the file to vendor/ when I run godep save (or with backends/.. as it was run lasttime by the looks of things.)

Also if I try to compile it now I get this. I'm guessing I did something wrong? (I have to jump through hoops to get golang.org/x/net/lex/httplex added, because golang.org/x/net/context was pinning x/net to an older version.)

root@6d7eb962daaf:/go/src/github.com/kelseyhightower/confd# ./build
Building confd...
# github.com/kelseyhightower/confd/vendor/k8s.io/client-go/pkg/apis/certificates
vendor/k8s.io/client-go/pkg/apis/certificates/types.generated.go:163: cannot use e (type *"github.com/kelseyhightower/confd/vendor/github.com/ugorji/go/codec".Encoder) as type *"k8s.io/client-go/vendor/github.com/ugorji/go/codec".Encoderin argument to yy10.CodecEncodeSelf

$ godep save backend/.. .
...
$ git grep tools/cache
Godeps/Godeps.json:                     "ImportPath": "k8s.io/client-go/tools/cache",
backends/kubernetes/client.go:  "k8s.io/client-go/tools/cache"
$ godep restore -v
...
godep: Checking dependency: k8s.io/client-go/rest
godep: Checking dependency: k8s.io/client-go/tools/cache
godep: Checking dependency: k8s.io/client-go/tools/clientcmd/api
godep: Checking dependency: k8s.io/client-go/tools/metrics
godep: Checking dependency: k8s.io/client-go/transport
root@6d7eb962daaf:/go/src/github.com/kelseyhightower/confd# ls vendor/k8s.io/client-go/tools/cache
ls: cannot access vendor/k8s.io/client-go/tools/cache: No such file or directory

[ashb]

Okay I think I finally got something by following the instructions here tools/godep#478 (comment) (I can at least run a ./build in a fresh docker container without error. Lets hope Travis agrees with me)

[selslack]

Sounds fancy! Any updates on this one?

[frittentheke]

Any chance for this PR to be accepted / merged? Having access to the Kubernetes resources as a basis for dynamic configuration would be awesome.

@ashb any chance you would extend your K8S client to

• map ingress resources to confd KVs as well
• support filtering by labels, like i.e. Traefik does it (https://docs.traefik.io/configuration/backends/kubernetes/)

[jfrabaute]

Hi,

I've also implemented a k8s backend for confd.
Here is the change: jfrabaute@fa5b0f4

It can monitor services, endpoints and statefulsets for now.
Having statefulsets is useful for our use case because we want to be able to expose the datanode/regionserver/namenodes for each instance, which gives useful information for debugging.

[okushchenko]

@frittentheke I'll review this PR in January.
@jfrabaute please, open a separate PR with your changes. I'll take a look.

[jfrabaute]

@okushchenko I sent a pull request: #649

I can rename the backend to something else so we can have two implementations of k8s backend if we want to merge both.
Having only one would be probably better in the long term.
I didn't see the other one when I implemented mine.
The one I have is running for several weeks without a problem so far, but I'm planning to clean code/add features when I'll have some time in 2018.
If you merge the other PR, I'll try to see if I can import my stuff to the other one so we have only one k8s backend.

Thanks.