Proposal: Credentials Helper Support in Reclient

[gkousik]

Background

Reclient has the following authentication mechanisms to talk to a corresponding RBE server:

1. Application default credentials (RBE_use_application_default_credentials)
2. GCE Credentials (RBE_use_gce_credentials)
3. External Authentication token (RBE_use_external_auth_token)

Methods (1) and (2) provide support for authenticating with an RBE server implementation running on GCP and method (3) is sufficient to authenticate with RBE servers that expect an “Authorization: Bearer <token>” token. While in method (3) the user can specify that they want to provide a custom token, there is no clear option in reclient for them to actually specify a token (creds_file can be used to specify the token, but its not clear how reproxy can refresh the credentials in this file).

In addition, there have been use cases where users want to provide custom HTTP headers for auth. Given this, the proposal in this doc is to implement support for a credentials helper interface in reclient.

Credentials Helper

A credentials helper would be a binary that can be provided to reclient. Reclient would call the binary to receive a set of headers to append to RBE RPCs along with information about when those headers expire.

Interface

Two new flags will be introduced:

Invoking the credentials helper

-creds_helper      - specifies the path to the credentials helper binary
-creds_helper_args - specifies the arguments to pass when invoking the
                     credentials helper binary

Return value of credentials helper

The credentials helper binary will be expected to return the following response.

{
    "headers": {
        "AuthorizationHeader_1": ["Value 1", "Value 2"],
        "AuthorizationHeader_2": ["Value 1", "Value 2"]
    },
    "expiry": "<timestamp>",
    "refresh_expiry": "<timestamp>"
}

Note that this is very similar to the interface of the Bazel credentials helper with two optional return options:

1. “expiry” - This will be a timestamp that indicates when the authorization headers expire. Reproxy will call the credentials helper to get new headers when the expiry is past. If no expiry is specified, the tokens in the headers are considered to NOT have an expiration.
2. “refreshExpiry” - This will be a timestamp that indicates the time until which reproxy can continue to call the credentials helper programmatically, without the user having to re authenticate themselves.
   This is useful in cases where the credentials helper relies on an authentication mechanism that requires the user to periodically reauthenticate themselves. If no refreshExpiry is specified, it is assumed to be infinite.
   This field is intended to only be used by the bootstrap binary. When bootstrap sees refresh_expiry to be valid, it would assume that reproxy would be able to obtain valid authorization headers from the credentials helper and continue with reproxy startup. Otherwise, bootstrap would fail the reproxy startup process and indicate to the user that they will need to reauthenticate.
   
   For both expiry and refreshExpiry, “YYYY-MM-DD HH:MM:SS.000 <TIMEZONE>” format will be used.

Implementation

Header Injection

RBE RPCs

Once the right authorization headers are returned from reclient, there needs to be a way to insert these headers into the RPCs being sent to RBE. With gRPC, a custom TokenSource can be created to insert custom headers per RPC request. Here’s a short example:

type customTokenSource struct {}

func (s customTokenSource) GetRequestMetadata(ctx context.Context, uri ...string) (map[string]string, error) {
	hdrs := make(map[string]string)
	hdrs["Authorization"] = "Bearer token" // Insert headers from creds helper here.
	return hdrs, nil
}

func (s customTokenSource) RequireTransportSecurity() bool {
	return true
}

// In reproxy
func main() {
   …
   		clientOpts = append(clientOpts, &client.PerRPCCreds{Creds: &customTokenSource{}})
		go func() {
			log.Infof("Creating a new SDK client")
			grpcClient, err := rflags.NewClientFromFlags(initCtx, clientOpts...)
    …
}

Credentials Helper Flag Location

There are two places where the creds helper flags can be defined:

Option (1) - In reclient (i.e., in bootstrap and reproxy binaries)

In this option, both bootstrap and reproxy binaries (and any other binary that relies on the credentials helper like remotetool) will define this flag. When it is set, bootstrap / reproxy would call the credentials helper to get the headers, then inject them into RBE RPCs using the mechanism described in this section.

Cons

1. Multiple definitions of the same flags across binaries.
2. In the future, any binary wanting to use the credentials helper to talk to RBE will also have to redefine these flags (e.g., remotetool in remote-apis-sdks repository).

Option (2) - In remote-apis-sdks repository

In this option, the flags will be defined in the remote-apis-sdks repository alongside the other authorization flags. When the creds helper flag is specified, the SDK will do the following:

1. Initialize the Client with a new TokenSource as described below. Initially, there won’t be any headers in the token source.
2. The first time this token source is asked for Request Metadata through “GetRequestMetadata()” call, there will be a blocking call to call the credentials helper and obtain the tokens. The obtained tokens will then be stored in memory along with the expiry.
3. Subsequent calls to GetRequestMetadata() within the expiration period will return the tokens stored in memory.
4. Whenever the headers are near expiration, they will be refreshed through a call to the credentials helper.

type credsHelperTokenSource struct {
    headers map[string]string
    headersLock sync.RWMutex
    expiry time.Time

    credsHelperPath string
    credsHelperArgs []string
}

func (s *credsHelperTokenSource) GetRequestMetadata(ctx context.Context, uri ...string) (map[string]string, error) {
    return updateToken()
}

func (s *credsHelperTokenSource) RequireTransportSecurity() bool {
    return true
}

func (s *credsHelperTokenSource) token(credsHelperPath string, credsHelperArgs []string) error {
    s.headersLock.Lock()
    defer s.headersLock.Unlock()

    if expiry.Sub(time.Now()) <= 5 * time.Minute {
        return s.headers, nil
    }
    ...
    // call creds helper to obtain new headers, save it to headers map.
    return s.headers, nil
}

Even with this approach, bootstrap binary in reclient would still need to define the credentials helper flags since it doesn’t initialize the SDK and will need to be able to say if the specified credentials helper can be used to obtain valid credentials or not to fail the build early.

Proposed solution: Pick Option (2) but move the auth related flags to a separate auth package within the remote-apis-sdks. This auth package can then be imported by the SDK and other clients binaries like bootstrap that don’t necessarily initialize the RBE client in remote-apis-sdks.

[Yannic]

LGTM in general. Would you mind opening an issue in github.com/EngFlow/credential-helper-spec for the new fields though please so we can standardize them? I‘d like to avoid having slightly different flavours of the protocol 😅

[Yannic]

We've recently standardized expiration of credentials in EngFlow/credential-helper-spec#2