Skip to content

Latest commit

 

History

History
372 lines (269 loc) · 11.9 KB

sts-private-bucket.md

File metadata and controls

372 lines (269 loc) · 11.9 KB

Short-lived Credentials with AWS Security Token Service using AWS CloudFront and private S3 bucket

NOTE: The steps described in this document, create a private S3 Bucket using CloudFront Distribution, was introduced to ccoctl using the flag --create-private-s3-bucket. We are keeping this document to provide an overview of the steps used in this solution.

To create an IAM OpenID Connect identity provider you should expose the OIDC config using a public HTTPS endpoint. The steps described here will guide you to create one CloudFront Distribution to expose the HTTPS endpoint, serving objects from a private S3 Bucket accessed by Origin Access Identity.

Summary:

Requirements

Steps to Create

Set the environment variables

considering that the installer directory is the same you are running this command

export CLUSTER_NAME=$(awk '/infrastructureName: / {print $2}' manifests/cluster-infrastructure-02-config.yml)
export CLUSTER_REGION=$(awk '/region: / {print $2}' manifests/cluster-infrastructure-02-config.yml)

export DIR_CCO="${PWD}/_output"
export OIDC_BUCKET_NAME="${CLUSTER_NAME}-oidc"
export OIDC_BUCKET_CONTENT="${DIR_CCO}/bucket-content"

mkdir -p ${OIDC_BUCKET_CONTENT}

Create the Origin Access Identity

Steps to create the Origin Access Identity (OAI) to be used to access the bucket through CloudFront Distribution:

  • Create the OAI and set the variable OAI_CLOUDFRONT_ID:
aws cloudfront create-cloud-front-origin-access-identity \
    --cloud-front-origin-access-identity-config \
    CallerReference="${OIDC_BUCKET_NAME}",Comment="OAI-${OIDC_BUCKET_NAME}"

export OAI_CLOUDFRONT_ID=$(aws cloudfront \
    list-cloud-front-origin-access-identities \
    --query "CloudFrontOriginAccessIdentityList.Items[?Comment==\`OAI-${OIDC_BUCKET_NAME}\`].Id" \
    --output text)

Create the Bucket

  • Create the private Bucket
aws s3api create-bucket \
    --bucket ${OIDC_BUCKET_NAME} \
    --region ${CLUSTER_REGION} \
    --create-bucket-configuration LocationConstraint=${CLUSTER_REGION} \
    --acl private
  • Create the respective tags on the Bucket (Recommended if you would like to use the ccoctl to delete resources)
aws s3api put-bucket-tagging \
    --bucket ${OIDC_BUCKET_NAME} \
    --tagging "TagSet=[{Key=Name,Value=${OIDC_BUCKET_NAME}},{Key=openshift.io/cloud-credential-operator/${CLUSTER_NAME},Value=owned}]"
  • Download the s3 bucket policy template that restricts access to CloudFront Origin Access Identity (OAI)
wget https://raw.githubusercontent.com/openshift/cloud-credential-operator/master/docs/sts-oidc-bucket-policy.json.tpl
  • Create the Bucket Policy configuration (cli-input-json) allowing OAI to retrieve objects
cat sts-oidc-bucket-policy.json.tpl \
   | envsubst \
   > ${DIR_CCO}/oidc-bucket-policy.json
  • Apply the policy to the Bucket to block public access
aws s3api put-bucket-policy \
    --bucket ${OIDC_BUCKET_NAME} \
    --policy file://${DIR_CCO}/oidc-bucket-policy.json

aws s3api put-public-access-block \
    --bucket ${OIDC_BUCKET_NAME} \
    --public-access-block-configuration \
    BlockPublicAcls=true,IgnorePublicAcls=true,BlockPublicPolicy=true,RestrictPublicBuckets=true

Create CloudFront Distribution

wget https://raw.githubusercontent.com/openshift/cloud-credential-operator/master/docs/sts-oidc-cloudfront.json.tpl
  • Create a CloudFront Distribution configuration (cli-input-json)
cat sts-oidc-cloudfront.json.tpl \
   | envsubst \
   > ${DIR_CCO}/oidc-cloudfront.json
  • Create the CloudFront Distribution with Tags
aws cloudfront create-distribution-with-tags \
    --distribution-config-with-tags \
    file://${DIR_CCO}/oidc-cloudfront.json

  • Wait until the Distribution resource has been created

  • Get the CloudFront's Distribution URL

export CLOUDFRONT_URI=$(aws cloudfront list-distributions \
    --query "DistributionList.Items[?Comment==\`${CLUSTER_NAME}\`].DomainName" \
    --output text)

echo ${CLOUDFRONT_URI}

Generate OIDC configuration and keys

  • Generate the key pair used to create the service account tokens
./ccoctl aws create-key-pair \
    --output-dir ${DIR_CCO}
  • Generate the OpenID configuration
./ccoctl aws create-identity-provider \
    --name=${CLUSTER_NAME} \
    --region=${CLUSTER_REGION} \
    --public-key-file=${DIR_CCO}/serviceaccount-signer.public \
    --output-dir=${DIR_CCO}/ \
    --dry-run
  • Update the CloudFront URI's endpoint to the CloudFront distribution's address:

A. Patch the issuer URL on the OpenID configuration file /.well-known/openid-configuration

mkdir -p ${OIDC_BUCKET_CONTENT}/.well-known
cat ${DIR_CCO}/02-openid-configuration \
    | sed "s/https:\/\/${CLUSTER_NAME}[a-z.-].*\//https:\/\/${CLOUDFRONT_URI}\//" \
    | sed "s/https:\/\/${CLUSTER_NAME}[a-z.-].*/https:\/\/${CLOUDFRONT_URI}\",/" \
    > ${OIDC_BUCKET_CONTENT}/.well-known/openid-configuration

B. Copy keys.json

cp -v ${DIR_CCO}/03-keys.json \
    ${OIDC_BUCKET_CONTENT}/keys.json

C. Patch the issuer url on Authentication CRD in cluster-authentication-02-config.yaml

sed -i "s/https:\/\/[a-z.-].*/https:\/\/${CLOUDFRONT_URI}/" \
    ${DIR_CCO}/manifests/cluster-authentication-02-config.yaml

D. Update the IdP OIDC object configuration

sed -i "s/https:\/\/[a-z.-].*/https:\/\/${CLOUDFRONT_URI}\",/" \
    ${DIR_CCO}/04-iam-identity-provider.json

Check the output of jq . ${DIR_CCO}/04-iam-identity-provider.json

  • Upload the bucket content
aws s3 sync ${OIDC_BUCKET_CONTENT}/ \
    s3://${OIDC_BUCKET_NAME}
  • Make sure you can access the content through the public URL

NOTE: CloudFront can take some time to deploy the distribution. Please be sure the distribution has been deployed and it's available before running this step (Status=Enabled and Last Modified!=Deploying). You can access the CloudFront Console to check it.

curl https://${CLOUDFRONT_URI}/keys.json
curl https://${CLOUDFRONT_URI}/.well-known/openid-configuration

Create the OpenID Connector identity provider

  • Create the IAM OpenID Connect identity provider
aws iam create-open-id-connect-provider \
    --cli-input-json file://${DIR_CCO}/04-iam-identity-provider.json \
    > ${DIR_CCO}/04-iam-identity-provider-object.json
  • Get the ARN of the IAM OpenID Connect identity provider created above
OIDC_ARN=$(jq -r .OpenIDConnectProviderArn \
    ${DIR_CCO}/04-iam-identity-provider-object.json)

echo ${OIDC_ARN}

Create IAM Roles

  • Extract CredentialRequests from the release image
./oc adm release extract \
    --credentials-requests \
    --cloud=aws \
    --to=${DIR_CCO}/credrequests \
    ${RELEASE_IMAGE}
  • Create IAM Roles for the OpenShift components
./ccoctl aws create-iam-roles \
    --name=${CLUSTER_NAME} \
    --region=${CLUSTER_REGION}\
    --credentials-requests-dir=${DIR_CCO}/credrequests \
    --identity-provider-arn=${OIDC_ARN} \
    --output-dir ${DIR_CCO}

We have now created IAM OpenID Connect identity provider and IAM roles, you can return to step 8 to continue with installation in STS mode.

Steps to Delete

These steps should be followed after you've removed the resources created by ccoctl described in the delete section.

Requirements:

  • You should have set the CloudFront Distribution Comment as ${CLUSTER_NAME}, as described in the section above. Otherwise, you should specify the value you've set when creating the CloudFront Distribution.
  • You should have set the CloudFront Origin Access Identity (OAI) with the field Comment with the value OAI-${OIDC_BUCKET_NAME}. Otherwise, you should specify the value you've set when creating the CloudFront OAI.

Remove the Bucket

If the ccoctl delete command failed due to non-empty Bucket (BucketNotEmpty), you should follow those steps to complete the Bucket removal.

  • Remove the Bucket objects
aws s3api delete-object \
    --bucket ${OIDC_BUCKET_NAME} \
    --key ".well-known/openid-configuration"

aws s3api delete-object \
    --bucket ${OIDC_BUCKET_NAME} \
    --key "keys.json"
  • Remove the Bucket using ccoctl
./ccoctl aws delete \
    --name=${CLUSTER_NAME} \
    --region=${CLUSTER_REGION}

Remove the CloudFront Distribution

CloudFront Distributions can be removed only when it is disabled. To do so, you need to get the current configuration, setting the field Enabled to false, apply the new configuration, then remove the Distribution.

  • Get the CloudFront Distribution ID
DISTRIBUTION_ID=$(aws cloudfront list-distributions \
    --query "DistributionList.Items[?Comment==\`${CLUSTER_NAME}\`].Id" \
    --output text)
  • Get the CloudFront Distribution Config ETag
ETAG=$(aws cloudfront get-distribution-config \
    --id ${DISTRIBUTION_ID} \
    | jq -r '.ETag')
  • Get the CloudFront Distribution Configuration, setting the Enabled field to false
aws cloudfront get-distribution-config --id ${DISTRIBUTION_ID} \
    | jq '.DistributionConfig' \
    | jq '.Enabled=false' \
    > ${DIR_CCO}/oidc-cloudfront-to-delete.json
  • Apply the new Distribution configuration
ETAG=$(aws cloudfront update-distribution \
    --id ${DISTRIBUTION_ID} \
    --if-match ${ETAG} \
    --distribution-config file://${DIR_CCO}/oidc-cloudfront-to-delete.json \
    | jq -r '.ETag')
  • Get the new ETag (it's also returned on the last command).

The last command updates the ETAG variable referencing to a new configuration (Disabled distribution). If you didn't note it you need to re-run the get-distribution-config command, as desribed above.

  • Delete the CloudFront Distribution
aws cloudfront delete-distribution \
    --id ${DISTRIBUTION_ID} \
    --if-match ${ETAG}

Remove the Origin Access Identity (OAI)

  • Get the OAI ID
OAI_CLOUDFRONT_ID=$(aws cloudfront \
    list-cloud-front-origin-access-identities \
    --query "CloudFrontOriginAccessIdentityList.Items[?Comment==\`OAI-${OIDC_BUCKET_NAME}\`].Id" \
    --output text)
  • Get the OAI ETag by ID
OAI_ETAG=$(aws cloudfront \
    get-cloud-front-origin-access-identity-config \
    --id ${OAI_CLOUDFRONT_ID} \
    | jq -r .ETag)
  • Remove the OAI by ID
aws cloudfront \
    delete-cloud-front-origin-access-identity \
    --id ${OAI_CLOUDFRONT_ID} \
    --if-match ${OAI_ETAG}