Skip to content

Latest commit

 

History

History
169 lines (122 loc) · 6.1 KB

README.md

File metadata and controls

169 lines (122 loc) · 6.1 KB

Tag Central

Creates Data Catalog Tag Templates from datacatalog-templates using terraform resources. It also sets up the IAM permissions for 3 personas:

  • Data Governor
  • Data Curator
  • Data Analyst

Follow the instructions and make sure you have set up the .tvars with your environment values before running terraform. The IAM members for the suggested personas can be any of: user:{emailid}, serviceAccount:{emailid}, group:{emailid} or domain:{domain}.

The datacatalog_tag_template.tf contains 4 Tag Templates:

  • Data Engineering Template
  • Derived Data Template
  • Data Governance Template
  • Data Quality Template

They are suggestions, change the Tag Template Fields to fit your needs.

Table of Contents

Detailed Permissions

Data Governor

N|Solid

Data Curator

N|Solid

Data Analyst

N|Solid

Terraform Configuration

Run Terraform

The sample works with a list of projects, if you want to use the recommended access controls at the folder or organization level, you can change the iam module: and switch the google_project_iam_member resource to google_folder_iam_member or google_organization_iam_member respectively.

Set terraform execution Service Account

At the moment this guide was created, Data Catalog does not support using end user credentials from the Google Cloud SDK. You need to set the service account before running terraform. As security best practices, we will not download the Service Account key, so we will use Service Account impersonation in the terraform execution.

Create a Service Account with the required permissions to create Data Catalog Tag Templates.

# Change the placeholder {tag-central-project-id}
export GOOGLE_CLOUD_PROJECT={tag-central-project-id}
export SA_NAME=terraform-dc-resources-sa

# Create Service Account
gcloud iam service-accounts create $SA_NAME \
--display-name  "Service Account to create DC Resources" \
--project $GOOGLE_CLOUD_PROJECT 

# Add Tag Template Owner role
gcloud projects add-iam-policy-binding $GOOGLE_CLOUD_PROJECT \
--member "serviceAccount:$SA_NAME@$GOOGLE_CLOUD_PROJECT.iam.gserviceaccount.com" \
--quiet \
--project $GOOGLE_CLOUD_PROJECT \
--role "roles/datacatalog.tagTemplateOwner"

# Change to your user
export TERRAFORM_USER_EMAIL={my-user-email}

# Give Service Account Token Creator Role for the SA to the user
# running terraform
gcloud iam service-accounts add-iam-policy-binding \
"$SA_NAME@$GOOGLE_CLOUD_PROJECT.iam.gserviceaccount.com" \
--member "user:$TERRAFORM_USER_EMAIL" \
--quiet \
--project $GOOGLE_CLOUD_PROJECT \
--role "roles/iam.serviceAccountTokenCreator"

❗ It takes about 1 minute to IAM propagate the Tag Template Owner role and Service Account Token Creator Roles used by in this script, so if you receive the following error when running terraform:

Error: googleapi: Error 403: The caller does not have permission, forbidden

Wait a few minutes and run it again.

❗ If you receive an error saying that Data Catalog API is not enabled, wait a few minutes or force it using gcloud:

gcloud services enable datacatalog.googleapis.com --project $GOOGLE_CLOUD_PROJECT

Set terraform variable placeholders

Go to .tfvars and change the placeholders:

  1. tag_central_project_id to your Tag Central Project
  2. datacatalog_analytics_projects_id to the list of Analytics Projects split by comma
  3. tag_template_region to the region you want the sample templates to be created
  4. datacatalog_data_governor_members to the list of data governor members split by comma
  5. datacatalog_data_curator_members to the list of data curator members split by comma
  6. datacatalog_data_analyst_members to the list of data curator analyst split by comma

Example of a valid configuration:

tag_central_project_id="tag-central-project"

datacatalog_analytics_projects_id=["my-analytics-project-1","my-analytics-project-2"]

datacatalog_resources_sa_name="terraform-dc-resources-sa"

tag_template_region="us"

datacatalog_data_governor_members=["user:john_data_governor@datacompany.com", "group:data_governors@datacompany.com"]

datacatalog_data_curator_members=["user:john_data_curator@datacompany.com", "group:data_curators@datacompany.com"]

datacatalog_data_analyst_members=["user:john_data_analytic@datacompany.com", "group:data_analytics@datacompany.com"]

member can be any of: user:{emailid}, serviceAccount:{emailid}, group:{emailid} or domain:{domain}

Run the following commands inside the terraform directory.

Init terraform

After that, let's get Terraform started. Run the following to pull in the providers.

terraform init

Execute terraform

With the providers downloaded and terraform variables set, you're ready to use Terraform. Go ahead!

terraform plan -input=false -out=tfplan -var-file=".tfvars" 
terraform apply tfplan

Cleanup

Run the following to remove the resources Terraform provisioned:

terraform destroy -var-file=".tfvars"
yes