📊 Bootstrap Lake Formation in Analytical Platform Compute and APDP Accounts

[julialawrence]

User Story

In order to deliver the QuickSight MVP backed by Lake Formation-managed Athena permissions, we would like to bootstrap and configure Lake Formation in our Analytical Platform Data Production and Analytical Platform Compute accounts.

Value / Purpose

Lake Formation is a superior approach to managing access to resources registered in Glue especially in scenarios where resources are shared cross-account. Therefore, in order to begin transitioning from using IAM for access to Lake Formation, QuickSight MVP will be a test bed for the approach, with assets managed solely in LF.

Useful Contacts

@julialawrence @MichaelJCollins @jamesstott

User Types

Data Engineers

Hypothesis

If we implement Lake Formation, this will reduce our and data engineering overhead in managing access as well as provide more granular permissions management approach.

Proposal

The purpose of this story is to configure Lake Formation in analytical-platform-data-production and analytical-platofrm-compute accounts to allow cross-account, cross-region sharing of assets using named resources method. Enabling sharing of tags is a stretch.

This will require implementing the following:
*A service IAM role in each account with broad LakeFormation, Athena, Glue and S3 permissions to be used as service roles for the UI to manage sharing.

• Due to [Enhancement]: LakeFormation - Support Cross Account Version Settings hashicorp/terraform-provider-aws#35773, follow these steps manually (or via a cli command) to set Lake Formation Data Sharing Version to 4 to allow named-resource sharing in hybrid mode.
• Using terraform, register the IAM roles as Lake Formation Admins in their respective accounts.
• Update Glue Resource policy in Analytical Platform Compute to allow ram.amazongaws.com to share resources into the account. Details here.

Additional Information

https://docs.google.com/document/d/1Xwbvc8ipI2m6nlK3et-TUcrksjhV9MmNqEVFtVbtkOQ/edit

Definition of Done

• Proposal Implemented
• Follow-on stories raised
• Another team member has reviewed
• Tests are green

[jacobwoffenden]

Blocked by #4358

[julialawrence]

Although it can be done with Analytical Platform account-hosted data, in order to share from and to another regular member account in Modernisation Platform, we will need some engineering support from that team. To make the process seamless, we will need to be able to assume roles from our MP accounts and that ability is currently severely restricted in MP.