⚗️ Propose a way forward on QuickSight DataSources

[julialawrence]

User Story

As a…
Platform engineer, I would like to understand how we can manage quicksight datasources and data sets based on Control Panel permissions without making it explicit to the data source admin, owner or user that a quicksight datasource is a different entity than raw AWS resource
So that…
Adoption and use of QS is seamless and additional infra-related complexity is abstrated.

Value / Purpose

The concept of a control panel datasource is not the same as QuickSight datasource. Having permissions on the AWS resources is insufficient. 2 additional QS specific resources need to be managed: a dataset and a data source

Thus purpose of this story is to explore to what degree the complexity around these concepts can be hidden from AP users.

Useful Contacts

@julialawrence

User Types

AP Users

Hypothesis

Any complexity we can abstract will remove a barrier from adoption of QS as a solution over more complex approaches such as RShiny.

Proposal

There are a couple of approaches that could be possible, and the chosen one must balance simplicity for user, manageable operational overhead and costs.

This list is not exhaustive

• Owner explicitly grants QS permissions to the user in the control panel just like other permissions are managed. Datasource is created in the backend once first permission is granted, datasets are created by the user.
• Datasources are created by the user based on their IAM permissions and QS permissions.
• Both dataset and datasource are created automatically and shared with the user. This could be done in response to owner granting permission or datasource being added to Control Panel.
• ???

Additional Information

• This story will require working with our service designer to arrive at a solution that balances user and ops needs.
• Explore the possibility of mapping and retrieving time-stamped version of the data
• Can Create a Derived Table be integrated as a datasource?

Definition of Done

• Approached investigated (including click-ops implementation if required)
• Proposal drafted and shared with the team
• Approach agreed
• Follow-on story raised.

[michaeljcollinsuk]

Some notes informed from our assumption testing around creating datasources from S3 buckets:

• Via quicksight, a datasource is created when the user creates a dataset
• A manifest.json is required. This file defines which files in the S3 bucket should be imported by quicksight.

If a datasource was created programatically by Control Panel at the point where a user marks it available in QuickSight, who's responsibility is it to define the manifest.json? It may not be possible for CP to do this, as it would not have knowledge of the content of the bucket, and the content of the bucket needs to match the defined manifest.json (e.g. .csv, .json) otherwise datasource creation will fail. Also the owner may only want to allow quicksight access to certain files in the bucket - CP would have no knowledge of this.

Therefore my thinking is that based on the three listed proposals above, the second is most feasible.

Datasources are created by the user based on their IAM permissions and QS permissions.

As the quicksight user that wants to use an S3 bucket to create a dataset should have knowledge of the contents of the bucket, so they are in position to create/manage the manifest.json.

[michaeljcollinsuk]

Some things that I think Control Panel will need to do to allow quicksight users to create a datasource/dataset from an S3 bucket:

• Update s3 bucket policy when a user allows it to be used in quicksight
• Create an iam policy for the bucket that can then be used with an iam-policy-assignment
• Create (and delete) iam-policy-assignment per user per bucket
  • This is required to allow a user to create a dataset in quicksight from the bucket
  • In assumption testing, this was done manually from within quicksight by an admin (via Manage Quicksight menu)
  • Can also be applied a group of users - so CP could create these groups based on the users that have access

QUESTIONS/PROBLEMS

• Should actions happen as soon as the bucket is marked as usable in quicksight?
• What if the user doesnt exist in quicksight yet? Should Control Panel create them?