Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: [M3-8953] - Object Storage Gen2 Updates and Improvements #11355

Merged
merged 22 commits into from
Dec 4, 2024
Merged
Show file tree
Hide file tree
Changes from 13 commits
Commits
Show all changes
22 commits
Select commit Hold shift + click to select a range
d4e7851
feat: [M3-8953] - OBJ Gen2 Updates
jaalah Dec 3, 2024
e03a55e
Add changeset
jaalah Dec 3, 2024
4d7799e
fix duplicate buckets landing page, may rework
coliu-akamai Dec 3, 2024
15e7bef
Update e2e tests
jaalah Dec 3, 2024
9559a04
Merge branch 'develop' of github.com:linode/manager into M3-8953
jaalah Dec 3, 2024
25f82d6
Merge branch 'M3-8953' of github.com:jaalah-akamai/manager into M3-8953
jaalah Dec 3, 2024
82da5de
Remove redundant beforeEach
jaalah Dec 3, 2024
0bd1c2c
Small adjustment to request
jaalah Dec 3, 2024
bcbe015
fix access keys e2e test
jaalah Dec 3, 2024
7f40ea4
Fix bucket-create-gen2.spec.ts failing tests @linode/frontend-sdet
jaalah Dec 4, 2024
7edebf9
E2E review updates @cliu-akamai @AzureLatte
jaalah Dec 4, 2024
597f46b
Fix issue where properties content was overriding SSL for non-gen2 ca…
jaalah Dec 4, 2024
26170f4
Remove CORS from object drawers @bnussman-akamai
jaalah Dec 4, 2024
03297f6
Update packages/manager/.changeset/pr-11355-upcoming-features-1733237…
jaalah-akamai Dec 4, 2024
ad14035
Update packages/manager/.changeset/pr-11355-upcoming-features-1733237…
jaalah-akamai Dec 4, 2024
1edce89
Remove double copy icon @dwiley-akamai
jaalah Dec 4, 2024
8755f08
Add changesets
jaalah Dec 4, 2024
0a0bfa5
Update packages/manager/src/features/ObjectStorage/AccessKeyLanding/A…
jaalah-akamai Dec 4, 2024
915e201
Pluralize the regions
jaalah Dec 4, 2024
d1d97d6
Fix E2E based in pluralize
jaalah Dec 4, 2024
2a0bb8c
Fix unit tests
jaalah Dec 4, 2024
c67db09
Fix e2e from removing toggle
jaalah Dec 4, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions packages/api-v4/src/object-storage/buckets.ts
Original file line number Diff line number Diff line change
Expand Up @@ -262,6 +262,11 @@ export const updateBucketAccess = (
setData(params, UpdateBucketAccessSchema)
);

/**
* getObjectStorageEndpoints
*
* Returns a list of Object Storage Endpoints.
*/
export const getObjectStorageEndpoints = ({ filter, params }: RequestOptions) =>
Request<Page<ObjectStorageEndpoint>>(
setMethod('GET'),
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
---
"@linode/manager": Upcoming Features
---

Updates Regions/S3 Hostnames interface to match new design guidelines with
jaalah-akamai marked this conversation as resolved.
Show resolved Hide resolved
improved visualization of multiple storage regions. ([#11355](https://github.com/linode/manager/pull/11355))
jaalah-akamai marked this conversation as resolved.
Show resolved Hide resolved
Original file line number Diff line number Diff line change
@@ -0,0 +1,54 @@
import { mockGetAccount } from 'support/intercepts/account';
import { mockAppendFeatureFlags } from 'support/intercepts/feature-flags';
import { ui } from 'support/ui';
import {
accountFactory,
objectStorageBucketFactory,
regionFactory,
} from 'src/factories';
import { randomLabel } from 'support/util/random';

describe('Object Storage Gen 1 Bucket Details Tabs', () => {
beforeEach(() => {
mockAppendFeatureFlags({
objMultiCluster: true,
objectStorageGen2: { enabled: false },
}).as('getFeatureFlags');
mockGetAccount(
accountFactory.build({
capabilities: ['Object Storage', 'Object Storage Access Key Regions'],
})
).as('getAccount');
});

const mockRegion = regionFactory.build({
capabilities: ['Object Storage'],
});

const mockBucket = objectStorageBucketFactory.build({
label: randomLabel(),
region: mockRegion.id,
});

describe('Properties tab without required capabilities', () => {
it(`confirms the Properties tab does not exist for users without 'Object Storage Endpoint Types' capability`, () => {
const { region, label } = mockBucket;

cy.visitWithLogin(
`/object-storage/buckets/${region}/${label}/properties`
);

cy.wait(['@getFeatureFlags', '@getAccount']);

// Confirm that expected tabs are visible.
ui.tabList.findTabByTitle('Objects').should('be.visible');
ui.tabList.findTabByTitle('Access').should('be.visible');
ui.tabList.findTabByTitle('SSL/TLS').should('be.visible');

// Confirm that "Properties" tab is absent.
cy.findByText('Properties').should('not.exist');

// TODO Confirm "Not Found" notice is present.
coliu-akamai marked this conversation as resolved.
Show resolved Hide resolved
});
});
});
Original file line number Diff line number Diff line change
Expand Up @@ -59,12 +59,12 @@ describe('Object Storage gen2 access keys tests', () => {
cy.findByText(mockAccessKey1.label).should('be.visible');
cy.findByText(mockAccessKey2.label).should('be.visible');
cy.findByText('US, Newark, NJ (E3): us-east.com').should('be.visible');
cy.findByText('US, Atlanta, GA (E3): us-southeast.com').should(
'be.visible'
);

// confirm endpoint types are present in the drawer
cy.findByText('and 3 more...').should('be.visible').click();
// Using contains since the text includes additional information, i.e. '| +2 regions | Show All'
cy.contains('US, Atlanta, GA (E3): us-southeast.com').should('be.visible');
cy.contains('+3 regions').should('be.visible');
cy.findByText('Show All').should('be.visible').click();

ui.drawer
.findByTitle('Regions / S3 Hostnames')
.should('be.visible')
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -233,9 +233,6 @@ describe('Object Storage Gen2 create bucket tests', () => {
.click();
});

// Wait for the newly 'created' mocked bucket to appear
cy.wait(['@getBuckets']);

// Confirm request body has expected data
cy.wait('@createBucket').then((xhr) => {
const requestPayload = xhr.request.body;
Expand Down Expand Up @@ -356,9 +353,6 @@ describe('Object Storage Gen2 create bucket tests', () => {
.click();
});

// Wait for the newly 'created' mocked bucket to appear
cy.wait(['@getBuckets']);
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So this was the culprit. We need to investigate why this wasn't an issue beforehand. πŸ€”


// Confirm request body has expected data
cy.wait('@createBucket').then((xhr) => {
const requestPayload = xhr.request.body;
Expand Down Expand Up @@ -483,9 +477,6 @@ describe('Object Storage Gen2 create bucket tests', () => {
.click();
});

// Wait for the newly 'created' mocked bucket to appear
cy.wait(['@getBuckets']);

// Confirm request body has expected data
cy.wait('@createBucket').then((xhr) => {
const requestPayload = xhr.request.body;
Expand Down Expand Up @@ -608,9 +599,6 @@ describe('Object Storage Gen2 create bucket tests', () => {
.click();
});

// Wait for the newly 'created' mocked bucket to appear
cy.wait(['@getBuckets']);

// Confirm request body has expected data
cy.wait('@createBucket').then((xhr) => {
const requestPayload = xhr.request.body;
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -27,28 +27,39 @@ export const HostNameTableCell = (props: Props) => {
if (!regionsLookup || !regionsData || !regions || regions.length === 0) {
return <TableCell>None</TableCell>;
}
const label = regionsLookup[storageKeyData.regions[0].id]?.label;
const s3Endpoint = storageKeyData?.regions[0]?.s3_endpoint;
const endpointType = storageKeyData?.regions[0]?.endpoint_type;
const formatEndpoint = (region: ObjectStorageKeyRegions) => {
const label = regionsLookup[region.id]?.label;
const endpointType = region.endpoint_type
? ` (${region.endpoint_type})`
: '';
return `${label}${endpointType}: ${region.s3_endpoint}`;
};

const firstRegion = regions[0];
const formattedFirstEndpoint = formatEndpoint(firstRegion);
const allEndpoints = regions.map(formatEndpoint).join('\n');

return (
<TableCell>
{label}
{endpointType && ` (${endpointType})`}: {s3Endpoint}&nbsp;
{storageKeyData?.regions?.length === 1 && (
<StyledCopyIcon text={s3Endpoint} />
{formattedFirstEndpoint}&nbsp;
{regions.length === 1 && (
<StyledCopyIcon text={firstRegion.s3_endpoint} />
)}
{storageKeyData.regions.length > 1 && (
<StyledLinkButton
onClick={() => {
setHostNames(storageKeyData.regions);
setShowHostNamesDrawers(true);
}}
type="button"
>
and {storageKeyData.regions.length - 1} more...
</StyledLinkButton>
{regions.length > 1 && (
<>
| +{regions.length - 1} regions |&nbsp;
jaalah-akamai marked this conversation as resolved.
Show resolved Hide resolved
<StyledLinkButton
onClick={() => {
setHostNames(regions);
setShowHostNamesDrawers(true);
}}
type="button"
>
Show All
</StyledLinkButton>
</>
)}
<StyledCopyIcon text={allEndpoints} />
</TableCell>
);
};
Expand All @@ -59,5 +70,5 @@ const StyledCopyIcon = styled(CopyTooltip)(({ theme }) => ({
top: 1,
width: 12,
},
marginLeft: theme.spacing(),
marginLeft: theme.spacing(0.5),
}));
Original file line number Diff line number Diff line change
Expand Up @@ -47,10 +47,10 @@ export const AccessSelect = React.memo((props: Props) => {

const { close: closeDialog, isOpen, open: openDialog } = useOpenClose();
const label = capitalize(variant);

// CORS is only available at a bucket level, not at an object level.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for this fix! πŸ™

const isCorsAvailable =
(variant === 'bucket' || variant === 'object') &&
endpointType !== 'E2' &&
endpointType !== 'E3';
variant === 'bucket' && endpointType !== 'E2' && endpointType !== 'E3';

const {
data: bucketAccessData,
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -64,7 +64,7 @@ export const BucketDetailLanding = React.memo((props: Props) => {

const { endpoint_type } = bucket ?? {};

const isSSLEnabled = endpoint_type !== 'E2' && endpoint_type !== 'E3';
const isGen2Endpoint = endpoint_type === 'E2' || endpoint_type === 'E3';

const tabs = [
{
Expand All @@ -75,15 +75,15 @@ export const BucketDetailLanding = React.memo((props: Props) => {
routeName: `${props.match.url}/access`,
title: 'Access',
},
...(flags.objectStorageGen2?.enabled
...(isObjectStorageGen2Enabled
jaalah-akamai marked this conversation as resolved.
Show resolved Hide resolved
? [
{
routeName: `${props.match.url}/properties`,
title: 'Properties',
},
]
: []),
...(isSSLEnabled
...(!isGen2Endpoint
? [
{
routeName: `${props.match.url}/ssl`,
Expand Down Expand Up @@ -136,7 +136,7 @@ export const BucketDetailLanding = React.memo((props: Props) => {
endpointType={endpoint_type}
/>
</SafeTabPanel>
{flags.objectStorageGen2?.enabled && bucket && (
{isObjectStorageGen2Enabled && bucket && (
<SafeTabPanel index={2}>
<BucketProperties bucket={bucket} />
</SafeTabPanel>
Expand Down
29 changes: 24 additions & 5 deletions packages/manager/src/queries/object-storage/requests.ts
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I didn't test this super thoroughly, but logic looks okay

Original file line number Diff line number Diff line change
Expand Up @@ -166,13 +166,29 @@ export const getAllBucketsFromEndpoints = async (
return { buckets: [], errors: [] };
}

// Initialize a Map to group endpoints by region for better error handling and flexibility.
const endpointsByRegion = new Map<string, ObjectStorageEndpoint[]>();

for (const endpoint of endpoints) {
const existingEndpoint = endpointsByRegion.get(endpoint.region) || [];

// Update the Map with the current endpoint, maintaining all endpoints per region.
endpointsByRegion.set(endpoint.region, [...existingEndpoint, endpoint]);
}

const results = await Promise.all(
endpoints.map((endpoint) =>
Array.from(endpointsByRegion.entries()).map(([region, regionEndpoints]) =>
getAll<ObjectStorageBucket>((params) =>
getBucketsInRegion(endpoint.region, params)
getBucketsInRegion(region, params)
)()
.then((data) => ({ buckets: data.data, endpoint }))
.catch((error) => ({ endpoint, error }))
.then((data) => ({
buckets: data.data,
endpoints: regionEndpoints,
}))
.catch((error) => ({
endpoints: regionEndpoints,
error,
}))
)
);

Expand All @@ -183,7 +199,10 @@ export const getAllBucketsFromEndpoints = async (
if ('buckets' in result) {
buckets.push(...result.buckets);
} else {
errors.push({ endpoint: result.endpoint, error: result.error });
// For each endpoint in the region, log the error to provide detailed error information.
result.endpoints.forEach((endpoint) => {
errors.push({ endpoint, error: result.error });
});
}
});

Expand Down
Loading