aws-cdk-lib/assertions: Cannot run testing against ProductStack objects

[pahud]

Reopening this issue for #24988 from thebillest

Discussed in #24989

^{Originally posted by thebillest April 7, 2023}

Describe the bug

When trying to perform testing with cdk-nag, according to the methodology listed on this AWS blog, it fails if the stack passed into Annotations.fromStack() is a ProductStack.

Expected Behavior

The expectation would be that the test would pass/fail based on the CDK-Nag output/findings.

Current Behavior

The mentioned code will fail with an error similar to: Unable to find artifact with id "TestStackTestProductStack

Reproduction Steps

import { Annotations, Match } from 'aws-cdk-lib/assertions';
import { App, Aspects, Stack } from 'aws-cdk-lib';
import { AwsSolutionsChecks } from 'cdk-nag';
import { CdkTestProductStack } from '../lib/cdk_test-product-stack';

describe('cdk-nag AwsSolutions Pack', () => {
  let productStack: ProductStack;
  let stack: Stack;
  let app: App;
  // In this case we can use beforeAll() over beforeEach() since our tests 
  // do not modify the state of the application 
  beforeAll(() => {
    // GIVEN
    app = new App();
    stack = new Stack(app, 'test-stack');
    productStack = new CdkTestProductStack(stack, 'test-productStack');

    // WHEN
    Aspects.of(app).add(new AwsSolutionsChecks());
  });

  // THEN
  test('No unsuppressed Warnings', () => {
    const warnings = Annotations.fromStack(productStack).findWarning(
      '*',
      Match.stringLikeRegexp('AwsSolutions-.*')
    );
    expect(warnings).toHaveLength(0);
  });

  test('No unsuppressed Errors', () => {
    const errors = Annotations.fromStack(productStack).findError(
      '*',
      Match.stringLikeRegexp('AwsSolutions-.*')
    );
    expect(errors).toHaveLength(0);
  });
});

Possible Solution

No response

Additional Information/Context

No response

CDK CLI Version

2.69.0

Framework Version

No response

Node.js Version

19.8.1

OS

macOS Ventura 13.2.1

Language

Typescript

Language Version

No response

Other information

No response

[pahud]

Hi @thebillest

It seems this only happens when Annotations.fromStack() on a ProductStack. Is it correct?

Do you have any possible solution or proposed changes?

[thebillest]

According to the ProductStack documentation:

This stack will not be treated as an independent deployment artifact (won't be listed in "cdk list" or deployable through "cdk deploy"), but rather only synthesized as a template and uploaded as an asset to S3

When I go to the aws-cdk-lib/assertions, look at the Annotations class, and work down to the toMessages method (which is what fromStack calls), it appears that it expects to synth and grab artifactIds. These won't work for ProductStack objects.

[wanjacki]

ProductStack works a bit differently from regular stacks:
https://docs.aws.amazon.com/cdk/api/v2/docs/aws-cdk-lib.aws_servicecatalog.ProductStack.html
This stack will not be treated as an independent deployment artifact (won't be listed in "cdk list" or deployable through "cdk deploy"), but rather only synthesized as a template and uploaded as an asset to S3.

Essentially it just generates a CFN template when you synthesize it. I suggest calling a synth, generating the template which should appear in cdk.out, then running testing against that template.

[thebillest]

While the alternative process of synthesizing a ProductStack and performing testing on that works, does it make sense to integrate fromProductStack() on Annotations or to expand the fromStack() method to handle a ProductStack object?

As it currently stands, it's not trivial to render annotations for any ProductStack object.

[bdoyle0182]

Any update on this? This is a pretty large gap in ability to unit test stacks?

Or at the very least provide a code sample that can achieve a test against a ProductStack even if it's not trivial

[bdoyle0182]

I've figured out a trivial solution to this at least for Template assertions for anyone that finds this. I'm essentially just doing what the aws-cdk class does for loading the template if it's a nested stack.

  const createTemplateFromProductStack = (props: StackProps): Template => {
    const parentStack = new Stack(app, 'ParentStack', props);
    const stack = new MyProductStack(parentStack, 'StackTemplate', props);
    const assembly = app.synth();
    const productStack = JSON.parse(fs.readFileSync(path.join(assembly.directory, stack.templateFile)).toString('utf-8'));
    return Template.fromJSON(productStack);
  };

And for the cdk team I think this can be trivially fixed with something like this where I've bolded the addition:

function toTemplate(stack: Stack): any {
  const stage = Stage.of(stack);
  if (!Stage.isStage(stage)) {
    throw new Error('unexpected: all stacks must be part of a Stage or an App');
  }

  const assembly = stage.synth();
  if (stack.nestedStackParent **|| stack instanceOf ProductStack**) {
    // if this is a nested stack (it has a parent), then just read the template as a string
    return JSON.parse(fs.readFileSync(path.join(assembly.directory, stack.templateFile)).toString('utf-8'));
  }
  return assembly.getStackArtifact(stack.artifactId).template;
}