W3C ACT Implementation

Overview - Accessibility Conformance Testing (ACT)

The W3C established a community of developers of accessibility evaluation methodologies and testing tools and some accessibility Subject Matter Experts (SMEs). The community establishes and documents ACT Rules for testing the conformance of WCAG and ARIA. These test rules address automated, semi-automated, and manual testing, but since the Checker is primarily an automated test tool, that is the primary interest. ACT’s goal is to make accessibility testing more transparent, and thus reduce confusion caused by different interpretations of the standards (WCAG Success Criteria). ACT members write & share with the community and implement ACT Rules into their products and services. They also test their tool against the ACT Rules and measure consistency and accuracy as reported in their ACT Implementation reports. ACT is not intended for end-users of accessibility tools, web developers, content authors, QA testers, etc. For more overview and background information, see Accessibility Conformance Testing (ACT) Overview.

ACT Rules

1. All ACT rules are listed by the standard or specification that they cover, including WCAG, ARIA, Proposed Rules, and beyond.

2. All ACT rules follow a review and publishing process.

3. Each ACT rule has a 6 character ID, description, applicability, expectations, assumptions, the WCAG criteria it maps to (reports against), and example test cases.

4. Each rule’s Test Cases include Passing examples, Failing examples, and Inapplicable examples

   • See an example - Rule ID: 2779a5 - HTML page has non-empty title [w3.org/WAI/standards-guidelines/act/rules/2779a5/].

   • The ACT Rule ID is found in the URLs, such as:

     • Rule description URL: e.g. w3.org/WAI/standards-guidelines/act/rules/97a4e1/

     • Rule test case URL: e.g. w3.org/WAI/content-assets/wcag-act-rules/testcases/97a4e1/….html

Checker mapping to ACT

1. Individual Checker rules may test and reports against (map to) one or more ACT rules by identifying the ACT rules ID in its checker_rule_name_.ts file using the act: [ _act_rule_ID_here_ ], declaration.

   1. See example img_alt_valid.ts, this Checker rule maps to act: "23a2a8" around line 53 in the rule’s logic code.

   2. See example aria_id_unique.ts where a single Checker rule maps to more than one ACT rule: act: ["59796f", "6a7281"],

   3. See example aria_graphic_labelled.ts where a single Checker rules maps to multiple ACT test cases depending on the Fail reasons: act: [{ "7d6734": {"Pass_0": "pass", "Fail_1": "inapplicable", "Fail_2": "fail", "Fail_3": "inapplicable" } }],

   4. See example aria_img_labelled.ts where a single Checker rule maps to multiple ACT Rules and multiple test cases:

act: ["23a2a8", { "7d6734": {
                  "Pass_0": "pass",
                  "Fail_1": "inapplicable",
                  "Fail_2": "fail",
                  "Fail_3": "inapplicable"
                 }
      }],

ACT Results

1. See test-act-w3 in the accessibility-checker repo that uses Puppeteer to run the ACT test cases and report the results.

2. See the ACT Test Results (act-report-v2.txt) Artifact created from Actions by each PR build that details the results of running the Checker against the ACT rule Test Cases. The results file is available in Actions for debugging purposes.

   1. Each ACT rule is listed, each test case is listed, and if there is a mismatch, then there is a message, such as “Expected passed, but returned earl:failed”

   2. If no Checker rule maps to an ACT rule, then a message: “? No checker rules“ is listed.

   3. See sample ACT Test Results (act-report-v2.txt) file saved to Box. But, it is best to view the Artifact file from Actions for the most current results.

ACT Implementation report

• After each deployment, the Checker sends the results to ACT.

• The “official results” are publicly available a day later from the ACT Implementation Reports page when ACT updates it. See the Equal Access Accessibility Checker after each deployment. Results from other test tools are also publicly available. Note the date in the report. Ignore the Checker version in the report until ACT fixes the issue.

• The Checker results sent to ACT are formatted by ACT into the common-looking report.

• Does the ACT Community still use the Git repo at act-rules.github.io/rules? The JSON file that indexes all the rules is at act-rules.github.io/testcases.json.

Background for contributors

The following statements help clear up the purpose and high level understanding of these related topics and specs:

• The Evaluation and Report Language (EARL) is a machine-readable format for expressing test results (not the rules) that tool developers use to send their results to ACT.

• EARL is an RDF vocabulary (not rules & not results), the terms of which are defined by Evaluation And Report Language (EARL) 1.0 Schema.

• Some of the terms defined by EARL are being used by the Accessibility Conformance Testing (ACT) Rules Format (v 1.0 Oct 31 2019).

• The ACT Rules Format defines a format for writing accessibility test rules (not the results). This format is intended to enable a consistent interpretation of how the test was / is to be done ... and promote consistent results between complementary and competing 3rd party engines in reporting accessibility testing. Example tools include Axe-core, SortSite, Alpha (SiteImprove), etc. As of 2022 it is not clear yet if/when WebAIM's WAVE engine plans to publish an ACT implementation report.  

• Accessibility Conformance Testing (ACT) Overview - introduces the ACT Rules Format, lists the ACT Rules formally published by W3C, Who ACT is for, and Who develops ACT rules.

• Each ACT rule has the following sections: these links go to an example rule: ARIA required context role): Applicability, Expectation, Assumptions, Accessibility Support, Background, Test Cases, Glossary, Implementations, Change log.

• ACT Community Git repo also includes: 

  • Same above sections and an additional section called Acknowledgements of who created / submitted the rule (see example at https://act-rules.github.io/rules/ff89c9#acknowledgments)
  • Both the ACT Community Git repo and the W3C list of ACT Rules contain an Implementation Report per rule showing which tool supports that particular ACT rule. One goal of the Checker squad would be to determine and eventually publish the support in the Checker for the particular ACT rule. Perhaps the squad could publish that eventual implementation support report on the IBM Equal Access Toolkit Tools (https://www.ibm.com/able/toolkit/tools) page, Equal Access Git repo (https://github.com/IBMa/equal-access), and/or in the user documentation.