How is this tool used?

[mgifford]

As I've already noted, it can be difficuclt to get started. 

Even understanding what are the intentions of using this multi-tool tool. 

Is this a CI/CD tool? 

Should I be able to scan a page from the command line? 

Do I need to create custom configs for each page I want scanned? 

I assume it doesn't act like https://unlighthouse.dev/ or https://github.com/GovTechSG/purple-a11y in scanning whole domains.

You're a lot of tools, but it would be interesting to see the results of a page.  Sometimes less is more in the context of accessibility.

[jrpool]

Thank you for describing ways in which the usage documentation is confusing. This is helpful for making the README file clearer.

Basically, Testaro makes one contribution: facilitating ensemble testing (testing with 11 different tools) for web accessibility. That contribution can be leveraged in various environments, such as CLIs, web UIs, development pipelines, and surveillance daemons. Testaro provides some help in setting up such environments, such as the call module for CLI usage, and for each Testaro module the README file shows how to use it programmatically and via a CLI. But integrating Testaro into any testing practice will require development work similar to the work for integrating a single testing tool; the difference is that this work gives you findings from 11 tools instead of 1.

In my own work putting Testaro to use, I am also making heavy use of the Testilo package. That streamlines the use of Testaro, but still requires development of a system to manage user requests, authorization, scheduling, recordkeeping, etc.

Your comment suggests it could be helpful to produce a few tutorials on typical use cases. I am working on one overview article now, but more specific examples might ease onboarding.

[kathari00]

@mgifford I second this. A quick start guide would be more beneficial than in-depth explainations for users. The idea is great, I want to use this tool for academic research.

Please provide more comprehensive and basic tutorials or get someone on board who can write them.

[jrpool]

Thank you, @kathari00, for confirming the desire for tutorial content.

The right strategy, it seems to me, is to produce, or encourage others to produce, software that applies Testaro and Testilo to common use cases. That software should be easy enough to use that a tutorial might not even be necessary.

Could you describe an example of the use of Testaro in academic research that you have in mind? That could be one such use case, so I’m curious about it.

[kathari00]

Hello,

my task involves evaluating university learning materials for accessibility compliance, particularly with the European Norm 301 549. During my research, I came across an article that proposed orchestrating automated testing. This led me to discover Testaro.

I have successfully produced results using Testaro with the help of Testilo. While exploring, I found Autotest, and its README provided a clearer overview of its usage. I understand why the repository was split, as it enables different usage scenarios.

The documentation contains the most essential information but it lacks intuitiveness. Additionally, some code examples do not seem to work as intended.

The excellent work on these tools could gain more visibility with a more accessible entry point. Providing a functional starter project accompanied by a basic tutorial could save users time (like Autotest). A detailed tutorial on setting up and using a server—including its subcases, such as splitting jobs—would be beneficial.

I am still trying to figure out how I can unhide all elements as described in the documentation.
To my understanding I would need to add it to the acts array of a job. However an error is being thrown:
ERROR making all elements visible (page.$$eval: ReferenceError: Can't find variable: act).

Thank you for your efforts!

[jrpool]

@kathari00 is correct that there was a bug in the testaro tool’s reveal act. The testing features of Testaro are currently too meager to have caught that bug. Your report of this bug is much appreciated.

I have corrected that bug, and now it seems to work without error, in version 53.0.8.

However, some haphazard testing with it makes me unsure that reveal makes the difference it would be expected to make. In some cases I am finding the total count of visible elements is lower after reveal than before it, the opposite of what I would expect. Rigorous testing with mocked data could address this, but has not yet been implemented.

Here is an example of a job performing such a haphazard test. The without-reveal version would be identical, except that the launch and reveal acts would be removed.

{
  "id": "221111T1111-aaa-1",
  "what": "Testing",
  "strict": false,
  "standard": "also",
  "observe": false,
  "device": {
    "id": "default",
    "windowOptions": {
      "reducedMotion": "no-preference"
    }
  },
  "browserID": "webkit",
  "timeLimit": 30,
  "creationTimeStamp": "240610T1529",
  "executionTimeStamp": "240610T1529",
  "sendReportTo": "",
  "target": {
    "what": "AccessiBe",
    "url": "https://accessibe.com"
  },
  "sources": {
    "script": "test",
    "batch": "test",
    "requester": ""
  },
  "acts": [
    {
      "type": "launch"
    },
    {
      "type": "reveal",
      "what": "Make all elements visible"
    },
    {
      "type": "test",
      "launch": {},
      "which": "testaro",
      "what": "Testaro",
      "rules": [
        "y",
        "bulk"
      ],
      "withItems": true,
      "stopOnFail": false
    }
  ]
}

[jrpool]

@kathari00: Can you give me an idea of whether the use case you describe would be best served by an API service, a service with a browser-based user interface, or a package that adds a front end to Testaro and Testilo that a user would install on a local computer?

[kathari00]

Hi it works better. Thanks so much. Regarding your example: there is a launch act before the reveal and another launch in the test act. When testing it like this multiple windows are openend for the same test because there are multiple launches. Is this expected behaviour?

I tried summarizing multiple scored reports but it is described here that

The summarize module summarizes one report when invoked by a module, but the call module invoked by a user can call summarize multiple times to summarize multiple reports and combine those summaries into a file.

Is there a way to summarize mutliple reports by module into one file?

[kathari00]

Since I want to process the results more efficiently, I would prefer an API service. At the moment, I am preparing the jobs with Testilo and then running them with Testaro (all by module), but I am not setting up a service yet.

[eta-orionis]

The discussion has gone a bit off-track, so I'd kindly remind everyone of the original question: How is this tool used?

• installing it with npm install testaro puts some files under node_modules, but they don't seem to export anything that can be used with import or require, like normal npm packages do.
• cloning the repo with git clone does not match the instructions in the README that "You can install Testaro as you would install any npm package."

I second @mgifford's initial comment. A step-by-step installation and usage guide, with the exact commands spelled out, would help immensely.

[jrpool]

I am grateful to @kathari00 , @eta-orionis , and @mgifford for the comments on this issue.

The blog entry on Testaro and Testilo advises that the critical missing piece to make efficient use of these packages is a “controller”. Our work implementing a company-specific controller indicates that it can provide a user-friendly and developer-friendly platform for ensemble accessibility testing and reporting. However, developing and maintaining one for an organization can be a substantial project, for two main reasons: (1) The organization has a digital infrastructure that the controller needs to conform to. (2) The potential users in the organization have various needs.

An external controller offering a service would have a challenge satisfying the various needs of users in multiple organizations, and would also be outside them, so internal web pages that are hidden from the public Internet could not be tested by it. An internal controller has access to those pages. Not all the tests can be run on them, but about 90% can.

Our proof-of-concept controller runs on a server inside the organization. It manages periodic tests on a classified inventory of publicly exposed pages both inside and outside the organization. It also manages on-demand testing of those or any other pages, including company-internal ones. The reports that Testaro produces get scored with the help of Testilo and then saved in a database. The controller has a UI that makes reports available on the existing results, in various formats, including numerical and graphical summarizations. It also accepts requests for on-demand tests via an API, so automated application-development workflows can include Testaro tests. That part is complicated, because ideally the workflow tentatively deploys the new version of an application before the controller starts to work, so the controller has actually operating pages to use for testing. Workstations with Testaro installed use the netWatch function to request jobs from the controller and return reports back to the controller after performing the tests.

A variation on this architecture is to run Testaro on the server itself. We are exploring that. A whole other strategy could be to submit Testaro results to an existing reporting platform, such as Lighthouse or Grafana.

The idea of a tutorial or an easy-to-use service is appealing, but the plethora of infrastructures and needs makes me unsure of the best initial step. I can more easily imagine a demo application running on a public server that lets a person test a publicly reachable page and get a user-friendly report of the results.

Your further comments on all this are welcome.

[jrpool]

Also, @kathari00 is right that the README does not explain when to clone the repository versus when to install the NPM package. Correcting that is on my agenda.

[jrpool]

The latest version of Testaro has a README file with substantial changes, especially in the Installation section. The intent is to correctly guide two different modes of use of Testaro. If you inspect this README and find anything amiss in it, I would be most grateful to learn what further improvements are necessary.

[jrpool]

I forgot to reply to “Is there a way to summarize mutliple reports by module into one file?” by @kathari00. Yes, in the controller I am developing there is a function that summarizes a report by extracting about 15 facts from it, and there is a function that combines the summaries of all reports into a single file. I can imagine that users would want different facts included in such summaries and would also want different criteria for which reports to summarize and which to omit. I am finding such a summary helpful in making reporting efficient. If you want to compare, say, the total scores from 100 reports, and if you have not indexed that property of reports in a database of reports, then getting the scores from a summary is faster than getting them from the original reports. Managing such a summary requires ensuring that, whenever a report gets added to the collection, its summary gets added to the collective summary, so they remain synchronized.

[eta-orionis]

Thanks @jrpool ! I finally managed to get it running and have a first useful report.