Data-navigator: Enabling navigable chart elements with alt text

[frankelavsky]

This PR is demonstrating a prototype of in-progress functionality and is subject to change.

Description

This work is part of an active consultation on the accessibility of React Spectrum Charts. Under the hood we want to enable accessible interaction and navigation of every chart in RSC, regardless of whether canvas or svg are used. In addition, we want to provide an API that enables core RSC developers to have opinionated design control over the navigation and interaction experience of each visualization type.

We are targeting smart defaults that make it easy for downstream developers to get accessible, rich, interactive data visualizations out of the box. But we also want to enable overrides for complex or advanced (but valid) use cases.

In general, this is a PR that is intended to demonstrate a working prototype (at minimum) and at best actually contribute working code into RSC across the ecosystem.

Related Issue

#482

Motivation and Context

If a chart can be hovered, clicked: a user should be able to do this with just a keyboard.
If a chart can't easily be summarized in just a couple of sentences: a screen reader user should be able to navigate the data and hear alt text for each element.
If a chart is being used for important decisions or is in an exploratory context: a user should be able to navigate and explore the data in a meaningful, structured way without requiring the use of a mouse or eyesight.

Presently, these experiences are not possible.

How Has This Been Tested?

Screenshots (if appropriate):

Element-level navigation

Alt text at the element-level

Group-level navigation

The goal will be to use Vega's signals to actually show the focus indication, while my new Navigator component will handle actually making accessible navigation possible (even if the visualization is rendered using canvas).

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

• I have signed the Adobe Open Source CLA.
• My code follows the code style of this project.
• My change requires a change to the documentation.
• I have updated the documentation accordingly.
• I have read the CONTRIBUTING document.
• I have added tests to cover my changes.
• All new and existing tests passed.

Current status

• = first pass, mostly working

Navigation (tested with screen reader and keyboard-only)

• Bar
• Dodge/stack
• Line
• Area
• Scatter
• Pie
• Combo

Semantics/labels

• Bar
• Dodge/stack
• Line
• Area
• Scatter
• Pie
• Combo

Interactivity (clicking/tooltips)

• Bar
• Dodge/stack
• Line
• Area
• Scatter
• Pie
• Combo

Focus Indication w/ signals

• Bar
• Dodge/stack
• Line
• Area
• Scatter
• Pie
• Combo

Bonus

• Exit element
• Help menu
• Key re-mapping?
• Dev-friendly API options?

[frankelavsky]

While this PR isn't ready to be merged, what I hope for is some feedback on the general approach before populating the prototype's approach across all other visualization types.

Right now the navigation capabilities are pretty cool, even with a never-before accomplished navigation type:

• Bar can be navigated left/right according to the left-right encoding (the "dimension" prop). In data navigator these encodings are all called "dimensions" via our dimensions structure generation api. This part isn't novel (pretty standard).
• However, the encoding for "metric" can also be turned into a navigable dimension. This means that for a visualization that might be more complex or have more going on, you can actually navigate from smallest -> largest value (or vice-versa) as well. This is the up-down navigation. Pretty much no other library does this right now (you typically have to resort and redraw visualizations to change nav order), save for tools like Olli but that only applies to scatterplots.
• For stacked/dodged you can also navigate across color using [ and ] keys. It may make more sense for these visualizations to use up/down and left/right for the categorical encodings and then [/] for metric-navigation.

The key bindings are set in the constants file, but obviously we can change these across charts and even allow end-users to set their own keybinds too (outside of the scope of this PR but worth thinking about!).

[marshallpete]

@frankelavsky sounds good. @c-lamoureux and I will try and provide feedback in the next couple of days. I'll try it out today and look through the approach.

[frankelavsky]

@marshallpete sounds good and thanks! Note that I recommend entering the chart via keyboard. It'll show a big group indicator and comma-periodgoes between high level groups. These are visually all the same at the high level (i use console logs to know where I'm at but a screen reader works too). But one is dimensions, one is metric, and (optionally) one is color. Enter goes down a level. If divisions exist (nested, for example) that is level 2. Otherwise, straight to the elements. Going back up is different depending on which "dimension" you want to traverse up.

I've got some design/ui ideas to help with wayfinding and keyboard commands we can think about too.

And any of this, from nav structure to keybinds to code quality can all be points of feedback. Nothing is sacred, it's a prototype after all.

[majornista]

• For stacked/dodged you can also navigate across color using [ and ] keys. It may make more sense for these visualizations to use up/down and left/right for the categorical encodings and then [/] for metric-navigation.

I think we agree here. I lean towards using left / right / up / down for navigation, Space/Enter for drilling in, and Escape for drilling out. The fewer unfamiliar interaction patterns, like [ and ] for metric-navigation, the better. I used to work at an educational publisher, building interactive learning materials and games for K-6, and a wise mentor would provide feedback like, "That's great, now make it make sense to a 2nd grader." We need to keep in mind that its rare for charts to be keyboard or screen reader accessible, so it will be more helpful to use familiar interaction patterns, rather than introducing something exciting and new that adds to the cognitive load of data navigating.

[marshallpete]

I like what you have here. I think the approach makes sense.

As this is intended to be a POC/Prototype, I'm not too worried about code structure and cleanliness.

The behavior lines up with what we had discussed and what I understood from our discussions.

The styling of things will probably be tweaked for a production version. We will have our designers provide feedback but once again, since this is a POC/prototype, I think it serves it's purpose.

[majornista]

@frankelavsky We have to account for the mobile/touch screen reader experience when navigating the chart. Javascript cannot handle swipe gestures to navigate from a mobile screen reader in the same way that it can handle keyboard or pointer events when the screen reader is not running, it's a privacy issue. The data navigator will need to include both the focused element and any item before or after it in the navigation order, so that as the user swipes to navigate, there is an element within the chart to navigate to.

Also, since javascript cannot interpret complicated gestures with a screen reader running, we should make sure that any ways to change the navigation mode are available to mobile screen reader users using simple double tap to activate gestures.

[frankelavsky]

The data navigator will need to include both the focused element and any item before or after it in the navigation order, so that as the user swipes to navigate, there is an element within the chart to navigate to.

@majornista - heck yes, absolutely agree. We used the 1-before, 1-after approach at Visa specifically for this reason (rendering these gives fallback support). I reckon we should default to the previous/next element that exists in the same dimension that we are assigning to the left/right directions.

I lean towards using left / right / up / down for navigation, Space/Enter for drilling in, and Escape for drilling out.

Also totally agree.

This means that we will limit dimensional navigation to 2 dimensions. So for cases like a color-encoded scatterplot we might run into some trouble! The left/right would move along x values, and up/down along y values.... but what do we do if there is also a legend that shows color categories?

In the Zong et al research project ("rich screen reader experiences", figure below), they had separate navigation spaces that were completely separated from each other: one was up/down/left/right and drill in/out (for x and y) and the other was via categories with left/right and drill in/out. Maybe we could let users choose which path at the start? Or maybe have a key command (like pressing X or something) that swaps between "modes?"

Also (in additional to the above stuff) I think that we need at least 2 things to make this prototype really usable:

1. a place for interaction instructions to live (like in a menu with each chart or via a link that can be discovered that points to a single adobe-federated website). I like the idea of a link because then you can have a single place for the instructions to live that won't be part of the charting library/maintenance/package size/etc and it will also reduce redundancy for users (rather than having a menu for every chart, like we did at Visa)
2. once we have 1, we should also consider on-demand interaction/navigation help (like pressing F1 or equivalent while on an element) to hear/see where/how to navigate (nobody really does this in other libraries right now but many users have requested this in past research studies I've ran). Maybe a modal, popup, or aria-live="polite" caption region under the chart?