-
Notifications
You must be signed in to change notification settings - Fork 0
/
Copy pathanalysis.qmd
135 lines (83 loc) · 10.1 KB
/
analysis.qmd
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
::: {.callout-important}
This version of the AQuA Book is a preliminary ALPHA draft. It is still in development, and we are still working to ensure that it meets user needs.
The draft currently has no official status. It is a work in progress and is subject to further revision and reconfiguration (possibly substantial change) before it is finalised.
:::
# Analysis
During the analysis stage, the planned analysis is undertaken and assured, and progress and relevance are monitored. The design may be amended to account for any changing circumstances, emerging information, unexpected difficulties or limitations that may be encountered. This stage also includes maintaining appropriate and traceable records of the analysis and assurance activities conducted, changes, decisions and assumptions made. In some cases changes or limitations encountered may mean that the design or scoping stage need to be revisited to address these issues.
## Roles and responsibilities in the analysis stage
### The commissioner's responsibilities
The commissioner should:
* be available to provide input and clarifications to the analyst
* review any changes in design or methodology that the analyst brings to their attention
### The analyst's responsibilities
The analyst shall:
* follow the conduct the verification and validation activities that were designed as part of the analytical plan in [the design stage](design.html#the-design-stage)
* provide traceable documentation of the assurance they have undertaken
* respond to recommendations from the assurer and act on them as appropriate
* proportionately follow [best practice for code development](#assurance-of-code), where relevant
* produce documentation of the data (as described in [The Government Data Quality Framework](https://www.gov.uk/government/publications/the-government-data-quality-framework/the-government-data-quality-framework)) and methods used
* ensure all documentation is sufficient for the assurer to understand the approach
* document any changes to the analytical plan in a proportionate manner
* maintain appropriate contact with commissioner and assurer to provide an opportunity for them to advise on whether the analysis is still meeting the commissioner's needs or whether there are any new requirements
### The assurer's responsibilities
The assurer shall:
* review the assurance completed by the analyst
* carry out any further validation and verification they may see as appropriate
* report errors and areas for improvement to the analyst
* review that the work proportionately adheres to [best practice for code development](#assurance-of-code), where relevant
The assurer may need to:
* re-review the analytical work completed, as required
* provide feedback on changes to the analytical plan,
* consider whether they are qualified to provide rigorous assurance on the revised methodology
### The approver's responsibilities
The approver should be aware of the progress of the analysis and ensure that they are available for approving the work at the delivery stage.
## Assurance activities
### Verification and validation
Verification that the implemented methodology meets the design requirements should be part the analysis. [Whitener and Balci (1989)](https://typeset.io/pdf/guidelines-for-selecting-and-using-simulation-model-143kzp6h5s.pdf) reviewed verification techniques in relation to simulation modelling but these techniques also extend to analysis more broadly. They include:
* informal analysis - techniques that rely on human reasoning and subjectivity
* static analysis - tests that the implementation of the analysis before it is run (for example, checking that code adheres to code conventions)
* dynamic analysis - tests the behaviour of the system, model or code to find errors that arise during execution, includes [unit testing](https://en.wikipedia.org/wiki/Unit_testing), [integration testing](https://en.wikipedia.org/wiki/Integration_testing) and [stress testing](https://en.wikipedia.org/wiki/Stress_testing_(computing))
* symbolic analysis - particularly relevant to modelling and tests the transformation of symbolic proxies of model inputs into outputs during [the execution of a model](https://typeset.io/pdf/guidelines-for-selecting-and-using-simulation-model-143kzp6h5s.pdf), includes path tracing and cause-effect testing
* constraint analysis - particularly relevant to modelling and tests the implementation of constraints during model execution, includes checking the assertions of the model and boundary analysis
* formal analysis - tests logical correctness through [formal verification](https://en.wikipedia.org/wiki/Formal_verification#Formal_verification_for_software) such as logic or mathematical proofs
Validation is testing whether the product meets the requirements of users. It is important to involve the users in the process. [Methods for validation](https://en.wikipedia.org/wiki/Verification_and_validation#Aspects_of_analytical_methods_validation_) include quantification and judgment of acceptable sensitivity, specificity, accuracy, precision and reproducibility.
Validation of models includes testing the validity of the conceptual model and the operational validity of any computerized model.
You can read more about [techniques that may be useful in validation of models](https://www.informs-sim.org/wsc11papers/016.pdf).
The analyst has primary responsibility for conducting verification and validation. The assurer is responsible for reviewing the verification and validation that is carried out by the analyst, and for conducting or recommending additional verification and validation as required. The assurer may refer to the [specification document](definitions_and_key_concepts.html#specification-documentation) to assure that the analysis meets the specification.
### Data validity and data considerations
Testing data validity (for example, ensuring that data meet the specification for which they are used) is a vital part of analysis. Procedures for assuring data validity include testing for internal consistency, screening for data characteristics such as outliers, trends and expected distributions, and assuring robust data management practices such as automating data creation and data sourcing.
It is rare to have the perfect dataset for an analytical commission. This could be because:
* the data is not available in the time frame required for the ideal analysis
* the data definition does not perfectly align with the commission
* there are data or coverage gaps
* the data may be experimental or there are other reasons why it is not mature
When no data is available that is directly and precisely relevant to the parameter and conditions of interest it is often possible to use surrogate data. This is the measurements of another parameter (or of the parameter of interest under different conditions) that is related to the parameter and conditions of interest. This implies extrapolating between parameters, or between conditions for the same parameter. Although the use of surrogate data introduces further uncertainty additional to that already associated with the data itself, it may be possible to quantify this additional uncertainty using expert knowledge of the relationship between the surrogate and the parameter of interest.
The effect of using a proxy dataset should be explored and if the uncertainty associated with the dataset has a large bearing on the analysis, its appropriateness should be revisited. This exploration and the decision to use a particular dataset or input should be recorded for the assurer to verify.
### Assurance of code
The [Duck Book](https://best-practice-and-impact.github.io/qa-of-code-guidance/intro.html) provides detailed guidance on developing and assurance for delivering quality code. This includes guidance on:
* structuring code
* producing documentation
* using version control
* data management
* testing
* peer review
* automation
The analyst shall follow the guidance for good quality code development in a proportionate manner and the assurer shall review this accordingly.
## Documentation
The analyst should:
* maintain appropriate records of the work
* fully document any code following agreed standards
* log the data, assumptions and inputs used in the analysis, and decisions made in appropriate [documentation](definitions_and_key_concepts.html/#documentation))
* record the verification and validation that has been undertaken, documenting any activities that are outstanding and noting what remedial action has been taken and its effect on the analysis
* produce [user and technical documentation](definitions_and_key_concepts.html#user-technical-documentation)
For modelling, the analyst may include a model map that describes data flows and transformations.
## Treatment of uncertainty
While the scoping and design stages identified and described risks and uncertainties, the analysis stage assesses and quantifies how uncertainty may influence the analytical outcome and their contribution to the range and likelihoods of possible outcomes. [The Uncertainty Toolkit for Analysts](hhttps://analystsuncertaintytoolkit.github.io/UncertaintyWeb/chapter_5.html) reviews methods of quantifying uncertainty. The verification and validation by the analyst and assurer should assure the appropriate treatment of uncertainty.
## Black box models
[Black box models](definitions_and_key_concepts.html/#black-box-models) such as Artificial Intelligence (AI) and machine learning models are not as transparent as traditionally coded models. This adds challenge to the assurance of these models as compared to other forms of analysis.
* should include the verification steps set out in the design stage
* should include validation and verification of automatic tests to ensure the model behave as expected
* may include performance testing in a live environment
You can read more in the [Introduction to AI Assurance](https://www.gov.uk/government/publications/introduction-to-ai-assurance).
## Multi-use models
In multi-use models, analysis and edits may be carried out on individual elements of the model at differing times. This requires mechanisms for assuring that the changes integrate into the larger model as expected. For example, through the use of test-suites.