diff --git a/analysis.qmd b/analysis.qmd
index d3214e8..56a4457 100644
--- a/analysis.qmd
+++ b/analysis.qmd
@@ -1,128 +1,135 @@
::: {.callout-important}
-This version of the AQuA book is a preliminary ALPHA draft. It is still in development, and we are still working to ensure that it meets user needs.
+This version of the AQuA Book is a preliminary ALPHA draft. It is still in development, and we are still working to ensure that it meets user needs.
The draft currently has no official status. It is a work in progress and is subject to further revision and reconfiguration (possibly substantial change) before it is finalised.
:::
# Analysis
-## Introduction and overview
-
-The analysis stage is where planned analysis is undertaken and assured, and progress and relevance are monitored. During this stage, the design may be amended to account for changing circumstances, emerging information or unexpected difficulties or limitations encountered. This stage also includes maintaining appropriate and traceable records of the analysis and assurance activities conducted, changes, decisions and assumptions made. In some cases, changes or limitations encountered may necessitate a return to either the design or scoping stage.
+During the analysis stage, the planned analysis is undertaken and assured, and progress and relevance are monitored. The design may be amended to account for any changing circumstances, emerging information, unexpected difficulties or limitations that may be encountered. This stage also includes maintaining appropriate and traceable records of the analysis and assurance activities conducted, changes, decisions and assumptions made. In some cases changes or limitations encountered may mean that the design or scoping stage need to be revisited to address these issues.
## Roles and responsibilities in the analysis stage
-### The Commissioner's responsibilities during the analysis stage
-
-* The Commissioner should be available to provide input and clarifications to the Analyst.
-
-* The Commissioner’s should review any changes in design or methodology that the Analyst brings to their attention.
-
-
-
+### The commissioner's responsibilities
-### The Analyst's responsibilities during the analysis stage
+The commissioner should:
-* The Analyst shall follow the conduct the verification and validation activities that were designed as part of the analytical plan in [the design stage](design.html#the-design-stage). They shall provide traceable documentation of the assurance they have undertaken. They shall respond to recommendations from the Assurer and act on them as appropriate.
+* be available to provide input and clarifications to the analyst
+* review any changes in design or methodology that the analyst brings to their attention
-* When the analysis includes coding, the Analyst shall proportionately follow [best practice for code development](#assurance-of-code).
-* The Analyst shall produce documentation of the data (see [The Government Data Quality Framework](https://www.gov.uk/government/publications/the-government-data-quality-framework/the-government-data-quality-framework)) and methods used. The Analyst shall ensure these are sufficient for the Assurer to understand the approach.
+### The analyst's responsibilities
-* The Analyst shall document any changes to the analytical plan in a proportionate manner.
+The analyst shall:
-* The Analyst shall maintain appropriate contact with Commissioner and Assurer. This provides and opportunity for them to advise on whether the analysis is still meeting the Commissioner's needs or whether there are any new requirements.
+* follow the conduct the verification and validation activities that were designed as part of the analytical plan in [the design stage](design.html#the-design-stage)
+* provide traceable documentation of the assurance they have undertaken
+* respond to recommendations from the assurer and act on them as appropriate
+* proportionately follow [best practice for code development](#assurance-of-code), where relevant
+* produce documentation of the data (as described in [The Government Data Quality Framework](https://www.gov.uk/government/publications/the-government-data-quality-framework/the-government-data-quality-framework)) and methods used
+* ensure all documentation is sufficient for the assurer to understand the approach
+* document any changes to the analytical plan in a proportionate manner
+* maintain appropriate contact with commissioner and assurer to provide an opportunity for them to advise on whether the analysis is still meeting the commissioner's needs or whether there are any new requirements
+### The assurer's responsibilities
-### The Assurer's responsibilities during the analysis stage
+The assurer shall:
-* The Assurer shall review the assurance completed by the Analyst, carry out any further validation and verification they may see as appropriate, and report errors and areas for improvement to the Analyst. The Assurer may then need to re-review the analytical work completed, as required.
+* review the assurance completed by the analyst
+* carry out any further validation and verification they may see as appropriate
+* report errors and areas for improvement to the analyst
+* review that the work proportionately adheres to [best practice for code development](#assurance-of-code), where relevant
-* When the analysis includes coding, the Assurer shall review that the work proportionately adheres to [best practice for code development](#assurance-of-code).
+The assurer may need to:
-* The Assurer may be required to provide feedback on changes to the analytical plan, and consider whether they are qualified to provide rigorous assurance on the revised methodology.
+* re-review the analytical work completed, as required
+* provide feedback on changes to the analytical plan,
+* consider whether they are qualified to provide rigorous assurance on the revised methodology
+### The approver's responsibilities
+The approver should be aware of the progress of the analysis and ensure that they are available for approving the work at the delivery stage.
-
-### The Approver's responsibilities during the analysis stage
-
-The Approver should be aware of the progress of the analysis and ensure that they are available for approving the work at the delivery stage.
-
-
-## Assurance activities in the analysis stage
+## Assurance activities
### Verification and validation
-Verification that the implemented methodology meets the design requirements should be incorporated as part the analysis. [Whitener and Balci (1989)](https://typeset.io/pdf/guidelines-for-selecting-and-using-simulation-model-143kzp6h5s.pdf) reviewed verification techniques in relation to simulation modelling. These techniques extend to analysis more broadly. These include:
-
-- Informal analysis: techniques that rely on human reasoning and subjectivity.
-- Static analysis: tests that the implementation of the analysis before it is run. For example, checking that code adheres to code conventions, structural analysis of the code by examining graphs of control and data flows, .
-- Dynamic analysis: tests the behaviour of the system, model or code to find errors that arise during execution. This includes [unit testing](https://en.wikipedia.org/wiki/Unit_testing), [integration testing](https://en.wikipedia.org/wiki/Integration_testing) and [stress testing](https://en.wikipedia.org/wiki/Stress_testing_(computing))
-- Symbolic analysis: particularly relevant to modelling and tests the transformation of symbolic proxies of model inputs into outputs during the execution of a model. Includes path tracing and cause-effect testing (see [Whitener and Balci (1989)](https://typeset.io/pdf/guidelines-for-selecting-and-using-simulation-model-143kzp6h5s.pdf) )
-- Constraint analysis: particularly relevant to modelling and tests the implementation of constraints during model execution. This includes checking the assertions of the model and boundary analysis.
-- Formal analysis: tests logical correctness through [formal verification](https://en.wikipedia.org/wiki/Formal_verification#Formal_verification_for_software) such as logic or mathematical proofs.
+Verification that the implemented methodology meets the design requirements should be part the analysis. [Whitener and Balci (1989)](https://typeset.io/pdf/guidelines-for-selecting-and-using-simulation-model-143kzp6h5s.pdf) reviewed verification techniques in relation to simulation modelling but these techniques also extend to analysis more broadly. They include:
+* informal analysis - techniques that rely on human reasoning and subjectivity
+* static analysis - tests that the implementation of the analysis before it is run (for example, checking that code adheres to code conventions)
+* dynamic analysis - tests the behaviour of the system, model or code to find errors that arise during execution, includes [unit testing](https://en.wikipedia.org/wiki/Unit_testing), [integration testing](https://en.wikipedia.org/wiki/Integration_testing) and [stress testing](https://en.wikipedia.org/wiki/Stress_testing_(computing))
+* symbolic analysis - particularly relevant to modelling and tests the transformation of symbolic proxies of model inputs into outputs during [the execution of a model](https://typeset.io/pdf/guidelines-for-selecting-and-using-simulation-model-143kzp6h5s.pdf), includes path tracing and cause-effect testing
+* constraint analysis - particularly relevant to modelling and tests the implementation of constraints during model execution, includes checking the assertions of the model and boundary analysis
+* formal analysis - tests logical correctness through [formal verification](https://en.wikipedia.org/wiki/Formal_verification#Formal_verification_for_software) such as logic or mathematical proofs
+Validation is testing whether the product meets the requirements of users. It is important to involve the users in the process. [Methods for validation](https://en.wikipedia.org/wiki/Verification_and_validation#Aspects_of_analytical_methods_validation_) include quantification and judgment of acceptable sensitivity, specificity, accuracy, precision and reproducibility.
-Validation refers to testing whether the product meets the requirements of users. Hence, it is important to involve the users in the process. [Methods for validation](https://en.wikipedia.org/wiki/Verification_and_validation#Aspects_of_analytical_methods_validation_) include quantification and judgment of acceptable sensitivity, specificity, accuracy, precision and reproducibility.
+Validation of models includes testing the validity of the conceptual model and the operational validity of any computerized model.
-Validation of models includes testing the validity of the conceptual model, and testing the operational validity of any computerized model. Techniques that may be useful in validation of models are reviewed by [Sargent (2011)](https://www.informs-sim.org/wsc11papers/016.pdf).
+You can read more about [techniques that may be useful in validation of models](https://www.informs-sim.org/wsc11papers/016.pdf).
-The Analyst has primary responsibility for conducting verification and validation. The Assurer is responsible for reviewing the verification and validation that is carried out by the Analyst, and for conducting or recommending additional verification and validation as required. The Assurer may refer to the [specification document](definitions_and_key_concepts.html#specification-documentation) to assure that the analysis meets the specification.
+The analyst has primary responsibility for conducting verification and validation. The assurer is responsible for reviewing the verification and validation that is carried out by the analyst, and for conducting or recommending additional verification and validation as required. The assurer may refer to the [specification document](definitions_and_key_concepts.html#specification-documentation) to assure that the analysis meets the specification.
### Data validity and data considerations
-Testing data validity (i.e. that data meet the specification for which they are used) is a vital part of analysis. Procedures for assuring data validity include testing for internal consistency, screening for data characteristics (outliers, trends, expected distributions etc), and assuring robust data management practices (e.g. automating data creation and data sourcing).
+Testing data validity (for example, ensuring that data meet the specification for which they are used) is a vital part of analysis. Procedures for assuring data validity include testing for internal consistency, screening for data characteristics such as outliers, trends and expected distributions, and assuring robust data management practices such as automating data creation and data sourcing.
-It is rare to have the perfect dataset for an analytical commission. Reasons for this include:
+It is rare to have the perfect dataset for an analytical commission. This could be because:
-* The data is not available in the time frame required for the ideal analysis;
-* The data definition does not perfectly align with the commission;
-* There are data or coverage gaps;
-* The data may be experimental or there are other reasons why it is not ‘mature’.
+* the data is not available in the time frame required for the ideal analysis
+* the data definition does not perfectly align with the commission
+* there are data or coverage gaps
+* the data may be experimental or there are other reasons why it is not mature
-Often, no data is available that are directly and precisely relevant to the parameter and conditions of interest. In such cases, it is often possible to use surrogate data. This is measurements of another parameter, or of the parameter of interest under different conditions, that is related to the parameter and conditions of interest. This implies an extrapolation between parameters, or between conditions for the same parameter. The use of surrogate data introduces further uncertainty, additional to that associated with the data itself. It may be possible to quantify this additional uncertainty using expert knowledge of the relationship between the surrogate and the parameter of interest.
+When no data is available that is directly and precisely relevant to the parameter and conditions of interest it is often possible to use surrogate data. This is the measurements of another parameter (or of the parameter of interest under different conditions) that is related to the parameter and conditions of interest. This implies extrapolating between parameters, or between conditions for the same parameter. Although the use of surrogate data introduces further uncertainty additional to that already associated with the data itself, it may be possible to quantify this additional uncertainty using expert knowledge of the relationship between the surrogate and the parameter of interest.
-The effect of using a proxy dataset should be explored and, if the uncertainty associated with the dataset has a large bearing on the analysis, its appropriateness should be revisited. This exploration, and the decision to use a particular dataset or input, should be recorded for the benefit of the Assurer.
+The effect of using a proxy dataset should be explored and if the uncertainty associated with the dataset has a large bearing on the analysis, its appropriateness should be revisited. This exploration and the decision to use a particular dataset or input should be recorded for the assurer to verify.
### Assurance of code
-The [Duck Book](https://best-practice-and-impact.github.io/qa-of-code-guidance/intro.html) provides detailed guidance on developing and assurance for delivering quality code. This includes guidance on structuring code, producing documentation, using version control, data management, testing, peer review, and automation. The Analyst shall follow the guidance for good quality code development in a proportionate manner, and the Assurer shall review this accordingly.
+The [Duck Book](https://best-practice-and-impact.github.io/qa-of-code-guidance/intro.html) provides detailed guidance on developing and assurance for delivering quality code. This includes guidance on:
+* structuring code
+* producing documentation
+* using version control
+* data management
+* testing
+* peer review
+* automation
-## Documentation in the analysis stage
+The analyst shall follow the guidance for good quality code development in a proportionate manner and the assurer shall review this accordingly.
-The Analyst should:
-* Maintain appropriate records of the work;
-* Fully document any code following agreed standards;
-* Log the data, assumptions and inputs used in the analysis, and decisions made (see [documentation](definitions_and_key_concepts.html/#documentation));
-* Record the verification and validation that has been undertaken, documenting any activities that are outstanding and noting what remedial action has been taken and its effect on the analysis;
-* Produce [user and technical documentation](definitions_and_key_concepts.html#user-technical-documentation). For modelling, the Analyst may include a model map that describes data flows and transformations.
+## Documentation
-## Treatment of uncertainty in the analysis stage
+The analyst should:
-While the Scoping and Design stages identified and described risks and uncertainties, the Analysis stage aims to assess and quantify how uncertainty may influence the analytical outcome and their contribution to the range and likelihoods of possible outcomes. [The Uncertainty Toolkit for Analysts](hhttps://analystsuncertaintytoolkit.github.io/UncertaintyWeb/chapter_5.html) reviews methods of quantifying uncertainty. The verification and validation by the Analyst and Assurer should assure the appropriate treatment of uncertainty.
+* maintain appropriate records of the work
+* fully document any code following agreed standards
+* log the data, assumptions and inputs used in the analysis, and decisions made in appropriate [documentation](definitions_and_key_concepts.html/#documentation))
+* record the verification and validation that has been undertaken, documenting any activities that are outstanding and noting what remedial action has been taken and its effect on the analysis
+* produce [user and technical documentation](definitions_and_key_concepts.html#user-technical-documentation)
+For modelling, the analyst may include a model map that describes data flows and transformations.
+## Treatment of uncertainty
-## Black box models and the analysis stage
+While the scoping and design stages identified and described risks and uncertainties, the analysis stage assesses and quantifies how uncertainty may influence the analytical outcome and their contribution to the range and likelihoods of possible outcomes. [The Uncertainty Toolkit for Analysts](hhttps://analystsuncertaintytoolkit.github.io/UncertaintyWeb/chapter_5.html) reviews methods of quantifying uncertainty. The verification and validation by the analyst and assurer should assure the appropriate treatment of uncertainty.
-[Black box models](definitions_and_key_concepts.html/#black-box-models) such as AI and ML models are not as transparent as traditionally coded models. This adds challenge to the assurance of these models as compared to other forms of analysis.
+## Black box models
-Assurance activities during the Analysis stage:
+[Black box models](definitions_and_key_concepts.html/#black-box-models) such as Artificial Intelligence (AI) and machine learning models are not as transparent as traditionally coded models. This adds challenge to the assurance of these models as compared to other forms of analysis.
-* may include performance testing in a live environment and
-* should include the verification steps set out in the Design Phase
+* should include the verification steps set out in the design stage
* should include validation and verification of automatic tests to ensure the model behave as expected
-See the [Introduction to AI Assurance](https://www.gov.uk/government/publications/introduction-to-ai-assurance) for further details.
-
+* may include performance testing in a live environment
+You can read more in the [Introduction to AI Assurance](https://www.gov.uk/government/publications/introduction-to-ai-assurance).
-## Multi-use models and the analysis stage
+## Multi-use models
-In multi-use models, analysis and edits may be carried out on individual elements of the model at differing times. This calls for mechanisms for assuring that the changes integrate into the larger model as expected, for example, through the use of test-suites.
+In multi-use models, analysis and edits may be carried out on individual elements of the model at differing times. This requires mechanisms for assuring that the changes integrate into the larger model as expected. For example, through the use of test-suites.
diff --git a/analytical_lifecycle.qmd b/analytical_lifecycle.qmd
index fdbd960..e75cc36 100644
--- a/analytical_lifecycle.qmd
+++ b/analytical_lifecycle.qmd
@@ -1,50 +1,52 @@
::: {.callout-important}
-This version of the AQuA book is a preliminary ALPHA draft. It is still in development, and we are still working to ensure that it meets user needs.
+This version of the AQuA Book is a preliminary ALPHA draft. It is still in development, and we are still working to ensure that it meets user needs.
The draft currently has no official status. It is a work in train and is subject to further revision and reconfiguration (possibly substantial change) before it is finalised.
:::
# Roles and the analytical lifecycle
-An analytical project can be viewed as a variation on an archetypal project defined by roles and stages. This chapter gives an overview of the roles and the project stages in the analytical lifecycle. Subsequent chapters go into the details of each stage, and the responsibilities of each role during a given stage.
+An analytical project can be viewed as a variation on an archetypal project defined by roles and stages. This chapter gives an overview of the roles and the project stages in the analytical lifecycle. Chapters 6 to 9 give more explanation of each stage and the responsibilities of each role during a given stage.
## Roles and responsibilities
-
Organisations may have their own titles for the main functional roles involved in analysis that are set out here.
-Each role may be fulfilled by a team or committee of people. However, a single individual will have overall accountability (such as the chair of a committee) for each role.
+Each role may be fulfilled by a team or committee of people. However, a single individual (for example, the chair of a committee) will have overall accountability for each role.
+The AQuA Book defines four roles.
-The AQuA book defines the following roles:
+The commissioner (may be known as customer):
-* **Commissioner** (may be known as customer)
- + Requests the analysis and sets out their requirements
- + Agrees what the analyst is going to do will satisfy the need
- + Accepts the analysis and assurance as fit for purpose
+* requests the analysis and sets out their requirements
+* agrees what the analyst is going to do will satisfy the need
+* accepts the analysis and assurance as fit for purpose
-* **Analyst**
- + Designs the approach, including the assurance, to meet the commissioner’s requirements
- + Agrees the approach with the Commissioner
- + Carries out the analysis
- + Carries out their own assurance
- + Acts on findings from the Assurer
- + Can be a group of analysts, in which case the lead analyst is responsible
+The analyst:
-* **Assurer** (may be known as Analytical Assurer, Assuring Analyst)
- + Reviews the assurance completed by the Analyst
- + Carries out any further validation and verification they may see as appropriate
- + Reports errors and areas for improvement to the analyst
- + Re-reviews as required
- + Confirms the work has been appropriately scoped, executed, validated and verified and documented to the Approver
- + Can be a group of assurers. In which case the leader of the group is responsible. They must be independent from the analysts.
+* designs the approach, including the assurance, to meet the commissioner’s requirements
+* agrees the approach with the commissioner
+* carries out the analysis
+* carries out their own assurance
+* acts on findings from the assurer
+* can be a group of analysts, in which case the lead analyst is responsible
-* **Approver** (may be known as Senior Analyst or Senior Responsible Officer (“SRO”))
- + Scrutinises the work of the Analyst and Assurer
- + Confirms (if necessary) to the Analyst, Assurer and Commissioner that the work has been appropriately assured
-
-The roles of Analyst and Assurer shall be distinct from each other. The Analyst should carry out their own assurance but responsibility for formal assurance to the Approver and Commissioner lies with the Assurer. In some instances, particularly for quick and / or simple analysis, an individual may deliver more than one of the roles apart from the Assurer and Analyst roles which shall be separate from one another in all cases.
+The assurer (may be known as the analytical assurer or assuring analyst):
+* reviews the assurance completed by the analyst
+* carries out any further validation and verification they may see as appropriate
+* reports errors and areas for improvement to the analyst
+* undertakes repeated reviews as required
+* confirms the work has been appropriately scoped, executed, validated and verified and documented to the approver
+* can be a group of assurers, in which case the leader of the group is responsible
+* must be independent from the analysts
+
+The approver (may be known as senior analyst or senior responsible officer):
+
+* scrutinises the work of the analyst and assurer
+* confirms (if necessary) to the analyst, assurer and commissioner that the work has been appropriately assured
+
+The roles of analyst and assurer shall be distinct from each other. The analyst should carry out their own assurance but responsibility for formal assurance to the approver and commissioner lies with the assurer. In some instances, particularly for quick or simple analysis, an individual may fulfill more than one of the roles, apart from the assurer and analyst roles which shall be separate from one another in all cases.
## The analytical lifecycle
@@ -54,52 +56,52 @@ Quality assurance activities should take place throughout all stages of an analy
Figure 2 is adapted from the [Government Functional Standard for Analysis](https://www.gov.uk/government/publications/government-analysis-functional-standard--2). Analytical quality assurance activities should take place during every phase of the cycle and should consider proportionality, although analytical quality considerations may vary depending on project governance and the specific phase of the cycle. All projects will involve some element of every phase of the cycle, even if this is not clearly defined.
-It is important that [proportionality](proportionality.qmd) is considered and that there is transparency of the analytical decisions, process, limitations and changes made at each stage to enable effective assurance and communication. This should be enabled by:
+It is important that [proportionality](proportionality.qmd) is considered and that there is transparency of the analytical decisions, process, limitations and changes made at each stage to enable effective assurance and communication. This should be clearly shown in:
-* Clear documentation of the analysis, assumptions and data,
-* Clear records of the analytical decisions made; and
-* Clear records of the quality assurance processes and checks completed.
+* documentation of the analysis, assumptions and data
+* records of the analytical decisions made
+* records of the quality assurance processes and checks completed
### Engagement and scoping
-Analytical projects typically start with customer engagement although other events may trigger analytical projects. Scoping ensures that an appropriate, common understanding of the problem is defined and that expectations are aligned with what can be delivered. During this phase the Commissioner plays an important role in communicating the questions to be addressed and working with the Analyst to ensure the requirements and scope are defined and understood.
+Analytical projects typically start with customer engagement although other events may trigger analytical projects. Scoping ensures that an appropriate, common understanding of the problem is defined and that expectations are aligned with what can be produced. During this phase the commissioner plays an important role in communicating the questions to be addressed and working with the analyst to ensure the requirements and scope are defined and understood.
-Where analysis requires multiple cycles, for example to develop, use and update analytical models, this phase may follow on from the Delivery and Communication phase. In these cases, the phase will concentrate on the scope of the questions to be addressed in the next stage of the analytical project.
+Where analysis requires multiple cycles (for example to develop, use and update analytical models), this phase may follow on from the delivery and communication phase. In these cases, the phase will concentrate on the scope of the questions to be addressed in the next stage of the analytical project.
-More effort may be needed to define the requirements and scope in this phase for research, evaluation or other projects that may need to seek a wider range of perspectives or for which subsequent phases and work may be delivered through a product or service.
+In this phase more effort may be needed to define the requirements and scope in this phase for research, evaluation or other projects that may need to seek a wider range of perspectives or for which subsequent phases and work may be provided through a product or service.
### Design
-During the design phase, the Analyst will convert the commission into an analytical plan, including the assurance required and ensuring it is sufficient to answer the questions posed. This phase includes the communication and approval of plans produced, and some iteration between the Commissioner and the Analyst is to be expected as the analytical solution is developed and limitations understood.
+During the design phase, the analyst will convert the commission into an analytical plan. This will include the assurance required and ensuring the analysis is sufficient to answer the questions posed. This phase includes the communication and approval of plans. Some iteration between the commissioner and the analyst is to be expected as the analytical solution is developed and limitations understood.
-For larger projects or those that require multiple cycles, the design phase may include consideration of the staging of work over the whole scope of the project as well as work required in each stage. Analysis plans for work that is dependent on insights from earlier stages may be high-level and necessitate a return to the design phase at a later date.
+For larger projects or those that require multiple cycles, the design phase may include consideration of the staging of work over the whole scope of the project as well as the work required in each stage. Analysis plans for work that is dependent on insights from earlier stages may be high-level and necessitate a return to the design phase at a later date.
### Analysis
-The analysis phase is where planned analysis is undertaken, and progress and relevance are monitored. During work, the design and plan may be amended to account for changing circumstances, emerging information or unexpected difficulties or limitations encountered, and this phase also includes maintaining appropriate records of the analysis conducted, changes, decisions and assumptions made. In some cases, changes or limitations encountered may necessitate a return to either the scoping or design phase.
+The analysis phase is where planned analysis is undertaken and progress and relevance are monitored. The design and plan may be amended to account for changing circumstances, emerging information or unexpected difficulties or limitations encountered. This phase also includes maintaining appropriate records of the analysis conducted, changes, decisions and assumptions made. In some cases the changes or limitations encountered may mean the scoping or design phase needes to be revisited.
-Throughout this phase, traceable documentation of the assurance activities undertaken shall also be produced.
+Throughout this phase traceable documentation of the assurance activities that have been undertaken shall also be produced.
-In larger analytical projects, some outputs of the analysis may be completed at different times as work develops, and aspects of other phases may therefore take place concurrently.
+In larger analytical projects, some outputs of the analysis may be completed at different times as work develops and aspects of other phases may therefore take place concurrently.
### Delivery, communication and sign-off
-During the delivery stage, insights and analytical assurances are communicated to the Approver and the Commissioner. The aim is ensuring that these are sufficiently understood in order for the Approver and Commissioner to determine whether the work has been appropriately assured and meets their requirements. This may then trigger additional analysis and further assurance as analytical projects frequently need further iteration or extension to satisfy the Commissioner's needs.
-
-Work in this stage can vary considerably depending on the commission, impact, approval processes and the nature of the project. Delivery and communication activities may include producing summary packs and reports, launching dashboards or websites and presentations to boards.
+During the delivery stage, insights and analytical assurances are communicated to the approver and the commissioner. These should be sufficiently understood for the approver and commissioner to determine whether the work has been appropriately assured and meets their requirements. Additional analysis and further assurance may be required as analytical projects frequently need further iteration or extension to satisfy the commissioner's needs.
-After analysis results have been determined to meet the requirements, they are formally approved for dissemination during sign-off. Sign-off includes confirmation that the commission was met, documentation and evidence was captured, and appropriate assurance was conducted. This approval may be phased as work develops and insights are produced.
+Work in this stage can vary considerably depending on the commission, impact, approval processes and the nature of the project. Delivery and communication activities may include producing summary packs and reports, launching dashboards or websites and presentations.
+After analysis results have been determined to meet the requirements, they are formally approved for dissemination during sign-off. Sign-off includes confirmation that the commission was met, documentation and evidence was captured and appropriate assurance was conducted. This approval may be phased as work develops and insights are produced.
## Maintenance and continuous review
-The analytical lifecycle is not a linear process. Where analysis is used on an ongoing basis, all aspects of the lifecycle should be regularly updated. For example, consideration should be made whether
+The analytical lifecycle is not a linear process. Where analysis is used on an ongoing basis all aspects of the lifecycle should be regularly updated. For example, consideration should be made as to whether:
+
+* the inputs used remain appropriate
+* the initial communication methods remain the best way to deliver the information
+* any software relied on continues to be supported and up to date
+* the model continues to be calibrated appropriately (this is particularly important for [black box models](definitions_and_key_concepts.html#black-box-models))
-*The inputs used remain appropriate
-*The initial communication methods remain the best way to deliver the information
-*Any software relied on continues to be supported and up to date
-*The model continues to be calibrated appropriately (this is particularly important for [black box models](definitions_and_key_concepts.html#black-box-models))
Additionally, a robust version control process should be in place to ensure any changes to the analysis are appropriately assured.
## Urgent analysis
-In some cases there may be a need for urgent analysis that cannot follow all the steps in this guide i.e. where the need for analysis outweighs the risk of poor quality. In this case analysts should follow the [Urgent data quality assurance guidance](https://www.gov.uk/government/publications/urgent-data-quality-assurance-guidance/urgent-data-quality-assurance-guidance).
+In some cases there may be a need for urgent analysis that cannot follow all the steps in this guide. For example, where the need for analysis outweighs the risk of poor quality. In this case analysts should follow the [Urgent data quality assurance guidance](https://www.gov.uk/government/publications/urgent-data-quality-assurance-guidance/urgent-data-quality-assurance-guidance).
diff --git a/definitions_and_key_concepts.qmd b/definitions_and_key_concepts.qmd
index a4d1df1..ee6a7a8 100644
--- a/definitions_and_key_concepts.qmd
+++ b/definitions_and_key_concepts.qmd
@@ -1,5 +1,5 @@
::: {.callout-important}
-This version of the AQuA book is a preliminary ALPHA draft. It is still in development, and we are still working to ensure that it meets user needs.
+This version of the AQuA Book is a preliminary ALPHA draft. It is still in development, and we are still working to ensure that it meets user needs.
The draft currently has no official status. It is a work in progress and is subject to further revision and reconfiguration (possibly substantial change) before it is finalised.
:::
@@ -10,8 +10,7 @@ This chapter sets out definitions and concepts that are used throughout the rest
## Analysis {.unnumbered}
-Analysis is the collection, manipulation and interpretation of information and data for use in decision making. Analysis can vary widely between situations and many different types of analysis may be used to form the evidence base that supports the decision-making process.
-
+Analysis is the collection, manipulation and interpretation of information and data for use in decision-making. Analysis can vary widely between situations and many different types of analysis may be used to form the evidence base that supports the decision-making process.
Examples of types of analysis that are frequently encountered in government are:
* actuarial
@@ -26,13 +25,13 @@ Examples of types of analysis that are frequently encountered in government are:
## Assurance {.unnumbered}
-Analytical assurance is the process and set of practices to ensure that the analysis is fit for purpose.
+Analytical assurance is the process and set of practices that ensure analysis is fit for purpose.
## Assurance activities {.unnumbered}
Assurance activities are any actions carried out in order to validate and verify analysis.
-For example:
+This may include:
* analyst testing
* peer review
@@ -40,46 +39,66 @@ For example:
## Artificial Intelligence {.unnumbered}
-Artificial intelligence (AI) attempts to simulate human intelligence using techniques and methods such as machine learning, natural language processing, and robotics. AI aims to perform tasks that typically require human intelligence, such as problem-solving, decision-making, and language understanding. Artificial Intelligence models are a subset of [black box models](#black_box_models)
-
+Artificial Intelligence (AI) attempts to simulate human intelligence using techniques and methods such as [machine learning](#Machine Learning), natural language processing, robotics, and generative AI. AI aims to perform tasks that typically require human intelligence, such as problem-solving, decision-making, and language understanding. AI models are usually considered [black box models](#black_box_models) and can be pre-trained models or custom built.
## Black box models {.unnumbered}
-Black box models internal workings are not visible or easily understood. These models take input and produce output without providing clarity about the process used to arrive at the output. [Artificial Intelligence](#Artificial Intelligence) models (including [Machine Learning](#Machine Learning)) are the most common type of black box models used today. Other forms of black box models may arise in future.
-
+Black box models internal workings are not visible, easily understood, or succinctly explained. These models take input and produce output without providing clarity about the process used to arrive at the output. This also includes proprietary models with protected intellectual property. [Artificial Intelligence](#Artificial Intelligence) models (including [Machine Learning](#Machine Learning)) can often be considered a type of black box models. Other forms of this type of black box models may arise in future.
## Business critical analysis {.unnumbered}
-Business critical analysis is analysis which plays such a role in decision making that it influences significant financial and funding decisions, is necessary to the achievement of a Departmental business plan, or where an error could have a significant reputational, economic or legal implications.
+Business critical analysis refers to analysis that has a significant influence over financial and funding decisions, is necessary to the achievement of a departmental business plan or is analysis where an error could have a significant reputational, economic or legal implications.
-The first edition of the AQuA book described business critical models. This has been generalised to business critical analysis, as it is possible for analysis to be business critical without including a model. Some departments may continue to use the term business critical models (BCM).
+The first edition of the AQuA Book described business critical models. This has been updated to the more general term 'business critical analysis' because analysis can be business critical without including a model. Some departments may still use the term business critical models (BCM).
## Documentation {.unnumbered}
### Specification documentation {.unnumbered}
-Specifications capture initial engagements with the commissioner. They describe the question, the context, and any boundaries of the analysis. This provides a definition of the scope and a mechanism for agreeing project constraints such as deadlines, available resources, and capturing what level of assurance is required by the commissioner.
+Specification documentation records the initial engagements with the commissioner. It describe the question, the context and any boundaries of the analysis. The specifications provide a definition of the scope of the project and a mechanism for agreeing project constraints (for example, deadlines and available resources) and define what level of assurance is required by the commissioner.
### Design documentation {.unnumbered}
-Design documents describe the analytical plan, including the methodology, inputs, and software. They also contain details of the planned [verification](#verification) and [validation](#validation) of the analysis. They provide a basis for the Analytical Assurer to verify whether the analysis meets the specified requirements. For more information on the design documentation, see the [Design](design.qmd) chapter.
+Design documents describe the analytical plan, including the methodology, inputs and software that will be used. They also contain details of the planned [verification](#verification) and [validation](#validation) of the analysis. They provide a basis for the analytical assurer to verify whether the analysis meets the specified requirements.
+
+You can read more about design documentation in the [Design](design.qmd) chapter.
### Assumptions log {.unnumbered}
-A register of assumptions, whether provided by the Commissioner or derived by the analysis, that have been risk assessed and signed off by an appropriate governance group or stakeholder. Assumption logs should describe each assumption, quantify its effect and reliability and set out when it was made, why it was made, who made it and who signed it off.
+The assumptions log is a register of assumptions, whether provided by the commissioner or gathered from the analysis, that have been risk assessed and signed off by an appropriate governance group or stakeholder. Assumption logs should:
+
+* describe each assumption
+* quantify its effect and reliability
+* set out when it was made
+* explain why it was made
+* explain who made the assumption and who signed it off
### Decisions log {.unnumbered}
-A register of decisions, whether provided by the Commissioner or derived by the analysis. Decisions logs should describe each decision and set out when it was made, why it was made, who made it and who signed it off.
+The decisions log is a register of decisions, whether provided by the commissioner or derived by the analysis. Decisions logs should:
+
+* describe each decision
+* set out when it was made
+* explain why it was made
+* explain who made the decision and who signed it off
### Data log {.unnumbered}
-A register of data provided by the Commissioner or derived by the analysis that has been risked assessed and signed-off by an appropriate governance group or stakeholder.
+A register of data provided by the commissioner or derived by the analysis that has been risked assessed and signed-off by an appropriate governance group or stakeholder.
+
+### User and technical documentation {.unnumbered}
+All analysis shall have user documentation, even if the only user is the analyst leading the analysis. This documentation should include:
-### User / technical documentation {.unnumbered}
+* a summary of the analysis including the context to the question being asked
+* what analytical methods were considered
+* what analysis was planned and why
+* what challenges were encountered and how they were overcome
+* what verification and validation steps were performed
-All analysis shall have user-documentation, even if the user is only the analyst leading the analysis. This is to ensure that they have captured sufficient material to assist them if the analysis is revisited in due course. For analysis that is likely to be revisited or updated in the future, documentation should be provided to assist a future analyst and should be more comprehensive. This documentation should include a summary of the analysis including the context to the question being asked, what analytical methods were considered, what analysis was planned and why, what challenges were encountered and how they were overcome and what verification and validation steps were performed. In addition, guidance on what should be considered if the analysis is to be revisited or updated is beneficial. For modelling, the Analyst may include a model map that describes data flows and transformations.
+Where relevant the analyst may include a model map that describes data flows and transformations.
+
+For analysis that is likely to be revisited or updated in the future, more comprehensive documentation should be provided to assist a future analyst. It may also be helpful to include guidance on what should be considered or updated.
### Assurance statement {.unnumbered}
@@ -87,53 +106,69 @@ A brief description of the analytical assurance that have been performed to assu
::: {.callout-tip}
# Example of publishing quality assurance tools
-The Department for Energy Security and Net Zero and Department for Business and Trade have published a range of quality assurance tools and guidance to help people with Quality Assurance of analytical models. [Modelling Quality Assurance tools and guidance](https://www.gov.uk/government/publications/energy-security-and-net-zero-modelling-quality-assurance-qa-tools-and-guidance) are used across the two departments to ensure analysis meets the standards set out in the AQuA book and provide assurance to users of the analysis that proportionate quality assurance has been completed.
+The Department for Energy Security and Net Zero (DESNZ) and Department for Business and Trade (DBT) have published a range of quality assurance tools and guidance to help people with the quality assurance of analytical models. [Modelling Quality Assurance tools and guidance](https://www.gov.uk/government/publications/energy-security-and-net-zero-modelling-quality-assurance-qa-tools-and-guidance) are used across the departments to ensure analysis meets the standards set out in the AQuA Book and provide assurance to users of the analysis that proportionate quality assurance has been completed.
:::
+## Machine Learning {.unnumbered}
+
+Machine Learning uses algorithms to learn from patterns in data without needing to programme explicit business rules. Some models are white box models and others are considered [black box models](#Black box models). Machine Learning is a subset of [Artificial Intelligence](#Artificial Intelligence).
+
## Multi-use models {.unnumbered}
-Some models, often complex and large, are used by more than one user or group of users for related but differing purposes, these are known as **multi-use models**.
+Multi-use models are used by more than one user or group of users for related but different purposes. These are often complex and large.
-Often, a Steering Group is created to oversee the analysis. This Steering Group would be chaired by the senior officer in charge of the area that maintains the model, and contain senior, ideally empowered, representatives of each major user area.
+A Steering Group may be created to oversee the analysis of these models. This Steering Group would be chaired by the senior officer in charge of the area that maintains the model and consist of senior representatives of each major user area. The members of the Steering Group would ideally have decision-making responsibilites in their area of work.
## Quality analysis {.unnumbered}
-Quality analysis is analysis which is fit for the purpose(s) it was commissioned to meet. It should be accurate, have undergone appropriate assurance, be evidenced, proportionate to its effect, adequately communicated, documented and accepted by its commissioners.
+Quality analysis is fit for the purpose it was commissioned to meet. It should be:
+
+* accurate
+* appropriately assured
+* evidenced
+* proportionate to its effect
+* adequately communicated
+* documented
+* accepted by its commissioners
## Roles and responsibilities {.unnumbered}
-The AQuA book defines the following roles:
+The AQuA Book defines the following roles:
-* **Commissioner**
-* **Analyst**
-* **Assurer**
-* **Approver**
+* commissioner
+* analyst
+* assurer
+* approver
-See [Roles and Responsibilities](analytical_lifecycle.qmd/#roles_and_responsibilities) for details.
+You can read more in the [Roles and Responsibilities](analytical_lifecycle.qmd/#roles_and_responsibilities) section.
## Third party {.unnumbered}
-Any individual, or group of individuals that is not a member of the same group as the those commissioning analysis. E.g. working for a different government department, a different function or an outside company.
+Any individual or group of individuals that is not a member of the same group as the those commissioning analysis. For example, they may be working for a different government department, a different function or an outside company.
## Uncertainty {.unnumbered}
-Uncertainties are things that are not known, or are in a state of doubt, or are things whose effect is difficult to know. They have the potential to have major consequences for a project, programme, piece of analysis meeting its objectives.[^1]
+Uncertainties are things that are not known, are in a state of doubt or are things whose effect is difficult to know. They have the potential to have major consequences for a project, programme or piece of analysis meeting its objectives.[^1]
[^1]: https://www.nao.org.uk/wp-content/uploads/2023/08/Good-practice-guide-Managing-uncertainty.pdf
-There are different types of uncertainty. A common classification divides uncertainty into known knowns, known unknowns, and unknown unknowns. The type of uncertainty will influence the analytical approach and assurance activities required.
+There are different types of uncertainty. A common classification divides uncertainty into known knowns, known unknowns and unknown unknowns. The type of uncertainty will influence the analytical approach and assurance activities required.
-The [Uncertainty Toolkit for Analysts in Government](https://analystsuncertaintytoolkit.github.io/UncertaintyWeb/index.html) is a tool produced by a cross government group to help assessing and communicating uncertainty.
+The [Uncertainty Toolkit for Analysts in Government](https://analystsuncertaintytoolkit.github.io/UncertaintyWeb/index.html) is a tool produced by a cross-government group to help assessing and communicating uncertainty.
## Validation {.unnumbered}
-Ensuring the analysis meets the needs of its intended users and the intended use environment. See @glover2014 for more information.
+Validation ensures the analysis meets the needs of its intended users and the intended use environment.
+
+You can read more in [Verification and validation for the AQuA Book](https://www.gov.uk/government/publications/verification-and-validation-for-the-aqua-book) by Paul Glover.
## Verification {.unnumbered}
-Ensuring the analysis meets it specified design requirements. See @glover2014 for more information.
+Verification ensures the analysis meets it specified design requirements.
+
+You can read more in [Verification and Validation for the AQuA Book](https://www.gov.uk/government/publications/verification-and-validation-for-the-aqua-book) by Paul Glover.
## Version control {.unnumbered}
-It is important to ensure that changes that have been made to analysis can be easily seen and quality assured by the analytical assurer, and the latest version of the analysis is being used. Tools and templates can be used to support with evidencing updates and the checks completed throughout a project providing a log of changes that have occurred, why, when, and by whom.
+It is important to ensure that the latest version of the analysis is being used and any changes made can be easily seen and quality assured by the analytical assurer. There are tools and templates that can be used to record any updates and checks made during a project. They can help to provide a log of the changes that have been made including why and when they were made, and who made them.
diff --git a/delivery_and_communication.qmd b/delivery_and_communication.qmd
index 4deff65..cdd6aac 100644
--- a/delivery_and_communication.qmd
+++ b/delivery_and_communication.qmd
@@ -1,203 +1,201 @@
::: {.callout-important}
-This version of the AQuA book is a preliminary ALPHA draft. It is still in development, and we are still working to ensure that it meets user needs.
+This version of the AQuA Book is a preliminary ALPHA draft. It is still in development, and we are still working to ensure that it meets user needs.
The draft currently has no official status. It is a work in progress and is subject to further revision and reconfiguration (possibly substantial change) before it is finalised.
:::
# Delivery, communication and sign-off
-## Introduction and overview
+The successful delivery of analysis to the commissioner marks its transition from being a product under development to one that is fit and ready to be used to inform decision-making in your organisation and, possibly, inform the public.
-The successful delivery of analysis to its Commissioner marks its transition from being a product under development to one that is fit and ready to be used to inform decision making in your organisation and possibly inform the public.
+This chapter provides information on the assurance of communication of analysis and delivery of analytical output.
-This chapter provides information on the processes around assurance of communication of analysis and delivery of analytical output.
+## Roles and responsibilities in delivery, communication and sign-off
+### The commissioner's responsibilities
-## Roles and responsibilities in delivery, communication and sign-off
+The commissioner shall:
+* confirm that the analysis is likely to meet their needs
+* use the analysis as specified
+* understand and apply any limitations to its use
-### The Commissioner's responsibilities during delivery, communication and sign-off
+### The analyst's responsibilities
-The Commissioner shall
-* confirm that the analysis is likely to meet their needs;
-* use the analysis as specified; and,
-* understand and apply any limitations to its use.
+The analyst shall:
+* follow organisational governance procedures for delivery and sign-off, including, where appropriate, updating the [business-critical analysis register](#business-critical-analysis-register) and making the analysis [publicly available](#open-publishing-of-analysis)
+* describe any limitations to using the analysis
-### The Analyst's responsibilities during delivery, communication and sign-off
+The analyst should:
-The Analyst
-- shall follow organisational governance procedures for delivery and sign-off, including, where appropriate, updating the [business-critical analysis register](#business-critical-analysis-register), and making the analysis [publicly available](#open-publishing-of-analysis);
-- shall decribe any limitations to using the analysis;
-- should ensure that communication meets audience requirements e.g. on accessibility;
-- should be prepared to respond to challenge from the Approver e.g. scruitiny from project or programme boards;
-- may be required to communicate the assurance state to the Approver, if not done directly by the Assurer.
+* ensure that communication meets audience requirements such as accessibility
+* be prepared to respond to challenge from the approver or scruitiny from project or programme boards
+The analyst may be required to communicate the assurance state to the approver if this is not done directly by the assurer.
-### The Assurer's responsibilities during delivery, communication and sign-off
+### The assurer's responsibilities
-The Assurer shall communicate the assurance state to the Approver. This includes confirmation that the work has been appropriately scoped, executed, validated, verified, documented, and provides adequate handling of uncertainty. This communication may go via the Analyst.
+The assurer shall communicate the assurance state to the approver. This includes confirmation that the work has been appropriately scoped, executed, validated, verified, documented and that it provides adequate handling of uncertainty. This communication may be undertaken by the analyst.
-### The Approver's responsibilities during delivery, communication and sign-off
-* The Approver shall review the assurance evidence that has been provided to them.
-* The Approver should provide sufficient challenge to the analysts to gain assurance that the analysis is fit for purpose.
-* The Approver shall be confident that the analysis meets the design requirements, is of sufficient quality and is adequately and proportionately documented.
-* When they are satisfied with the validity and robustness of the analysis, the Approver should provide the Analyst with evidence that the analysis outputs have been properly reviewed and formally approved.
-* The Approver shall follow organisation governance procedures for sign-off, including updating of the [business-critical analysis register](#business-critical-analysis-register), where appropriate.
+### The approver's abilities
+The approver shall:
+* review the assurance evidence that has been provided to them
+* be confident that the analysis meets the design requirements, is of sufficient quality and is adequately and proportionately documented
+* follow organisation governance procedures for sign-off, including updating of the [business-critical analysis register](#business-critical-analysis-register), where appropriate
-## Assurance activities in the delivery, communication and sign-off stage
+The approver should:
+* provide sufficient challenge to the analysts to gain assurance that the analysis is fit for purpose
+* provide the analyst with evidence that the analysis outputs have been properly reviewed and formally approved when they are satisfied with the validity and robustness of the analysis
+## Assurance activities
### Delivery
-When delivering a piece of analysis, the Analyst and/or Assurer should communicate its assurance state to the Approver and provide evidence that the analysis and associated outputs have undergone proportionate quality assurance and to demonstrate that the analysis is ready for delivery, for example:
+When delivering a piece of analysis, the analyst (or assurer), should communicate its assurance state to the approver and provide evidence that the analysis and associated outputs have undergone proportionate quality assurance. They should also demonstrate that the analysis is ready for delivery. This may include confirming that the analysis:
-* It uses suitable data and assumptions;
-* It has provisions for regular review;
-* It meets the purpose of its commission;
-* It has been carried out correctly and to its agreed specification;
-* It has a risk assessment and statement against the programme risk register;
-* It meets analytical standards, such as those around coding standards and documentation;
-* It adheres to any professional codes of practice (e.g. [The Code of Practice for Statistics](https://code.statisticsauthority.gov.uk/)
-* Where appropriate the analysis is accompanied by a completed [assurance statement](https://view.officeapps.live.com/op/view.aspx?src=https%3A%2F%2Fassets.publishing.service.gov.uk%2Fmedia%2F65c5021f9c5b7f0012951b83%2F20231101-analysis-evidence-quality-assurance-report-gov-uk-E02.docx&wdOrigin=BROWSELINK).
-
-Though not strictly assurance, the analyst should also consider areas such as security ratings, retention policies, intellectual property, ethics and related concerns.
-
+* uses suitable data and assumptions
+* has provisions for regular review
+* meets the purpose of its commission
+* has been carried out correctly and to its agreed specification
+* has a risk assessment and statement against the programme risk register
+* meets analytical standards, such as those around coding standards and documentation
+* adheres to any professional codes of practice (for example, [The Code of Practice for Statistics](https://code.statisticsauthority.gov.uk/)
+* is accompanied by a completed [assurance statement](https://view.officeapps.live.com/op/view.aspx?src=https%3A%2F%2Fassets.publishing.service.gov.uk%2Fmedia%2F65c5021f9c5b7f0012951b83%2F20231101-analysis-evidence-quality-assurance-report-gov-uk-E02.docx&wdOrigin=BROWSELINK), where appropriate
+Though not strictly assurance, the analyst should also consider areas such as:
-The Approver should scrutinise the evidence delivered and approve the work if the analysis meets the required standard, which considers the [proportionality](proportionality.html) of the work. The Approver should then feedback the outcome of any approval activities to the analyst so that the analysis can be updated if required.
+* security ratings
+* retention policies
+* intellectual property
+* ethics and related concerns
+The approver should scrutinise the evidence delivered and approve the work if the analysis meets the required standard. The approver should then feedback the outcome of any approval activities to the analyst so that the analysis can be updated if required.
-The exact nature of any scrutiny made by the Approver should be proportionate to the effect the analysis is likely to have, the governance process of their programme/ organisation, and follow the principles of proportionality described in [Chapter 3](proportionality.html) of this document.
+The exact nature of any scrutiny made by the approver should be proportionate to the effect the analysis is likely to have, the governance process of their programme/ organisation, and follow the [principles of proportionality](proportionality.html).
-To ensure that the analysis is used as intended, the Commissioner should use the analysis as specified at the start of the analytical cycle, applying any limitations to its use as described by the Analyst.
+To ensure that the analysis is used as intended, the commissioner should use the analysis as specified at the start of the analytical cycle, applying any limitations to its use as described by the analyst.
### Communication
-The effective and transparent communication is essential to enable analysis to be adopted and trusted by the Commissioner and onward users. Depending on its final use and likelihood of publication, any analysis may be communicated to a wide audience including:
+The effective and transparent communication is essential to ensure analysis is adopted and trusted by the commissioner and onward users. Depending on its final use and likelihood of publication, any analysis may be communicated to a wide audience including:
-* Commissioners and users of the analysis;
-* External scrutiny including the [Public Accounts Committee](https://committees.parliament.uk/committee/127/public-accounts-committee/), the [National Audit Office](https://www.nao.org.uk/), internal and external audit;
-* The public, through publications and [Freedom of Information Act](https://www.legislation.gov.uk/ukpga/2000/36/contents) requests;
-* Academic experts, possibly through a departmental [Areas of Research Interest](https://www.gov.uk/government/publications/dwp-areas-of-research-interest-2023/dwp-areas-of-research-interest-2023) document.
-* Governmental partners - both national and international
+* commissioners and users of the analysis
+* external scrutiny including the [Public Accounts Committee](https://committees.parliament.uk/committee/127/public-accounts-committee/), the [National Audit Office](https://www.nao.org.uk/), internal and external audit
+* the public, through publications and [Freedom of Information Act](https://www.legislation.gov.uk/ukpga/2000/36/contents) requests
+* academic experts, possibly through a departmental [Areas of Research Interest](https://www.gov.uk/government/publications/dwp-areas-of-research-interest-2023/dwp-areas-of-research-interest-2023) document
+* national and international governmental partners
The form of communication should be tailored to the audience. The communication should be quality assured in a proportionate manner to ensure an accurate reflection of the analytical results.
-The Analysis Function's [Making Analytical Publications Accessible Toolkit](https://analysisfunction.civilservice.gov.uk/policy-store/making-analytical-publications-accessible/) gives guidance to help ensure that any that websites, tools, and technologies produced from analysis are designed and developed so that people with disabilities can use them. More specifically, people can: perceive, understand, navigate, and interact with the web.
+The Analysis Function's [Making Analytical Publications Accessible Toolkit](https://analysisfunction.civilservice.gov.uk/policy-store/making-analytical-publications-accessible/) gives guidance to help ensure that any that websites, tools, and technologies produced from analysis are designed and developed so that they meet accessibility guidelines.
-If publishing the outcome of any analysis is required, the Analyst should follow departmental and statutory guidance. Some examples are given below:
+If the outcome of any analysis needs to be published the analyst should follow departmental and statutory guidance. There are different guidelines depending on the work that is being published. These are:
-* If you are publishing statistics you will need to follow your organisation's guidance and the [regulatory guidance for publishing official statistics and national statistics](https://osr.statisticsauthority.gov.uk/publication/regulatory-guidance-publishing-official-statistics-and-national-statistics/) ;
-* If you are publishing research, you shall follow your organisations guidance and the [Government Social Research Publication protocol](https://www.gov.uk/government/publications/government-social-research-publication-protocols);
-* If you are publishing an evaluation, refer to any recommendations from the [Evaluation Task Force](https://www.gov.uk/government/organisations/evaluation-task-force);
+* [The regulatory guidance for publishing official statistics and national statistics](https://osr.statisticsauthority.gov.uk/publication/regulatory-guidance-publishing-official-statistics-and-national-statistics/)
+* [The Government Social Research Publication protocol](https://www.gov.uk/government/publications/government-social-research-publication-protocols)
+* any recommendations from the [Evaluation Task Force](https://www.gov.uk/government/organisations/evaluation-task-force) for evaluations
### Sign-off
-The exact nature of the approval process may vary depending on:
-* The effect the analysis is likely to have;
-* The approval process of the organisation, and
-* The nature of the programme, project or board approving the analysis.
+The exact nature of the approval process may vary depending on the:
-The formality of the sign-off process should be governed by organisational procedures, and be proportionate to the analysis.
+* effect the analysis is likely to have
+* approval process of the organisation
+* nature of the programme, project or board approving the analysis
-The Approver should provide the Analyst with evidence that the analysis outputs have been properly reviewed and formally approved. For example, through the notes of a project / programme board where the decision to approve the analysis was made or similar.
+The formality of the sign-off process should be governed by organisational procedures and be proportionate to the analysis.
+The approver should provide the analyst with evidence that the analysis outputs have been properly reviewed and formally approved. For example, through the notes of a project or programme board where the decision to approve the analysis was made.
-## Documentation in the delivery, communication and sign-off stage
-When the Analyst and Assurer are satisfied that the analysis is ready to hand over to the Commissioner, they should ensure that any associated documentation supporting the analysis is ready and has also undergone quality assurance. Supporting documentation may include:
+## Documentation
-* [Specification](definitions_and_key_concepts.html#specification-documentation) and [design documentation](definitions_and_key_concepts.html#design-documentation)
-* Logs of [data](definitions_and_key_concepts.html#data-log), [assumptions](definitions_and_key_concepts.html#assumptions-log) and [decisions](definitions_and_key_concepts.html#decisions-log) including their source, ownership, reliability and any sensitivity analysis carried out;
-* [User and technical documentation](definitions_and_key_concepts.html#user-technical-documentation)
-* Advice on uncertainty and its affect on the outputs of the analysis;
-* A description of the limits of the analysis and what it can and cannot be used for;
-* Any materials for presenting the analysis to the Commissioner, for example slide decks or reports;
-* A record of the analysis including methods used, dependencies, process maps, change and version control logs and error reporting;
-* The code-base, when it has been agreed to [publish the analysis openly](#open-publishing-of-analysis)
-* The test plan and results of the tests made against that plan;
-* A statement of assurance;
-* A statement that ethical concerns have been addressed, especially in cases that include the application of [black-box models](definitions_and_key_concepts.html#black-box-models);
+When the analyst and assurer are satisfied that the analysis is ready to hand over to the commissioner, they should ensure that any associated documentation supporting the analysis is ready and has also undergone quality assurance. Supporting documentation may include:
+* [specification](definitions_and_key_concepts.html#specification-documentation) and [design documentation](definitions_and_key_concepts.html#design-documentation)
+* logs of [data](definitions_and_key_concepts.html#data-log), [assumptions](definitions_and_key_concepts.html#assumptions-log) and [decisions](definitions_and_key_concepts.html#decisions-log) including their source, ownership, reliability and any details of any sensitivity analysis carried out
+* [user and technical documentation](definitions_and_key_concepts.html#user-technical-documentation)
+* advice on uncertainty and its affect on the outputs of the analysis
+* a description of the limits of the analysis and what it can and cannot be used for
+* any materials for presenting the analysis to the commissioner (for example, slide decks or reports)
+* a record of the analysis including methods used, dependencies, process maps, change and version control logs and error reporting
+* the code-base, when it has been agreed to [publish the analysis openly](#open-publishing-of-analysis)
+* the test plan and results of the tests made against that plan
+* a statement of assurance
+* a statement confirming that ethical concerns have been addressed, especially in cases that include the application of [black-box models](definitions_and_key_concepts.html#black-box-models)
-
+## Treatment of uncertainty
-## Treatment of uncertainty in the delivery, communication and sign-off stage
+Government has produced a range of guidance to support analysts in presenting and communicating uncertainty in analysis, providing valuable advice on how to estimate and present uncertainty when describing the limitations of use of a piece of analysis. This includes:
-Government has produced a range of guidance to support analysts in presenting and communicating uncertainty in analysis. This includes:
+* The Office for Statistical Regulation’s [Approaches to presenting uncertainty in the statistical system](https://osr.statisticsauthority.gov.uk/publication/approaches-to-presenting-uncertainty-in-the-statistical-system/)
+* The [Uncertainty Toolkit for Analysts](https://analystsuncertaintytoolkit.github.io/UncertaintyWeb/chapter_6.html)
+* The Government Analysis Function guidance note on [Communicating quality, uncertainty and change](https://analysisfunction.civilservice.gov.uk/policy-store/communicating-quality-uncertainty-and-change/)
-* The Office for Statistical Regulation’s [Approaches to presenting uncertainty in the statistical system](https://osr.statisticsauthority.gov.uk/publication/approaches-to-presenting-uncertainty-in-the-statistical-system/);
-* The [Uncertainty Toolkit for Analysts](https://analystsuncertaintytoolkit.github.io/UncertaintyWeb/chapter_6.html);
-* The Government Analysis Function guidance note [Communicating quality, uncertainty and change](https://analysisfunction.civilservice.gov.uk/policy-store/communicating-quality-uncertainty-and-change/);
+## Black-box models
+
+The approver is responsible for the sign-off that confirms that all risks and ethical considerations around the use of black-box models have been addressed.
-Each provides valuable advice on how to estimate and present uncertainty when describing the limitations of use of a piece of analysis.
+This may include:
## Black-box models and the delivery, communication and sign-off stage
-
The Approver is responsible for signing-off that all risks and ethical considerations around the use of black-box models have been addressed.
This may include
-* formal consultation and approval by an ethics committee or similar
-* provisions for regular review
-* communicating the "health" of the model at regular intervals to the commissioner i.e. is it continuing to behave as expected
+* formal consultation and approval by an ethics committee or similar depending on internal governance structures
+* provisions for regular review, including whether on-going peer review is required to ensure the lastest guidance and assurance methodology is taken into account
+* communicating the "health" of the model at regular intervals to the commissioner i.e. is it continuing to behave as expected or has there been data drift i.e. is the model's performance decreasing over time? This can happen when a model is trained on historical data, but then uses current data when it is being used in production. The model may become less effective because it is no longer conditioned on the current state of the data.
The aspects to be considered are detailed in the [Introduction to AI assurance](https://www.gov.uk/government/publications/introduction-to-ai-assurance).
+You can read more in the [Introduction to AI assurance](https://www.gov.uk/government/publications/introduction-to-ai-assurance).
-## Multi-use models and the delivery, communication and sign-off stage
+## Multi-use models
-There is a greater risk that multi-use models may be used for purposes outside the intended scope. This places a greater onus on the Analyst to clearly communicate to all users the limitations and intended use. The Analyst may consider testing communication with different user groups to ensure that the analytical outputs are used as intended.
+There is a greater risk that multi-use models may be used for purposes outside the intended scope. This means it is important that the analyst very clearly communicates to all users the limitations and intended use. The analyst may consider testing communication with different user groups to ensure that the analytical outputs are used as intended.
## Analytical transparency
-Enabling the public to understand and scrutinise analysis promotes public confidence in decisions. This includes providing the public with information on models used for [business-critical decisions](#business-critical-analysis-register) and [making analysis open](#open-publishing-of-analysis). Further guidance on transparency can be found [here](https://osr.statisticsauthority.gov.uk/publication/regulatory-guidance-on-intelligent-transparency/).
+Supporting and encouraging the public to understand and scrutinise analysis promotes public confidence in decisions. This includes providing the public with information on models used for [business-critical decisions](#business-critical-analysis-register), [making analysis open](#open-publishing-of-analysis) and ensuring [transparency](https://osr.statisticsauthority.gov.uk/publication/regulatory-guidance-on-intelligent-transparency/).
### Business-critical analysis register
-This section applies to publishing lists of [business critical analysis](definitions_and_key_concepts.html#business-critical-analysis) (BCA), including models.
-
-1. Departments and Arm’s Length Bodies[^1] (ALBs) should publish a list of BCA
-in use within their organisations at least annually.
-2. Each department and ALB should decide what is defined as business
-critical based on the extent to which they influence significant financial and funding
-decisions; are necessary to the achievement of a departmental business plan, or
-where an error could lead to serious financial, legal or reputational damage.
-3. Departments and ALBs should align their definitions and thresholds of business
-criticality with their own risk framework respectively. The thresholds should be
-agreed by the Director of Analysis or equivalent.
-4. ALB’s are responsible for publishing their own BCA list, unless agreed otherwise
-with the department. The ALB’s Accounting Officer is accountable for
-ensuring publication and the sponsor department’s AO oversees this.
-5. The BCA lists should include all business-critical analysis unless there is an
-internally documented reason that the analysis should be excluded, agreed with the Director
-of Analysis (or equivalent) and the agreement documented.
-6. Justification for not publishing a model in the list may include exemptions
-under the Freedom of Information (FOI) Act 2000 where relevant, for example,
-including, but not limited to: National Security, policy under development or
-prejudicing commercial interests.
-7. In addition to these exemptions, there may be further reasons where the risk of
-negative consequence is deemed to outweigh the potential benefits resulting from
-publication of the model. One example is where population behaviour may change
-in response to awareness of a model or modelling.
-8. For clarity, the name of model/analysis and what it is used for should be included,
-alongside links to published material.
-9. To ensure the list is accessible, content and structure should follow guidance
-for [writing plainly](https://www.gov.uk/guidance/content-design/writing-for-gov-uk)
+Departments and Arm’s Length Bodies[^1] (ALBs) should publish a list of [business critical analysis](definitions_and_key_concepts.html#business-critical-analysis) (BCA) in use within their organisations at least annually. This includes models. The list should meet accessibility guidelines.
+
+Each department and ALB should decide what is defined as business critical based on the extent to which they influence significant financial and funding decisions, are necessary to the achievement of a departmental business plan, or where an error could lead to serious financial, legal or reputational damage.
+
+The definitions and thresholds of business criticality should be aligned with their organisation's own risk framework. The thresholds should be agreed by the director of analysis or equivalent.
+
+ALB’s are responsible for publishing their own BCA list, unless agreed otherwise with the department. The ALB’s accounting officer is accountable for ensuring publication and the sponsor department’s accounting officer oversees this.
+
+The BCA list should include all business-critical analysis unless there is an internally documented reason that the analysis should be excluded. This should be agreed with the director of analysis (or equivalent) and that agreement should be documented.
+
+Justification for not publishing a model in the list may include, but is not limited to:
+
+* exemptions under the Freedom of Information (FOI) Act 2000
+* national security
+* policy under development
+* commercial interests
+
+In addition to these exemptions there may be further reasons where the risk of negative consequence is deemed to outweigh the potential benefits resulting from publication of the model. For example, where population behaviour may change in response to awareness of a model or modelling.
+
+For clarity, the name of the analysis or model and what it is used for should be included alongside links to any published material.
### Open publishing of analysis
-To facilitate public scrutiny, departments may choose to make analysis/models (e.g. source code or spreadsheets) and details which may include data, assumptions, methodology and outputs open to the public. Open publishing source code and other elements of analysis allows others to reuse and build on the work (https://www.gov.uk/service-manual/service-standard/point-12-make-new-source-code-open). Practical guidance to open coding can be found [here](https://www.gov.uk/service-manual/technology/making-source-code-open-and-reusable).
+To facilitate public scrutiny departments may choose to make the analysis or model (for example, source code or spreadsheets) and any relevant data, assumptions, methodology and outputs open to the public. [Open publishing source code](https://www.gov.uk/service-manual/service-standard/point-12-make-new-source-code-open) and other parts of the analysis allows others to reuse and build on the work.
-Publication of analysis should also draw on the [guidance for BCA lists](#business-critical-analysis-register) related to accessibility and justification for when publication may not be appropriate. For analysis that is extremely complex in nature, it may be more appropriate to publish summary information instead, to aid the accessibility.
+You can read more about making [source code open and reusable](https://www.gov.uk/service-manual/technology/making-source-code-open-and-reusable).
+The [guidance for publishing BCA lists](#business-critical-analysis-register) should be applied to the publication of analysis as in some cases it might not be appropriate to publish the work. For example, if the analysis is extremely complex it may be more appropriate to publish summary information to make the analysis more accessible.
-[^1]: ALBs include executive agencies, non-departmental public bodies and non-ministerial departments, please see [Cabinet Office guidance on Classification of Public Bodies](https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/519571/Classification-of-Public_Bodies-Guidance-for-Departments.pdf)
+[^1]: ALBs include executive agencies, non-departmental public bodies and non-ministerial departments, you can read more in the [Cabinet Office guidance on Classification of Public Bodies](https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/519571/Classification-of-Public_Bodies-Guidance-for-Departments.pdf)
diff --git a/design.qmd b/design.qmd
index 33851d2..0247e58 100644
--- a/design.qmd
+++ b/design.qmd
@@ -1,130 +1,123 @@
::: {.callout-important}
-This version of the AQuA book is a preliminary ALPHA draft. It is still in development, and we are still working to ensure that it meets user needs.
+This version of the AQuA Book is a preliminary ALPHA draft. It is still in development, and we are still working to ensure that it meets user needs.
The draft currently has no official status. It is a work in progress and is subject to further revision and reconfiguration (possibly substantial change) before it is finalised.
:::
# Design
-## Introduction and overview
-
-The design stage is where the Analyst translates the scope for the analysis agreed with the Commissioner into an actionable analytical plan. This chapter sets out recommended practices around designing the analysis and associated assurance activities, documenting the design and assuring the design. It also discusses considerations around the treatment of uncertainty in design, and design of multi-use and AI models.
+During the design stage the analyst creates an actionable [analytical plan]((#the-analytical-plan) from the scope for the analysis agreed with the commissioner. This chapter sets out recommended practices around designing the analysis, deeciding on the associated assurance activities, documenting the design and assuring the design. It also discusses considerations around the treatment of uncertainty in design and the design of multi-use and Artifical Intelligence (AI) models.
### The analytical plan
The development of the analytical plan should consider:
-* Methodology for producing results, including the treatment of uncertainty;
-* Project management approach (for example Agile, Waterfall or a combination of approaches);
-* Sourcing of inputs and assumptions;
-* Data and file management;
-* Change management and version control;
-* Programming language and/or software;
-* Code management, documentation and testing;
-* Communication between stakeholders;
-* Verification and validation procedures during the project lifetime;
-* Documentation to be delivered;
-* Process for updating the analytical plan;
-* Process for ongoing review and maintenance of models, including reviewing inputs and calibrations and ensuring that software relied on continues to be supported and up to date;
-* Ethics;
-* Reporting;
-* Downstream application.
-
-The use of [Reproducible Analytical Pipelines (RAP)](#rap) is encouraged as a means of effective project design.
-
-Iteration between the Commissioner and the Analyst is normal and expected whilst the analytical design develops.
+* methodology for producing results, including the treatment of uncertainty
+* project management approach (for example agile, waterfall or a combination of approaches)
+* sourcing of inputs and assumptions
+* data and file management
+* change management and version control
+* programming language and software
+* code management, documentation and testing
+* communication between stakeholders
+* verification and validation procedures during the project lifetime
+* documentation to be delivered
+* process for updating the analytical plan
+* process for ongoing review and maintenance of models, including reviewing inputs and calibrations and ensuring that software relied on continues to be supported and up to date
+* ethics
+* reporting
+* downstream application
+
+Iteration of the plan between the commissioner and the analyst is normal and expected while the analytical design develops.
::: {.callout-tip}
# Reproducible analytical pipelines
-The recommended approach for developing analysis in code is to use a [Reproducible Analytical Pipeline (RAP)](https://analysisfunction.civilservice.gov.uk/support/reproducible-analytical-pipelines/). Reproducible Analytical Pipelines shall:
+The recommended approach for developing analysis in code is to use a [Reproducible Analytical Pipeline (RAP)](https://analysisfunction.civilservice.gov.uk/support/reproducible-analytical-pipelines/). RAPs shall:
-* Follow the practices set out in the [Analysis Function Quality Assurance of Code for Analysis and Research manual](https://best-practice-and-impact.github.io/qa-of-code-guidance/intro.html).
-* Meet the requirements of the [Reproducible Analytical Pipelines minimum viable product](https://github.com/best-practice-and-impact/rap_mvp_maturity_guidance/blob/master/Reproducible-Analytical-Pipelines-MVP.md).
+* follow the practices set out in the [Analysis Function Quality Assurance of Code for Analysis and Research manual](https://best-practice-and-impact.github.io/qa-of-code-guidance/intro.html)
+* meet the requirements of the [RAPs minimum viable product](https://github.com/best-practice-and-impact/rap_mvp_maturity_guidance/blob/master/Reproducible-Analytical-Pipelines-MVP.md)
:::
-
-
## Roles and responsibilities in the design stage
-### The Commissioner's responsibilities during the design stage
-The Commissioner should confirm that the analytical approach will satisfy their needs. To assist in this, the Commissioner may review the analytical plan.
-
-The Commissioner's domain expertise can be a useful resource for the analyst in the design stage. The Commissioner might provide information regarding the input assumptions, data requirements and the most effective ways to present the outputs, all of which can inform the design.
-
+### The commissioner's responsibilities
+The commissioner should confirm that the analytical approach will satisfy their needs. To assist in this, the commissioner may review the analytical plan.
-### The Analyst's responsibilities during the design stage
-The Analyst should:
+The commissioner's expertise can be a useful resource for the analyst in the design stage. The commissioner might provide information regarding the input assumptions, data requirements and the most effective ways to present the outputs. All of these can inform the design.
-* develop the method and plan to address the Commissioner’s needs;
-* establish assurance requirements;
-* develop the plan for proportionate verification and validation - see [National Audit Office Framework to review models](https://www.nao.org.uk/wp-content/uploads/2016/03/11018-002-Framework-to-review-models_External_4DP.pdf);
-* plan in sufficient time for the assurance activity;
-* document the analytical plan in a proportionate manner; and,
-* follow any organisation governance procedures for project design.
+### The analyst's responsibilities
+The analyst should:
-### The Assurer's responsibilities during the design stage
-The Assurer should review the analytical plan to ensure that they are able to conduct the required assurance activities. They may provide feedback on the analytical plan. The Assurer should plan sufficient time for the assurance activity.
+* develop the method and plan to address the commissioner’s needs
+* establish assurance requirements
+* develop a plan for proportionate verification and validation as described in the [National Audit Office Framework to review models](https://www.nao.org.uk/wp-content/uploads/2016/03/11018-002-Framework-to-review-models_External_4DP.pdf);
+* plan in sufficient time for the assurance activity
+* document the analytical plan in a proportionate manner
+* follow any organisation governance procedures for project design
+### The assurer's responsibilities
+The assurer should review the analytical plan to ensure that it is able to conduct the required assurance activities. They may provide feedback on the analytical plan. The assurer should plan sufficient time for the assurance activity.
-### The Approver's responsibilities during the design stage
-In smaller projects, the Approver may not be heavily involved in the design stage. However, for business critical analysis, the Approver may want to confirm that organisational governance procedures for design have been followed.
+### The approver's responsibilities
-## Assurance activities in the design stage
+In smaller projects, the approver may not be heavily involved in the design stage. However, for business critical analysis, the approver may want to confirm that organisational governance procedures for design have been followed.
+## Assurance activities
-On completion of the design stage, the Assurer should be aware of the quality assurance tasks that will be required of them during the project lifetime and have assured the necessary elements of the analytical plan.
+When the design stage has been completed the assurer should be aware of the quality assurance tasks that will be required of them during the project lifetime and have assured the necessary elements of the analytical plan.
The assurance of the design stage should consider whether the analytical plan is likely to:
-* Address commissioner's requirements - validation;
-* Deliver as intended - verification;
-* Be robust i.e. well-structured, data driven, with a sound overall design.
+* address commissioner's requirements (validation)
+* deliver as intended (verification)
+* be robust (for exmaple, provide a well-structured, data driven plan with a sound overall design)
-The assurance of the design stage may be carried out by the Assurer. For more complex analysis, it is good practice to engage subject matter experts to provide independent assurance, and to ensure the accuracy and limitations of the chosen methods are understood, ideally with tests baselining their response against independent reference cases.
+The assurance of the design stage may be carried out by the assurer. For more complex analysis, it is good practice to engage subject matter experts to provide independent assurance and to ensure the accuracy and limitations of the chosen methods are understood, ideally with tests baselining their response against independent reference cases.
-## Documentation in the design stage
+## Documentation
-The design process should be documented in a proportionate manner. A design document that records the [analytical plan](#the-analytical-plan) should be produced by the Analyst and signed-off by the Commissioner. The design document may be reviewed by the Assurer.
-
-For modelling, an initial model map may be produced that describes data flows and transformations. This can be updated as the project progresses through the Analysis stage.
+The design process should be documented in a proportionate manner. A design document that records the analytical plan should be produced by the analyst and signed-off by the commissioner. The design document may be reviewed by the assurer.
+For modelling, an initial model map may be produced that describes data flows and transformations. This can be updated as the project progresses through the Analysis stage.
It is best practice to use formal version control to track changes in the design document.
-
-
## Treatment of uncertainty in the design stage
-During the design stage, Analysts should examine the planned analysis systematically for possible sources and types of uncertainty, to maximise the chance of identifying all that are sufficiently large to breach the acceptable margin of error. This is discussed in Chapter 3 of the [Uncertainty Toolkit for Analysts](https://analystsuncertaintytoolkit.github.io/UncertaintyWeb/chapter_3.html)
+During the design stage, analysts should examine the planned analysis systematically for possible sources and types of uncertainty. This is to maximise the chance of identifying all that are sufficiently large to breach the acceptable margin of error.
+You can read more in Chapter 3 of the [Uncertainty Toolkit for Analysts](https://analystsuncertaintytoolkit.github.io/UncertaintyWeb/chapter_3.html).
-## Black box models and the design stage
+
+## Black box models
Using [black box models](definitions_and_key_concepts.qmd/#black-box-models) places greater weight on the design of the analysis and the assurance and validation of outputs by domain experts.
This [guidance on AI assurance](https://www.gov.uk/government/publications/introduction-to-ai-assurance/introduction-to-ai-assurance) outlines considerations for the design of AI models, including risk assessment, impact assessment, bias audits and compliance audits.
-In the Design of AI/ML models, the Analyst should:
-
+In the design of AI and machine learning models, the analyst should:
* define the situation they wish to model;
* the prediction they wish to make;
-* the data that could be used to make the prediction;
-* carry out a literature review to identify appropriate modelling, valuation verification methods and document the rationale for selecting their approach;
-* consider how to separate the data for the design and testing of models - it's usual to design a model with a fraction of the data and then test it with the data that was not used in the design;
-* consider developing automatic checks to identify if the model is behaving unexpectedly, this is important if the model is likely to be used frequently to make regular decisions; and,
-* consider whether to refer the model to their ethics committee, or a similar group - see the [Data Ethics Framework](https://www.gov.uk/government/publications/data-ethics-framework/data-ethics-framework-2020).
+* assess the quality of data that could be used to make the prediction, this includes the data used in any pretrained models
+* carry out a literature review to identify appropriate modelling, valuation verification methods and document the rationale for selecting their approach
+* consider the appropriate data training, validation and testing strategy for your models - it's usual to design a model with a fraction of the data, validate with a separate portion and then test the final model with the data that was not used in the design;
+* consider your strategy when testing a pre-trained model, including appropriate validation methods for the models such as calculating similarity to labelled images or ground truths for generative AI
+* consider developing automatic checks to identify if the model is behaving unexpectedly, this is important if the model is likely to be used frequently to make regular decisions or is deployed into a production environment
+* consider the plan for maintenance and continuous review, including the thresholds or timeline to retrain the model and the resources required to support this - see the [maintenance and continuous review](analytical_lifecycle.qmd/#mainenance-and-continuous-review) section
+* consider referring the model to their ethics committee, or a similar group dependent on your interal governance structures - see the [Data Ethics Framework](https://www.gov.uk/government/publications/data-ethics-framework/data-ethics-framework-2020)
+* consider setting up a peer or academic review process to test the methodologies and design decisions
-## Multi-use models and the design stage
+## Multi-use models
Designing multi-use models should take into account the needs of all users of the analysis. An Analysis Steering Group may be an effective means for communication about the design with a range of user groups.
-The design of multi-use models may entail a modular structure with different Analysts and Assurers responsible for different elements. The design of assurance activities should capture both the assurance of individual modules and their integration.
+The design of multi-use models may entail a modular structure with different analysts and assurers responsible for different elements. The design of assurance activities should capture both the assurance of individual modules and their integration.
diff --git a/docs/index.html b/docs/index.html
index aa41b79..3955e40 100644
--- a/docs/index.html
+++ b/docs/index.html
@@ -7,7 +7,7 @@
-
+
The AQuA Book