sample_assumptions_table.qmd

---
title: "Example assumptions log"
format:
  html:
    embed-resources: true
    css: apd_style.css
    html-table-processing: none
---

::: {.callout-note collapse="false"}
This guidance is an ALPHA draft. It is in development and we are still working 
to ensure that it meets user needs.

Please get in touch with feedback to support the guidance by creating a [GitHub 
Issue](https://github.com/best-practice-and-impact/analysis_project_documentation/issues) or [emailing us](mailto:ASAP@ons.gov.uk).
:::

This log contains a list of assumptions used in an example analysis of data from UK universities. It also provides a score for each assumption.

## Definitions

Assumptions are scored as Red, Amber or Green (a RAG rating) depending on quality. In RAG, green denotes a favourable value, red unfavourable and amber neutral. The quality of an assumption measures both how certain and robust an assumption is and how appropriate it is for its intended use. 

For example, we would usually consider a well documented assumption drawn from published evidence to be very robust, but if it needs to be transformed or adapted significantly to fit the analysis, the quality rating might need downgrading. 

You would normally lower the quality rating of an assumption if you cannot get technical sign-off (for example because of lack of technical knowledge) or if the information on which it is based is incomplete or poor quality. You would also normally lower the quality if the confidence interval or uncertainty range is wide (i.e. you wouldn’t be surprised if the value was 50% different from what you measure because of uncertainty).

| **RAG Rating** | **Assumption quality** |
|----------------|------------------------|
| **GREEN**      | Based on validated data; Methodology is robust; No or few transformations, or transformation methodology is fully verified and robust; Data is current and signed off by experts; Confidence intervals are narrow. |
| **AMBER**      | The methodology is robust but based on limited data; Data required significant transformation to fit the model; Confidence interval is quite wide; Data has not been reviewed recently. |
| **RED**        | Unclear/unreliable data source or no data source provided; Based on limited data and methodology not robust; Data is not current; Confidence interval is wide or quality is unknown. |

---
title: "Sample assumptions log"
format:
  html:
    embed-resources: true
    css: apd_style.css
    html-table-processing: none
---

<div style="overflow-x:auto; float=left;">
<table class="assumptionTable">
<tr VALIGN=BOTTOM TEXT-ALIGN=LEFT>
    <th scope = "col"> Assumption ID</th>
    <th scope = "col"> Depends on Assumptions</th>
    <th scope = "col"> Location in code, documentation or publication</th>
    <th scope = "col"> Plain English description of assumption</th>
    <th scope = "col"> Basis for assumption</th>
    <th scope = "col"> Numerical value of the assumption</th>
    <th scope = "col"> Range around the estimated value</th>
    <th scope = "col"> Estimated distribution</th>
    <th scope = "col"> Links to supporting analysis</th>
    <th scope = "col"> Documentation dependencies</th>
    <th scope = "col"> Date of last review/update</th>
    <th scope = "col"> Externally reviewed by</th>
    <th scope = "col"> Date of external review</th>
    <th scope = "col"> Next review/update due on</th>
    <th scope = "col"> Quality rating</th>
    <th scope = "col"> Sensitivity score</th>
    <th scope = "col"> Risk score</th>
</tr>
<tr>
    <td>1</td>
    <td>2,3,4,5,6,7,8,9,10</td>
    <td>Assumption log</td>
    <td>We assume that the dataset is representative of the population.</td>
    <td>Team opinion</td>
    <td>N/A</td>
    <td>N/A</td>
    <td>N/A</td>
    <td>Descriptive statistics (link), comparison to existing data source/publication (link)</td>
    <td>Final report: methods, caveats</td>
    <td>14/02/2024</td>
    <td>John Doe</td>
    <td>14/02/2024</td>
    <td>14/05/2024</td>
    <td class="RAGgreen">GREEN</td>
    <td>High</td>
    <td>High</td>

</tr>
<tr>
    <td>2</td>
    <td>3,4,5,6,8,9,10</td>
    <td>Assumption log</td>
    <td>We assume that the data does not exclude any population groups based on their demographic 
        and socio-economic characteristics.</td>
    <td>Team opinion</td>
    <td>N/A</td>
    <td>N/A</td>
    <td>N/A</td>
    <td>Descriptive statistics (link), comparison to existing data source/publication (link)</td>
    <td>Final report: methods, descriptive statistics, caveats</td>
    <td>14/02/2024</td>
    <td>Jane Roe</td>
    <td>14/02/2024</td>
    <td>14/05/2024</td>
    <td class="RAGgreen">GREEN</td>
    <td>High</td>
    <td>High</td>
</tr>
<tr>
    <td>3</td>
    <td></td>
    <td>Correspondence with data provider</td>
    <td>We assume that all UK universities report data to the Higher Education Statistics Agency (HESA).</td>
    <td>Validated from data provider, coverage check against university list</td>
    <td>N/A</td>
    <td>N/A</td>
    <td>N/A</td>
    <td>N/A</td>
    <td>N/A</td>
    <td>14/02/2024</td>
    <td>Jane Roe</td>
    <td>14/02/2024</td>
    <td>14/05/2024</td>
    <td class="RAGgreen">GREEN</td>
    <td>High</td>
    <td>Medium</td>
</tr>

<tr>
    <td>4</td>
    <td>3</td>
    <td>Correspondence with data provider</td>
    <td>We assume that our list of UK universities is correct, current and comprehensive.</td>
    <td>Team opinion, reliable source (HESA list)</td>
    <td>N/A</td>
    <td>N/A</td>
    <td>N/A</td>
    <td>N/A</td>
    <td>N/A</td>
    <td>14/02/2024</td>
    <td>Jane Roe</td>
    <td>14/02/2024</td>
    <td>14/05/2024</td>
    <td class="RAGgreen">GREEN</td>
    <td>Low</td>
    <td>Medium</td>
</tr>

<tr>
    <td>5</td>
    <td>2,3,4</td>
    <td>Assumption log</td>
    <td>We assume that all universities accurately report the number of students enrolled during the academic year.</td>
    <td>Expert opinion</td>
    <td>N/A</td>
    <td>N/A</td>
    <td>N/A</td>
    <td>N/A</td>
    <td>Final report: Caveats</td>
    <td>14/02/2024</td>
    <td>Jane Roe</td>
    <td>14/02/2024</td>
    <td>14/05/2024</td>
    <td class="RAGamber">AMBER</td>
    <td>High</td>
    <td>High</td>
</tr>
<tr>
    <td>6</td>
    <td>3,4</td>
    <td>Assumption log</td>
    <td>We assume that all universities accurately report the number of students who dropped out during the academic year.</td>
    <td>Expert opinion</td>
    <td>N/A</td>
    <td>N/A</td>
    <td>N/A</td>
    <td>Publication on drop-out rates</td>
    <td>Final report: Caveats</td>
    <td>14/02/2024</td>
    <td>Jane Roe</td>
    <td>14/02/2024</td>
    <td>12/05/2024</td>
    <td class="RAGamber">AMBER</td>
    <td>High</td>
    <td>High</td>
</tr>

<tr>
    <td>7</td>
    <td>3</td>
    <td>Correspondence with data providers.</td>
    <td>We assume that the academic year is consistently measured across UK universities.</td>
    <td>Validated from data provider</td>
    <td>N/A</td>
    <td>N/A</td>
    <td>N/A</td>
    <td>Quality assurance based on sampling universities from their websites</td>
    <td>Quality assurance log, final report: caveats</td>
    <td>14/02/2024</td>
    <td>Jane Roe</td>
    <td>14/02/2024</td>
    <td>14/05/2024</td>
    <td class="RAGred">RED</td>
    <td>High</td>
    <td>High</td>
</tr>

<tr>
    <td>8</td>
    <td>9</td>
    <td>Correspondence with data providers.</td>
    <td>We assume that students who receive special education services are excluded from the calculation of dropout rates.</td>
    <td>Validated from data provider</td>
    <td>N/A</td>
    <td>N/A</td>
    <td>N/A</td>
    <td>Sensitivity analysis comparing dropout rates with and without this population included.</td>
    <td>Quality assurance log, final report: methods, caveats</td>
    <td>14/02/2024</td>
    <td>Jane Roe</td>
    <td>14/02/2024</td>
    <td>14/05/2024</td>
    <td class="RAGred">RED</td>
    <td>High</td>
    <td>High</td>
</tr>

<tr>
    <td>9</td>
    <td>2,3,4,10</td>
    <td> Exploratory data analysis notebook (link)</td>
    <td>We assume that there is complete information for all the variables in the analysis.</td>
    <td>Robustness testing</td>
    <td>N/A</td>
    <td>N/A</td>
    <td>N/A</td>
    <td>Descriptive statistics notebook link</td>
    <td>Desk instructions, final report: methods, summarising the sample</td>
    <td>14/02/2024</td>
    <td>Jane Roe</td>
    <td>14/02/2024</td>
    <td>14/05/2024</td>
    <td class="RAGamber">AMBER</td>
    <td>High</td>
    <td>High</td>
</tr>

<tr>
    <td>10</td>
    <td>2,3,4,9</td>
    <td>Correspondence with data provider</td>
    <td>We assume that the data collection process has not changed at all over time.</td>
    <td>Validated from data provider (link)</td>
    <td>N/A</td>
    <td>N/A</td>
    <td>N/A</td>
    <td>Methods documents from prior runs of this work, supplier data specifications and quality reports</td>
    <td>Data supply specification, data quality report, methods document</td>
    <td>Not yet assigned</td>
    <td>No external review</td>
    <td>14/02/2024</td>
    <td>14/05/2024</td>
    <td class="RAGred">RED</td>
    <td>High</td>
    <td>High</td>
</tr>
<tr>
    <td>11</td>
    <td>2,3,4,5,6,7,8</td>
    <td>Assumption log</td>
    <td>We assume that the correlation coefficient between the dropout rate and social grade of local authority of origin is 0.7</td>
    <td>Statistical analysis of past data</td>
    <td>0.7</td>
    <td>+-0.1</td>
    <td>Normal</td>
    <td>Correlation analysis report m(link), comparison to domain knowledge (link)</td>
    <td>Data analysis documentation</td>
    <td>05/12/2023</td>
    <td>John Doe</td>
    <td>05/12/2023</td>
    <td>01/05/2024</td>
    <td class="RAGgreen">GREEN</td>
    <td>Medium</td>
    <td>Low</td>
</tr>
</table>
</div>