Reorg jetstream into deep dives folder, add autogeneration for sideba…

…r. Small FAQ cleanup.

docs/jetstream/add-new-platform.md → docs/deep-dives/jetstream/adding-a-platform.md

@@ -1,4 +1,9 @@
-# Adding a Platform
+---
+id: adding-a-platform
+sidebar_position: 10
+title: Adding a Platform
+slug: adding-a-platform 
+---
 
 Jetstream runs analyses for experiments launched on several different platforms, such as Fenix or Firefox Desktop. When adding a new platform in Experimenter, the new platform also needs to be configured in Jetstream to enable automated analyses for launched experiments.
 

docs/jetstream/configuration.md → docs/deep-dives/jetstream/configuration.md

@@ -1,4 +1,6 @@
----
+--- 
+sidebar_position: 5
+slug: configuration
 id: configuration
 title: Configuring Jetstream
 ---
@@ -17,7 +19,7 @@ which should match the experiment slug, like `my-experiment-slug.toml`.
 This works for both Normandy and Nimbus slugs. 
 
 [partybal]: https://protosaur.dev/partybal/
-[outcome]: jetstream/outcomes.md
+[outcome]: ./outcomes
 [metric-hub]: https://github.com/mozilla/metric-hub/tree/main/jetstream
 
 ## Landing configurations
@@ -215,7 +217,7 @@ depends_on = ["moos", "clicks"]
 You should also add some sections to describe how your new metrics should be summarized for reporting.
 You can do this by adding a statistics section to the metric for the statistic you want.
 
-See [Statistics](jetstream/statistics.md) for a list of supported statistics and details about implementing your own.
+See [Statistics](./statistics) for a list of supported statistics and details about implementing your own.
 
 This looks like:
 
@@ -459,7 +461,7 @@ Results for exposure based metrics are currently not visualized in Experimenter.
 
 ## Testing configurations
 
-For more information on how to test configurations see [Testing Jetstream Configs](jetstream/testing.md)
+For more information on how to test configurations see [Testing Jetstream Configs](./testing)
 
 
 [Nimbus exposure event]: ../mobile-feature-api#recording-exposure-events

docs/jetstream/data-products.md → docs/deep-dives/jetstream/data-products.md

@@ -1,6 +1,9 @@
 ---
 id: data-products
 title: Jetstream Data Products
+sidebar_position: 6
+sidebar_label: Data products
+slug: data-products
 ---
 
 Jetstream writes analysis results and enrollments information to BigQuery. Statistics data and metadata gets exported to GCS to make it accessible to the Experimenter console.

docs/jetstream/metrics.md → docs/deep-dives/jetstream/metrics.md

@@ -1,6 +1,8 @@
 ---
 id: metrics
 title: Metrics
+sidebar_position: 2
+slug: metrics
 ---
 
 A *metric* describes an aggregation of user activity,
@@ -12,9 +14,9 @@ So is "a user's average page load time."
 
 Metrics are applied over a window of time,
 called an analysis window.
-The [Jetstream overview](jetstream.md) has more details about analysis windows.
+The [Jetstream overview](./overview) has more details about analysis windows.
 
-:::note What's in a name?
+:::tip What's in a name?
 
 [Glean](https://mozilla.github.io/glean/book/index.html)
 also has a concept called
@@ -25,7 +27,7 @@ Jetstream metrics represent aggregations of the recorded values of Glean metrics
 
 :::
 
-[Statistics](statistics.md) summarize the distribution of metrics
+[Statistics](./statistics) summarize the distribution of metrics
 within and between branches of an experiment.
 
 ## How do I add a metric to my experiment?
@@ -34,11 +36,11 @@ A small set of critical guardrail metrics (called Core Firefox Metrics in Nimbus
 specific to each platform is run by default for each experiment.
 These are defined for each platform in the metric-hub repository in [jetstream/defaults/](https://github.com/mozilla/metric-hub/tree/main/jetstream/defaults/). Look at the file for your platform. At the top you will see the metrics collected and the timeframe they are available: daily, weekly, or overall (at the end of the experiment).  For help understanding any aspect of guardrail metrics on your platform - link to the file and ask in #ask-experimenter.
 
-[Outcomes](outcomes.md) are collections of metrics that relate to each other.  You can associate an Outcome with your experiment in the Experiment Console - Metrics section.
+[Outcomes](./outcomes) are collections of metrics that relate to each other.  You can associate an Outcome with your experiment in the Experiment Console - Metrics section.
 Data scientists can extend and define outcomes in the outcomes path of the
 [`metric-hub`](https://github.com/mozilla/metric-hub/tree/main/jetstream/outcomes) repository.  See what [Outcomes are available](https://mozilla.github.io/metric-hub/outcomes/fenix/default-browser/).
 
-If the metrics you need are not covered by Guardrail/Core metrics or existing Outcomes - you can add a metric to your experiment by working with a data scientist to write a [custom configuration](configuration.md) for your experiment.
+If the metrics you need are not covered by Guardrail/Core metrics or existing Outcomes - you can add a metric to your experiment by working with a data scientist to write a [custom configuration](./configuration) for your experiment.
 
 
 

docs/jetstream/operations.md → docs/deep-dives/jetstream/operations.md

@@ -1,9 +1,12 @@
 ---
 id: operations
 title: Jetstream Architecture and Operations
+slug: operations
+sidebar_position: 7
+sidebar_label: Architecture and Operations
 ---
 
-**[Jetstream]** is part of the Cirrus ecosystem and depends on some external services.
+[Jetstream](./overview) is part of the Cirrus ecosystem and depends on some external services.
 
 ![Cirrus overview](/img/jetstream/cirrus.png)
 *High-level overview of Cirrus*
@@ -111,7 +114,7 @@ This date will again be based on the last updated timestamp of the enrollments t
 
 When making changes to experiment-specific configs, jetstream will automatically rerun the affected experiments which will result in the enrollments table getting updated and the most recent configs in metric-hub being used.
 
-More information on how to use the most recent tooling and metric versions can be found [here](jetstream.md/#how-to-use-the-latest-tooling-and-metric-definitions).
+More information on how to use the most recent tooling and metric versions can be found [here](./overview#how-to-use-the-latest-tooling-and-metric-definitions).
 
 [jetstream]: https://github.com/mozilla/jetstream
 [jetstream error dashboard]: https://mozilla.cloud.looker.com/dashboards/246

docs/jetstream/outcomes.md → docs/deep-dives/jetstream/outcomes.md

@@ -1,6 +1,8 @@
 ---
 id: outcomes
 title: Outcomes
+slug: outcomes
+sidebar_position: 4
 ---
 
 An Outcome is a reusable configuration snippet
@@ -50,7 +52,7 @@ If it passes, a data scientist can merge the Outcome definition without addition
 
 Experimenter will need to be re-deployed to pick up a new Outcome. Please ask in #nimbus-project if you're in a hurry!
 
-[Configuring Jetstream]: jetstream/configuration.md
+[Configuring Jetstream]: ./configuration
 
 ## Parameterizing Outcomes
 
@@ -94,5 +96,5 @@ and then copy-paste the metrics into an Outcome for later use.
 ## What happens if an Outcome changes?
 
 Changing an Outcome does not re-run any experiments.
-A commit hash associated with the version of each Outcome is captured in the [experiment metadata](jetstream/data-products.md) published to GCS,
+A commit hash associated with the version of each Outcome is captured in the [experiment metadata](./data-products) published to GCS,
 so it's possible to understand which version of an Outcome was associated with an experiment analysis.

docs/jetstream/jetstream.md → docs/deep-dives/jetstream/overview.md

@@ -1,7 +1,8 @@
 ---
-id: jetstream
-title: Jetstream overview
-slug: /jetstream/jetstream
+id: overview 
+title: Overview
+slug: overview
+sidebar_position: 1
 ---
 
 **[Jetstream]** is an analysis framework for experiments.

docs/jetstream/statistics.md → docs/deep-dives/jetstream/statistics.md

@@ -1,6 +1,8 @@
 ---
 id: statistics
 title: Statistics
+slug: statistics
+sidebar_position: 3
 ---
 
 Statistics reduce observations of many clients

docs/jetstream/testing.md → docs/deep-dives/jetstream/testing.md

@@ -1,6 +1,8 @@
 ---
 id: testing
 title: Testing Jetstream Configs
+slug: testing
+sidebar_position: 9
 ---
 
 # Testing Jetstream Configs

docs/jetstream/troubleshooting.md → docs/deep-dives/jetstream/troubleshooting.md

@@ -1,6 +1,9 @@
 ---
 id: troubleshooting
 title: Troubleshooting Jetstream
+slug: troubleshooting
+sidebar_label: Troubleshooting
+sidebar_position: 8
 ---
 
 ### How can I see what Jetstream is doing? 

docs/faq/feature-restart/feature-restart.mdx

@@ -0,0 +1,18 @@
+---
+id: feature-restart 
+title: Feature restart
+slug: /faq/feature-restart
+---
+
+### Question
+Is feature restart supported?
+
+### Answer
+
+Yes! Feature restart is supported by Nimbus. This is how it works when a feature requires a restart:
+* **Session 1:** Client enrolls, applies feature configuration, but the feature itself won't implement the changes until the next restart
+* **Session 2:** Client is enrolled, feature configuration is applied, feature now shows changes
+
+:::tip Important
+Please share that this experiment needs a restart in your data science Jira ticket and that there is custom work needed at analysis to exlude the pre-restart data.
+:::

docs/faq/general-faq/general-faq.mdx

@@ -10,4 +10,25 @@ What's the difference between "Experimenter" and "Nimbus"?
 
 ### Answer
 
-Nimbus is the whole experimentation system (handles enrolling/unenrolling users, sending branch configurations, etc).  Nimbus was the code name of the experimentation project.  If you hear Jetstream, that is the whole analysis system (interfaces with data, runs analysis, formats results).  Experimenter is the customer interface to both Nimbus and Jetstream - that you access at experimenter.services.mozilla.com/.  In Experimenter users can configure experiments, manage them (launch, monitor, end), and can see the results.
+Nimbus is the code name of the experimentation project, so Nimbus refers to **the whole experimentation system** (handles enrolling/unenrolling users, sending branch configurations, etc). Experimenter is the **customer interface to both Nimbus and Jetstream** that you access at http://experimenter.services.mozilla.com/. In Experimenter, users can configure experiments, manage them (launch, monitor, end), and can see the results. 
+
+---
+
+### Question
+
+What is Jetstream?
+
+### Answer
+
+[Jetstream](/jetstream/overview) is the whole **analysis system** (interfaces with data, runs analysis, formats results).
+
+---
+
+### Question
+
+What is Normandy?
+
+### Answer
+
+Normandy (https://experimenter.services.mozilla.com/legacy/) is the legacy experimentation platform for Firefox. It is being deprecated in favor of Nimbus.
+

docs/getting-started/data-scientists/auto-sizing-cli.md

@@ -12,7 +12,7 @@ The sizing CLI relies on a local TOML file to configure the job. This TOML file
 
 ### TOML file layout
 
-The TOML configuration file must contain a `metrics`, `data_sources`, `segments`, and `segments.data_sources` section, each containing the definitions for those of interest for the experiments. The definition of each of these follows the same patterns as [Jetstream], and details on how to define your own inside of the TOML file can be found [here](https://experimenter.info/jetstream/configuration#defining-metrics).
+The TOML configuration file must contain a `metrics`, `data_sources`, `segments`, and `segments.data_sources` section, each containing the definitions for those of interest for the experiments. The definition of each of these follows the same patterns as [Jetstream], and details on how to define your own inside of the TOML file can be found [here](/deep-dives/jetstream/configuration#defining-metrics).
 
 The TOML file can also contain references to metrics, segments, data sources, and segment data sources that are already contained in [metric-hub]. To reference these pre-defined objects, an `import_from_metric_hub` list can be included in the TOML file. For instance, to import the `active_hours` metric for Firefox Desktop, the following is included in the TOML config file:
 ```
@@ -52,7 +52,7 @@ The results JSON will include an entry for each combination of power and effect
 ```
 
 
-[Jetstream]: jetstream/jetstream.md
+[Jetstream]: /deep-dives/jetstream/overview
 [metric-hub]: https://github.com/mozilla/metric-hub
 [mozanalysis]: https://github.com/mozilla/mozanalysis
-[auto-sizing]: https://github.com/mozilla/auto-sizing
+[auto-sizing]: https://github.com/mozilla/auto-sizing

docs/getting-started/data-scientists/experiment-sizing.md

@@ -2,6 +2,7 @@
 id: experiment-sizing
 title: Sizing Experiments Using Mozanalysis
 slug: /experiment-sizing
+sidebar_label: Sizing using Mozanalysis
 ---
 
 This page gives an overview of how Mozanalysis can be used to do experiment sizing. **[Mozanalysis]** is a library used to standardize experiment analysis at Mozilla. Mozanalysis contains the statistical tools used by [Jetstream] to produce results from experiments, as well as tools to pull historical data from BigQuery to calculate necessary sample sizes to carry out experiments. The results of the tool are the sample size and percent of the total target population required per branch of an experiment with a balanced design to achieve a given power.
@@ -38,7 +39,7 @@ The time series data returned will contain a row for each client for each time s
 
 Selecting clients for analysis is accomplished by using the [`Segment`](https://github.com/mozilla/mozanalysis/tree/main/src/mozanalysis/segments) objects in Mozanalysis; users can either reuse segments that currently exist in Mozanalysis or define one at runtime. Segments consist of a data source, either a table or view in BigQuery, and a SELECT expression to filter that data source; this SELECT expression must include a SQL aggregate function.
 
-Similarly, Mozanalysis experiment sizing reuses `Metric` objects from [Jetstream](jetstream/metrics.md), and users can reuse metrics that are currently implemented in [Mozanalysis](https://github.com/mozilla/mozanalysis/tree/main/src/mozanalysis/metrics) or in [Jetstream configs](https://github.com/mozilla/metric-hub/tree/main/jetstream), or users can define their own at runtime.
+Similarly, Mozanalysis experiment sizing reuses [`Metric` objects](/deep-dives/jetstream/metrics) from Jetstream, and users can reuse metrics that are currently implemented in [Mozanalysis](https://github.com/mozilla/mozanalysis/tree/main/src/mozanalysis/metrics) or in [Jetstream configs](https://github.com/mozilla/metric-hub/tree/main/jetstream), or users can define their own at runtime.
 
 Metrics and target segments are passed to Mozanalysis experiment sizing as lists of `Segment` or `Metric` objects; users may include multiple of each in the analysis. When multiple `Segment` objects are used, Mozanalysis identifies clients that satisfy the conditions of **all** targets in the list. If users would like to run analyses where `Segment`s should be joined with OR rather than AND, multiple experiment sizing tasks should be completed, for each condition in the OR statement, and aggregate the returned results from each separate study.
 
@@ -60,5 +61,5 @@ Mozanalysis contains functions that take the results from pulling historical dat
 4. Replicating sizing for Waldo, which uses continuous enrollment [(link)](https://colab.research.google.com/drive/1_R4zBUnucRPmHwIVLlPTYInDZwTbCn-F?usp=sharing)
 5. Pulling time series historical data and sizing with empirical sample size calculation [(link)](https://colab.research.google.com/drive/1-XT2DMfGSqiCS18yGPIGCs_YWg75qZzn?usp=sharing)
 
-[Jetstream]: jetstream/jetstream.md
+[Jetstream]: /deep-dives/jetstream/overview
 [mozanalysis]: https://github.com/mozilla/mozanalysis

docs/getting-started/data-scientists/data-scientists-root.md → docs/getting-started/data-scientists/overview.md

@@ -1,14 +1,16 @@
 ---
-id: data-scientists-root
+id: overview
 title: Experimentation for data scientists
-slug: /data-scientists
+slug: /overview
+sidebar_position: 1
+sidebar_label: Overview
 ---
 
 This page describes the roles data scientists play in experimentation at Mozilla.
 
 Some other things you may be looking for are:
 
-* Documentation about using [Jetstream](jetstream/jetstream.md), Mozilla's experiment analysis tool
+* Documentation about using [Jetstream](/deep-dives/jetstream/overview), Mozilla's experiment analysis tool
 * Technical documentation about [datasets used in experimentation](https://docs.telemetry.mozilla.org/tools/experiments.html)
 * [Process documentation](https://mozilla-hub.atlassian.net/wiki/spaces/DATA/overview) for the Mozilla data science organization
   (internal link)
@@ -35,14 +37,14 @@ The support that experiment owners need from data scientists during experiment s
 * validating that the experimental design will answer their questions
 * consulting on telemetry specifications
 * sample size recommendations
-* writing [custom metrics](jetstream/metrics.md) if needed
+* writing [custom metrics](/deep-dives/jetstream/metrics) if needed
 * guidance about interpretation
 
 ## Sampling framework
 
 Each experiment begins with an enrollment period with fixed start and end dates during which clients can enroll in the experiment. After the declared enrollment period ends, we modify the recipe to instruct clients to stop enrolling, and ignore clients that report enrollment anyway. Because enrollment depends on the client checking for software updates, samples will be skewed towards active users at the beginning of the enrollment period. We typically leave enrollment open for a week to account for weekly seasonality (e.g. weekday vs. weekend users) and to give clients who are active less often a chance to enroll. This makes our experiment population essentially a sample of the weekly active users (WAU).
 
-For each client, experiment metrics are analyzed over a defined period of time from enrollment. We report results for an analysis period (e.g. the first day after enrollment) after all clients have had a chance to experience the treatment for that duration. The [Jetstream overview](jetstream/jetstream.md#analysis-paradigm) describes the analysis paradigm in more depth and how it relates to the length of an experiment.
+For each client, experiment metrics are analyzed over a defined period of time from enrollment. We report results for an analysis period (e.g. the first day after enrollment) after all clients have had a chance to experience the treatment for that duration. The [Jetstream overview](/deep-dives/jetstream/overview#analysis-paradigm) describes the analysis paradigm in more depth and how it relates to the length of an experiment.
 
 For more nuances about sampling, enrollment and exposure (whether or not the client actually saw the treatment or control), see [the experiment lifecycle overview](deep-dives/specifications/client-sdk-states-and-lifecycle.mdx).
 
@@ -93,7 +95,7 @@ Most of our telemetry metrics are not normally distributed. A couple approaches
 
 Two weeks after the beginning of the enrollment period, Jetstream will begin to produce auto-generated reports summarizing the results of the experiment. [Here is one such report](https://experimenter.services.mozilla.com/nimbus/custom-messaging-in-aboutwelcome-for-chrome-users-to-import/results) (internal link).
 
-To see which metrics are included by default to this auto-generated report as well as information on adding custom and default metrics, statistics and confidence, see the [Jetstream documentation](jetstream/jetstream.md).
+To see which metrics are included by default to this auto-generated report as well as information on adding custom and default metrics, statistics and confidence, see the [Jetstream documentation](/deep-dives/jetstream/overview).
 
 If you want to perform some analysis by hand, [Jetstream datasets](https://docs.telemetry.mozilla.org/datasets/jetstream.html) are also available in BigQuery. Many telemetry datasets also include an `experiments` field, which when filtered on the experiment slug, can identify rows in the dataset enrolled in the experiment.