Polishing and add fides to docs environment.

docs/rtd_environment.yml

@@ -40,3 +40,4 @@ dependencies:
       - types-jinja2  # dev, tests
       - sqlalchemy-stubs  # dev, tests
       - sphinxcontrib-mermaid  # dev, tests, docs
+      - fides==0.7.4  # dev, tests

docs/source/how_to/how_to_algorithm_selection.ipynb

@@ -35,6 +35,45 @@
     "\n",
     "These steps work well for most problems. Sometimes you need [variations](four-steps-variations).\n",
     "\n",
+    "\n",
+    "## A decision tree \n",
+    "\n",
+    "This is a practical guide for narrowing down the set of algorithms to experiment with:\n",
+    "\n",
+    "```{mermaid}\n",
+    "graph LR\n",
+    "    classDef highlight fill:#FF4500;\n",
+    "    A[\"Do you have<br/>nonlinear constraints?\"] -- yes --> B[\"differentiable?\"]\n",
+    "    B[\"Is your objective function differentiable?\"] -- yes --> C[\"'ipopt', 'nlopt_slsqp', 'scipy_trust_constr', ...\"]\n",
+    "    B[\"differentiable?\"] -- no --> D[\"'scipy_cobyla', 'nlopt_cobyla', ...\"]\n",
+    "\n",
+    "    A[\"Do you have<br/>nonlinear constraints?\"] -- no --> E[\"Can you exploit<br/>a least-squares<br/>structure?\"]\n",
+    "    E[\"Can you exploit<br/>a least-squares<br/>structure?\"] -- yes --> F[\"differentiable?\"]\n",
+    "    E[\"Can you exploit<br/>a least-squares<br/>structure?\"] -- no --> G[\"differentiable?\"]\n",
+    "\n",
+    "    F[\"differentiable?\"] -- yes --> H[\"'scipy_ls_lm', 'scipy_ls_trf', 'scipy_ls_dogleg', ...\"]\n",
+    "    F[\"differentiable?\"] -- no --> I[\"'nag_dflos', 'pounders', 'tao_pounders', ...\"]\n",
+    "\n",
+    "    G[\"differentiable?\"] -- yes --> J[\"'scipy_lbfgsb', 'nlopt_lbfgsb', 'fides', ...\"]\n",
+    "    G[\"differentiable?\"] -- no --> K[\"'nlopt_bobyqa', 'nlopt_neldermead', 'neldermead_parallel', ...\"]\n",
+    "```\n",
+    "\n",
+    "Going through the different questions will give you a list of candidate algorithms. \n",
+    "All algorithms in that list are designed for the same problem class but use different \n",
+    "approaches to solve the problem. Which of them works best for your problem can only be \n",
+    "found out through experimentation.\n",
+    "\n",
+    "```{note}\n",
+    "Many books on numerical optimization focus strongly on the inner workings of algorithms.\n",
+    "They will, for example, describe the difference between a trust-region algorithm and a \n",
+    "line-search algorithm in a lot of detail. We have an [intuitive explanation](../explanation/explanation_of_numerical_optimizers.md) of this too. However, these details are not \n",
+    "very relevant for algorithm selection. For example, If you have a scalar, differentiable \n",
+    "problem without nonlinear constraints, the decision tree suggests `fides` and `lbfgsb`.\n",
+    "`fides` is a trust-region algorithm, `lbfgsb` is a line-search algorithm. Both are \n",
+    "designed to solve the same kinds of problems and which one works best needs to be \n",
+    "found out through experimentation.\n",
+    "```\n",
+    "\n",
     "## An example problem\n",
     "\n",
     "As an example we use the [Trid function](https://www.sfu.ca/~ssurjano/trid.html). The Trid function has no local minimum except \n",
@@ -84,29 +123,11 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "## Step 1: Theory\n",
+    "### Step 1: Theory\n",
     "\n",
-    "This is a practical guide for narrowing down the set of algorithms to experiment with:\n",
-    "\n",
-    "```{mermaid}\n",
-    "graph LR\n",
-    "    classDef highlight fill:#FF4500;\n",
-    "    A[\"Do you have<br/>nonlinear constraints?\"] -- yes --> B[\"differentiable?\"]\n",
-    "    B[\"Is your objective function differentiable?\"] -- yes --> C[\"'ipopt', 'nlopt_slsqp', 'scipy_trust_constr', ...\"]\n",
-    "    B[\"differentiable?\"] -- no --> D[\"'scipy_cobyla', 'nlopt_cobyla', ...\"]\n",
     "\n",
-    "    A[\"Do you have<br/>nonlinear constraints?\"] -- no --> E[\"Can you exploit<br/>a least-squares<br/>structure?\"]\n",
-    "    E[\"Can you exploit<br/>a least-squares<br/>structure?\"] -- yes --> F[\"differentiable?\"]\n",
-    "    E[\"Can you exploit<br/>a least-squares<br/>structure?\"] -- no --> G[\"differentiable?\"]\n",
-    "\n",
-    "    F[\"differentiable?\"] -- yes --> H[\"'scipy_ls_lm', 'scipy_ls_trf', 'scipy_ls_dogleg', ...\"]\n",
-    "    F[\"differentiable?\"] -- no --> I[\"'nag_dflos', 'pounders', 'tao_pounders', ...\"]\n",
-    "\n",
-    "    G[\"differentiable?\"] -- yes --> J[\"'scipy_lbfgsb', 'nlopt_lbfgsb', 'fides', ...\"]\n",
-    "    G[\"differentiable?\"] -- no --> K[\"'nlopt_bobyqa', 'nlopt_neldermead', 'neldermead_parallel', ...\"]\n",
-    "```\n",
     "\n",
-    "Let's go through the steps for the Trid function:\n",
+    "Let's go through the decision tree for the Trid function:\n",
     "\n",
     "1. **No** nonlinear constraints our solution needs to satisfy\n",
     "2.  **No** no least-squares structure we can exploit \n",
@@ -117,7 +138,7 @@
     "`fides`.\n",
     "\n",
     "\n",
-    "## Step 2: Experiments\n",
+    "### Step 2: Experiments\n",
     "\n",
     "To find out which algorithms work well for our problem, we simply run optimizations with\n",
     "all algorithms in a loop and store the result in a dictionary. We limit the number of \n",
@@ -146,7 +167,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "## Step 3: Comparison\n",
+    "### Step 3: Comparison\n",
     "\n",
     "Next we plot the optimizer histories to find out which optimizer worked best:"
    ]
@@ -170,7 +191,7 @@
     "better than the others, so we will select it for the next step. In more difficult\n",
     "examples, the difference between optimizers can be much more pronounced.\n",
     "\n",
-    "## Step 4: Optimization \n",
+    "### Step 4: Optimization \n",
     "\n",
     "All that is left to do is to run the optimization until convergence with the best \n",
     "optimizer. To avoid duplicated calculations, we can already start from the previously \n",