Why-derivative-Y=XW+b-on-X=W-transpose.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "3178a5f0-1d17-4a9b-a6d0-d4d15dfb7ac6",
   "metadata": {},
   "source": [
    "# 神经网络Y=XW + b中的矩阵求导和转置扫盲\n",
    "## 标量对矩阵求导\n",
    "\n",
    "在深度学习或数值优化中，如果 L 是一个标量损失函数，而 $\\boldsymbol{Y}$ 是一个形状为 ($N \\times m$) 的输出（通常是网络的某层输出，或模型的预测结果），那么\n",
    "\n",
    "$$\n",
    "\\frac{\\partial L}{\\partial \\boldsymbol{Y}}\n",
    "$$\n",
    "\n",
    "的形状也为 ($N \\times m$)。这是由「标量对矩阵求导」的一般原则决定的。\n",
    "\n",
    "1. 一般原则：标量对矩阵求导\n",
    "\n",
    "若 L（标量）依赖于一个矩阵 $\\boldsymbol{Y}$ 中的每个元素 $Y_{i,j}$（总共有 $N \\times m$ 个元素），那么从微积分的角度看，$\\frac{\\partial L}{\\partial \\boldsymbol{Y}}$ 就是记录了对矩阵中每个元素 $Y_{i,j}$ 的偏导数：\n",
    "\n",
    "$$\n",
    "\\frac{\\partial L}{\\partial \\boldsymbol{Y}}\n",
    "\\;=\\;\n",
    "\\biggl[\n",
    "\\frac{\\partial L}{\\partial Y_{i,j}}\n",
    "\\biggr]{1\\le i\\le N,\\;1\\le j\\le m}\n",
    "\\;\\;\\in\\;\\mathbb{R}^{N\\times m}.\n",
    "$$\n",
    "\n",
    "换言之，$\\tfrac{\\partial L}{\\partial \\boldsymbol{Y}}$的$(i,j)$元素就是$\\tfrac{\\partial L}{\\partial Y{i,j}}$。由于 $i$ 从 $1$ 到 $N$，$j$ 从 $1$ 到 $m$，可见它自然是一个 $N\\times m$ 的矩阵。\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "292f4df8-f806-4ac9-b089-6cba1909b3c6",
   "metadata": {},
   "source": [
    "## Y=WX+B\n",
    "\n",
    "前向传播公式：\n",
    "\n",
    "$$\n",
    "Y = XW + b\n",
    "$$\n",
    "\n",
    "其中：\n",
    "* $X$ 是输入矩阵，形状为 ($N \\times d$)\n",
    "* $W$ 是权重矩阵，形状为 ($d \\times m$)\n",
    "* $b$ 是偏置向量，形状为 ($1 \\times m$)\n",
    "* $Y$ 是输出矩阵，形状为 ($N \\times m$)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1e5b5331-2b9c-44c7-aeb6-bc4f6a66db9b",
   "metadata": {},
   "source": [
    "1. 从标量情形类比\n",
    "\n",
    "假设有一个最简单的标量情形：\n",
    "\n",
    "$$\n",
    "y = xw\n",
    "$$\n",
    "\n",
    "其中 $x \\in \\mathbb{R}, w \\in \\mathbb{R}$ 都是标量。显然，\n",
    "\n",
    "$$\n",
    "\\frac{\\partial y}{\\partial x} \\;=\\; w.\n",
    "$$\n",
    "\n",
    "如果我们把 $x$ 和 $w$ 都“升格”到列向量，但仍保持“输出是标量”，比如\n",
    "\n",
    "$$\n",
    "y \\;=\\; x^\\top w,\n",
    "$$\n",
    "\n",
    "其中 $x\\in \\mathbb{R}^d,\\,w\\in \\mathbb{R}^d$。此时，\n",
    "\n",
    "$$\n",
    "y \\;=\\; \\sum_{k=1}^{d} x_k\\,w_k,\n",
    "$$\n",
    "\n",
    "那么对 x 求偏导（梯度）就得到\n",
    "\n",
    "$$\n",
    "\\nabla_x\\,y\n",
    "\\;=\\;\n",
    "w\n",
    "\\;\\;\\in \\mathbb{R}^d.\n",
    "$$\n",
    "\n",
    "这说明：从 $\\mathbb{R}^d$ 到 $\\mathbb{R}$ 的线性映射，Jacobian梯度就是那个“权重向量”本身。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0fcebcda-66dc-4b79-9a1e-b25a0e55c42f",
   "metadata": {},
   "source": [
    "2. 将输出扩展为向量：小型 2D 示例\n",
    "\n",
    "现在我们让输出也变成向量（多维），并且暂时使用一个小例子来避免符号上的抽象。设\n",
    "\n",
    "$$\n",
    "x \\;=\\; (x_1,\\;x_2)\n",
    "\\;\\in \\mathbb{R}^2,\n",
    "\\quad\n",
    "W \\;=\\;\n",
    "\\begin{pmatrix}\n",
    "w_{1,1} & w_{1,2} \\\\[6pt]\n",
    "w_{2,1} & w_{2,2}\n",
    "\\end{pmatrix}\n",
    "\\;\\in \\mathbb{R}^{2\\times 2}.\n",
    "$$\n",
    "\n",
    "令\n",
    "$$\n",
    "y\n",
    "\\;=\\;\n",
    "x \\,W\n",
    "\\;=\\;\n",
    "(x_1,\\;x_2)\n",
    "\\begin{pmatrix}\n",
    "w_{1,1} & w_{1,2} \\\\\n",
    "w_{2,1} & w_{2,2}\n",
    "\\end{pmatrix}\n",
    "\\;=\\;\n",
    "\\bigl(y_1,\\;y_2\\bigr)\n",
    "\\;\\in \\mathbb{R}^2.\n",
    "$$\n",
    "\n",
    "那么\n",
    "$$\n",
    "\\begin{cases}\n",
    "y_1 = x_1\\,w_{1,1} + x_2\\,w_{2,1},\\\\\n",
    "y_2 = x_1\\,w_{1,2} + x_2\\,w_{2,2}.\n",
    "\\end{cases}\n",
    "$$\n",
    "\n",
    "我们要构造它的 Jacobian 矩阵 J。在多数数学与工程文献中，对映射 $f:\\,\\mathbb{R}^d \\to \\mathbb{R}^m$，其 Jacobian $J$ 的形状是\n",
    "\n",
    "$$\n",
    "m \\,\\times\\, d,\n",
    "$$\n",
    "\n",
    "其中\n",
    "\n",
    "$$\n",
    "J_{i,j}\n",
    "\\;=\\;\n",
    "\\frac{\\partial\\,y_i}{\\partial\\,x_j}.\n",
    "$$\n",
    "\n",
    "具体到 2D-2D 的例子中：\n",
    "$$\n",
    "J\n",
    "\\;=\\;\n",
    "\\begin{pmatrix}\n",
    "\\dfrac{\\partial y_1}{\\partial x_1} & \\dfrac{\\partial y_1}{\\partial x_2} \\\\[6pt]\n",
    "\\dfrac{\\partial y_2}{\\partial x_1} & \\dfrac{\\partial y_2}{\\partial x_2}\n",
    "\\end{pmatrix}.\n",
    "$$\n",
    "\n",
    "据上式可得\n",
    "\n",
    "$$\n",
    "\\frac{\\partial y_1}{\\partial x_1} = w_{1,1},\n",
    "\\quad\n",
    "\\frac{\\partial y_1}{\\partial x_2} = w_{2,1};\n",
    "\\quad\n",
    "\\frac{\\partial y_2}{\\partial x_1} = w_{1,2},\n",
    "\\quad\n",
    "\\frac{\\partial y_2}{\\partial x_2} = w_{2,2}.\n",
    "$$\n",
    "\n",
    "所以\n",
    "\n",
    "$$\n",
    "J\n",
    "\\;=\\;\n",
    "\\begin{pmatrix}\n",
    "w_{1,1} & w_{2,1}\\\\\n",
    "w_{1,2} & w_{2,2}\n",
    "\\end{pmatrix}.\n",
    "$$\n",
    "\n",
    "而 $W$ 本身是\n",
    "\n",
    "$$\n",
    "W\n",
    "\\;=\\;\n",
    "\\begin{pmatrix}\n",
    "w_{1,1} & w_{1,2} \\\\\n",
    "w_{2,1} & w_{2,2}\n",
    "\\end{pmatrix}.\n",
    "$$\n",
    "\n",
    "显然\n",
    "\n",
    "$$\n",
    "J\n",
    "\\;=\\;\n",
    "W^\\top,\n",
    "$$\n",
    "\n",
    "这就是最简单的 $2×2$ 例子所展示的Jacobian = $W^T$现象。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fed6f69e-7b77-461b-91c0-d074158d49a8",
   "metadata": {},
   "source": [
    "3. 一般情形：$x \\in \\mathbb{R}^d \\to y \\in \\mathbb{R}^m$\n",
    "\n",
    "推广到一般情形，如果我们把 x 当作列向量（这是最常见的数学约定），维度为 $d\\times 1$；把 $W$ 视作 $d\\times m$ 的矩阵；那么\n",
    "\n",
    "$$\n",
    "y\n",
    "\\;=\\;\n",
    "x^\\top W\n",
    "\\;\\;\\in \\mathbb{R}^m\n",
    "\\;\\;(\\text{或等价地 }y = W^\\top x \\in \\mathbb{R}^m),\n",
    "$$\n",
    "\n",
    "这里我们先用 $y = x^\\top W$ 来与「行向量乘 $W$」的场景对齐。此时的函数是\n",
    "\n",
    "$$\n",
    "f:\\,\\mathbb{R}^d \\;\\to\\; \\mathbb{R}^m,\n",
    "\\quad\n",
    "y = f(x).\n",
    "$$\n",
    "\n",
    "* 输出 $y$ 的每个分量 $y_i$ 都是 $x_j$ 的线性组合；\n",
    "* 根据$(\\tfrac{\\partial y_i}{\\partial x_j} = W_{j,i}$可知，Jacobian $J = [W_{j,i}]_{i,j}$，也就是把 W 在行列上做一个调换，等于 $W^\\top$，形状为 $m \\times d$。\n",
    "\n",
    "若你在某些深度学习框架或某些教材里，看到说$\\frac{\\partial y}{\\partial x} = W^T$并且标注形状 $d\\times m$的情形，往往是因为它们把 $x$ 当作行向量($1×d$)，然后希望得到一个同样能与“行向量形状匹配”的梯度张量，从而在实现(autograd)层面会把该矩阵写成$d\\times m$。实际上，这跟上文的数学定义下 Jacobian 通常是$m\\times d$在本质上并不矛盾——只是行向量 / 列向量在编程实现与数学记号上做了不同的约定或转置。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "16382258-f15f-49e4-9e00-8c824012fa14",
   "metadata": {},
   "source": [
    "4. 为什么「线性映射」的导数必然是“那个矩阵的转置”？\n",
    "\n",
    "从微分/导数的本质出发：对于线性映射（或仿射映射），输出对输入的变化率是一个“恒定不变的矩阵”。若前向映射是\n",
    "$$\n",
    "y = xW \\quad \\bigl(\\text{无论 }x\\text{是行还是列}\\bigr),\n",
    "$$\n",
    "那么对输入 x 做一个「无穷小」变化 $\\mathrm{d}x$，输出的变化 $\\mathrm{d}y$ 便是\n",
    "\n",
    "$$\n",
    "\\mathrm{d}y = \\mathrm{d}x \\,W\n",
    "\\quad\\text{(或 }W^\\top\\,\\mathrm{d}x\\text{, 取决于 }x\\text{形状)},\n",
    "$$\n",
    "\n",
    "而“Jacobian”就是将 $\\mathrm{d}x$ 转换为 $\\mathrm{d}y$ 的那个映射本身。但因为我们在索引 $(i,j)$ 上要写成$\\partial y_i/\\partial x_j$，行列索引会对调，所以出现了“转置”这一现象。\n",
    "\n",
    "简而言之：前向乘法用 $W$，反向（求导)“看矩阵”时就自带一次转置。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "880a7ea6-a768-4169-9945-3e0c7f6f0b39",
   "metadata": {},
   "source": [
    "5. 回到问题：“为什么是 $W^T$，且形状是 $d\\times m$ 或 $m\\times d$”\n",
    "* 数学上最常见的 Jacobian 定义：若 $f:\\,\\mathbb{R}^d \\to \\mathbb{R}^m$，那么 Jacobian $J$ 形状是 $m\\times d$。\n",
    "在此约定下，$J = W^\\top（形状 m\\times d）$，因为\n",
    "\n",
    "$$\n",
    "J_{i,j}\n",
    "\\;=\\;\n",
    "\\frac{\\partial y_i}{\\partial x_j}\n",
    "\\;=\\;\n",
    "W_{j,i}.\n",
    "$$\n",
    "\n",
    "* 在部分编程/深度学习框架中的写法：如果将 $x$ 视作$1×d$行向量，输出 $y$ 视作$1×m$行向量，那么在实现自动求导时，常把\n",
    "\n",
    "$$\n",
    "\\frac{\\partial (1\\times m)}{\\partial (1\\times d)}\n",
    "$$\n",
    "\n",
    "写成一个$d\\times m$的矩阵，以便与$\\Delta X$（$1×d$）的维度做矩阵乘法时符合形状规则。\n",
    "这时也会看到$\\frac{\\partial Y}{\\partial X} = W^T$但它们往往会标注 “($d\\times m$)”。\n",
    "这里的$d\\times m$和前面数学定义里的$m\\times d$其实仅仅是一个行向量 / 列向量的转置约定差异，本质内容相同。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b59ef9de-3285-4d64-8c65-4295079632b2",
   "metadata": {},
   "source": [
    "<span style=\"color: red;\">**一句话总结**</span>\n",
    "\n",
    "凡是“前向”用 $W$ 做乘法的线性映射，求对输入的导数（Jacobian）时，就会冒出 $W^\\top$。\n",
    "出现 $d\\times m$ 还是 $m\\times d$，主要取决于你把输入、输出当作行向量还是列向量来写，以及对应的 Jacobian 在行、列索引上的定义。\n",
    "\n",
    "这就是为什么\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "\\text{对线性映射 } y = xW,\\;\\; \\nabla_x\\,y = W^\\top\n",
    "}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "83d90f56-865f-4b99-bb9e-2105e698cd24",
   "metadata": {},
   "source": [
    "## XW对W的偏导数\n",
    "\n",
    "$$\n",
    "Y = XW + b,\n",
    "$$\n",
    "\n",
    "* X 形状为 ($N \\times d$)，\n",
    "* W 形状为 ($d \\times m$)，\n",
    "* Y 形状为 ($N \\times m$)。\n",
    "\n",
    "我们已经知道\n",
    "\n",
    "$$\n",
    "\\frac{\\partial Y}{\\partial X} = W^T.\n",
    "$$\n",
    "\n",
    "那么，如果不再是标量损失函数 L，而是直接问$\\frac{\\partial Y}{\\partial W} 是什么？$，会遇到一个概念上的小“坑”：$Y$ 本身是矩阵，$W$ 也是矩阵，严格来说它们之间的导数在最一般的定义下会是一个四阶张量（4D Tensor），形状为 ($N \\times m \\times d \\times m$)。这是因为：\n",
    "* $Y$ 里有 $N\\times m$ 个元素 $Y_{i,j}$；\n",
    "* $W$ 里有 $d\\times m$ 个元素 $W_{k,\\ell}$；\n",
    "* 对任意 $(i,j)$ 和 $(k,\\ell)$，$\\frac{\\partial\\,Y_{i,j}}{\\partial\\,W_{k,\\ell}}$ 都是一个标量。\n",
    "\n",
    "在很多应用（尤其是训练神经网络）时，我们并不会直接用到这样的四阶张量，而是配合「链式法则」把它和 $\\frac{\\partial L}{\\partial Y}$（标量损失对输出的梯度）等结合，得到 $\\frac{\\partial L}{\\partial W}$。只有在纯粹的矩阵微积分探讨中，才会把 $\\frac{\\partial Y}{\\partial W}$ 当作“矩阵对矩阵的 Jacobian 张量”来详细研究。\n",
    "\n",
    "下面分几步来讲："
   ]
  },
  {
   "cell_type": "markdown",
   "id": "59cacf7a-c80c-47f5-a34b-3604125809db",
   "metadata": {},
   "source": [
    "1. 元素级别的推导\n",
    "\n",
    "先不考虑偏置 b，令\n",
    "\n",
    "$$\n",
    "Y = X W\n",
    "$$\n",
    "\n",
    "则\n",
    "$$\n",
    "Y_{i,j}\n",
    "\\;=\\;\n",
    "\\sum_{k=1}^{d} \\underbrace{X_{i,k}}{\\text{第 }i\\text{ 行 }k\\text{ 列}}\n",
    "\\,\\underbrace{W{k,j}}{\\text{第 }k\\text{ 行 }j\\text{ 列}}.\n",
    "$$\n",
    "\n",
    "问：$\\frac{\\partial Y{i,j}}{\\partial W_{p,q}}$ 等于多少？\n",
    "* 若 $(p,q)$ 与 $(k,j)$ 不匹配，则导数为 0；\n",
    "* 只有当 $p=k$ 且 $q=j$ 时，$\\frac{\\partial Y_{i,j}}{\\partial W_{p,q}} = \\frac{\\partial}{\\partial W_{p,q}}(X_{i,p}W_{p,j}) = X_{i,p}$.\n",
    "\n",
    "用“克罗内克 $δ$ 符号”记，就是\n",
    "$$\n",
    "\\frac{\\partial Y_{i,j}}{\\partial W_{p,q}}=\n",
    "X_{i,p}\\,\\delta_{j,q},\n",
    "$$\n",
    "\n",
    "其中 $\\delta_{j,q} = 1$ 当 $j=q$，否则为 $0$。\n",
    "\n",
    "这说明：在四维索引 $(i,j,p,q)$ 的位置上，导数的值是 $X_{i,p}$ 或 $0$。\n",
    "\n",
    "从而我们知道这个$\\frac{\\partial Y}{\\partial W}$在最一般的形式下是一个形如 ($N \\times m \\times d \\times m$) 的四阶张量。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "92ef90d8-5436-47c9-9bbc-65df03c4fdf6",
   "metadata": {},
   "source": [
    "2. 若我们想得到与 W 形状一致的“梯度矩阵”怎么办？\n",
    "\n",
    "如果问题是“如何在反向传播中计算 $\\frac{\\partial L}{\\partial W}$ ”，其中 $L$ 是一个标量损失，则可以利用链式法则：\n",
    "\n",
    "$$\n",
    "\\frac{\\partial L}{\\partial W_{k,\\ell}}\n",
    "\\;=\\;\n",
    "\\sum_{i=1}^{N}\\sum_{j=1}^{m}\n",
    "\\frac{\\partial L}{\\partial Y_{i,j}}\n",
    "\\cdot\n",
    "\\frac{\\partial Y_{i,j}}{\\partial W_{k,\\ell}}\n",
    "\\;=\\;\n",
    "\\sum_{i=1}^{N}\\sum_{j=1}^{m}\n",
    "\\frac{\\partial L}{\\partial Y_{i,j}}\n",
    "\\,\n",
    "X_{i,k}\n",
    "\\,\n",
    "\\delta_{\\ell,j}.\n",
    "$$\n",
    "\n",
    "$$\n",
    "=\\;\\sum_{i=1}^{N}\n",
    "\\Bigl(\n",
    "X_{i,k}\n",
    "\\sum_{j=1}^{m}\n",
    "\\delta_{\\ell,j}\\,\\frac{\\partial L}{\\partial Y_{i,j}}\n",
    "\\Bigr)\n",
    "\\;=\\;\n",
    "\\sum_{i=1}^{N}\n",
    "X_{i,k}\n",
    "\\,\n",
    "\\frac{\\partial L}{\\partial Y_{i,\\ell}}.\n",
    "$$\n",
    "\n",
    "把索引拼一拼，就能得到常见的结果（写成矩阵形式）：\n",
    "\n",
    "$$\n",
    "\\frac{\\partial L}{\\partial W}\n",
    "\\;=\\;\n",
    "X^\\top\n",
    "\\,\\frac{\\partial L}{\\partial Y}\n",
    "\\;\\;\\in \\mathbb{R}^{d\\times m}.\n",
    "$$\n",
    "\n",
    "其中\n",
    "$\\frac{\\partial L}{\\partial Y}$ 形状为 ($N\\times m$)。\n",
    "\n",
    "\n",
    "这正是我们在神经网络反向传播中常用的公式：若前向 $Y=XW$，反向就有\n",
    "\n",
    "$$\n",
    "\\tfrac{\\partial L}{\\partial W} = X^\\top\\,\\tfrac{\\partial L}{\\partial Y}\n",
    "$$\n",
    "\n",
    "注意这一步已经用到了$\\frac{\\partial Y}{\\partial W}$ 与 $\\frac{\\partial L}{\\partial Y}$ 的乘积”——也就是把四阶张量$\\frac{\\partial Y}{\\partial W}$与矩阵$\\frac{\\partial L}{\\partial Y}$挤压/合并成了最终一个与 W 形状相同的矩阵。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5326a42f-dbc9-4d99-a41a-a16db5efe91e",
   "metadata": {},
   "source": [
    "<span style=\"color: red;\">**小结**</span>\n",
    "\n",
    "1. 矩阵对矩阵求导，严格意义上会得到一个高阶张量，因为输出、权重都是多维对象。\n",
    "2. 在深度学习场景，我们几乎总是关注“标量损失 L 对参数 W 的梯度”，借助链式法则会把高阶张量与 $\\tfrac{\\partial L}{\\partial Y}$ 相乘，最终得到与 $W$ 同形状的 $\\tfrac{\\partial L}{\\partial W}$。这就是常用的\n",
    "\n",
    "$$\n",
    "\\tfrac{\\partial L}{\\partial W}\n",
    "\\;=\\;\n",
    "X^\\top\\,\n",
    "\\tfrac{\\partial L}{\\partial Y}.\n",
    "$$\n",
    "\n",
    "3. 如果硬要问$\\tfrac{\\partial Y}{\\partial W}$ 的形状和表达是什么”，答案是：形状 $N\\times m \\times d \\times m$的四阶张量，元素级计算可写为\n",
    "\n",
    "$$\n",
    "\\frac{\\partial Y_{i,j}}{\\partial W_{k,\\ell}}\n",
    "\\;=\\;\n",
    "X_{i,k}\\,\\delta_{j,\\ell}.\n",
    "$$\n",
    "\n",
    "或者使用向量化，将其变成 Kronecker 积 \\(\\bigl(I_m \\otimes X\\bigr)\\) 的一个 (Nm)\\times(dm) 矩阵。\n",
    "\n",
    "记住：之所以在大多数教材/推导里不常看到$\\frac{\\partial Y}{\\partial W} = \\dots$的完整四阶张量形式，是因为它极少直接使用；我们最终要的是$\\frac{\\partial L}{\\partial W}$，而那正好绕过了显式写出四阶张量。只要记得\n",
    "$$\n",
    "\\displaystyle\n",
    "\\frac{\\partial L}{\\partial W}=X^\\top\n",
    "\\bigl(\\frac{\\partial L}{\\partial Y}\\bigr)\n",
    "$$\n",
    "\n",
    "就够用了。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d4a39702-b113-4609-9da4-e16c4739ca64",
   "metadata": {},
   "source": [
    "## 线性映射的转置问题\n",
    "\n",
    "$$\n",
    "\\begin{aligned}\n",
    "\\frac{\\partial L}{\\partial \\boldsymbol{X}} &= \\frac{\\partial L}{\\partial \\boldsymbol{Y}} \\cdot \\boldsymbol{W}^{\\mathrm{T}}\n",
    "\\end{aligned}\n",
    "$$\n",
    "\n",
    "$$\n",
    "\\begin{aligned}\n",
    "\\frac{\\partial L}{\\partial \\boldsymbol{W}} &= \\boldsymbol{X}^{\\mathrm{T}} \\cdot \\frac{\\partial L}{\\partial \\boldsymbol{Y}}\n",
    "\\end{aligned}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e02cbdfc-d719-46a8-a8a4-f2acc87dcac5",
   "metadata": {},
   "source": [
    "### 前置背景\n",
    "* 前向计算（忽略偏置）\n",
    "\n",
    "$$\n",
    "\\boldsymbol{Y} \\;=\\; \\boldsymbol{X}\\,\\boldsymbol{W}.\n",
    "$$\n",
    "\n",
    "* $\\boldsymbol{X}$ 形状 ($N\\times d$)，\n",
    "* $\\boldsymbol{W}$ 形状 ($d\\times m$)，\n",
    "* $\\boldsymbol{Y}$ 形状 ($N\\times m$)。\n",
    "\n",
    "\n",
    "* 反向传播时，我们往往先得到\n",
    "\n",
    "$\\tfrac{\\partial L}{\\partial \\boldsymbol{Y}}$（形状 ($N\\times m$)），\n",
    "\n",
    "然后使用链式法则去求\n",
    "$$\n",
    "\\tfrac{\\partial L}{\\partial \\boldsymbol{X}} \n",
    "$$\n",
    "和\n",
    "$$\n",
    "\\tfrac{\\partial L}{\\partial \\boldsymbol{W}}\n",
    "$$\n",
    "\n",
    "最终结论是：\n",
    "$$\n",
    "\\frac{\\partial L}{\\partial \\boldsymbol{X}}\n",
    "\\;=\\;\n",
    "\\frac{\\partial L}{\\partial \\boldsymbol{Y}}\n",
    "\\;\\cdot\\;\n",
    "\\boldsymbol{W}^\\top\n",
    "\\quad(\\text{得到形状 }N\\times d),\n",
    "$$\n",
    "\n",
    "$$\n",
    "\\frac{\\partial L}{\\partial \\boldsymbol{W}}\n",
    "\\;=\\;\n",
    "\\boldsymbol{X}^\\top\n",
    "\\;\\cdot\\;\n",
    "\\frac{\\partial L}{\\partial \\boldsymbol{Y}}\n",
    "\\quad(\\text{得到形状 }d\\times m).\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "afa854b9-9cc6-492b-8f47-1a50316ae333",
   "metadata": {},
   "source": [
    "1. 从矩阵维度的角度看「转置」\n",
    "\n",
    "先只看维度匹配：\n",
    "1. $\\tfrac{\\partial L}{\\partial \\boldsymbol{X}}$ 应该和 $\\boldsymbol{X}$ 同形状 ($N\\times d$)。\n",
    "2. 在右手边，$\\tfrac{\\partial L}{\\partial \\boldsymbol{Y}}$ 形状 ($N\\times m$)。若要得到 ($N\\times d$)，必须右乘一个 ($m\\times d$) 的矩阵。\n",
    "\n",
    "* $\\boldsymbol{W}$ 本身是 ($d\\times m$)，它的转置 $\\boldsymbol{W}^\\top$ 才是 ($m\\times d$)。\n",
    "\n",
    "所以\n",
    "\n",
    "$$\n",
    "(N\\times m)\n",
    "\\;\\times\\;\n",
    "(m\\times d)\n",
    "\\;=\\;\n",
    "(N\\times d),\n",
    "$$\n",
    "\n",
    "正好匹配 $\\tfrac{\\partial L}{\\partial \\boldsymbol{X}}$ 的维度。\n",
    "\n",
    "3. $\\tfrac{\\partial L}{\\partial \\boldsymbol{W}}$ 应该和 $\\boldsymbol{W}$ 同形状 ($d\\times m$)。\n",
    "\n",
    "4. 在右手边，$\\boldsymbol{X}^\\top$ 形状 ($d\\times N$)，$\\tfrac{\\partial L}{\\partial \\boldsymbol{Y}}$ 形状 ($N\\times m$)，则\n",
    "\n",
    "$$\n",
    "(d\\times N)\n",
    "\\;\\times\\;\n",
    "(N\\times m)\n",
    "\\;=\\;\n",
    "(d\\times m),\n",
    "$$\n",
    "\n",
    "刚好得到所需形状。\n",
    "\n",
    "结论：从矩阵乘法的维度约束上，我们就能看出必须要出现转置，否则维度对不上。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2ac403c7-a207-434c-b4ab-74b831df6f0e",
   "metadata": {},
   "source": [
    "2. 从索引法/元素级的角度推导\n",
    "\n",
    "下面用更细致的元素索引来看看为什么一定会有转置。\n",
    "\n",
    "2.1 先回顾：$\\boldsymbol{Y} = \\boldsymbol{X} \\boldsymbol{W}$\n",
    "\n",
    "元素形式：\n",
    "\n",
    "$$\n",
    "Y_{i,j}\n",
    "\\;=\\;\n",
    "\\sum_{k=1}^{d}\n",
    "X_{i,k}\\,W_{k,j},\n",
    "$$\n",
    "\n",
    "其中 $i=1,\\dots,N$ 表示第 $i$ 个样本，$j=1,\\dots,m$ 表示第 $j$ 维输出特征。\n",
    "\n",
    "2.2 计算 $\\tfrac{\\partial L}{\\partial \\boldsymbol{X}}$\n",
    "\n",
    "用链式法则：\n",
    "\n",
    "$$\n",
    "\\frac{\\partial L}{\\partial X_{p,q}}\n",
    "\\;=\\;\n",
    "\\sum_{i=1}^N \\sum_{j=1}^m\n",
    "\\frac{\\partial L}{\\partial Y_{i,j}}\n",
    "\\;\\cdot\\;\n",
    "\\frac{\\partial Y_{i,j}}{\\partial X_{p,q}}.\n",
    "$$\n",
    "\n",
    "但只有在 $i=p$ 时才有非零，因为 $Y_{i,j}$ 仅依赖于同一行 $X_{i,:}$。于是\n",
    "\n",
    "$$\n",
    "\\frac{\\partial L}{\\partial X_{p,q}}\n",
    "\\;=\\;\n",
    "\\sum_{j=1}^m\n",
    "\\frac{\\partial L}{\\partial Y_{p,j}}\n",
    "\\;\\cdot\\;\n",
    "\\frac{\\partial}{\\partial X_{p,q}}\n",
    "\\Bigl(\\sum_{k=1}^{d} X_{p,k} \\, W_{k,j}\\Bigr).\n",
    "$$\n",
    "\n",
    "这里对 $X_{p,q}$ 有非零贡献的只有那一项 $k=q$。故\n",
    "\n",
    "$$\n",
    "\\frac{\\partial L}{\\partial X_{p,q}}\n",
    "\\;=\\;\n",
    "\\sum_{j=1}^m\n",
    "\\frac{\\partial L}{\\partial Y_{p,j}}\n",
    "\\;\\cdot\\;\n",
    "W_{q,j}.\n",
    "$$\n",
    "\n",
    "把索引按矩阵乘法的方式来“收集”，就得到：\n",
    "* 对固定的行 $p$，$\\tfrac{\\partial L}{\\partial X_{p,:}}$ 作为一个行向量（大小 $1\\times d$），等于\n",
    "$$\n",
    "\\bigl[\\tfrac{\\partial L}{\\partial Y_{p,:}}\\bigr]{(1\\times m)}\n",
    "\\;\\times\\;\n",
    "[W]^\\top{(m\\times d)}.\n",
    "$$\n",
    "\n",
    "* 把所有行 p 组合起来，就得到：\n",
    "\n",
    "$$\n",
    "\\frac{\\partial L}{\\partial \\boldsymbol{X}}=\n",
    "\\frac{\\partial L}{\\partial \\boldsymbol{Y}}\n",
    "\\;\\cdot\\;\n",
    "\\boldsymbol{W}^\\top.\n",
    "$$\n",
    "\n",
    "这就是为什么出现 $\\boldsymbol{W}^\\top$：\n",
    "在 $\\tfrac{\\partial Y_{i,j}}{\\partial X_{i,k}}$ 时，“列”索引 $k$ 跟 $\\boldsymbol{W}$ 的“行”索引 $k$ 对应，而 $\\boldsymbol{Y}$ 的列索引 $j$ 与 $\\boldsymbol{W}$ 的列索引 $j$ 一起被相乘累加；这在矩阵乘法汇总时会体现在「$\\boldsymbol{W}$ 的行列要翻转」的效果上。\n",
    "\n",
    "2.3 计算 $\\tfrac{\\partial L}{\\partial \\boldsymbol{W}}$\n",
    "\n",
    "类似地：\n",
    "\n",
    "$$\n",
    "\\frac{\\partial L}{\\partial W_{p,q}}\n",
    "\\;=\\;\n",
    "\\sum_{i=1}^N\\sum_{j=1}^m\n",
    "\\frac{\\partial L}{\\partial Y_{i,j}}\n",
    "\\cdot\n",
    "\\frac{\\partial Y_{i,j}}{\\partial W_{p,q}}.\n",
    "$$\n",
    "\n",
    "$$\n",
    "Y_{i,j}=\\sum_{k=1}^d X_{i,k}\\,W_{k,j},\n",
    "$$\n",
    "\n",
    "只有当 $k=p$ 且 $j=q$ 时该项与 $W_{p,q}$ 相关。所以\n",
    "\n",
    "$$\n",
    "\\frac{\\partial Y_{i,j}}{\\partial W_{p,q}}\n",
    "\\;=\\;\n",
    "X_{i,p} \\;\\;\\delta_{j,q}\n",
    "\\quad(\\delta_{j,q}=1\\text{ 当 }j=q,\\text{否则}0).\n",
    "$$\n",
    "\n",
    "则\n",
    "$$\n",
    "\\frac{\\partial L}{\\partial W_{p,q}}\n",
    "\\;=\\;\n",
    "\\sum_{i=1}^N\n",
    "\\sum_{j=1}^m\n",
    "\\frac{\\partial L}{\\partial Y_{i,j}}\n",
    "\\,X_{i,p}\\,\\delta_{j,q}\n",
    "\\;=\\;\n",
    "\\sum_{i=1}^N\n",
    "\\Bigl(\n",
    "X_{i,p}\n",
    "\\,\\frac{\\partial L}{\\partial Y_{i,q}}\n",
    "\\Bigr)\n",
    "$$\n",
    "\n",
    "对固定 $(p,q)$，$\\frac{\\partial L}{\\partial Y_{i,q}}$ 按 $i$ 累加； 同时 $X_{i,p}$ 也按 i 累加。用矩阵乘法收集可写成\n",
    "\n",
    "$$\n",
    "\\frac{\\partial L}{\\partial W}\n",
    "\\;=\\;\n",
    "X^\\top\n",
    "\\,\\frac{\\partial L}{\\partial Y}.\n",
    "$$\n",
    "\n",
    "所以得到 $\\boldsymbol{X}^\\top$（而不是 $\\boldsymbol{X}$）——同理，是因为在元素级别上，$X_{i,p}$ 的索引 $i$ 变成了“列-行”翻转的角色。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e7bb92c3-515d-4128-8cf9-61f2cdf51e70",
   "metadata": {},
   "source": [
    "<span style=\"color: red;\">**小结**</span>\n",
    "* $\\tfrac{\\partial L}{\\partial \\boldsymbol{X}} = \\tfrac{\\partial L}{\\partial \\boldsymbol{Y}} \\cdot \\boldsymbol{W}^{\\mathrm{T}}$\n",
    "转置的原因：$\\boldsymbol{Y} = \\boldsymbol{X}\\,\\boldsymbol{W}$ 时，\n",
    "$\\tfrac{\\partial Y_{i,j}}{\\partial X_{i,k}} = W_{k,j}$。在进行矩阵化合并时，$\\boldsymbol{W}$ 的行、列索引要互换才与$\\tfrac{\\partial L}{\\partial Y}$正确乘法匹配。\n",
    "* $\\tfrac{\\partial L}{\\partial \\boldsymbol{W}} = \\boldsymbol{X}^{\\mathrm{T}} \\cdot \\tfrac{\\partial L}{\\partial \\boldsymbol{Y}}$\n",
    "同理，因为 $\\tfrac{\\partial Y_{i,j}}{\\partial W_{p,q}} = X_{i,p}\\,\\delta_{j,q}$。索引拼起来后，对应矩阵乘法要用 $\\boldsymbol{X}^\\top$。\n",
    "* 从维度观点看：\n",
    "    1. $\\boldsymbol{X}$ 是 ($N\\times d$)，要得到同维度的 $\\tfrac{\\partial L}{\\partial \\boldsymbol{X}}$，只好让 ($N\\times m$) 的 $\\tfrac{\\partial L}{\\partial \\boldsymbol{Y}}$ 去乘一个 ($m\\times d$) 矩阵，即 $\\boldsymbol{W}^\\top$。\n",
    "    2. $\\boldsymbol{W}$ 是 ($d\\times m$)，要得到同维度的 $\\tfrac{\\partial L}{\\partial \\boldsymbol{W}}$，只好让 ($d\\times N$) 的 $\\boldsymbol{X}^\\top$ 和 ($N\\times m$) 的 $\\tfrac{\\partial L}{\\partial \\boldsymbol{Y}}$ 相乘。\n",
    "\n",
    "这就是这两条公式及其中转置出现的根本原因。换言之，前向用 $\\boldsymbol{X}\\cdot\\boldsymbol{W}$时，反向对输入或权重的梯度就会出现 $\\boldsymbol{W}^\\top$ 或 $\\boldsymbol{X}^\\top$。这是最经典也最重要的「线性层反向传播」公式。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a16568f2-656d-4bea-b14c-822a4b91308b",
   "metadata": {},
   "source": [
    "## 对B求导\n",
    "\n",
    "在仿射变换\n",
    "\n",
    "$$\n",
    "\\mathbf{Y} \\;=\\; \\mathbf{X}\\,\\mathbf{W} \\;+\\; \\mathbf{b}\n",
    "$$\n",
    "\n",
    "中，$\\mathbf{b}$ 形状为 $1 \\times m$，即它是一个行向量（可视作“偏置”）。当我们要计算损失 $L$（标量）对 $\\mathbf{b}$ 的梯度\n",
    "$$\n",
    "\\tfrac{\\partial L}{\\partial \\mathbf{b}}\n",
    "$$\n",
    "其结果应与 $\\mathbf{b}$ 形状相同，即 $1 \\times m$。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "baa5de7c-383b-4121-a687-4d7b624a3d70",
   "metadata": {},
   "source": [
    "1. 元素级推导\n",
    "\n",
    "令\n",
    "* $\\mathbf{Y}$ 形状 ($N\\times m$)，\n",
    "* $\\mathbf{b}$ 形状 ($1\\times m$)。\n",
    "\n",
    "则对任意 $i\\in \\{1,\\dots,N\\}$ 和 $j\\in \\{1,\\dots,m\\}$，有\n",
    "\n",
    "$$\n",
    "Y_{i,j} \\;=\\; \\sum_{k=1}^{d} X_{i,k}\\,W_{k,j} \\;+\\; b_{1,j}.\n",
    "$$\n",
    "\n",
    "由于 $\\mathbf{b}$ 只影响同列的输出 $Y_{i,j}$（并且对每个样本行 $i$ 都一样），故\n",
    "\n",
    "$$\n",
    "\\frac{\\partial Y_{i,j}}{\\partial b_{1,\\ell}}\n",
    "$$\n",
    "\n",
    "$$\n",
    "\\begin{cases}\n",
    "1,& \\ell = j,\\\\[5pt]\n",
    "0,& \\ell \\neq j.\n",
    "\\end{cases}\n",
    "$$\n",
    "\n",
    "接下来，用链式法则对损失 L（标量）求偏导：\n",
    "\n",
    "$$\n",
    "\\frac{\\partial L}{\\partial b_{1,\\ell}}\n",
    "\\;=\\;\n",
    "\\sum_{i=1}^N \\sum_{j=1}^m\n",
    "\\frac{\\partial L}{\\partial Y_{i,j}}\n",
    "\\;\\cdot\\;\n",
    "\\frac{\\partial Y_{i,j}}{\\partial b_{1,\\ell}}\n",
    "\\;=\\;\n",
    "\\sum_{i=1}^N \\sum_{j=1}^m\n",
    "\\frac{\\partial L}{\\partial Y_{i,j}}\n",
    "\\;\\cdot\\;\n",
    "\\delta_{\\ell,j}\n",
    "\\quad\n",
    "\\bigl(\\delta_{\\ell,j}=1 \\text{ 当 } \\ell=j\\bigr).\n",
    "$$\n",
    "只有当 $j = \\ell$ 时这项才非 $0$，于是\n",
    "\n",
    "$$\n",
    "\\frac{\\partial L}{\\partial b_{1,\\ell}}\n",
    "\\;=\\;\n",
    "\\sum_{i=1}^N\n",
    "\\frac{\\partial L}{\\partial Y_{i,\\ell}}.\n",
    "$$\n",
    "\n",
    "这就是对 $b_{1,\\ell}$ 的偏导。把 $\\ell$ 从 $1$ 到 $m$ 整理放成一个行向量，就得到\n",
    "\n",
    "$$\n",
    "\\frac{\\partial L}{\\partial \\mathbf{b}}\n",
    "\\;=\\;\n",
    "\\biggl(\n",
    "\\sum_{i=1}^N \\frac{\\partial L}{\\partial Y_{i,1}},\n",
    "\\;\\sum_{i=1}^N \\frac{\\partial L}{\\partial Y_{i,2}},\n",
    "\\;\\dots,\n",
    "\\;\\sum_{i=1}^N \\frac{\\partial L}{\\partial Y_{i,m}}\n",
    "\\biggr)\n",
    "\\;\\;\\in \\mathbb{R}^{1\\times m}.\n",
    "$$\n",
    "\n",
    "也可以写成更紧凑的「按列求和」形式：\n",
    "\n",
    "\n",
    "$$\n",
    "\\frac{\\partial L}{\\partial \\mathbf{b}}=\n",
    "\\sum_{i=1}^N\n",
    "\\left(\n",
    "\\frac{\\partial L}{\\partial \\mathbf{Y}}\n",
    "\\right)_{i,:}=\n",
    "\\mathbf{1}^\\top\n",
    "\\;\\frac{\\partial L}{\\partial \\mathbf{Y}}\n",
    "\\quad\n",
    "(\\text{其中 }\\mathbf{1}\\in \\mathbb{R}^{N\\times 1}\\text{，元素全为 }1)\n",
    "$$\n",
    "}\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5b47effa-ef5b-4a2f-88de-545a01a211e5",
   "metadata": {},
   "source": [
    "2. 向量 / 矩阵形式的直观理解\n",
    "\n",
    "从深度学习常见的批量处理(batch)视角看：\n",
    "* $\\mathbf{X}$ 是 $N$ 个样本拼起来的矩阵 ($N \\times d$)；\n",
    "* 每个样本都有一个输出行向量 $\\mathbf{Y}[i, :]\\in \\mathbb{R}^m$；\n",
    "* $\\mathbf{b}$ 是偏置，广播到每个样本的输出上。\n",
    "\n",
    "因此，对于同一个输出特征列 $j$，$\\mathbf{b}[1, j]$ 都会加到 $\\mathbf{Y}[i, j]$（$i$ 从 $1$ 到 $N$）上，产生同样的加法影响；所以损失对偏置第 $j$ 个分量的梯度会是该列在所有样本上的梯度之和。写成矩阵形式，就是把$\\tfrac{\\partial L}{\\partial \\mathbf{Y}}$ (形状 $N\\times m$)\n",
    "在行维度上进行“列求和”，最后得到 $1\\times m$。\n",
    "\n",
    "这正好对应于公式\n",
    "\n",
    "$$\n",
    "\\frac{\\partial L}{\\partial \\mathbf{b}}=\\mathbf{1}^\\top\n",
    "\\,\\frac{\\partial L}{\\partial \\mathbf{Y}}\n",
    "$$\n",
    "\n",
    "的含义：对每列特征求和（“累加所有样本”）。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d62ef742-e126-4b25-a10b-5890f8eea8f2",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}