1-Basics.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### <div>\n",
    "<img src=\"https://discuss.pytorch.org/uploads/default/original/2X/3/35226d9fbc661ced1c5d17e374638389178c3176.png\" width=\"400\" style=\"margin: 50px auto; display: block; position: relative; left: -30px;\" />\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!--NAVIGATION-->\n",
    "# | Basics | [Data >](2-Data.ipynb)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Basics\n",
    "\n",
    "In this first notebook, we will introduce Tensors, which are the base element in PyTorch.  \n",
    "We will see different ways to create Tensors and check their properties. Then, we will briefly go through all the different kind of operations they support, such as mathematical operations, indexing, reshaping, expansion, masking, type conversion, etc.\n",
    "\n",
    "### Table of Contents\n",
    "\n",
    "#### 1. [Introduction](#Introduction)  \n",
    "#### 2. [Tensor Creation](#Tensor-Creation)\n",
    "#### 3. [Tensor Properties](#Tensor-Properties)  \n",
    "#### 4. [Tensor Operations](#Tensor-Operations)  \n",
    "#### 5. [Tensor Conversions](#Tensor-Conversions)  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Introduction"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### What is PyTorch ?\n",
    "Python-based scientific computing library, similar to NumPy.  \n",
    "Differentiates from NumPy in 3 main aspects:\n",
    "- It allows to use the power of **GPU** computing\n",
    "- It comes with an **automatic differentiation** module\n",
    "- It is a fully-fledged **deep learning research platform**\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"PyTorch Version:\", torch.__version__)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "____"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### What is a tensor?\n",
    "\n",
    "A **matrix** is a grid of numbers, let's say (3x5).  \n",
    "In simple terms, a **tensor** can be seen as a generalization of a matrix to higher dimension.  \n",
    "It can be of arbitrary shape, e.g. (3 x 6 x 2 x 10). \n",
    "\n",
    "You can think of tensors as multidimensional arrays."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X = torch.tensor([1, 2, 3, 4, 5])\n",
    "X"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X = torch.tensor([[1, 2, 3], [4, 5, 6]])\n",
    "X"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### PyTorch vs NumPy "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`torch.tensor` behaves like `numpy.array` under mathematical operations.  \n",
    "The syntax is very similar between the two libraries.  \n",
    "If you are familiar with NumPy, you can [browse here](https://github.com/wkentaro/pytorch-for-numpy-users#types) to check what are NumPy functions equivalent in PyTorch.\n",
    "\n",
    "For example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.eye(2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "torch.eye(2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.arange(1,5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "torch.arange(1,5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we said, `torch.tensor` additionally keeps track of the computation graphs (see next notebook) and provides GPU support."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "# Tensor Creation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "torch.zeros(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "torch.ones(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "torch.eye(3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "torch.empty((3, 5))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    " torch.rand((5, 3))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "torch.arange(3, 9, 2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "torch.linspace(0, 1, 11)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Most of these creation operations have a `_like` counterpart that creates a tensor of the same size, dtype and device as the given tensor."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "A = torch.ones((2,3))    # A is a float tensor of size 2x3 located on cpu\n",
    "B = torch.zeros_like(A)  # So B is a float tensor of size 2x3 located on cpu\n",
    "B"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This list is not exhaustive but gives you an idea of the diversity of way to create a Tensor."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div style=\"background-color:lightblue;padding:1rem;border-radius: 0.015rem 0.015rem 0.03rem 0.03rem;\">\n",
    "<h3 style=\"display: inline; font-weight:bold\">Your turn!</h3>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**_Create the tensor:_**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\n",
    "\\begin{bmatrix}\n",
    "5 & 7 & 9 & 11 & 13 & 15 & 17 & 19\n",
    "\\end{bmatrix}\n",
    "$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div style=\"background-color:lightblue;padding:1rem;border-radius: 0.03rem 0.03rem 0.015rem 0.015rem;\">\n",
    "<h3 style=\"display: inline\"></h3>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "# Tensor Properties"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x = torch.Tensor([[0,1,2], [3,4,5]])\n",
    "\n",
    "print(\"x.shape: \\n%s\\n\" % (x.shape,))\n",
    "print(\"x.size(): \\n%s\\n\" % (x.size(),))\n",
    "print(\"x.size(1): \\n%s\\n\" % x.size(1))\n",
    "print(\"x.dim(): \\n%s\\n\" % x.dim())\n",
    "print(\"x.numel(): \\n%s\\n\" % x.numel())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"x.dtype: \\n%s\\n\" % x.dtype)\n",
    "print(\"x.device: \\n%s\\n\" % x.device)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `nonzero` function returns indices of the non zero elements."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x = torch.Tensor([[0,1,2], [3,4,5]])\n",
    "\n",
    "print(\"x.nonzero(): \\n%s\\n\" % x.nonzero())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "# Tensor Operations"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Unlike in NumPy, there are two ways to performs most operations in PyTorch:\n",
    " - using **`torch.op(tensor)`**\n",
    " - using **`tensor.op()`**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X = torch.rand(3, 2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "torch.exp(X)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X.exp()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can easily chain operators :"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X.sqrt().std()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "(X.exp() + 2).sqrt() - 2 * X.log().sigmoid()  # be creative :-)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Many more functions are available: sin, cos, tanh, bmm, cumsum, dot, etc."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div style=\"background-color:lightblue;padding:1rem;border-radius: 0.015rem 0.015rem 0.03rem 0.03rem;\">\n",
    "<h3 style=\"display: inline; font-weight:bold\">Your turn!</h3>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Compute the norms of the row-vectors in matrix **X** without using `torch.norm()`.\n",
    "\n",
    "Remember: $$||\\vec{v}||_2 = \\sqrt{x_1^2 + x_2^2 + \\dots + x_n^2}$$\n",
    "\n",
    "Hint: `X**2` computes the element-wise square."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X = torch.eye(4) + torch.arange(4).repeat(4, 1).float()\n",
    "\n",
    "# YOUR TURN\n",
    "\n",
    "# SOLUTION: tensor([3.8730, 4.1231, 4.3589, 4.5826])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div style=\"background-color:lightblue;padding:1rem;border-radius: 0.03rem 0.03rem 0.015rem 0.015rem;\">\n",
    "<h3 style=\"display: inline\"></h3>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Reductions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X = torch.rand(3, 2)\n",
    "X"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X.sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X.max()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X.mean(dim=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X.norm(p=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div style=\"background-color:lightblue;padding:1rem;border-radius: 0.015rem 0.015rem 0.03rem 0.03rem;\">\n",
    "<h3 style=\"display: inline; font-weight:bold\">Your turn!</h3>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For $X_{i,j}$ , compute $Y$ such that $Y_j = \\log \\big[ \\sum_j \\exp (X_{i,j}) \\big]$."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X = torch.eye(4) + torch.arange(4).repeat(4, 1).float()\n",
    "\n",
    "# YOUR TURN\n",
    "\n",
    "# SOLUTION: tensor([3.4938, 3.5797, 3.7817, 4.1852])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div style=\"background-color:lightblue;padding:1rem;border-radius: 0.03rem 0.03rem 0.015rem 0.015rem;\">\n",
    "<h3 style=\"display: inline\"></h3>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Linear Algebra"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "Y = torch.rand(2, 3)\n",
    "Y"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Matrix multiplication\n",
    "Y.t() @ Y"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "Y.t().matmul(Y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# CAUTION: Operator '*' does element-wise multiplication, just like in numpy!\n",
    "# Y.t() * Y  # error, dimensions do not match for element-wise multiplication"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "torch.inverse(Y.t() @ Y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "Y = torch.rand(3, 3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "Y.det()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "Y.eig()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## In-place operators (mutations)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Functions that mutate the object end with an underscore, e.g. *add_*, *div_*, etc."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "A = torch.eye(3)\n",
    "A"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "A.add(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "A"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "A.add_(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "A"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "A.uniform_()  # fills the tensor with random uniform numbers in [0, 1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "A"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Also note the difference:\n",
    "```python\n",
    "A = A + 1  # After this operation, A is a new variable and memory has been copied\n",
    "A += 1     # After this operation, A stayed the same variable and memory has been changed in place\n",
    "```\n",
    "\n",
    "Compare the outputs:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "A = torch.ones(1)\n",
    "\n",
    "A_before = A\n",
    "A = A + 1\n",
    "\n",
    "print(A, A_before)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "A = torch.ones(1)\n",
    "\n",
    "A_before = A\n",
    "A += 1\n",
    "\n",
    "print(A, A_before)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Indexing & Masking"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Again, it works just like in NumPy."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Indexing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "A = torch.randint(100, (3, 3))\n",
    "A"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "A[0,2]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "A[:, 1:2], A[:, 1:2].shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "_Note: You can use `...` to mark any number of dimension_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Masking"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X = torch.randint(100, (5, 3))\n",
    "X"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mask = (X > 25) & (X < 75)\n",
    "mask"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X[mask]  # returns all elements matching the criteria in a 1D-tensor"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X[mask] = 0  # You can assign new values only to indices matching the condition:\n",
    "X"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div style=\"background-color:lightblue;padding:1rem;border-radius: 0.015rem 0.015rem 0.03rem 0.03rem;\">\n",
    "<h3 style=\"display: inline; font-weight:bold\">Your turn!</h3>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X = torch.arange(40).view(5,8)\n",
    "X"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**_Extract this vector from the tensor X:_**\n",
    "\n",
    "$ \\begin{bmatrix}\n",
    "17 & 19 & 21 & 23 \\\\\n",
    "\\end{bmatrix}  $\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# YOUR TURN"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X = torch.tensor([[1., 0., 2.], [4., 6., 0.]])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Get the number of non-zeros in X**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# YOUR TURN"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Compute the sum of all entries in X that are larger than the mean of all values in X**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# YOUR TURN"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div style=\"background-color:lightblue;padding:1rem;border-radius: 0.015rem 0.015rem 0.03rem 0.03rem;\">\n",
    "<h3 style=\"display: inline; font-weight:bold\"></h3>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Reshaping & Expanding"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**View**  \n",
    "**`view`** is the equivalent of `reshape` in NumPy, but `view` does not allocate new memory: the output tensor shares the same data!\n",
    "\n",
    "The number of arguments of `view` will be the number of dimensions of the output tensor."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X = torch.tensor([1, 2, 3, 4, 5, 6])\n",
    "X"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "Y = X.view(2, 3)  # view tensor X on 2 dimensions, with a size 2 on dimension 1 and a size 3 on dimension 2\n",
    "Y"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "Y = X.view(2, -1)  ## -1 tells PyTorch to infer the number of elements along that dimension\n",
    "Y, Y.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Expand**  \n",
    "**`expand`** creates a new view of the tensor with dimensions of size 1 expanded to a larger size. This does not allocate new memory either !"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "Y = torch.ones(5)\n",
    "Y"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "Y.expand(5, 5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "_Note: There also exists `reshape` and `repeat` functions in PyTorch. They work similarly to `view` and `expand` but **do** copy memory_."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Squeeze  and  Unsqueeze**  \n",
    "**`squeeze`** removes all dimensions of size 1, while **`unsqueeze`** adds a new dimension at a given position"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X = torch.eye(4)\n",
    "Y = X[:1]\n",
    "Y, Y.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "Y = Y.squeeze()  # removes all dimensions of size '1'\n",
    "Y, Y.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "Y = Y.unsqueeze(1)  # add a new dimension in position 1\n",
    "Y, Y.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Flatten**  \n",
    "We use **`flatten`** to reshape the input into a lower-dimensional tensor."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X = torch.rand(3,2,4)\n",
    "print(X.shape)\n",
    "X"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "Y = X.flatten()\n",
    "Y.shape, Y"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "Y = X.flatten(start_dim=1)\n",
    "Y.shape, Y"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div style=\"background-color:lightblue;padding:1rem;border-radius: 0.015rem 0.015rem 0.03rem 0.03rem;\">\n",
    "<h3 style=\"display: inline; font-weight:bold\">Your turn!</h3>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**_Create the tensor:_**\n",
    "\n",
    "$ \\begin{bmatrix}\n",
    "7 & 5 & 5 & 5 & 5 \\\\\n",
    "5 & 7 & 5 & 5 & 5 \\\\\n",
    "5 & 5 & 7 & 5 & 5 \\\\\n",
    "5 & 5 & 5 & 7 & 5 \\\\\n",
    "5 & 5 & 5 & 5 & 7 \n",
    "\\end{bmatrix}  $\n",
    "\n",
    "Hint: You can use matrix sum and scalar multiplication"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# YOUR TURN"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**_Create the tensor:_**\n",
    "\n",
    "$ \\begin{bmatrix}\n",
    "2 & 2 & 2 & 2 & 2 \\\\\n",
    "4 & 4 & 4 & 4 & 4 \\\\\n",
    "6 & 6 & 6 & 6 & 6 \\\\\n",
    "8 & 8 & 8 & 8 & 8\n",
    "\\end{bmatrix}  $"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# YOUR TURN"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div style=\"background-color:lightblue;padding:1rem;border-radius: 0.015rem 0.015rem 0.03rem 0.03rem;\">\n",
    "<h3 style=\"display: inline; font-weight:bold\"></h3>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## And many more ..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# press tab to autocomplete\n",
    "x."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "# Tensor Conversions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Type conversions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Using `.to()`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To cast a Tensor to a different type, we can use the `Tensor.to(type)` function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "Y = 4 * torch.rand((2,4))\n",
    "Y"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "Y.dtype"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "Y.to(torch.float16)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "Y.to(torch.int64)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "Y.to(torch.bool)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note the automatic type promotion :"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "torch.LongTensor([1, 2]) + torch.FloatTensor([1.1, 2.2])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Using dedicated function per type"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can use `.bool()`, `.short()`, `.int()`, `.long()`, `.float()`, `.double()` to convert the tensor to the required type."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "Y = 4 * torch.rand((2,4))\n",
    "Y"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "Y.int()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "Y.float()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Converting between PyTorch and NumPy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X = np.random.random((5,3))\n",
    "X"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# numpy ---> torch\n",
    "Y = torch.from_numpy(X)  # Y is actually a DoubleTensor (i.e. 64-bit representation)\n",
    "Y"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "Y = torch.rand((2,4))\n",
    "Y"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# torch ---> numpy\n",
    "X = Y.numpy()\n",
    "X"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Converting between GPU and CPU"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, you may want to check: \n",
    " - that cuda can actually be used : `torch.cuda.is_available()`\n",
    " - how many gpus are available : `torch.cuda.device_count()`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "torch.cuda.is_available()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "torch.cuda.device_count()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x = torch.Tensor([[1,2,3], [4,5,6]])\n",
    "print(x)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### `torch.device`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The best way to easily move a tensor from a device to another is again by using the **`Tensor.to(...)`** function.  \n",
    "You need to pass as argument a **`torch.device`** object.\n",
    "\n",
    "A **`torch.device`** is an object representing the device on which a torch.tensor is or will be allocated.\n",
    "\n",
    "_Note : If you don't have Cuda on the machine, the following examples won't work_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cpu = torch.device('cpu')\n",
    "cuda_0 = torch.device('cuda:0')\n",
    "\n",
    "x = x.to(cpu)\n",
    "print(x.device)\n",
    "x = x.to(cuda_0)\n",
    "print(x.device)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It is flexible since you can check if cuda exists only once in your code"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Your Turn**  \n",
    "Define the device to be a gpu if available or to fallback on cpu if not"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "device = None # YOUR TURN\n",
    "\n",
    "x = x.to(device)  # We don't need to worry anymore about whether cuda is available or not in the rest of the code\n",
    "print(x.device)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### `Tensor.cuda` and `Tensor.cpu`\n",
    "\n",
    "You may also see the use of the `Tensor.cuda()` and `Tensor.cpu()` functions.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x = torch.Tensor([[1,2,3], [4,5,6]])\n",
    "print(x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x.cuda()\n",
    "print(x.device) # Note x is still on cpu\n",
    "\n",
    "x = x.cuda()\n",
    "print(x.device)\n",
    "\n",
    "x = x.cuda(1) # This will fail if you have only one gpu\n",
    "print(x.device)\n",
    "\n",
    "x = x.cpu()\n",
    "print(x.device)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x = torch.Tensor([[1,2,3], [4,5,6]])\n",
    "\n",
    "# This will generate an error since you cannot do operation on tensor that are not on the same device\n",
    "x + x.cuda()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**In general, the more flexible `Tensor.to(...)` function should be used preferably.**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "___"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!--NAVIGATION-->\n",
    "# | Basics | [Data >](2-Data.ipynb)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}