GenePattern Notebook Tutorial.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# GenePattern Notebook Tutorial\n",
    "\n",
    "The GenePattern Notebook Environment provides a variety of features for both basic and advanced users. This tutorial will familiarize you with some of its most important features. \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-info\">\n",
    "<b>All instructions for you to follow will appear in a blue panel like this one.</b>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## GenePattern Notebook Introduction Video\n",
    "\n",
    "Below is a brief video introduction to the GenePattern Notebook Environment. This video introduces many of the basic concepts and features provided by the tool. If you would perfer a more \"hands on\" introduction, scroll down and follow the subsequent interactive tutorial."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-info\"><strong><p>To view the video, click the Play button in the middle of the video cell.</p><p>If the video is not visible, highlight the cell below and press the Run Cell ( <i class=\"fa-step-forward fa\"></i> ) button in the toolbar to see the video.</strong></p></div>\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"854\"\n",
       "            height=\"480\"\n",
       "            src=\"https://www.youtube.com/embed/8npzyGLpUHU\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.IFrame at 0x7f6f6dd57358>"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from IPython.display import IFrame\n",
    "\n",
    "IFrame('https://www.youtube.com/embed/8npzyGLpUHU', width=\"854\", height=\"480\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Basic Features\n",
    "\n",
    "These are the most commonly used features in the GenePattern Notebook environment. They also form the building blocks for most advanced use cases."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Notebook Cells\n",
    "\n",
    "All notebooks consist of some number of cells. These cells may interleave text, images, tables, code or interactive widgets. Try clicking different sections of this notebook and will you notice the different cells as they are selected."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Inserting cells\n",
    "\n",
    "New cells can be inserted by clicking the Insert Cell button ( <i class=\"fa-plus fa\"></i> ) on the toolbar or by using the Insert menu at the top of the screen."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-info\">\n",
    "<b>1. Select this cell by clicking to the left of the text.<br>\n",
    "2. Click the Insert Cell button ( <i class=\"fa-plus fa\"></i> ) to add a new cell below this one.</b>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Executing cells\n",
    "\n",
    "Cells can be executed by clicking the Run Cell button ( <i class=\"fa-step-forward fa\"></i> ) on the toolbar or using the menu at the top of the screen to select *Cell > Run Cells*. Depending on the type of cell, when a cell executes it will run any code contained in the cell or render any HTML/markdown as formatted text."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-info\">\n",
    "<b>\n",
    "1. Select this cell by clicking to the left of the text.<br>\n",
    "2. Create a new cell below this one and type or paste <code> 3 + 7 </code> into the input box. <br>\n",
    "3. Click the Run Cell button or type Shift+Enter to execute the cell.</b>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Changing cell type\n",
    "\n",
    "Every cell has a type. Cell types include code cells, markdown cells and GenePattern cells. Code cells contain code that can be executed. Markdown cells contain either markdown or HTML that is rendered when the cell is executed. GenePattern cells contain interactive widgets that allow you to access GenePattern's many analyses.\n",
    "\n",
    "To change cell type, select a cell and then use the dropdown menu on the toolbar above. Alternatively, you can use the menu at the top of the page to select *Cell > Cell Type*."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### GenePattern Cells\n",
    "\n",
    "The GenePattern Notebook environment provides a number of graphical widgets that make performing analyses easy, even for non-programming users. These widgets take the form of GenePattern Cells that allow a user to prepare analyses, launch jobs and visualize results.\n",
    "\n",
    "To insert a GenePattern Cell, insert a new cell, then select the cell and then change the cell type to GenePattern either by using the Cell > Cell Type > GenePattern menu or by going to the dropdown menu in the notebook toolbar and selecting GenePattern from the list of options."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-info\">\n",
    "<b>1. Insert a new cell.<br/>\n",
    "2. Change the cell type to GenePattern. <br/>\n",
    "3. You should now have a new cell that looks exactly like the one shown below.</b>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<b>Below is an example GenePattern authentication cell. The cell you have just created above should look identical.</b>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "genepattern": {
     "name": "Login",
     "server": "https://cloud.genepattern.org/gp",
     "type": "auth"
    }
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "4b6ba1684ab949619402a8f313f212b5",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "GPAuthWidget()"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Requires GenePattern Notebook: pip install genepattern-notebook\n",
    "import gp\n",
    "import genepattern\n",
    "\n",
    "# Username and password removed for security reasons.\n",
    "genepattern.display(genepattern.session.register(\"https://cloud.genepattern.org/gp\", \"\", \"\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Authentication Cells\n",
    "\n",
    "The first GenePattern cell that you have encountered is an Authentication Cell. This cell allows a user to sign into a GenePattern server. Doing this allows GenePattern to keep a user's results private, and to remember a user's settings.\n",
    "\n",
    "Authentication cells look like a login form with the additional option of selecting which GenePattern server to sign into. If the user has already authenticated, such as when usng the GenePattern Notebook Repository, the user will instead be prompted to either sign in as the current user or to cancel and sign in as a different user"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-info\">\n",
    "<b>1. Sign into the authentication cell above.<br/>\n",
    "</b>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Analysis Cells\n",
    "\n",
    "After a user has signed in using an Authentication Cell, the next task is usually to insert an Analysis Cell to perform an analysis. To do this click the <em><i class=\"fa-th fa\"></i> Tools</em> button in the notebook toolbar. This wll open a dialog box with a list of available GenePattern analyses. Search or browse through the list and then click one of the options to insert an Analysis Cell of that type.\n",
    "\n",
    "Every Analysis Cell has a number of parameters, which can be used to upload data and to select other options for the analysis. Once you fill in these parameters, click Run to submit them as a job on the GenePattern server.\n",
    "\n",
    "Once the Run button has been clicked, all selected files will upload and then a Job Cell will be inserted below to indicate the status of the job in GenePattern’s queue (see Job Cells below)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<b>Below is an example GenePattern analysis cell for a module that does some basic formatting. While this analysis isn't particually biologically interesting, it nevertheless serves as a concise example of analysis cell interface.</b>\n",
    "\n",
    "<b>You are are instead seeing an error message stating \"You must be authenticated...,\" you likely are not yet logged into the GenePattern server. You will first need to scroll up and sign into the GenePattern authentication cell before you can view the available analyses.</b>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "genepattern": {
     "param_values": {
      "input.filename": null,
      "job.cpuCount": null,
      "job.memory": null,
      "job.queue": null,
      "job.walltime": null,
      "output.file": null
     },
     "type": "task"
    }
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "75a432e60972417881d5cd9e656f04f1",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "GPTaskWidget(lsid='urn:lsid:broad.mit.edu:cancer.software.genepattern.module.analysis:00002')"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "convertlineendings_task = gp.GPTask(genepattern.get_session(0), 'urn:lsid:broad.mit.edu:cancer.software.genepattern.module.analysis:00002')\n",
    "convertlineendings_job_spec = convertlineendings_task.make_job_spec()\n",
    "convertlineendings_job_spec.set_parameter(\"input.filename\", \"\")\n",
    "convertlineendings_job_spec.set_parameter(\"output.file\", \"<input.filename_basename>.cvt.<input.filename_extension>\")\n",
    "genepattern.display(convertlineendings_task)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-info\" style=\"font-weight: bold;\">\n",
    "<ol>\n",
    "<li>Highlight this cell and then click the <i><i class=\"fa-th fa\"></i> Tools</i> button in the toolbar. This should open a dialog with a list of the available GenePattern analyses.</li>\n",
    "<li>Scroll through the list or use the search box in the upper right of the dialog to find the <i>PreprocessDataset</i> module. This module is used to preprocess data ahead of further analysis.</li>\n",
    "<li>Click the listing for <i>PreprocessDataset</i>. This should insert a new analysis cell below, representing the preprocessing analysis.</li>\n",
    "</ol>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Job Cells\n",
    "\n",
    "The third and final kind of GenePattern cell are Job Cells. These cells represent an analysis job that has been submitted to GenePattern queue for processing. Jobs progress through the states of Pending, Running and finally either to Completed or Error.\n",
    "\n",
    "Once a job has completed or experienced an error, the cell will update to display a list of outputs. These outputs are files which can be displayed in the browser, downloaded or sent as input to another GenePattern analysis. Outputs are indicated by the <i class=\"fa fa-info-circle\" style=\"color: rgb(128, 128, 128);\"></i> icon.\n",
    "\n",
    "To see a list of all output file options, simply click that output in the list and a menu wll open, displaying the available options.\n",
    "\n",
    "If this analysis includes visualization, the visualization will load and appear inside the Job Cell as well."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<b>Below is a screenshot of a GenePattern job cell. We have not embedded one directly because GenePattern jobs are private to the user running the analysis.</b>\n",
    "\n",
    "<img src=\"http://genepattern-notebook.org/wp-content/uploads/2017/02/content_screen_shot_2015-08-24_at_10_33_20.png\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-info\" style=\"font-weight: bold;\">\n",
    "<ol>\n",
    "<li>We are going to run the ProprocessDataset analysis inserted earlier. To do this we need to provide the analysis with a data file on which to perform processing. A suitable file is linked below. This file contains gene expression data comparing ALL samples with AML samples.\n",
    "<a style=\"display: block; padding: 10px;\" href=\"https://datasets.genepattern.org/data/all_aml/all_aml_test.gct\">https://datasets.genepattern.org/data/all_aml/all_aml_test.gct</a>\n",
    "</li>\n",
    "<li>Copy the URL for this file, then scroll back up to the <i>ProprocessDataset</i> cell and paste the URL into the <i>input filename</i> parameter. Alternatively, you could download this file and then use the <i>Upload File</i> button next to the <i>input filename</i> parameter</li>\n",
    "<li>The default values of the other input settings should be sufficient for this file. If you want more information about them you can read the documentation by going to the Gear menu ( <span class=\"fa fa-cog\"></span> ) on the cell and selecting <i>Documentation</i>.</li>\n",
    "<li>Click the <i>Run</i> button to begin the analysis. This will create a new GenePattern job cell.</li>\n",
    "</ol>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Markdown Cells\n",
    "\n",
    "Markdown cells are another cell type available in the GenePattern Notebook environment. They allow a notebook author to take notes, document methods or embed images in a notebook document.\n",
    "\n",
    "To insert a markdown cell, first insert a new cell, either through the Insert > Insert Cell Below menu or by clicking the <i class=\"fa-plus fa\"></i> button in the notebook toolbar. Once a new cell has been inserted, you can select the cell and then change the cell type to Markdown either by using the Cell > Cell Type > Markdown menu or by going to the dropdown menu in the notebook toolbar and selecting Markdown from the list of options.\n",
    "\n",
    "Markdown cells allow the user to format text using either <a href=\"https://developer.mozilla.org/en-US/docs/Web/HTML\" target=\"_blank\">HTML</a> or the <a href=\"https://daringfireball.net/projects/markdown/syntax\" target=\"_blank\">Markdown</a> syntax. Additionally, we provide a \"What You See is What You Get\" rich text editor (see the feature below)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-info\" style=\"font-weight: bold;\">\n",
    "<ol>\n",
    "<li>Insert a new cell below</li>\n",
    "<li>Use the dropdown menu in the toolbar to change the cell type to <i>Markdown</i>.</li>\n",
    "</ol>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Rich Text Editor\n",
    "The GenePattern Rich Text Editor allows a user to format notes and documentation in a notebook in much the same way that one might use Microsoft Word or Libre Office — without the need to write a single HTML tag or line of markdown.\n",
    "\n",
    "To use the Rich Text Editor, first insert a markdown cell (see the instructions above). Once a cell has been changed to the markdown type, two buttons should appear to the left of the cell. The <i class=\"fa fa-file-code-o\"></i> button opens the Rich Text Editor and the <i class=\"fa-step-forward fa\"></i> button finalizes the cell and renders the text.\n",
    "\n",
    "Click the <i class=\"fa fa-file-code-o\"></i> button and the editor will appear. A toolbar will show above the cell, allowing the user to format text, insert headers or add links. Style the text as desired, and when finished, click the <i class=\"fa-step-forward fa\"></i> button to finish editing."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-info\" style=\"font-weight: bold;\">\n",
    "<ol>\n",
    "<li>Highlight the markdown cell you created above and click the <i class=\"fa fa-file-code-o\"></i> button on the left side of the cell to activate the Rich Text Editor. If you don't see the button, make sure the cell is in editing mode by double-clicking the cell.</li>\n",
    "<li>Edit the text of the cell and then click the <i class=\"fa-step-forward fa\"></i> button to display the rendered text.</li>\n",
    "</ol>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Publishing & Sharing Features\n",
    "\n",
    "When using the <a href=\"https://notebook.genepattern.org\" target=\"_blank\">GenePattern Notebook Repository</a>, users have the ability to publicly share their notebooks and to run notebooks shared by others."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Browsing Public Notebooks\n",
    "\n",
    "* To view the available notebooks, click the \"Public Notebooks\" tab and browse through the list.\n",
    "* To obtain a copy and run the shared notebook, click the notebook in the list and then select \"Get a Copy\" on the confirmation dialog that pops up. This will make a copy of the chosen notebook in your current directory, accessible by clicking on the Files tab.\n",
    "* To run the notebook, click the \"Click here if you would like to open this notebook\" link in the resulting dialog, or go back to the \"Files\" tab and run the notebook as normal from there."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-info\" style=\"font-weight: bold;\">\n",
    "<ol>\n",
    "<li>To browse public notebooks you will need to be logged into the GenePattern Notebook Repository and on the index page displaying your list of notebook files. To view this page <a href=\"https://notebook.genepattern.org\" target=\"_blank\">click here</a> and log in if required.</li>\n",
    "<li>On that page click the \"Public Notebooks\" tab found near the top. This will display the list of available notebooks. Mouse over the name of a notebook to read a description.</li>\n",
    "<li>Find a notebook in the list you want to run and then click it. This will open a dialog with the full description. Click the \"Get a Copy\" button to get a copy of the selected notebook.</li>\n",
    "<li>Click \"OK\" to close the dialog, then click on the \"Files\" tab at the top of the page. This will bring you back to your list of notebooks. You should see that the selected notebook is now in the list. You can click on it to run it from here, just as you would any other notebook.</li>\n",
    "</ol>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "### Publicly Sharing Notebooks\n",
    "* To publicly share a notebook, click the checkbox next to that notebook and then click the \"Publish\" button on the toolbar above.\n",
    "* This will open a dialog with a short form, prompting you to enter the author's name, the quality and a brief description. Fill out this information and then click the \"Publish\" button on the dialog.\n",
    "* Your notebook should now be available to the public and should appear on the \"Public Notebooks\" tab. The published version of the notebook is checkpoint of the notebook at the time it was published. Any changes you make to the notebook in the future are not copied over to the published version unless you explicitly choose to update the notebook in the public repository.\n",
    "\n",
    "<img src=\"http://genepattern-notebook.org/wp-content/uploads/2017/02/content_share-notebook.jpg\" style=\"border: black solid 1px;\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "### Updating or Removing a Public Notebook\n",
    "* You can update or remove a notebook that you have made public by first clicking the checkbox next to the notebook and then clicking the \"Publish\" button.\n",
    "* In the dialog that comes up you will have the option to edit any of the notebook's metadata, such as author name, description, etc.\n",
    "* To update or remove the notebook, click the \"Update\" or \"Unpublish\" buttons, respectively."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Programmatic Features\n",
    "\n",
    "In addition to the basic and publishing features intended for use by both non-programming and programming users alike, the GenePattern Notebook environment also provides a variety of features intended primarily for use by coders."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Code Cells\n",
    "\n",
    "Code cells allow a user to type in execute Python code. They are the default type of cell. Simply enter the Python code you want to run and then click the <i class=\"fa-step-forward fa\"></i> (run cell) button in the notebook toolbar.\n",
    "\n",
    "More information on code cells and their various options is available in the <a href=\"https://jupyter.readthedocs.io/en/latest/\">Jupyter documentation</a>."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### GenePattern Python Library\n",
    "\n",
    "A library is available for using Python to interact with available GenePattern's service. This allows a user to programmatically prepare analyses, launch jobs and retrieve results.\n",
    "\n",
    "A complete tutorial on using the GenePattern Python Library is available as a public notebook in the GenePattern Notebook Repository. Additional information is available in our <a href=\"http://software.broadinstitute.org/cancer/software/genepattern/programmers-guide#_Using_GenePattern_from_Python\" target=\"_blank\">documentation</a>."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Send to Code\n",
    "\n",
    "The GenePattern Python Library seamlessly integrates with GenePattern cells. Code examples of how to reference GenePattern jobs or GenePattern result files are available in GenePattern Job Cells by clicking a job result and selecting “Send to Code” in the menu.\n",
    "\n",
    "<img src=\"http://genepattern-notebook.org/wp-content/uploads/2017/02/content_screen-shot-2015-10-15-at-13_50.jpg\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Send to Dataframe\n",
    "\n",
    "The GenePattern Python Library also provides functionality for common GenePattern file formats, allowing them seamlessly integrate with <a href=\"http://pandas.pydata.org/\">Pandas</a>, a popular Python data analysis library.\n",
    "\n",
    "Both the <a href=\"http://software.broadinstitute.org/cancer/software/genepattern/file-formats-guide\" target=\"_blank\">GCT and ODF file formats</a> are easily loaded as Pandas Dataframes. Code examples of how to load these files are available in GenePattern Job Cells by clicking a GCT or ODF job result and selecting “Send to Dataframe” in the menu."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Python Variable Input\n",
    "\n",
    "As part of the seamless integration between Python and GenePattern, Python variables may be directly used as input in GenePattern Analysis Cells. To use a Python variable as input, first define the variable in a code cell, then in a GenePattern Analysis cell, enter the variable name surrounded by two sets of curly brackets. For example, using the variable <code> foo </code> would appear as <code> {{ foo }} </code>.\n",
    "\n",
    "When the Run button is clicked to launch the analysis job, the notebook will first obtain the value of the variable used as input and then send that to the GenePattern server. As the GenePattern services expect a string, non-string variables will be cast as strings when they are evaluated."
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3.6",
   "language": "python",
   "name": "python3.6"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.7"
  },
  "widgets": {
   "state": {
    "305366c16ab14aae9886f40da7231b11": {
     "views": [
      {
       "cell_index": 20
      }
     ]
    },
    "6427149a01a440b79e7b790c9c94ae03": {
     "views": [
      {
       "cell_index": 15
      }
     ]
    },
    "d40ab328d5ff44c09440b9dcbf3c60d2": {
     "views": [
      {
       "cell_index": 20
      }
     ]
    },
    "e816afbceda54d1398c1e0a1bb1543f9": {
     "views": [
      {
       "cell_index": 15
      }
     ]
    }
   },
   "version": "1.2.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}