04 Deploy for Batch Scoring.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "c9a8c692-9562-4dd1-bf9a-fd61d204fc24",
     "showTitle": false,
     "title": ""
    }
   },
   "source": [
    "For the batch scoring, we will persist the values in a new global persistent Databricks table. In production data workloads, you may save the scored data to Blob Storage, Azure Cosmos DB, or other serving layer. Another implementation detail we are skipping for the lab is processing only new files. This can be accomplished by creating a widget in the notebook that accepts a path parameter that is passed in from Azure Data Factory."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "9abaaef2-16d1-4a69-bef0-7c46d6639690",
     "showTitle": false,
     "title": ""
    }
   },
   "outputs": [
    {
     "output_type": "display_data",
     "data": {
      "text/html": [
       "<style scoped>\n  .ansiout {\n    display: block;\n    unicode-bidi: embed;\n    white-space: pre-wrap;\n    word-wrap: break-word;\n    word-break: break-all;\n    font-family: \"Source Code Pro\", \"Menlo\", monospace;;\n    font-size: 13px;\n    color: #555;\n    margin-left: 4px;\n    line-height: 19px;\n  }\n</style>\n<div class=\"ansiout\"></div>"
      ]
     },
     "metadata": {
      "application/vnd.databricks.v1+output": {
       "addedWidgets": {},
       "arguments": {},
       "data": "<div class=\"ansiout\"></div>",
       "datasetInfos": [],
       "metadata": {},
       "removedWidgets": [],
       "type": "html"
      }
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "from pyspark.ml import Pipeline, PipelineModel\n",
    "from pyspark.ml.feature import OneHotEncoder, StringIndexer, VectorAssembler, Bucketizer\n",
    "from pyspark.sql.functions import array, col, lit\n",
    "from pyspark.sql.types import *"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "0d4c94ec-d46f-4b94-a3ac-45ce1d06829b",
     "showTitle": false,
     "title": ""
    }
   },
   "source": [
    "Replace STORAGE-ACCOUNT-NAME with the name of your storage account. You can find this in the Azure portal by locating the storage account that you created in the lab setup, within your resource group. The container name is set to the default used for this lab. If yours is different, update the containerName variable accordingly."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "17e8718b-9b9a-45f2-ac75-b7a8522bc4ad",
     "showTitle": false,
     "title": ""
    }
   },
   "outputs": [
    {
     "output_type": "display_data",
     "data": {
      "text/html": [
       "<style scoped>\n  .ansiout {\n    display: block;\n    unicode-bidi: embed;\n    white-space: pre-wrap;\n    word-wrap: break-word;\n    word-break: break-all;\n    font-family: \"Source Code Pro\", \"Menlo\", monospace;;\n    font-size: 13px;\n    color: #555;\n    margin-left: 4px;\n    line-height: 19px;\n  }\n</style>\n<div class=\"ansiout\"></div>"
      ]
     },
     "metadata": {
      "application/vnd.databricks.v1+output": {
       "addedWidgets": {},
       "arguments": {},
       "data": "<div class=\"ansiout\"></div>",
       "datasetInfos": [],
       "metadata": {},
       "removedWidgets": [],
       "type": "html"
      }
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "accountName = \"STORAGE-ACCOUNT-NAME\"\n",
    "containerName = \"sparkcontainer\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "5faeb479-e3bb-4725-88b8-02e24f5d146a",
     "showTitle": false,
     "title": ""
    }
   },
   "source": [
    "Define the schema for the CSV files"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "da5c12bc-1645-4033-b31f-b89df0c9e168",
     "showTitle": false,
     "title": ""
    }
   },
   "outputs": [
    {
     "output_type": "display_data",
     "data": {
      "text/html": [
       "<style scoped>\n  .ansiout {\n    display: block;\n    unicode-bidi: embed;\n    white-space: pre-wrap;\n    word-wrap: break-word;\n    word-break: break-all;\n    font-family: \"Source Code Pro\", \"Menlo\", monospace;;\n    font-size: 13px;\n    color: #555;\n    margin-left: 4px;\n    line-height: 19px;\n  }\n</style>\n<div class=\"ansiout\"></div>"
      ]
     },
     "metadata": {
      "application/vnd.databricks.v1+output": {
       "addedWidgets": {},
       "arguments": {},
       "data": "<div class=\"ansiout\"></div>",
       "datasetInfos": [],
       "metadata": {},
       "removedWidgets": [],
       "type": "html"
      }
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "data_schema = StructType([\n",
    "        StructField('OriginAirportCode',StringType()),\n",
    "        StructField('Month', IntegerType()),\n",
    "        StructField('DayofMonth', IntegerType()),\n",
    "        StructField('CRSDepHour', IntegerType()),\n",
    "        StructField('DayOfWeek', IntegerType()),\n",
    "        StructField('Carrier', StringType()),\n",
    "        StructField('DestAirportCode', StringType()),\n",
    "        StructField('DepDel15', IntegerType()),\n",
    "        StructField('WindSpeed', DoubleType()),\n",
    "        StructField('SeaLevelPressure', DoubleType()),  \n",
    "        StructField('HourlyPrecip', DoubleType())])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "7a9ad4d9-7372-4a8f-9a5a-8d08b27261d7",
     "showTitle": false,
     "title": ""
    }
   },
   "source": [
    "Create a new DataFrame from the CSV files, applying the schema"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "1fce6879-d0cc-4075-ab26-a23885eb2083",
     "showTitle": false,
     "title": ""
    }
   },
   "outputs": [
    {
     "output_type": "display_data",
     "data": {
      "text/html": [
       "<style scoped>\n  .ansiout {\n    display: block;\n    unicode-bidi: embed;\n    white-space: pre-wrap;\n    word-wrap: break-word;\n    word-break: break-all;\n    font-family: \"Source Code Pro\", \"Menlo\", monospace;;\n    font-size: 13px;\n    color: #555;\n    margin-left: 4px;\n    line-height: 19px;\n  }\n</style>\n<div class=\"ansiout\"></div>"
      ]
     },
     "metadata": {
      "application/vnd.databricks.v1+output": {
       "addedWidgets": {},
       "arguments": {},
       "data": "<div class=\"ansiout\"></div>",
       "datasetInfos": [],
       "metadata": {},
       "removedWidgets": [],
       "type": "html"
      }
     },
     "output_type": "display_data"
    },
    {
     "output_type": "display_data",
     "data": {
      "text/html": [
       "<style scoped>\n  .ansiout {\n    display: block;\n    unicode-bidi: embed;\n    white-space: pre-wrap;\n    word-wrap: break-word;\n    word-break: break-all;\n    font-family: \"Source Code Pro\", \"Menlo\", monospace;;\n    font-size: 13px;\n    color: #555;\n    margin-left: 4px;\n    line-height: 19px;\n  }\n</style>\n<div class=\"ansiout\"><span class=\"ansi-red-fg\">---------------------------------------------------------------------------</span>\n<span class=\"ansi-red-fg\">Py4JJavaError</span>                             Traceback (most recent call last)\n<span class=\"ansi-green-fg\">&lt;command-1036921577453799&gt;</span> in <span class=\"ansi-cyan-fg\">&lt;module&gt;</span>\n<span class=\"ansi-green-intense-fg ansi-bold\">      2</span>                     schema<span class=\"ansi-blue-fg\">=</span>data_schema<span class=\"ansi-blue-fg\">,</span>\n<span class=\"ansi-green-intense-fg ansi-bold\">      3</span>                     sep<span class=\"ansi-blue-fg\">=</span><span class=\"ansi-blue-fg\">&#34;,&#34;</span><span class=\"ansi-blue-fg\">,</span>\n<span class=\"ansi-green-fg\">----&gt; 4</span><span class=\"ansi-red-fg\">                     header=True)\n</span>\n<span class=\"ansi-green-fg\">/databricks/spark/python/pyspark/sql/readwriter.py</span> in <span class=\"ansi-cyan-fg\">csv</span><span class=\"ansi-blue-fg\">(self, path, schema, sep, encoding, quote, escape, comment, header, inferSchema, ignoreLeadingWhiteSpace, ignoreTrailingWhiteSpace, nullValue, nanValue, positiveInf, negativeInf, dateFormat, timestampFormat, maxColumns, maxCharsPerColumn, maxMalformedLogPerPartition, mode, columnNameOfCorruptRecord, multiLine, charToEscapeQuoteEscaping, samplingRatio, enforceSchema, emptyValue, locale, lineSep, pathGlobFilter, recursiveFileLookup)</span>\n<span class=\"ansi-green-intense-fg ansi-bold\">    533</span>             path <span class=\"ansi-blue-fg\">=</span> <span class=\"ansi-blue-fg\">[</span>path<span class=\"ansi-blue-fg\">]</span>\n<span class=\"ansi-green-intense-fg ansi-bold\">    534</span>         <span class=\"ansi-green-fg\">if</span> type<span class=\"ansi-blue-fg\">(</span>path<span class=\"ansi-blue-fg\">)</span> <span class=\"ansi-blue-fg\">==</span> list<span class=\"ansi-blue-fg\">:</span>\n<span class=\"ansi-green-fg\">--&gt; 535</span><span class=\"ansi-red-fg\">             </span><span class=\"ansi-green-fg\">return</span> self<span class=\"ansi-blue-fg\">.</span>_df<span class=\"ansi-blue-fg\">(</span>self<span class=\"ansi-blue-fg\">.</span>_jreader<span class=\"ansi-blue-fg\">.</span>csv<span class=\"ansi-blue-fg\">(</span>self<span class=\"ansi-blue-fg\">.</span>_spark<span class=\"ansi-blue-fg\">.</span>_sc<span class=\"ansi-blue-fg\">.</span>_jvm<span class=\"ansi-blue-fg\">.</span>PythonUtils<span class=\"ansi-blue-fg\">.</span>toSeq<span class=\"ansi-blue-fg\">(</span>path<span class=\"ansi-blue-fg\">)</span><span class=\"ansi-blue-fg\">)</span><span class=\"ansi-blue-fg\">)</span>\n<span class=\"ansi-green-intense-fg ansi-bold\">    536</span>         <span class=\"ansi-green-fg\">elif</span> isinstance<span class=\"ansi-blue-fg\">(</span>path<span class=\"ansi-blue-fg\">,</span> RDD<span class=\"ansi-blue-fg\">)</span><span class=\"ansi-blue-fg\">:</span>\n<span class=\"ansi-green-intense-fg ansi-bold\">    537</span>             <span class=\"ansi-green-fg\">def</span> func<span class=\"ansi-blue-fg\">(</span>iterator<span class=\"ansi-blue-fg\">)</span><span class=\"ansi-blue-fg\">:</span>\n\n<span class=\"ansi-green-fg\">/databricks/spark/python/lib/py4j-0.10.9-src.zip/py4j/java_gateway.py</span> in <span class=\"ansi-cyan-fg\">__call__</span><span class=\"ansi-blue-fg\">(self, *args)</span>\n<span class=\"ansi-green-intense-fg ansi-bold\">   1303</span>         answer <span class=\"ansi-blue-fg\">=</span> self<span class=\"ansi-blue-fg\">.</span>gateway_client<span class=\"ansi-blue-fg\">.</span>send_command<span class=\"ansi-blue-fg\">(</span>command<span class=\"ansi-blue-fg\">)</span>\n<span class=\"ansi-green-intense-fg ansi-bold\">   1304</span>         return_value = get_return_value(\n<span class=\"ansi-green-fg\">-&gt; 1305</span><span class=\"ansi-red-fg\">             answer, self.gateway_client, self.target_id, self.name)\n</span><span class=\"ansi-green-intense-fg ansi-bold\">   1306</span> \n<span class=\"ansi-green-intense-fg ansi-bold\">   1307</span>         <span class=\"ansi-green-fg\">for</span> temp_arg <span class=\"ansi-green-fg\">in</span> temp_args<span class=\"ansi-blue-fg\">:</span>\n\n<span class=\"ansi-green-fg\">/databricks/spark/python/pyspark/sql/utils.py</span> in <span class=\"ansi-cyan-fg\">deco</span><span class=\"ansi-blue-fg\">(*a, **kw)</span>\n<span class=\"ansi-green-intense-fg ansi-bold\">    125</span>     <span class=\"ansi-green-fg\">def</span> deco<span class=\"ansi-blue-fg\">(</span><span class=\"ansi-blue-fg\">*</span>a<span class=\"ansi-blue-fg\">,</span> <span class=\"ansi-blue-fg\">**</span>kw<span class=\"ansi-blue-fg\">)</span><span class=\"ansi-blue-fg\">:</span>\n<span class=\"ansi-green-intense-fg ansi-bold\">    126</span>         <span class=\"ansi-green-fg\">try</span><span class=\"ansi-blue-fg\">:</span>\n<span class=\"ansi-green-fg\">--&gt; 127</span><span class=\"ansi-red-fg\">             </span><span class=\"ansi-green-fg\">return</span> f<span class=\"ansi-blue-fg\">(</span><span class=\"ansi-blue-fg\">*</span>a<span class=\"ansi-blue-fg\">,</span> <span class=\"ansi-blue-fg\">**</span>kw<span class=\"ansi-blue-fg\">)</span>\n<span class=\"ansi-green-intense-fg ansi-bold\">    128</span>         <span class=\"ansi-green-fg\">except</span> py4j<span class=\"ansi-blue-fg\">.</span>protocol<span class=\"ansi-blue-fg\">.</span>Py4JJavaError <span class=\"ansi-green-fg\">as</span> e<span class=\"ansi-blue-fg\">:</span>\n<span class=\"ansi-green-intense-fg ansi-bold\">    129</span>             converted <span class=\"ansi-blue-fg\">=</span> convert_exception<span class=\"ansi-blue-fg\">(</span>e<span class=\"ansi-blue-fg\">.</span>java_exception<span class=\"ansi-blue-fg\">)</span>\n\n<span class=\"ansi-green-fg\">/databricks/spark/python/lib/py4j-0.10.9-src.zip/py4j/protocol.py</span> in <span class=\"ansi-cyan-fg\">get_return_value</span><span class=\"ansi-blue-fg\">(answer, gateway_client, target_id, name)</span>\n<span class=\"ansi-green-intense-fg ansi-bold\">    326</span>                 raise Py4JJavaError(\n<span class=\"ansi-green-intense-fg ansi-bold\">    327</span>                     <span class=\"ansi-blue-fg\">&#34;An error occurred while calling {0}{1}{2}.\\n&#34;</span><span class=\"ansi-blue-fg\">.</span>\n<span class=\"ansi-green-fg\">--&gt; 328</span><span class=\"ansi-red-fg\">                     format(target_id, &#34;.&#34;, name), value)\n</span><span class=\"ansi-green-intense-fg ansi-bold\">    329</span>             <span class=\"ansi-green-fg\">else</span><span class=\"ansi-blue-fg\">:</span>\n<span class=\"ansi-green-intense-fg ansi-bold\">    330</span>                 raise Py4JError(\n\n<span class=\"ansi-red-fg\">Py4JJavaError</span>: An error occurred while calling o327.csv.\n: shaded.databricks.org.apache.hadoop.fs.azure.AzureException: shaded.databricks.org.apache.hadoop.fs.azure.AzureException: Unable to access container sparkcontainer in account STORAGE-ACCOUNT-NAME.blob.core.windows.net using anonymous credentials, and no credentials found for them  in the configuration.\n\tat shaded.databricks.org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.createAzureStorageSession(AzureNativeFileSystemStore.java:1063)\n\tat shaded.databricks.org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.initialize(AzureNativeFileSystemStore.java:507)\n\tat shaded.databricks.org.apache.hadoop.fs.azure.NativeAzureFileSystem.initialize(NativeAzureFileSystem.java:1374)\n\tat org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2669)\n\tat org.apache.hadoop.fs.FileSystem.get(FileSystem.java:370)\n\tat org.apache.hadoop.fs.Path.getFileSystem(Path.java:295)\n\tat org.apache.spark.sql.execution.datasources.DataSource$.$anonfun$checkAndGlobPathIfNecessary$1(DataSource.scala:812)\n\tat scala.collection.TraversableLike.$anonfun$flatMap$1(TraversableLike.scala:245)\n\tat scala.collection.immutable.List.foreach(List.scala:392)\n\tat scala.collection.TraversableLike.flatMap(TraversableLike.scala:245)\n\tat scala.collection.TraversableLike.flatMap$(TraversableLike.scala:242)\n\tat scala.collection.immutable.List.flatMap(List.scala:355)\n\tat org.apache.spark.sql.execution.datasources.DataSource$.checkAndGlobPathIfNecessary(DataSource.scala:810)\n\tat org.apache.spark.sql.execution.datasources.DataSource.checkAndGlobPathIfNecessary(DataSource.scala:618)\n\tat org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:438)\n\tat org.apache.spark.sql.DataFrameReader.loadV1Source(DataFrameReader.scala:384)\n\tat org.apache.spark.sql.DataFrameReader.$anonfun$load$2(DataFrameReader.scala:373)\n\tat scala.Option.getOrElse(Option.scala:189)\n\tat org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:373)\n\tat org.apache.spark.sql.DataFrameReader.csv(DataFrameReader.scala:825)\n\tat sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\n\tat sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)\n\tat sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\n\tat java.lang.reflect.Method.invoke(Method.java:498)\n\tat py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)\n\tat py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:380)\n\tat py4j.Gateway.invoke(Gateway.java:295)\n\tat py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)\n\tat py4j.commands.CallCommand.execute(CallCommand.java:79)\n\tat py4j.GatewayConnection.run(GatewayConnection.java:251)\n\tat java.lang.Thread.run(Thread.java:750)\nCaused by: shaded.databricks.org.apache.hadoop.fs.azure.AzureException: Unable to access container sparkcontainer in account STORAGE-ACCOUNT-NAME.blob.core.windows.net using anonymous credentials, and no credentials found for them  in the configuration.\n\tat shaded.databricks.org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.connectUsingAnonymousCredentials(AzureNativeFileSystemStore.java:814)\n\tat shaded.databricks.org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.createAzureStorageSession(AzureNativeFileSystemStore.java:1058)\n\t... 30 more\nCaused by: hadoop_azure_shaded.com.microsoft.azure.storage.StorageException: \n\tat hadoop_azure_shaded.com.microsoft.azure.storage.StorageException.translateException(StorageException.java:87)\n\tat hadoop_azure_shaded.com.microsoft.azure.storage.core.ExecutionEngine.executeWithRetry(ExecutionEngine.java:220)\n\tat hadoop_azure_shaded.com.microsoft.azure.storage.blob.CloudBlobContainer.exists(CloudBlobContainer.java:744)\n\tat hadoop_azure_shaded.com.microsoft.azure.storage.blob.CloudBlobContainer.exists(CloudBlobContainer.java:731)\n\tat shaded.databricks.org.apache.hadoop.fs.azure.StorageInterfaceImpl$CloudBlobContainerWrapperImpl.exists(StorageInterfaceImpl.java:218)\n\tat shaded.databricks.org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.connectUsingAnonymousCredentials(AzureNativeFileSystemStore.java:807)\n\t... 31 more\nCaused by: java.net.UnknownHostException: STORAGE-ACCOUNT-NAME.blob.core.windows.net\n\tat java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:184)\n\tat java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)\n\tat java.net.Socket.connect(Socket.java:613)\n\tat sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:293)\n\tat sun.net.NetworkClient.doConnect(NetworkClient.java:175)\n\tat sun.net.www.http.HttpClient.openServer(HttpClient.java:463)\n\tat sun.net.www.http.HttpClient.openServer(HttpClient.java:558)\n\tat sun.net.www.protocol.https.HttpsClient.&lt;init&gt;(HttpsClient.java:264)\n\tat sun.net.www.protocol.https.HttpsClient.New(HttpsClient.java:367)\n\tat sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.getNewHttpClient(AbstractDelegateHttpsURLConnection.java:203)\n\tat sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1167)\n\tat sun.net.www.protocol.http.HttpURLConnection$6.run(HttpURLConnection.java:1051)\n\tat sun.net.www.protocol.http.HttpURLConnection$6.run(HttpURLConnection.java:1049)\n\tat java.security.AccessController.doPrivileged(Native Method)\n\tat java.security.AccessController.doPrivilegedWithCombiner(AccessController.java:784)\n\tat sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1048)\n\tat sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:189)\n\tat sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1577)\n\tat sun.net.www.protocol.http.HttpURLConnection.access$200(HttpURLConnection.java:97)\n\tat sun.net.www.protocol.http.HttpURLConnection$9.run(HttpURLConnection.java:1497)\n\tat sun.net.www.protocol.http.HttpURLConnection$9.run(HttpURLConnection.java:1495)\n\tat java.security.AccessController.doPrivileged(Native Method)\n\tat java.security.AccessController.doPrivilegedWithCombiner(AccessController.java:784)\n\tat sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1494)\n\tat java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480)\n\tat sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:352)\n\tat hadoop_azure_shaded.com.microsoft.azure.storage.core.ExecutionEngine.executeWithRetry(ExecutionEngine.java:115)\n\t... 35 more\n</div>"
      ]
     },
     "metadata": {
      "application/vnd.databricks.v1+output": {
       "arguments": {},
       "data": "<div class=\"ansiout\"><span class=\"ansi-red-fg\">---------------------------------------------------------------------------</span>\n<span class=\"ansi-red-fg\">Py4JJavaError</span>                             Traceback (most recent call last)\n<span class=\"ansi-green-fg\">&lt;command-1036921577453799&gt;</span> in <span class=\"ansi-cyan-fg\">&lt;module&gt;</span>\n<span class=\"ansi-green-intense-fg ansi-bold\">      2</span>                     schema<span class=\"ansi-blue-fg\">=</span>data_schema<span class=\"ansi-blue-fg\">,</span>\n<span class=\"ansi-green-intense-fg ansi-bold\">      3</span>                     sep<span class=\"ansi-blue-fg\">=</span><span class=\"ansi-blue-fg\">&#34;,&#34;</span><span class=\"ansi-blue-fg\">,</span>\n<span class=\"ansi-green-fg\">----&gt; 4</span><span class=\"ansi-red-fg\">                     header=True)\n</span>\n<span class=\"ansi-green-fg\">/databricks/spark/python/pyspark/sql/readwriter.py</span> in <span class=\"ansi-cyan-fg\">csv</span><span class=\"ansi-blue-fg\">(self, path, schema, sep, encoding, quote, escape, comment, header, inferSchema, ignoreLeadingWhiteSpace, ignoreTrailingWhiteSpace, nullValue, nanValue, positiveInf, negativeInf, dateFormat, timestampFormat, maxColumns, maxCharsPerColumn, maxMalformedLogPerPartition, mode, columnNameOfCorruptRecord, multiLine, charToEscapeQuoteEscaping, samplingRatio, enforceSchema, emptyValue, locale, lineSep, pathGlobFilter, recursiveFileLookup)</span>\n<span class=\"ansi-green-intense-fg ansi-bold\">    533</span>             path <span class=\"ansi-blue-fg\">=</span> <span class=\"ansi-blue-fg\">[</span>path<span class=\"ansi-blue-fg\">]</span>\n<span class=\"ansi-green-intense-fg ansi-bold\">    534</span>         <span class=\"ansi-green-fg\">if</span> type<span class=\"ansi-blue-fg\">(</span>path<span class=\"ansi-blue-fg\">)</span> <span class=\"ansi-blue-fg\">==</span> list<span class=\"ansi-blue-fg\">:</span>\n<span class=\"ansi-green-fg\">--&gt; 535</span><span class=\"ansi-red-fg\">             </span><span class=\"ansi-green-fg\">return</span> self<span class=\"ansi-blue-fg\">.</span>_df<span class=\"ansi-blue-fg\">(</span>self<span class=\"ansi-blue-fg\">.</span>_jreader<span class=\"ansi-blue-fg\">.</span>csv<span class=\"ansi-blue-fg\">(</span>self<span class=\"ansi-blue-fg\">.</span>_spark<span class=\"ansi-blue-fg\">.</span>_sc<span class=\"ansi-blue-fg\">.</span>_jvm<span class=\"ansi-blue-fg\">.</span>PythonUtils<span class=\"ansi-blue-fg\">.</span>toSeq<span class=\"ansi-blue-fg\">(</span>path<span class=\"ansi-blue-fg\">)</span><span class=\"ansi-blue-fg\">)</span><span class=\"ansi-blue-fg\">)</span>\n<span class=\"ansi-green-intense-fg ansi-bold\">    536</span>         <span class=\"ansi-green-fg\">elif</span> isinstance<span class=\"ansi-blue-fg\">(</span>path<span class=\"ansi-blue-fg\">,</span> RDD<span class=\"ansi-blue-fg\">)</span><span class=\"ansi-blue-fg\">:</span>\n<span class=\"ansi-green-intense-fg ansi-bold\">    537</span>             <span class=\"ansi-green-fg\">def</span> func<span class=\"ansi-blue-fg\">(</span>iterator<span class=\"ansi-blue-fg\">)</span><span class=\"ansi-blue-fg\">:</span>\n\n<span class=\"ansi-green-fg\">/databricks/spark/python/lib/py4j-0.10.9-src.zip/py4j/java_gateway.py</span> in <span class=\"ansi-cyan-fg\">__call__</span><span class=\"ansi-blue-fg\">(self, *args)</span>\n<span class=\"ansi-green-intense-fg ansi-bold\">   1303</span>         answer <span class=\"ansi-blue-fg\">=</span> self<span class=\"ansi-blue-fg\">.</span>gateway_client<span class=\"ansi-blue-fg\">.</span>send_command<span class=\"ansi-blue-fg\">(</span>command<span class=\"ansi-blue-fg\">)</span>\n<span class=\"ansi-green-intense-fg ansi-bold\">   1304</span>         return_value = get_return_value(\n<span class=\"ansi-green-fg\">-&gt; 1305</span><span class=\"ansi-red-fg\">             answer, self.gateway_client, self.target_id, self.name)\n</span><span class=\"ansi-green-intense-fg ansi-bold\">   1306</span> \n<span class=\"ansi-green-intense-fg ansi-bold\">   1307</span>         <span class=\"ansi-green-fg\">for</span> temp_arg <span class=\"ansi-green-fg\">in</span> temp_args<span class=\"ansi-blue-fg\">:</span>\n\n<span class=\"ansi-green-fg\">/databricks/spark/python/pyspark/sql/utils.py</span> in <span class=\"ansi-cyan-fg\">deco</span><span class=\"ansi-blue-fg\">(*a, **kw)</span>\n<span class=\"ansi-green-intense-fg ansi-bold\">    125</span>     <span class=\"ansi-green-fg\">def</span> deco<span class=\"ansi-blue-fg\">(</span><span class=\"ansi-blue-fg\">*</span>a<span class=\"ansi-blue-fg\">,</span> <span class=\"ansi-blue-fg\">**</span>kw<span class=\"ansi-blue-fg\">)</span><span class=\"ansi-blue-fg\">:</span>\n<span class=\"ansi-green-intense-fg ansi-bold\">    126</span>         <span class=\"ansi-green-fg\">try</span><span class=\"ansi-blue-fg\">:</span>\n<span class=\"ansi-green-fg\">--&gt; 127</span><span class=\"ansi-red-fg\">             </span><span class=\"ansi-green-fg\">return</span> f<span class=\"ansi-blue-fg\">(</span><span class=\"ansi-blue-fg\">*</span>a<span class=\"ansi-blue-fg\">,</span> <span class=\"ansi-blue-fg\">**</span>kw<span class=\"ansi-blue-fg\">)</span>\n<span class=\"ansi-green-intense-fg ansi-bold\">    128</span>         <span class=\"ansi-green-fg\">except</span> py4j<span class=\"ansi-blue-fg\">.</span>protocol<span class=\"ansi-blue-fg\">.</span>Py4JJavaError <span class=\"ansi-green-fg\">as</span> e<span class=\"ansi-blue-fg\">:</span>\n<span class=\"ansi-green-intense-fg ansi-bold\">    129</span>             converted <span class=\"ansi-blue-fg\">=</span> convert_exception<span class=\"ansi-blue-fg\">(</span>e<span class=\"ansi-blue-fg\">.</span>java_exception<span class=\"ansi-blue-fg\">)</span>\n\n<span class=\"ansi-green-fg\">/databricks/spark/python/lib/py4j-0.10.9-src.zip/py4j/protocol.py</span> in <span class=\"ansi-cyan-fg\">get_return_value</span><span class=\"ansi-blue-fg\">(answer, gateway_client, target_id, name)</span>\n<span class=\"ansi-green-intense-fg ansi-bold\">    326</span>                 raise Py4JJavaError(\n<span class=\"ansi-green-intense-fg ansi-bold\">    327</span>                     <span class=\"ansi-blue-fg\">&#34;An error occurred while calling {0}{1}{2}.\\n&#34;</span><span class=\"ansi-blue-fg\">.</span>\n<span class=\"ansi-green-fg\">--&gt; 328</span><span class=\"ansi-red-fg\">                     format(target_id, &#34;.&#34;, name), value)\n</span><span class=\"ansi-green-intense-fg ansi-bold\">    329</span>             <span class=\"ansi-green-fg\">else</span><span class=\"ansi-blue-fg\">:</span>\n<span class=\"ansi-green-intense-fg ansi-bold\">    330</span>                 raise Py4JError(\n\n<span class=\"ansi-red-fg\">Py4JJavaError</span>: An error occurred while calling o327.csv.\n: shaded.databricks.org.apache.hadoop.fs.azure.AzureException: shaded.databricks.org.apache.hadoop.fs.azure.AzureException: Unable to access container sparkcontainer in account STORAGE-ACCOUNT-NAME.blob.core.windows.net using anonymous credentials, and no credentials found for them  in the configuration.\n\tat shaded.databricks.org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.createAzureStorageSession(AzureNativeFileSystemStore.java:1063)\n\tat shaded.databricks.org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.initialize(AzureNativeFileSystemStore.java:507)\n\tat shaded.databricks.org.apache.hadoop.fs.azure.NativeAzureFileSystem.initialize(NativeAzureFileSystem.java:1374)\n\tat org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2669)\n\tat org.apache.hadoop.fs.FileSystem.get(FileSystem.java:370)\n\tat org.apache.hadoop.fs.Path.getFileSystem(Path.java:295)\n\tat org.apache.spark.sql.execution.datasources.DataSource$.$anonfun$checkAndGlobPathIfNecessary$1(DataSource.scala:812)\n\tat scala.collection.TraversableLike.$anonfun$flatMap$1(TraversableLike.scala:245)\n\tat scala.collection.immutable.List.foreach(List.scala:392)\n\tat scala.collection.TraversableLike.flatMap(TraversableLike.scala:245)\n\tat scala.collection.TraversableLike.flatMap$(TraversableLike.scala:242)\n\tat scala.collection.immutable.List.flatMap(List.scala:355)\n\tat org.apache.spark.sql.execution.datasources.DataSource$.checkAndGlobPathIfNecessary(DataSource.scala:810)\n\tat org.apache.spark.sql.execution.datasources.DataSource.checkAndGlobPathIfNecessary(DataSource.scala:618)\n\tat org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:438)\n\tat org.apache.spark.sql.DataFrameReader.loadV1Source(DataFrameReader.scala:384)\n\tat org.apache.spark.sql.DataFrameReader.$anonfun$load$2(DataFrameReader.scala:373)\n\tat scala.Option.getOrElse(Option.scala:189)\n\tat org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:373)\n\tat org.apache.spark.sql.DataFrameReader.csv(DataFrameReader.scala:825)\n\tat sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\n\tat sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)\n\tat sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\n\tat java.lang.reflect.Method.invoke(Method.java:498)\n\tat py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)\n\tat py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:380)\n\tat py4j.Gateway.invoke(Gateway.java:295)\n\tat py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)\n\tat py4j.commands.CallCommand.execute(CallCommand.java:79)\n\tat py4j.GatewayConnection.run(GatewayConnection.java:251)\n\tat java.lang.Thread.run(Thread.java:750)\nCaused by: shaded.databricks.org.apache.hadoop.fs.azure.AzureException: Unable to access container sparkcontainer in account STORAGE-ACCOUNT-NAME.blob.core.windows.net using anonymous credentials, and no credentials found for them  in the configuration.\n\tat shaded.databricks.org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.connectUsingAnonymousCredentials(AzureNativeFileSystemStore.java:814)\n\tat shaded.databricks.org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.createAzureStorageSession(AzureNativeFileSystemStore.java:1058)\n\t... 30 more\nCaused by: hadoop_azure_shaded.com.microsoft.azure.storage.StorageException: \n\tat hadoop_azure_shaded.com.microsoft.azure.storage.StorageException.translateException(StorageException.java:87)\n\tat hadoop_azure_shaded.com.microsoft.azure.storage.core.ExecutionEngine.executeWithRetry(ExecutionEngine.java:220)\n\tat hadoop_azure_shaded.com.microsoft.azure.storage.blob.CloudBlobContainer.exists(CloudBlobContainer.java:744)\n\tat hadoop_azure_shaded.com.microsoft.azure.storage.blob.CloudBlobContainer.exists(CloudBlobContainer.java:731)\n\tat shaded.databricks.org.apache.hadoop.fs.azure.StorageInterfaceImpl$CloudBlobContainerWrapperImpl.exists(StorageInterfaceImpl.java:218)\n\tat shaded.databricks.org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.connectUsingAnonymousCredentials(AzureNativeFileSystemStore.java:807)\n\t... 31 more\nCaused by: java.net.UnknownHostException: STORAGE-ACCOUNT-NAME.blob.core.windows.net\n\tat java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:184)\n\tat java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)\n\tat java.net.Socket.connect(Socket.java:613)\n\tat sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:293)\n\tat sun.net.NetworkClient.doConnect(NetworkClient.java:175)\n\tat sun.net.www.http.HttpClient.openServer(HttpClient.java:463)\n\tat sun.net.www.http.HttpClient.openServer(HttpClient.java:558)\n\tat sun.net.www.protocol.https.HttpsClient.&lt;init&gt;(HttpsClient.java:264)\n\tat sun.net.www.protocol.https.HttpsClient.New(HttpsClient.java:367)\n\tat sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.getNewHttpClient(AbstractDelegateHttpsURLConnection.java:203)\n\tat sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1167)\n\tat sun.net.www.protocol.http.HttpURLConnection$6.run(HttpURLConnection.java:1051)\n\tat sun.net.www.protocol.http.HttpURLConnection$6.run(HttpURLConnection.java:1049)\n\tat java.security.AccessController.doPrivileged(Native Method)\n\tat java.security.AccessController.doPrivilegedWithCombiner(AccessController.java:784)\n\tat sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1048)\n\tat sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:189)\n\tat sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1577)\n\tat sun.net.www.protocol.http.HttpURLConnection.access$200(HttpURLConnection.java:97)\n\tat sun.net.www.protocol.http.HttpURLConnection$9.run(HttpURLConnection.java:1497)\n\tat sun.net.www.protocol.http.HttpURLConnection$9.run(HttpURLConnection.java:1495)\n\tat java.security.AccessController.doPrivileged(Native Method)\n\tat java.security.AccessController.doPrivilegedWithCombiner(AccessController.java:784)\n\tat sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1494)\n\tat java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480)\n\tat sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:352)\n\tat hadoop_azure_shaded.com.microsoft.azure.storage.core.ExecutionEngine.executeWithRetry(ExecutionEngine.java:115)\n\t... 35 more\n</div>",
       "errorSummary": "shaded.databricks.org.apache.hadoop.fs.azure.AzureException: shaded.databricks.org.apache.hadoop.fs.azure.AzureException: Unable to access container sparkcontainer in account STORAGE-ACCOUNT-NAME.blob.core.windows.net using anonymous credentials, and no credentials found for them  in the configuration.",
       "errorTraceType": "html",
       "metadata": {},
       "type": "ipynbError"
      }
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "dfDelays = spark.read.csv(\"wasbs://\" + containerName + \"@\" + accountName + \".blob.core.windows.net/FlightsAndWeather/*/*/FlightsAndWeather.csv\",\n",
    "                    schema=data_schema,\n",
    "                    sep=\",\",\n",
    "                    header=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "577e9746-92df-4f1e-aa8d-f35479d8d8ed",
     "showTitle": false,
     "title": ""
    }
   },
   "source": [
    "Load the trained machine learning model you created earlier in the lab"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "2bcea984-f39f-4db9-a284-0839a085a02e",
     "showTitle": false,
     "title": ""
    }
   },
   "outputs": [
    {
     "output_type": "display_data",
     "data": {
      "text/html": [
       "<style scoped>\n  .ansiout {\n    display: block;\n    unicode-bidi: embed;\n    white-space: pre-wrap;\n    word-wrap: break-word;\n    word-break: break-all;\n    font-family: \"Source Code Pro\", \"Menlo\", monospace;;\n    font-size: 13px;\n    color: #555;\n    margin-left: 4px;\n    line-height: 19px;\n  }\n</style>"
      ]
     },
     "metadata": {
      "application/vnd.databricks.v1+output": {
       "arguments": {},
       "data": "",
       "errorSummary": "Command skipped",
       "errorTraceType": "html",
       "metadata": {},
       "type": "ipynbError"
      }
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Load the saved pipeline model\n",
    "model = PipelineModel.load(\"/flightDelayModel\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "69f14192-9349-43ff-a749-cf981c33e8b3",
     "showTitle": false,
     "title": ""
    }
   },
   "source": [
    "Make a prediction against the loaded data set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "0368152f-e893-4df1-8df9-b7b291733ef4",
     "showTitle": false,
     "title": ""
    }
   },
   "outputs": [
    {
     "output_type": "display_data",
     "data": {
      "text/html": [
       "<style scoped>\n  .ansiout {\n    display: block;\n    unicode-bidi: embed;\n    white-space: pre-wrap;\n    word-wrap: break-word;\n    word-break: break-all;\n    font-family: \"Source Code Pro\", \"Menlo\", monospace;;\n    font-size: 13px;\n    color: #555;\n    margin-left: 4px;\n    line-height: 19px;\n  }\n</style>"
      ]
     },
     "metadata": {
      "application/vnd.databricks.v1+output": {
       "arguments": {},
       "data": "",
       "errorSummary": "Command skipped",
       "errorTraceType": "html",
       "metadata": {},
       "type": "ipynbError"
      }
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Make a prediction against the dataset\n",
    "prediction = model.transform(dfDelays)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "876d4509-f7c8-4864-a79b-318e96550bfe",
     "showTitle": false,
     "title": ""
    }
   },
   "source": [
    "Save the scored data into a new global table called **scoredflights**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "440984c3-02ad-4e0f-8a0e-2c9cbe0097aa",
     "showTitle": false,
     "title": ""
    }
   },
   "outputs": [
    {
     "output_type": "display_data",
     "data": {
      "text/html": [
       "<style scoped>\n  .ansiout {\n    display: block;\n    unicode-bidi: embed;\n    white-space: pre-wrap;\n    word-wrap: break-word;\n    word-break: break-all;\n    font-family: \"Source Code Pro\", \"Menlo\", monospace;;\n    font-size: 13px;\n    color: #555;\n    margin-left: 4px;\n    line-height: 19px;\n  }\n</style>"
      ]
     },
     "metadata": {
      "application/vnd.databricks.v1+output": {
       "arguments": {},
       "data": "",
       "errorSummary": "Command skipped",
       "errorTraceType": "html",
       "metadata": {},
       "type": "ipynbError"
      }
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "prediction.write.mode(\"overwrite\").saveAsTable(\"scoredflights\")"
   ]
  }
 ],
 "metadata": {
  "application/vnd.databricks.v1+notebook": {
   "dashboards": [],
   "language": "python",
   "notebookMetadata": {
    "pythonIndentUnit": 4
   },
   "notebookName": "01 Deploy for Batch Scoring",
   "notebookOrigID": 1036921577453791,
   "widgets": {}
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}