Deployed 068093e with MkDocs version: 1.5.3

CS231n/lec10/lec/index.html

@@ -3924,6 +3924,17 @@
 
 
 
+        <label class="md-nav__link md-nav__link--active" for="__toc">
+
+
+  <span class="md-ellipsis">
+    Lecture 10 Video Understanding
+  </span>
+
+
+          <span class="md-nav__icon md-icon"></span>
+        </label>
+
       <a href="./" class="md-nav__link md-nav__link--active">
 
 
@@ -3934,6 +3945,121 @@
 
       </a>
 
+
+
+<nav class="md-nav md-nav--secondary" aria-label="目录">
+
+
+
+
+
+
+    <label class="md-nav__title" for="__toc">
+      <span class="md-nav__icon md-icon"></span>
+      目录
+    </label>
+    <ul class="md-nav__list" data-md-component="toc" data-md-scrollfix>
+
+        <li class="md-nav__item">
+  <a href="#late-fusion-approach" class="md-nav__link">
+    Late Fusion Approach
+  </a>
+
+</li>
+
+        <li class="md-nav__item">
+  <a href="#early-fusion-approach" class="md-nav__link">
+    Early Fusion Approach
+  </a>
+
+</li>
+
+        <li class="md-nav__item">
+  <a href="#comparison" class="md-nav__link">
+    Comparison
+  </a>
+
+    <nav class="md-nav" aria-label="Comparison">
+      <ul class="md-nav__list">
+
+          <li class="md-nav__item">
+  <a href="#general-comparison" class="md-nav__link">
+    General Comparison
+  </a>
+
+</li>
+
+          <li class="md-nav__item">
+  <a href="#2d-conv-early-fusion-vs-3d-conv-3d-cnn" class="md-nav__link">
+    2D Conv (Early Fusion) vs 3D Conv (3D CNN)
+  </a>
+
+</li>
+
+          <li class="md-nav__item">
+  <a href="#c3d-the-vgg-of-3d-cnns" class="md-nav__link">
+    C3D : The VGG of 3D CNNs
+  </a>
+
+</li>
+
+          <li class="md-nav__item">
+  <a href="#measuring-motion-optical-flow" class="md-nav__link">
+    Measuring Motion : Optical Flow
+  </a>
+
+    <nav class="md-nav" aria-label="Measuring Motion : Optical Flow">
+      <ul class="md-nav__list">
+
+          <li class="md-nav__item">
+  <a href="#two-stream-networks" class="md-nav__link">
+    Two Stream Networks
+  </a>
+
+</li>
+
+          <li class="md-nav__item">
+  <a href="#modeling-long-term-temporal-structure" class="md-nav__link">
+    Modeling long-term temporal structure
+  </a>
+
+</li>
+
+          <li class="md-nav__item">
+  <a href="#recurrent-convolutional-network" class="md-nav__link">
+    Recurrent Convolutional Network
+  </a>
+
+</li>
+
+          <li class="md-nav__item">
+  <a href="#spatio-temporal-self-attention-non-local-block" class="md-nav__link">
+    Spatio-Temporal Self-Attention (Non Local Block)
+  </a>
+
+</li>
+
+          <li class="md-nav__item">
+  <a href="#inflating-2d-networks-to-3d-i3d" class="md-nav__link">
+    Inflating 2D Networks to 3D (I3D)
+  </a>
+
+</li>
+
+      </ul>
+    </nav>
+
+</li>
+
+      </ul>
+    </nav>
+
+</li>
+
+    </ul>
+
+</nav>
+
     </li>
 
 
@@ -4214,6 +4340,110 @@
 
 
 
+    <label class="md-nav__title" for="__toc">
+      <span class="md-nav__icon md-icon"></span>
+      目录
+    </label>
+    <ul class="md-nav__list" data-md-component="toc" data-md-scrollfix>
+
+        <li class="md-nav__item">
+  <a href="#late-fusion-approach" class="md-nav__link">
+    Late Fusion Approach
+  </a>
+
+</li>
+
+        <li class="md-nav__item">
+  <a href="#early-fusion-approach" class="md-nav__link">
+    Early Fusion Approach
+  </a>
+
+</li>
+
+        <li class="md-nav__item">
+  <a href="#comparison" class="md-nav__link">
+    Comparison
+  </a>
+
+    <nav class="md-nav" aria-label="Comparison">
+      <ul class="md-nav__list">
+
+          <li class="md-nav__item">
+  <a href="#general-comparison" class="md-nav__link">
+    General Comparison
+  </a>
+
+</li>
+
+          <li class="md-nav__item">
+  <a href="#2d-conv-early-fusion-vs-3d-conv-3d-cnn" class="md-nav__link">
+    2D Conv (Early Fusion) vs 3D Conv (3D CNN)
+  </a>
+
+</li>
+
+          <li class="md-nav__item">
+  <a href="#c3d-the-vgg-of-3d-cnns" class="md-nav__link">
+    C3D : The VGG of 3D CNNs
+  </a>
+
+</li>
+
+          <li class="md-nav__item">
+  <a href="#measuring-motion-optical-flow" class="md-nav__link">
+    Measuring Motion : Optical Flow
+  </a>
+
+    <nav class="md-nav" aria-label="Measuring Motion : Optical Flow">
+      <ul class="md-nav__list">
+
+          <li class="md-nav__item">
+  <a href="#two-stream-networks" class="md-nav__link">
+    Two Stream Networks
+  </a>
+
+</li>
+
+          <li class="md-nav__item">
+  <a href="#modeling-long-term-temporal-structure" class="md-nav__link">
+    Modeling long-term temporal structure
+  </a>
+
+</li>
+
+          <li class="md-nav__item">
+  <a href="#recurrent-convolutional-network" class="md-nav__link">
+    Recurrent Convolutional Network
+  </a>
+
+</li>
+
+          <li class="md-nav__item">
+  <a href="#spatio-temporal-self-attention-non-local-block" class="md-nav__link">
+    Spatio-Temporal Self-Attention (Non Local Block)
+  </a>
+
+</li>
+
+          <li class="md-nav__item">
+  <a href="#inflating-2d-networks-to-3d-i3d" class="md-nav__link">
+    Inflating 2D Networks to 3D (I3D)
+  </a>
+
+</li>
+
+      </ul>
+    </nav>
+
+</li>
+
+      </ul>
+    </nav>
+
+</li>
+
+    </ul>
+
 </nav>
                   </div>
                 </div>
@@ -4228,13 +4458,70 @@
 
 
 <h1 id="video-understanding">Video Understanding<a class="headerlink" href="#video-understanding" title="Permanent link">&para;</a></h1>
+<h2 id="late-fusion-approach">Late Fusion Approach<a class="headerlink" href="#late-fusion-approach" title="Permanent link">&para;</a></h2>
+<ul>
+<li>Fusing the features from different modalities at the end of the network.<blockquote>
+<p>FC layers may result in overfitting</p>
+</blockquote>
+</li>
+</ul>
+<p><img alt="7" src="../7.png" />
+<img alt="1" src="../1.png" />
+<img alt="2" src="../2.png" /></p>
+<h2 id="early-fusion-approach">Early Fusion Approach<a class="headerlink" href="#early-fusion-approach" title="Permanent link">&para;</a></h2>
+<ul>
+<li>Stakcing through channel dimension.</li>
+<li>Fuse the features Using 2D convolutions.(<span class="arithmatex">\(3T\times D\)</span> )</li>
+</ul>
+<p><img alt="3" src="../3.png" />
+* One layer of temporal processing may not be enough!
+<img alt="4" src="../4.png" /></p>
+<h2 id="comparison"><strong>Comparison</strong><a class="headerlink" href="#comparison" title="Permanent link">&para;</a></h2>
+<h3 id="general-comparison">General Comparison<a class="headerlink" href="#general-comparison" title="Permanent link">&para;</a></h3>
+<p><img alt="5" src="../5.png" />
+<img alt="6" src="../6.png" /></p>
+<h3 id="2d-conv-early-fusion-vs-3d-conv-3d-cnn">2D Conv (Early Fusion) vs 3D Conv (3D CNN)<a class="headerlink" href="#2d-conv-early-fusion-vs-3d-conv-3d-cnn" title="Permanent link">&para;</a></h3>
+<blockquote>
+<p>Details see Slides.
+* 2D : perceptive of features of time at once and slide over the space feature. No temporal shift-invariance! Needs to learn separate filters for the same motion at different times in the clip.</p>
+</blockquote>
+<ul>
+<li>3D : Temporal shift-invariant since each filter slides <strong>over time !</strong></li>
+</ul>
+<h3 id="c3d-the-vgg-of-3d-cnns">C3D : The VGG of 3D CNNs<a class="headerlink" href="#c3d-the-vgg-of-3d-cnns" title="Permanent link">&para;</a></h3>
+<p><img alt="8" src="../8.png" /></p>
+<h3 id="measuring-motion-optical-flow">Measuring Motion : Optical Flow<a class="headerlink" href="#measuring-motion-optical-flow" title="Permanent link">&para;</a></h3>
+<h4 id="two-stream-networks">Two Stream Networks<a class="headerlink" href="#two-stream-networks" title="Permanent link">&para;</a></h4>
+<p><img alt="9" src="../9.png" /></p>
+<h4 id="modeling-long-term-temporal-structure">Modeling long-term temporal structure<a class="headerlink" href="#modeling-long-term-temporal-structure" title="Permanent link">&para;</a></h4>
+<p><img alt="10" src="../10.png" />
+* Sometimes don’t backprop to CNN to save memory; pretrain and use it as a feature extractor : Like using pretrained C3D as a feature extractor.</p>
+<h4 id="recurrent-convolutional-network">Recurrent Convolutional Network<a class="headerlink" href="#recurrent-convolutional-network" title="Permanent link">&para;</a></h4>
+<p><img alt="11" src="../11.png" />
+<img alt="12" src="../12.png" /></p>
+<blockquote>
+<p>Actually not too much used because of the time complexity.(Not good for parallelization) 
+<strong>Sequential processing is not good for parallelization.</strong></p>
+</blockquote>
+<h4 id="spatio-temporal-self-attention-non-local-block">Spatio-Temporal Self-Attention (Non Local Block)<a class="headerlink" href="#spatio-temporal-self-attention-non-local-block" title="Permanent link">&para;</a></h4>
+<p><img alt="13" src="../13.png" /></p>
+<blockquote>
+<p>Trick:Add non-local blocks to 3D CNNs , initilize the weights of the non-local block with all zeros, and fine-tune the network. 
+<img alt="14" src="../14.png" /></p>
+</blockquote>
+<h4 id="inflating-2d-networks-to-3d-i3d">Inflating 2D Networks to 3D (I3D)<a class="headerlink" href="#inflating-2d-networks-to-3d-i3d" title="Permanent link">&para;</a></h4>
+<blockquote>
+<p>Refer to the slides for details.
+Trick: Pretrain 2D CNNs and inflate them to 3D CNNs by repeating the weights along the temporal dimension.</p>
+</blockquote>
+<p><img alt="15" src="../15.png" /></p>
 
   <hr>
 <div class="md-source-file">
   <small>
 
       最后更新:
-      <span class="git-revision-date-localized-plugin git-revision-date-localized-plugin-datetime">2024年4月17日 10:15:44</span>
+      <span class="git-revision-date-localized-plugin git-revision-date-localized-plugin-datetime">2024年4月21日 10:20:14</span>
 
         <br>
         创建日期: