Ca2-VDM - Efficient Autoregressive Video Diffusion Model with Causal Generation and Cache Sharing

Gao et al. 2025

code

---

Hozy Summary

---

Limits of the Existing VDMs & Ca2-VDM’s Suggestion

1. Cache Computation
   • Problem Summary
     • Let \(z_t^i\) be the \(i\)-th token from the \(L\)-length sequence at the denoising timestep of \(t\).
     • Then, \(z_t^{3,4}\) are denoised conditioned on \(z_t^{0,1,2}\).
     • Also, the key, value features of \(z_t^{0,1,2}\) are computed conditioned on \(z_t^{3,4}\).
     • Thus, when computing the cache of \(z_t^{0,1,2}\), \(z_t^{3,4}\) is still unavailable.
   • Paper’s Solution : Causal Generation
     • Replace full temporal attention in each block with causal temporal attention.
       • i.e.) Each generated frame only depends on its prefix frames.
     • Propose prefix-enhanced spatial attention.
       • cf.) Enhances the guidance from the prefix frames.
     • The above two makes cache computations obtainable at early steps.
2. Cache Storage
   • Problem Summary
     • Majority of the existing models use the same timestep embedding for both conditional and noisy frames.
     • Thus, it requires respective cache values for each timestep \(t\).
   • Paper’s Solution : Cache Sharing
     • Compute cache only with non-noisy preceding conditional frames.
     • Subsequent noisy frames should not affect the cache.
     • To accomplish this, assign \(t=0\) as a distinct timestep embedding fo the conditional frames, and share them in both training and inference.

3. Method

3.1 Preliminaries

• Diffusion : DDPM
• Problem Formulation : VDM
  • VAE
    • Encoding
      • \(z_0 = \mathcal{E}(x_0)\).
      • Forward \(q(z_t\mid z_{t-1})\) and denoising process \(p_\theta(z_{t-1}\mid z_{t})\) in the latent space
    • Decoding
      • \(\hat{x}_0 = \mathcal{D}(\hat{z}_0)\) for the denoised latent \(\hat{z}_0\).
  • Latent Space
    • \(z_0^{0:L} = \left[ z_0^0, \ldots, z_0^{L-1} \right] \in\mathbb{R}^{L\times H\times W\times C}\) where
      • \(L\) : number of frames
      • \(H\times W\) : downsampled resolution
      • \(C\) : channel size
  • Parameterization
    • \(p_\theta(z_0^{P:L} \mid z_0^{0:P})\) : the learning distribution where
      • \(P\) : the number of prefix frames serves as condition (clean prefix)
        • \(L-P\) frames are the denoising targets.

3.2 Causal Generation and Training Objectives

• Settings)
  • Randomly keep \(P\) consecutive frames uncorrupted as clean prefix.
    • Remaining frames are the denoising targets.
  • Separated timesteps
    • \(\text{tEmb}(0)\) for the clean prefix
    • \(\text{tEmb}(t)\) for the denoising target
• Training Objective)
  • \(\tilde{\mathcal{L}}_{\text{simple}}(\theta) = \mathbb{E}_{z,\epsilon, t} \bigg[ \Big\Vert \Big( \epsilon_\theta\big( \underbrace{\left[ z_0^{0:P}, z_t^{P:L} \right]}_{\text{concat}}, t \big) - \epsilon \Big) \odot m \Big\Vert_2^2 \bigg]\) where
    • \([\cdot, cdot]\) : concatenation along the temporal axis
    • \(t\) : the timestep vector with \(t_i = \begin{cases} t & \text{if } i\ge P \\ 0 \text{otherwise} \end{cases}\)
  • In practice, they trained the combination of \(\tilde{\mathcal{L}}_{\text{simple}}\) and ELBO \(\mathcal{L}_{\text{vlb}}\)
• Causal Temporal Attention)

• Prefix-Enhanced Spatial Attention)

3.3 Autoregressive Inference with Cache Sharing