ACDiT - Interpolating Autoregressive Conditional Modeling and Diffusion Transformer

Hu et al. 2025

code

---

Hozy Summary

• Settings)
  • \(B\) : the number of block, i.e. a group of frames
  • \(L\) : the length of the sequence composed with blocks
  • \(T\) : the diffusion timestep

• Model setting)
  • AR Diffusion
  • Conditioned on the ground truth image of the previous blocks
    • i.e.) Teacher Forcing
    • Implemented with a causal mask that attends…
      1. All previous clean ground truth images : \(c_{\lt i}\)
      2. Current noisy generating image : \(n_{i}\)
• Analysis)
  • Comparison with CA2-VDM
    • CA2-VDM ensures that the \(i\)-th frame should be conditioned only on \(\lt i\) frames.
    • In ACDiT, frames in a same block refers to each other, so some earlier frames attends to the future frames in its block.

---

3. Prerequisite

3.1 Autoregressive Modeling

• Def.)
  • \((x_1, \ldots, x_n)\) : sequence of tokens
    • where the likelihood of \(x_i\) is described as \(\displaystyle p(x_1, \ldots, x_N) = \prod_{i=1}^N p(x_i\mid x_{\lt i})\)

3.2 Diffusion

• Def.)
  • \(x_0\sim p(x)\) : the initial data
  • Forward Process
    • \(x^{(t)} = \sqrt{\alpha^{(t)}} x^{(t-1)} + \sqrt{1-\alpha^{(t)}}\epsilon^{(t)}\) : the noised data at \(t\)
      • where \(\epsilon^{(t)}\sim\mathcal{N}(\mathbf{0,I})\)
      • i.e.) sampled with \(q\left( x^{(t)}\mid x^{(t-1)} \right) = \mathcal{N}\left( x^{(t)};\; \sqrt{\alpha^{(t)}} x^{(0)}, (1-\alpha^{(t)})\mathbf{I} \right)\)
  • Parameterized Reverse Process
    • \(p_\theta\left( x^{(t-1)} \mid x^{(t)} \right) = \mathcal{N}\left( \mu_\theta(x^{(t)}), \beta^{(t)}\mathbf{I} \right)\).
  • \(\epsilon\)-prediction Objective
    • \(\mathcal{L}_\theta = \mathbb{E}_{t\sim\mathcal{U}[0,1], \epsilon\sim\mathcal{N}(\mathbf{0,I})} \left\Vert \epsilon_\theta (x^{(t)}, t) - \epsilon^{(t)} \right\Vert_2\).

4. ACDiT

4.1 Desiderata for Autoregressive Diffusion Model

• Def.)
  • \(\mathcal{L}_\theta = \displaystyle\mathbb{E}_{t\sim\mathcal{U}[0,1], \epsilon\sim\mathcal{N}(\mathbf{0,I})} \sum_{i=1}^N \bigg\Vert \epsilon_\theta (x_i^{(t)}, t, \underbrace{x_{\lt i}}_{\text{cond'ed}}) - \epsilon^{(t)} \bigg\Vert_2\).
    • where \(\{x_1, \ldots, x_N \}\) are \(N\) autoregressive units.
• Desiderata)
  1. The generation of future elements should be predicated on a precise representation of antecedent sequences.
     • cf.)
       • Diffusion Forcing does not satisfy this.
  2. Both the autoregressive modeling and the denoising process should optimally utilize the entire parameter space of the neural network.
  3. The denoising process should directly attend comprehensively to the entire sequence of past sequences.
     • cf.)
       • MAR does not satisfy this.

4.2 Framework

• Settings)
  • \(x_i\) : an AR unit
    • \(c_i\) : the clean version \(x_i\)
    • \(n_i\) : the corrupted version \(x_i^{(t)}\)
• Model)
  • Then the model learns \(\epsilon_\theta\left( n_i^{(t)};\; t, c_{\lt i} \right)\).
    • Both \(n_i\) and \(c_i\) attend to all preceding clean blocks \(\{c_j\mid j=0,\ldots,i-1\}\)
    • Architecturally implemented using the causal mask below.
      • KV cache on \(\{c_j\mid j=0,\ldots,i-1\}\)
    • When \(c_i\) is fully denoised, it is added to the clean block (KV cache) for the denoising of \(n_{i+1}\)

• Sampling)

4.3 RoPE-ND

4.4 Block Size Choice

4.5 Implementation Details