Illustrated portrait of Taishi Nakashima Taishi Nakashima

Assistant Professor
Tokyo Metropolitan University
Department of Computer Science
Ono Laboratory

Search this site

Implementing online source separation

Frequency-domain inputs and outputs

Let the STFT observation be xf,t∈CM\mathbf{x} _ {f,t} \in \mathbb{C}^{M}. Here ff is the frequency bin, tt is the frame and MM is the microphone count. For NN outputs, define the demixing matrix Wf,t∈CN×M\mathbf{W} _ {f,t} \in \mathbb{C}^{N \times M} with the following orientation:

yf,t=Wf,txf,t.\mathbf{y} _ {f,t} = \mathbf{W} _ {f,t} \mathbf{x} _ {f,t} .

Each row already contains the conjugate-transposed coefficients in this convention. The code therefore applies the matrix directly, without conjugating it again.

Array shapes

Removing the time axis from a single-frame function makes its state interface easier to follow.

QuantityArrayShape
Observationx(F, M)
Demixing matrixW(F, N, M)
Observation covariancecovariance(F, M, M)
Outputy(F, N)

State carried between frames

An exponential average of observation outer products updates the observation covariance as follows. The forgetting factor satisfies 0≤β<10 \leq \beta \lt 1.

Cf,t=βCf,t−1+(1−β)xf,txf,tH.\mathbf{C} _ {f,t} = \beta \mathbf{C} _ {f,t-1} + (1 - \beta) \mathbf{x} _ {f,t} \mathbf{x} _ {f,t}^{\mathrm{H}} .

The following example updates that covariance and applies a supplied demixing matrix. It does not contain a learning rule for the matrix and cannot learn source separation by itself. The code requires Python and NumPy.

import numpy as np


def process_frame(x, W, covariance, beta=0.98):
    """x: (F, M), W: (F, N, M), covariance: (F, M, M)."""
    if not 0.0 <= beta < 1.0:
        raise ValueError("beta must satisfy 0 <= beta < 1")

    outer = x[:, :, None] * x[:, None, :].conj()
    covariance = beta * covariance + (1.0 - beta) * outer
    y = np.einsum("fnm,fm->fn", W, x)
    return y, covariance

Initial state

This example uses five frequency bins, two microphones and two outputs. Identity initialization makes the output equal to the observation. The input checks connectivity and shapes; it is not audio for evaluating separation quality.

F, M, N = 5, 2, 2
x = np.ones((F, M), dtype=np.complex128)
W = np.broadcast_to(np.eye(N, M), (F, N, M)).copy()
covariance = np.zeros((F, M, M), dtype=np.complex128)

y, covariance = process_frame(x, W, covariance)
print(y.shape, covariance.shape)

Streaming boundaries

Reset the demixing matrix, statistics, and STFT and inverse-STFT buffers when moving to a different recording. Carry those states across chunks of the same recording. Define how the final chunk is padded and how the output is trimmed.

To check online behavior, compare inputs with the same past and different futures. Use the same permitted lookahead and check whether earlier outputs change when samples beyond it change. Inspect centered STFT and normalization for unintended access to future samples.

Records to retain for evaluation

Save input and output channel ordering, scale restoration, the warm-up interval and per-frame processing time. Associate audio quality metrics and timing with the same input identity. See Managing audio signal processing experiments for recording execution conditions.

References

Search this site

Enter words to search this site.

    Search this site →