Fixing the comparison conditions
Record the sampling rate, STFT window and hop lengths, and input ordering for every experiment. For randomized operations, also record the seed, generator and execution environment. A shared seed alone does not guarantee identical results across libraries or machines.
sample_rate = 16000
fft_size = 1024
hop_size = 256
seed = 42
[evaluation]
warmup_frames = 100
Save this configuration as config.toml.
These values are examples for the lesson; choose them for the recordings and method being evaluated.
Identifying the inputs
Record both the path and SHA-256 of each audio file to track content changes even when filenames stay the same. Associate microphone ordering, excerpt boundaries, resampling and normalization settings with the processed audio.
Saving an execution record
The following code uses only the Python 3.11+ standard library.
It saves the configuration and input hash in a new run directory.
Run it with an actual recording, for example python record_run.py config.toml mixture.wav.
import hashlib
import json
import platform
import sys
import tomllib
import uuid
from datetime import datetime, timezone
from pathlib import Path
config_path, audio_path = map(Path, sys.argv[1:])
config_bytes = config_path.read_bytes()
config = tomllib.loads(config_bytes.decode("utf-8"))
with audio_path.open("rb") as stream:
audio_hash = hashlib.file_digest(stream, "sha256").hexdigest()
run_id = uuid.uuid4().hex
run_dir = Path("runs") / run_id
run_dir.mkdir(parents=True, exist_ok=False)
record = {
"run_id": run_id,
"created_at": datetime.now(timezone.utc).isoformat(),
"config": config,
"input": {"path": str(audio_path), "sha256": audio_hash},
"python": platform.python_version(),
}
(run_dir / "config.toml").write_bytes(config_bytes)
(run_dir / "run.json").write_text(
json.dumps(record, ensure_ascii=False, indent=2) + "\n",
encoding="utf-8",
)
Keep the code revision, dependency lock file and execution command in the same directory to document the conditions needed for a rerun.
Associate subsequent results with this run_id and the input identity.
Aggregating processing times
For audio duration and processing time , define the real-time factor (RTF) as
State whether timing includes file loading and STFT, and use the same scope when comparing methods. For online processing, save per-frame processing times and their maximum in addition to the mean RTF. A mean RTF below one can still hide individual frames that miss their deadlines.
The frame processing budget
With hop length and sampling rate , frames arrive every . The example configuration gives 16 ms. Treat this interval separately from windowing and input/output buffer latency.