Audio Generation

Audio generation is the use of generative machine-learning models to synthesise sound — including speech, music, sound effects, and ambient audio — from inputs such as text, symbolic scores, or learned latent representations. Models learn the statistical structure of audio either in the raw waveform domain or in compressed time-frequency representations such as spectrograms, then sample new outputs that are perceptually realistic. The field encompasses text-to-speech, music generation, voice cloning, and general audio synthesis, drawing on autoregressive, diffusion, and adversarial generative architectures.

Semantic Classification

Content

Compositional Relationships (Components)

SubClassOf(ai:AudioGeneration
  ObjectSomeValuesFrom(ai:hasPart ai:NeuralVocoder))
SubClassOf(ai:AudioGeneration
  ObjectSomeValuesFrom(ai:hasPart ai:AudioCodec))
SubClassOf(ai:AudioGeneration
  ObjectSomeValuesFrom(ai:hasPart ai:Spectrogram))
SubClassOf(ai:AudioGeneration
  ObjectSomeValuesFrom(ai:hasPart ai:ConditioningModule))
SubClassOf(ai:AudioGeneration
  ObjectSomeValuesFrom(ai:hasPart ai:LatentEncoder))

Dependency Relationships

SubClassOf(ai:AudioGeneration
  ObjectSomeValuesFrom(ai:requires ai:DeepLearning))
SubClassOf(ai:AudioGeneration
  ObjectSomeValuesFrom(ai:requires ai:AudioProcessing))
SubClassOf(ai:AudioGeneration
  ObjectSomeValuesFrom(ai:requires ai:GPUCompute))
SubClassOf(ai:AudioGeneration
  ObjectSomeValuesFrom(ai:requires ai:TrainingCorpus))
SubClassOf(ai:AudioGeneration
  ObjectSomeValuesFrom(ai:dependsOn ai:NeuralNetwork))
SubClassOf(ai:AudioGeneration
  ObjectSomeValuesFrom(ai:dependsOn ai:AudioFeatureExtraction))
SubClassOf(ai:AudioGeneration
  ObjectSomeValuesFrom(ai:dependsOn ai:ResidualVectorQuantisation))

Capability Relationships

SubClassOf(ai:AudioGeneration
  ObjectSomeValuesFrom(ai:enables ai:TextToSpeech))
SubClassOf(ai:AudioGeneration
  ObjectSomeValuesFrom(ai:enables ai:SpeechSynthesis))
SubClassOf(ai:AudioGeneration
  ObjectSomeValuesFrom(ai:enables ai:VoiceCloning))
SubClassOf(ai:AudioGeneration
  ObjectSomeValuesFrom(ai:enables ai:MusicGeneration))
SubClassOf(ai:AudioGeneration
  ObjectSomeValuesFrom(ai:enables ai:SpatialAudioSynthesis))
SubClassOf(ai:AudioGeneration
  ObjectSomeValuesFrom(ai:supports ai:MultimodalAI))
SubClassOf(ai:AudioGeneration
  ObjectSomeValuesFrom(ai:supports ai:ExtendedReality))

Implementation Relationships

SubClassOf(ai:AudioGeneration
  ObjectSomeValuesFrom(ai:implements ai:GenerativeModel))
SubClassOf(ai:AudioGeneration
  ObjectSomeValuesFrom(ai:implements ai:DiffusionModel))
SubClassOf(ai:AudioGeneration
  ObjectSomeValuesFrom(ai:implements ai:AutoregressiveModel))
SubClassOf(ai:AudioGeneration
  ObjectSomeValuesFrom(ai:implements ai:GenerativeAdversarialNetwork))
SubClassOf(ai:AudioGeneration
  ObjectSomeValuesFrom(ai:uses ai:TransformerArchitecture))
SubClassOf(ai:AudioGeneration
  ObjectSomeValuesFrom(ai:uses ai:FlowMatching))

Reduction Relationships

SubClassOf(ai:AudioGeneration
  ObjectSomeValuesFrom(ai:reducesTo ai:WaveformModelling))
SubClassOf(ai:AudioGeneration
  ObjectSomeValuesFrom(ai:reducesTo ai:SpectrogramSynthesis))
SubClassOf(ai:AudioGeneration
  ObjectSomeValuesFrom(ai:reducesTo ai:TokenPrediction))
SubClassOf(ai:AudioGeneration
  ObjectSomeValuesFrom(ai:reducesTo ai:LatentDiffusion))
SubClassOf(ai:AudioGeneration
  ObjectSomeValuesFrom(ai:reducesTo ai:AdversarialTraining))

About

• Audio generation is one of the most technically demanding modalities in Generative AI because audio is a high-dimensional, temporally structured signal with perceptual requirements spanning multiple timescales simultaneously — from the fine-grained micro-structure of individual phonemes (millisecond resolution) to large-scale musical phrase structure (tens of seconds), and from instantaneous spectral balance to long-range semantic coherence. A human listener’s auditory system is exquisitely sensitive to artefacts — phase discontinuities, formant distortions, unnatural prosody, pitch instabilities — making perceptual quality assessment substantially harder than in image generation. These constraints have driven an unusually rapid succession of architectural innovations over the past decade, with four broad paradigms now coexisting: autoregressive waveform generation, GAN-based adversarial synthesis, diffusion-based denoising, and flow-matching continuous normalising flows.
• The field stratifies functionally into three major application clusters. Speech-domain generation — Text-to-Speech (TTS), Voice Cloning, prosody transfer, and speaker adaptation — is the most mature cluster, commercially deployed at massive scale by ElevenLabs (which closed a $500MSeriesDatan$11B valuation in February 2026, serving 41% of the Fortune 500 and processing hundreds of millions of synthesis requests monthly), Google, Amazon, and Microsoft. Music-domain generation — Music Generation, loop synthesis, accompaniment generation, and source-conditioned remixing — has seen explosive activity since 2023 through systems such as Meta’s MusicGen and AudioCraft suite, Stability AI’s Stable Audio 3.0, Suno v5, Udio v4, and the open-source ACE-Step v1.5 (which generates a full song in under 2 seconds on an A100 GPU as of early 2026). General audio generation — environmental sounds, foley, sound effects, and ambient soundscapes — has matured as a distinct task with dedicated systems including Meta’s AudioBox and AudioGen. The cross-domain challenge of maintaining prosodic naturalness, long-range coherence, stylistic consistency, and real-time latency simultaneously continues to drive fundamental research at the intersection of signal processing, generative modelling, and Psychoacoustics.
• A decisive architectural shift occurred in 2023–2024 with the near-universal adoption of Residual Vector Quantisation (RVQ) neural audio codecs — notably Meta’s EnCodec and Descript Audio Codec (DAC) — as the discrete representation layer beneath language-model-style autoregressive generators. This replaces direct mel-spectrogram or waveform modelling with a two-stage pipeline: a neural codec compresses audio into discrete token sequences at 50–150 tokens per second, which a language model or diffusion backbone then generates. Systems such as AudioLM, VALL-E, and SoundStorm adopt this codec-language-model architecture, achieving naturalness scores competitive with human speech while enabling zero-shot speaker generalisation from 3-second references.

Components / Architecture

• Waveform Modelling
  • Autoregressive Autoregressive Model approaches (WaveNet 2016; WaveRNN 2018) predict each audio sample conditioned on all previous samples using dilated causal convolutions or gated recurrent units.
  • WaveNet receptive field: 2^(9) × 3 = 1,536 samples at 16 kHz ≈ 96ms — sufficient to capture phoneme transitions. WaveNet achieves near-human naturalness (MOS 4.21 vs human 4.55 on TTS samples) but requires O(N) sequential steps for N samples.
  • WaveRNN reduces the sequential generation bottleneck to a single GRU step per sample via subscale generation (generating 16 samples per step in parallel), achieving ~3ms/sample on V100 versus ~250ms/sample for WaveNet.
  • Parallel WaveNet (van den Oord et al., 2018) uses inverse autoregressive flow (IAF) to enable parallel synthesis at inference time while maintaining WaveNet’s quality, at the cost of two-model training (teacher-student knowledge distillation).
  • These autoregressive waveform approaches have largely been superseded in production TTS by GAN vocoders and flow-matching models for latency-sensitive applications, but remain theoretically important for high-quality offline synthesis.
• Spectrogram Synthesis and Neural Vocoding
  • The dominant speech synthesis pipeline separates acoustic modelling (predicting a mel-spectrogram from text) from vocoding (reconstructing the waveform from the spectrogram), enabling independent optimisation of each stage.
  • Acoustic models: Tacotron 2 (seq2seq attention RNN, 2018, MOS 4.44); FastSpeech 2 (feed-forward with duration/pitch/energy predictors, 2021, 38x faster inference); VITS (end-to-end VAE + flow + GAN, 2021, MOS 4.43, single model for full pipeline).
  • Neural vocoders (GAN-based): HiFi-GAN (MPD + MSD discriminators, 2020, MOS 4.37, RTF >100x on V100); UnivNet (multi-resolution spectrogram discriminator, 2021, better high-frequency reconstruction); BigVGAN (periodic activation functions, anti-aliased multi-period discriminator, 2022, SOTA MOS 4.42 on LJSpeech).
  • Neural vocoders (flow-based): WaveGlow (normalising flow, 2019, real-time); Vocos (frequency-domain ISTFT decoder, 2023, closes the gap between time-domain and Fourier vocoders, MOS 4.44).
  • The two-stage pipeline decouples the high-level linguistic/prosodic modelling (text → mel) from the fine acoustic detail reconstruction (mel → waveform), enabling both stages to be optimised independently and mixed across models.
• Diffusion-Based Generation
  • Diffusion Model and Latent Diffusion architectures encode audio into a compact latent space via a Variational Autoencoder, then train a U-Net or Diffusion Transformer (DiT) to iteratively denoise latent vectors conditioned on text embeddings.
  • AudioLDM (Liu et al., ICML 2023): text-to-audio latent diffusion using CLAP audio embeddings as conditioning signal; SOTA FAD on AudioCaps at publication.
  • AudioLDM 2 (IEEE/ACM TASLP 2024): unified holistic audio (speech + music + effects) via Language of Audio (LOA) semantic encoder + GPT-2 bridge network that translates any modality to LOA representation, then conditions a latent diffusion U-Net.
  • Stable Audio (Stability AI, 2024): 907M-parameter DiT, timing-conditioned on start/end time enabling precise audio length control; 44.1 kHz stereo; 512-dim latent space from 64x compressing VAE.
  • Stable Audio 3.0 (2025, arXiv:2605.17991): replaces DDPM with flow-matching DiT, 1.3B parameters, 64x Deep Compression AutoEncoder (DCAE); generates coherent 6-minute stereo tracks. Open weights (medium variant) on Hugging Face.
  • Key advantage over autoregressive models: diffusion models can generate variable-length outputs with global structural coherence, essential for long-form music with verse-chorus-bridge structure.
• Flow Matching: Flow Matching continuous normalising flows (Voicebox, 2023; Matcha-TTS, 2024; F5-TTS, 2024) offer a deterministic alternative to diffusion — training an ODE vector field from noise to data distribution. F5-TTS achieves naturalness competitive with Voicebox while reducing inference to 10 NFE (function evaluations), enabling real-time synthesis. Stable Audio 3.0 adopts a flow-matching DiT as its core generation backbone.
• Adversarial Synthesis: Generative Adversarial Network vocoders (MelGAN, 2019; HiFi-GAN, 2020; EnCodec discriminators) use multi-resolution spectral discriminators and feature matching losses to achieve fast, high-fidelity waveform synthesis without iterative sampling. GAN vocoders remain the fastest path to high-quality waveform reconstruction and are used in production TTS at ElevenLabs and Google.
• Neural Audio Codecs and Codec Language Models
  • Residual Vector Quantisation (RVQ) codecs compress 44.1 kHz audio to 8–12 kbps using multi-stage vector quantisation, producing discrete token sequences that language-model-style decoders can predict autoregressively.
  • EnCodec (Meta, 2022, arXiv:2210.13438): 24/48 kHz mono/stereo variants, 1.5–24 kbps, trained with multi-scale STFT discriminator + adversarial discriminators. Real-time at 12x on CPU. Standard discrete vocabulary for codec LMs.
  • Descript Audio Codec — DAC (NeurIPS 2023): 44.1 kHz universal codec for speech, music, and effects at 8 kbps; improved RVQGAN training; improved tonal artefact handling versus EnCodec.
  • FlexiCodec (2025, arXiv:2510.00981): dynamic frame-rate codec supporting multiple bitrates from a single model; optimised for low-frame-rate streaming applications.
  • U-Codec (2025, arXiv:2510.16718): ultra-low frame-rate neural speech codec for fast high-fidelity generation; reduces sequence length for codec LMs, enabling faster autoregressive inference.
  • Codec Language Models: VALL-E (Microsoft, 2023) — GPT-style AR decoder over EnCodec tokens, zero-shot voice cloning MOS 4.58; SoundStorm (Google DeepMind, 2023) — parallel masked diffusion decoder over EnCodec tokens, 30x faster than VALL-E; VoiceCraft (ACL 2024) — token-rearrangement codec LM enabling in-context speech editing.
• Conditioning Mechanisms: All modern architectures condition generation on external signals via cross-Attention Mechanism over embeddings from Transformer Architecture text encoders (CLAP, FLAN-T5, T5), speaker embeddings (x-vectors, d-vectors), MIDI/chord-progressions, tempo, key, and genre tags. Reinforcement Learning from Human Feedback (RLHF) and direct preference optimisation (DPO) fine-tune generation to human aesthetic preferences and prosodic naturalness scores.
• Evaluation Metrics: Fréchet Audio Distance (FAD), Kernel Inception Distance for audio (KID-audio), Mean Opinion Score (MOS) from human raters, DNSMOS, UTMOS, Speaker Encoder Cosine Similarity (SECS) for voice cloning, and task-specific metrics (BLEU for TTS alignment, beat tracking F-score for music) form the standard evaluation battery.

Use Cases / Major Families

• Text-to-Speech and Conversational AI
  • Neural TTS underpins voice assistants (Google Assistant, Amazon Alexa, Siri), audiobook production, customer service automation, accessibility tools for visually impaired users, and real-time conversational AI.
  • ElevenLabs’ Flash v2.5 model achieves sub-75ms end-to-end latency in production, enabling seamless conversational turn-taking. OpenAI’s Voice Mode uses GPT-4o as a joint audio-language model that processes and generates speech directly without a TTS intermediary.
  • Key technical requirements: sub-100ms time-to-first-audio-byte, MOS ≥ 4.0 on target language, prosody naturalness across sentence types (declarative, interrogative, exclamatory), graceful handling of out-of-vocabulary words and domain-specific terminology (medical, legal, technical).
  • Accessibility impact: screen reader voices (NVDA, VoiceOver, JAWS) are transitioning from legacy formant synthesisers to neural TTS. The UK’s RNIB (Royal National Institute of Blind People) advocates for natural neural TTS in assistive technology as part of its AI strategy.
• Voice Cloning and Personalisation
  • Zero-shot voice cloning from 3-second reference audio enables personalised TTS, dubbed content localisation (VoiceCraft-Dub), and — controversially — voice deepfake generation.
  • Technical pipeline: reference audio → speaker encoder (d-vector or x-vector) → conditioning embedding → codec language model or TTS acoustic model → neural vocoder → synthesised speech in target voice.
  • Commercial services: ElevenLabs Voice Library (300,000+ cloned voices), Resemble AI, Respeecher (entertainment industry focus), Eleven Dubbing (video localisation).
  • Legal developments: RIAA filed lawsuits against Suno and Udio in 2024 alleging training data infringement; Warner Music Group settled with Suno by 2025. The US NO FAKES Act (introduced April 2025) and EU AI Act (applicable from August 2026) impose consent requirements and disclosure obligations.
• Music Generation
  • MusicGen (Meta, 2023) generates up to 30 seconds of coherent instrumental music from text prompts at the 3.5B parameter scale. Stable Audio 3.0 generates full 6-minute tracks with structural coherence using a flow-matching DiT.
  • Suno v5 and Udio v4 are commercial systems generating full vocal songs with genre, mood, and lyric conditioning; both have reached millions of monthly active users. Udio v4 adds inpainting, stem separation, and section-by-section regeneration — prefiguring DAW-integrated AI production.
  • ACE-Step v1.5 (February 2026), an open-source foundation model with a 0.6–4B LM planner driving a DiT synthesiser, generates 10-minute compositions in under 10 seconds on a consumer RTX 3090, and supports lyric2vocal, singing2accompaniment, and voice cloning conditioning.
  • Open-source ecosystem: MusicGen, AudioCraft, ACE-Step, audio-diffusion-pytorch, and Riffusion (spectrogram-based diffusion) are available on Hugging Face with permissive or research-only licences.
• Environmental and Foley Audio
  • AudioBox (Meta, 2023) and AudioGen generate environmental sounds and sound effects from natural-language descriptions (e.g. “a dog barking near a busy city street on a rainy day”).
  • Video-to-audio generation systems (Seeing and Hearing, CVPR 2024; Kling-Foley DiT, 2025; V2M-Zero, 2026) synchronise generated audio to video frames, enabling automated foley for film and game production pipelines.
  • Sound design for Extended Reality: procedurally generated ambient audio that responds to user position and gaze in real-time, using generation models conditioned on 3D scene graphs.
  • AudioMoG (2025) introduces mixture-of-guidance conditioning that blends multiple guidance signals (text, image, video) for richer, more controllable environmental audio generation.
• Spatial Audio Generation
  • Spatial Audio synthesis — directional audio rendered for binaural or multi-channel playback — integrates with Extended Reality pipelines.
  • Diff-SAGe (arXiv 2410.11299, 2024) uses end-to-end Diffusion Model for spatial audio generation conditioned on room geometry and source position, enabling dynamic procedural soundscapes in VR/AR.
  • Binaural TTS — spatially localised speech rendered to appear at specific 3D positions relative to the listener — extends conventional TTS into the spatial domain, critical for XR conference systems and virtual presence applications.
  • Higher-order Ambisonics (HOA) generation from mono or stereo sources using neural up-mixing networks enables spatial audio for broadcast and streaming platforms without object-based authoring tools.

Mathematical and Algorithmic Foundations

• Autoregressive Waveform Modelling: WaveNet parametrises the joint distribution of an audio waveform x = (x₁, x₂, …, xT) as a product of conditional distributions p(x) = ∏ p(xₜ | x₁, …, xₜ₋₁), implemented by a stack of dilated causal convolution layers with exponentially increasing dilation rates (1, 2, 4, …, 512, repeated in blocks), covering a receptive field of ≥32,000 samples at 16 kHz. Each layer applies a gated activation function z = tanh(Wf * x) ⊙ σ(Wg * x) — the same gating mechanism as LSTM cells applied in the convolutional domain. Computational complexity is O(T × L × k) for sequence length T, L layers, and kernel size k. At 16 kHz, generating one second of audio requires 16,000 sequential forward passes, making real-time WaveNet inference on CPU infeasible without model distillation.
• GAN Vocoder Objective: HiFi-GAN’s generator G learns a mapping from mel-spectrogram M to waveform x̂ = G(M; θ_G). Training minimises a combination of: (i) adversarial loss L_adv = E[(D(x)-1)² + D(x̂)²] using multiple discriminators D; (ii) feature matching loss L_fm = Σ_l ||D_l(x) - D_l(x̂)||₁ penalising mismatches in intermediate discriminator features; (iii) mel-spectrogram reconstruction loss L_mel = ||φ(x) - φ(x̂)||₁ where φ is the mel-filterbank transform. The multi-period discriminator (MPD) evaluates waveform segments at periods p ∈ {2, 3, 5, 7, 11} to capture different temporal structures. HiFi-GAN-V1 (93M parameters) achieves MOS of 4.37 at inference speeds exceeding 167x real-time on a V100 GPU.
• Diffusion Score Matching: Score-based generative models define a forward noising process q(xₜ|x₀) = N(xₜ; √ᾱₜx₀, (1-ᾱₜ)I) that progressively adds Gaussian noise across T timesteps. The model learns to reverse this by predicting the noise ε̂ = ε_θ(xₜ, t) minimising the denoising score matching objective L_DDPM = Eₜ,x₀,ε[||ε - ε_θ(√ᾱₜx₀ + √(1-ᾱₜ)ε, t)||²]. In the latent domain (AudioLDM), the denoising operates on compressed latent representations z = E(x) from a pre-trained Variational Autoencoder with encoder E and decoder D, reducing the spatial dimension by a factor of 4–16 and making generation 4–16× more compute-efficient than pixel-/sample-space diffusion.
• Flow Matching: Continuous normalising flows parametrise a time-dependent vector field vₜ(x; θ) whose integration x₁ = x₀ + ∫₀¹ vₜ(x; θ)dt transforms a Gaussian source distribution p₀ into the data distribution p₁. The flow matching loss L_FM = E_{t,x₀,x₁}[||vₜ(xₜ; θ) - (x₁ - x₀)||²] trains the vector field by interpolating between source and target samples along linear paths xₜ = (1-t)x₀ + tx₁. At inference, an ODE solver (Euler with 10–50 steps, or Heun’s method) traces the flow from noise to audio. Flow matching converges faster than DDPM and requires far fewer NFEs (number of function evaluations) for high-quality synthesis.
• Residual Vector Quantisation: RVQ compresses a continuous audio embedding e ∈ ℝᵈ to K codebooks of size N, iteratively quantising residuals: r₀ = e, qₖ = argmin_cⱼ ||rₖ - cⱼ||, rₖ₊₁ = rₖ - qₖ. The discrete token sequence (q₁, …, qK) at 75 frames/second with K=8 codebooks of 1024 codes each encodes 44.1 kHz audio at ≈8 kbps. The codebook is trained jointly with a convolutional encoder-decoder using commitment loss, adversarial loss from multi-scale spectral discriminators, and reconstruction loss in both time and frequency domains. RVQ bottleneck tokens are the input vocabulary for codec language model generation systems.

Training Data, Corpora, and Benchmarks

• Speech Synthesis Datasets: LJSpeech (24 hours, single English female speaker, MOS ground-truth 4.44) is the canonical TTS development benchmark. LibriSpeech (960 hours, 2484 speakers) and VCTK (44 hours, 110 speakers) support multi-speaker TTS research. For multilingual TTS, MAILABS (700+ hours, 9 languages) and Mozilla Common Voice (50,000+ hours, 120+ languages, crowd-sourced) provide training data. Large-scale weakly-supervised corpora — such as Whisper’s 5M-hour training set drawn from internet audio — enable zero-shot cross-lingual generalisation.
• Music Generation Datasets: FMA (Free Music Archive, 343 hours, 104 genres), MagnaTagATune (25,863 clips, 188 tags), NSynth (300,000 musical notes, 1006 instruments), MusicNet (330 classical recordings with MusicXML alignment), and AudioSet’s music subset provide diverse training material. Proprietary datasets used by Suno, Udio, and Stable Audio are estimated to be 100,000+ hours of commercially licensed or scraped music. MUSDB18HQ (150 tracks, 10-song test set, stereo stems) is the standard music source separation benchmark.
• General Audio Datasets: AudioSet (2M 10-second clips, 632 event classes from YouTube) and FreeSound (500,000+ user-contributed clips) are the primary training and evaluation resources for environmental sound generation. VGGSound (200k clips, 309 classes) and ESC-50 (2000 clips, 50 classes) are used for event classification evaluations that inform generation model conditioning.
• Evaluation Benchmarks: AudioCaps (46,000 captioned 10-second AudioSet clips) and Clotho (6974 clips with 5 human-written captions each) are standard text-to-audio generation evaluation sets. The DCASE Challenge (annual, since 2013) maintains task-specific evaluations of audio generation, captioning, and scene synthesis. Codec-SUPERB (SLT 2024) provides a standardised multi-task benchmark comparing 11 neural audio codecs across 11 tasks covering ASR, speaker verification, emotion recognition, and reconstruction quality.

Ethical Dimensions and Risks

• Audio generation raises acute ethical, legal, and safety concerns that distinguish it from other generative modalities. Voice is personally identifying — voice biometrics underpin fraud detection, authentication systems, and identity verification — making unconsented voice cloning both a privacy violation and an identity fraud vector. The deepfake voice attack surface has expanded dramatically: voice phishing (vishing) using AI-generated voices impersonating bank officials, family members, or corporate executives has been documented in multiple cases since 2024. Academic research (arXiv 2602.20061, 2025) demonstrates that human listeners cannot reliably distinguish AI-generated speech from genuine recordings even under direct comparison conditions.
• For music generation, Intellectual Property concerns are central. The RIAA’s 2024 lawsuits against Suno and Udio allege that training on copyrighted musical recordings without licensing constitutes direct infringement. The music industry’s position — articulated in submissions to the US Copyright Office in 2024 — is that AI music generation systems that memorise and reproduce copyrighted works violate the reproduction rights of the original recording and underlying composition rights holders. Proposed legislative remedies include the US TRAIN Act (transparency in AI training data) and mandatory licensing frameworks.
• Provenance and Watermarking of generated audio is an active technical and policy area. AudioSeal (Meta, 2023) embeds imperceptible watermarks into generated speech that survive transcoding and moderate signal processing. SynthID-Audio (Google DeepMind) applies a similar approach to music. The ArtifactNet framework (2026) takes a different approach — detecting residual physical artefacts in the compression and generation pipeline that function as model fingerprints — enabling attribution without pre-embedded watermarks. The EU AI Act’s transparency requirements mandate disclosure of AI-generated content, and audio is explicitly included in the Act’s definition of deep fakes subject to labelling obligations.
• The Ethical AI implications extend to workforce displacement: AI music generation platforms directly compete with session musicians, voiceover artists, foley recordists, and sound designers. Labour organisations including the Screen Actors Guild (SAG-AFTRA) and the British Equity union have negotiated AI voice use provisions into collective agreements as of 2024-2025, requiring consent and compensation for AI training on members’ voice recordings.

Academic Context

• The theoretical foundations of audio generation trace to source-filter models of speech production (Fant, 1960), hidden Markov model (HMM) based synthesis systems (HTS, 2000s), and statistical parametric speech synthesis. The deep-learning era was initiated by WaveNet (van den Oord et al., 2016, DeepMind), which demonstrated that a convolutional autoregressive model could match human naturalness for the first time. The score-based generative model framework (Song & Ermon, 2020, 2021) provided the theoretical foundation for diffusion-based audio models, formalising the reverse stochastic differential equation that underpins AudioLDM and Stable Audio. CLAP (Contrastive Language-Audio Pre-training, Wu et al., 2023) extended the CLIP paradigm to audio, providing the text-audio embedding space used as conditioning in most current systems. The codec language model paradigm was crystallised by VALL-E (Wang et al., Microsoft, 2023), which demonstrated that a GPT-style model over EnCodec tokens could zero-shot clone any voice from 3 seconds of reference audio.
• Key research venues are ICASSP (IEEE International Conference on Acoustics, Speech, and Signal Processing), Interspeech, ISMIR (International Society for Music Information Retrieval), NeurIPS, ICLR, and IEEE/ACM Transactions on Audio, Speech, and Language Processing (TASLP). The DCASE (Detection and Classification of Acoustic Scenes and Events) challenge community maintains annual evaluations of audio captioning and generation. Key industrial research groups include Meta AI (AudioCraft, EnCodec, AudioBox, AudioSeal), Google DeepMind (WaveNet, Voicebox-adjacent work, SynthID), Microsoft Research (VALL-E, VoiceCraft-Dub), Stability AI (Stable Audio), and ElevenLabs (proprietary production TTS). Academic groups at Stanford CCRMA (Centre for Computer Research in Music and Acoustics), Carnegie Mellon University, Edinburgh CSTR, and Queen Mary C4DM provide fundamental research underpinning commercial systems.

Computational Infrastructure and Deployment Constraints

• Audio generation at scale requires substantial infrastructure investment. Training a music generation foundation model at the MusicGen 3.5B parameter scale requires approximately 2,000 GPU-hours on A100s for the transformer backbone alone, exclusive of the codec pre-training. Production TTS at ElevenLabs scale — hundreds of millions of requests per month — is served from cloud GPU clusters (primarily NVIDIA A10G and H100 instances) with aggressive caching of speaker embeddings and model KV-cache. Real-Time Inference is the key deployment constraint for conversational TTS: sub-100ms end-to-end latency (text input to first audio byte) requires model quantisation (INT8 or INT4 weights), speculative decoding, and parallel codec decoding of multiple codebook levels simultaneously.
• On-device deployment of audio generation is progressing rapidly. Apple’s on-device TTS (used in iOS text-to-speech accessibility features and offline Siri) runs a lightweight FastSpeech-2-style acoustic model with a GAN vocoder at <5MB model size. GPU Compute constraints on mobile devices (Apple A18, Qualcomm Snapdragon 8 Elite NPUs with 45+ TOPS) now permit real-time 24 kHz mono TTS with minimal battery impact. Music generation on-device remains impractical at state-of-the-art quality levels but is projected to become feasible for short clips (up to 30 seconds) by 2027-2028 as model compression techniques (knowledge distillation, pruning, activation quantisation) mature.
• Distributed inference for large generative audio models follows the same patterns as LLM serving: tensor parallelism across multiple GPUs for the transformer backbone, with the codec encoder/decoder running on separate, lighter compute units. Latency-optimised kernels (FlashAttention-2, custom CUDA implementations of RVQ decoding) are essential for production deployment. Streaming TTS — where audio is synthesised and transmitted incrementally as the acoustic model generates mel-spectrogram frames — reduces perceived latency from 2-3 seconds (wait for full synthesis then play) to under 300ms (first audio chunk in-ear while synthesis continues).

Standards, Formats, and Interoperability

• No single standard governs the audio generation domain, but several format and protocol standards shape interoperability. The W3C Web Speech API (SpeechSynthesis interface) defines a browser-native TTS abstraction that delegates to browser-embedded or OS-provided synthesis engines; it does not yet expose neural TTS at the API level. REST/WebSocket APIs — such as ElevenLabs’ TTS API, OpenAI’s Audio API, and Google Cloud Text-to-Speech — use HTTP streaming of PCM or Opus Codec encoded audio in response to text POST requests. The SSML (Speech Synthesis Markup Language, W3C) provides an XML-based notation for prosody control (pitch, rate, volume, emphasis, pauses) that is supported by all major commercial TTS APIs.
• For music generation, MIDI (Musical Instrument Digital Interface, MIDI 1.0 1983; MIDI 2.0 2020) remains the dominant symbolic format for conditioning generation models on musical structure. MusicXML (W3C Music Notation Community Group) encodes sheet music digitally and is used in automatic music transcription pipelines. The Audiocraft library (Meta) and ACE-Step expose Python APIs for programmatic access. The Hugging Face Diffusers library provides a standard pipeline interface for AudioLDM 2 and Stable Audio, enabling interoperability of these models in research workflows.
• Audio file format interoperability: WAV (PCM, uncompressed, 44.1–192 kHz), FLAC (lossless compression), and Opus Codec (lossy, standardised) are the primary interchange formats for audio generation systems. 44.1 kHz 24-bit WAV is the standard output format for music generation systems targeting professional use. 16 kHz 16-bit PCM is standard for speech synthesis in conversational AI due to telephony heritage. MPEG-4 Audio (AAC, HE-AAC) is used for streaming distribution of generated content.

Current Landscape (2026)

• As of mid-2026, audio generation has crossed the threshold from research prototype to mass-market deployment. ElevenLabs holds the dominant position in commercial TTS and voice cloning with $330M ARR and 41% Fortune 500 penetration as of February 2026. Google’s Gemini integrates native audio input/output via a joint audio-language model. OpenAI’s GPT-4o processes speech natively in the audio domain without TTS/ASR intermediaries. Meta releases AudioCraft and its successors as open-source, making MusicGen and AudioGen freely available to researchers and developers. The AI voice synthesis infrastructure market is projected to reach tens of billions of dollars by 2034 driven by conversational AI, content localisation, and accessibility applications.
• The neural audio codec has become the universal substrate for audio generation: EnCodec, DAC, FlexiCodec, and U-Codec all achieve perceptually transparent compression at 8–12 kbps, and all major generative systems now operate in the codec token domain rather than in raw waveforms or mel-spectrograms. Flow matching has largely displaced DDPM-style diffusion for latency-critical applications because it converges to high quality in 10–20 denoising steps versus 1000 for the original DDPM formulation.
• Regulatory and legal pressure is intensifying. The EU AI Act’s high-risk classification for biometric AI systems creates compliance requirements for voice-identity-based generation. The US NO FAKES Act (introduced April 2025) targets unconsented voice replicas. Watermarking standards — AudioSeal (Meta), SynthID-Audio (Google DeepMind) — are being integrated into generation pipelines to enable forensic attribution of synthetic audio. The ArtifactNet forensic residual physics framework (arXiv 2604.16254, 2026) demonstrates model-fingerprinting-level detection of AI-generated music.

UK Context

• Academic Institutions
  • Queen Mary University of London’s Centre for Digital Music (C4DM), led by Professor Simon Dixon and with Professor Emmanouil Benetos as Deputy Director of the UKRI AI Centre for Doctoral Training in Artificial Intelligence and Music (AIM CDT), is the UK’s primary hub for Music Information Retrieval and AI-driven Music Generation. The AIM CDT (funded by UKRI EPSRC/AHRC) trains 50+ PhD researchers in generative music, machine listening, and music AI, and began new industry research partnerships in 2024.
  • The University of Edinburgh’s Centre for Speech Technology Research (CSTR) has historically been one of the world’s leading speech synthesis groups, contributing the Festival speech synthesis system, HTS/Merlin neural TTS toolkit, and multiple Interspeech and ICASSP best-paper contributions. The Edinburgh Futures Institute hosted the “AI and Digital Innovations for Voice and Vocal Music” event series in 2024-2025.
  • Imperial College London’s Department of Computing and Cambridge’s Machine Intelligence Lab contribute to generative audio through speech and signal processing research groups. UCL’s Ear Institute has research relevance to perceptual evaluation of generated audio.
  • Stability AI, though restructured in 2025, was UK-headquartered (London) and produced the Stable Audio open-source series, demonstrating the UK AI startup ecosystem’s ability to produce globally-deployed audio generation systems.
• Northern England
  • The University of Sheffield’s Speech and Hearing Research Group conducts research in speech synthesis, voice pathology, and speech technology for healthcare applications.
  • The University of Leeds’ School of Electronic and Electrical Engineering contributes audio signal processing research. Leeds also hosts cultural institutions (Leeds International Film Festival, Opera North) that are potential early adopters of AI music and audio generation for creative production.
  • Manchester’s music industry heritage — the Haçienda, Factory Records, The Hallé Orchestra, Manchester Camerata — and its dense music technology startup ecosystem (companies clustered around MediaCity Salford, proximity to BBC R&D Salford) position Greater Manchester as a natural testbed for AI music generation tools in professional creative contexts.
  • Newcastle University’s Digital Institute and Northumbria University’s audio and music research groups contribute to the northern England research ecosystem.
• Policy and Industry
  • The UK’s Online Safety Act 2023 and proposed AI regulation create a compliance context for audio deepfake detection and provenance watermarking of synthetic voices published on UK platforms.
  • The Creative Industries Council, Ukie (UK Interactive Entertainment), and BPI (British Phonographic Industry) have all produced policy positions on AI-generated music affecting licensing, attribution, and performer rights in 2024-2025.
  • Shazam (now Apple-owned, UK-founded), Focusrite, SSL (Solid State Logic), and Native Instruments (which acquired several UK audio AI companies) are industry actors bridging professional audio production and AI generation.

Future Directions (2026–2030)

• Real-time joint audio-language models
  • The architectural trajectory — exemplified by GPT-4o’s native audio mode and Google Gemini’s audio integration — points toward end-to-end multimodal models that perceive and generate audio, text, and potentially video in a unified latent space, eliminating the TTS/ASR abstraction layer entirely.
  • Sub-100ms round-trip latency for conversational voice AI is a key 2026-2028 target. Current bottlenecks include KV-cache memory bandwidth for long context, codec decoding throughput, and network round-trip time (RTT) to cloud inference endpoints.
  • Streaming autoregressive generation — producing audio tokens in real-time as the language model generates, rather than completing the full sequence before decoding — is the primary latency reduction technique and is deployed in ElevenLabs’ Flash models and OpenAI’s Real-Time API (2024).
• Personalised on-device synthesis
  • Lightweight codec language models (sub-500M parameters) and flow-matching vocoders targeting real-time synthesis on mobile NPUs will enable fully on-device voice personalisation without cloud round-trips.
  • Apple’s investment in on-device ML (Core ML, Apple Neural Engine), Google’s on-device Gemini Nano, and Qualcomm’s Snapdragon Neural Processing SDK all provide the inference substrate for on-device TTS with personalised voices.
  • Privacy-preserving voice personalisation — where the user’s voice model is stored exclusively on-device and never transmitted to cloud servers — is a regulatory-driven design requirement under GDPR and forthcoming UK AI regulation.
• Music production AI
  • Generative systems will move from clip-level generation toward full production-quality mix creation — arrangement, mixing, mastering — integrating with DAWs via plugin APIs (VST3, AU, AAX).
  • ACE-Step’s modular architecture (lyric2vocal, singing2accompaniment) prefigures DAW-integrated AI tools. Music Information Retrieval capabilities — beat tracking, chord estimation, key detection — will be integrated as constraints on the generation process to ensure musical coherence with existing track material.
  • MIDI-conditioned generation from piano roll or score inputs, combined with style transfer from reference recordings, will enable AI-assisted arrangement and orchestration for composers working in notation software.
• Controllable and explainable generation
  • Attribute-specific control sliders (tempo, key, instrumentation, speaker affect, speech rate) over generation outcomes, combined with interpretable conditioning representations, will be required for professional creative workflows and regulatory compliance.
  • Disentangled latent spaces — where independent latent dimensions control separable perceptual attributes (pitch, timbre, rhythm, affect) — are a central research challenge, analogous to face disentanglement in image generation.
  • Classifier-free guidance (CFG) and classifier guidance on attribute embeddings already provide coarse controllability; fine-grained attribute control at the level of individual bars or phonemes remains an open problem.
• Deepfake detection and watermarking standardisation
  • As synthetic audio proliferates, standardised watermarking and detection frameworks — building on AudioSeal, SynthID-Audio, and ArtifactNet — will be required for broadcast, journalism, and legal proceedings.
  • UK-specific regulatory frameworks aligned with the Online Safety Act 2023 and EU AI Act provisions will mandate provenance metadata for published synthetic audio. Ofcom’s content moderation guidelines may require platform-level synthetic audio labelling from 2027.
  • The ADD (Audio Deep Synthesis Detection) Challenge community and the ASVspoof challenge (anti-spoofing and deepfake detection) maintain annual evaluation benchmarks that track the state of the detection/generation arms race.
• Spatialised and immersive audio generation
  • Generating 6DOF (six-degrees-of-freedom) Spatial Audio directly conditioned on 3D scene descriptions and listener head tracking will be essential for Extended Reality presence.
  • Diff-SAGe and its successors will integrate with XR rendering pipelines, replacing static HRTF convolution with learned, content-aware spatial synthesis that adapts to estimated room acoustics and listener position in real-time.
  • Object-based audio generation — where individual sound objects are generated with 3D position metadata for renderer-independent playback — aligns with the MPEG-H Audio object model and will enable AI-generated audio to interoperate with existing broadcast and streaming spatial audio infrastructure.

Training Datasets and Scale

• Speech Corpora: LJSpeech (single speaker, 24h, 22.05 kHz mono) — standard TTS development benchmark; VCTK (110 speakers, 44h, 48 kHz) — multi-speaker; LibriTTS (585h, 2456 speakers, 24 kHz, cleaned subset of LibriSpeech) — large-scale multi-speaker; Common Voice (120+ languages, 50,000+ hours, crowd-sourced) — multilingual coverage; Whisper training set (5M+ hours of weakly-supervised internet audio) — largest publicly disclosed speech training corpus.
• Music Corpora: FMA (343h, 104 genres, Creative Commons licensed) — standard open dataset; MagnaTagATune (25,863 clips, 188 tags) — music tagging evaluation; Slakh2100 (145 hours of synthesised multi-instrument MIDI-rendered audio with stems) — source separation; MUSDB18-HQ (150 professional tracks, 10-song test set, 44.1 kHz stereo WAV stems) — music separation gold standard; NSynth (300,000 single-note recordings, 1006 instruments, Google). Suno, Udio, and commercial music generation systems use proprietary datasets estimated at 100,000–1,000,000+ hours of licensed or scraped music; the exact composition is not publicly disclosed and is the subject of the RIAA 2024 litigation.
• Environmental/Foley Audio: AudioSet (2M clips, 632 event classes from YouTube) — foundational audio classification and generation dataset; FreeSound (500,000+ user-contributed clips under various Creative Commons licences) — primary resource for sound effect generation research; ESC-50 (2000 clips, 50 environmental classes) — classification benchmark; VGGSound (200,000 clips, 309 classes, YouTube) — large-scale audio-visual event dataset.
• Text-to-Audio Evaluation: AudioCaps (46,000 captioned 10-second AudioSet clips) and Clotho (6974 clips, 5 human-written captions each, 15-30 seconds) are the standard evaluation sets for text-to-audio generation quality assessment. Evaluation metrics: FAD (Fréchet Audio Distance) — lower is better; CLAP-score — cosine similarity between generated audio and text embeddings; human MOS on relevance and quality.

Model Architecture Deep Dive: Diffusion Transformer

• The Diffusion Transformer (DiT), introduced by Peebles & Xie (2023) for image generation, has become the dominant backbone for music and general audio generation as of 2024-2026, replacing U-Net architectures in Stable Audio 3.0, ACE-Step, and proposed successors to AudioLDM.
• Architecture: A DiT processes flattened latent patches through L layers of transformer blocks (self-attention + feed-forward), each conditioned on timestep t and optional guidance c via adaptive layer norm (AdaLN): [α, β, γ] = MLP([embed(t), embed(c)]), with the block output scaled and shifted as y = γ ⊙ LayerNorm(x) + β + α × Attention(x). This conditioning mechanism allows a single backbone to handle variable conditioning signals without architecture modification.
• Scalability: DiT-XL (675M parameters, 28 layers, 1152 hidden dim) outperforms U-Net architectures of equivalent parameter count on FID for image generation; the same scaling laws apply to audio DiTs. Stable Audio 3.0’s 1.3B-parameter DiT operates on highly compressed latent audio (64x compression via DCAE), reducing sequence length from millions of samples to thousands of tokens, making self-attention over full audio tracks computationally feasible.
• Inference: Stable Audio 3.0 uses flow matching ODE integration with 100 steps at training; distilled to 25-50 steps at inference. A 44.1 kHz, 6-minute stereo track (15.9M samples) is processed as a latent sequence of ~2,500 tokens (64x compression ratio), enabling full-sequence self-attention in the DiT at practical GPU memory cost.

Key Systems Comparison (2024–2026)

• WaveNet (DeepMind, 2016): Autoregressive dilated causal CNN, 24 kHz, ~40ms/sample on GPU, MOS 4.21. Landmark system — first neural vocoder at human naturalness. Not real-time without distillation. Architecture: 30 layers × 10 dilations × 3 blocks = 30 dilated convolution layers with 512 residual channels.
• WaveRNN (DeepMind, 2018): Single GRU autoregressive vocoder, ~3ms/sample. Distilled to Subscale WaveRNN (20 samples/step in parallel), achieves 11ms/frame on mobile CPU. Key efficiency advance: sparse weight matrices reduce FLOPS by 96%.
• HiFi-GAN (V1, 2020, NeurIPS): GAN vocoder, 22.05 kHz, real-time factor 167x on V100, MOS 4.37. Multi-period (MPD) and multi-scale (MSD) discriminators. Generator: 7 residual blocks with dilations [1,3,5]. Foundational for all subsequent production TTS vocoders.
• Tacotron 2 (Google, 2018, ICASSP): Seq2seq acoustic model (encoder CNN + attention + decoder RNN) predicting 80-dim log-mel spectrograms from character sequences. Combined with WaveNet: MOS 4.44, matching human recorded speech on LJSpeech for the first time.
• FastSpeech 2 (Microsoft, 2021): Fully feed-forward acoustic model using duration, pitch, and energy predictors — no attention, 38x faster inference than Tacotron 2, MOS 3.87. Standard production TTS backbone for latency-constrained deployment.
• VITS (Kim et al., 2021, ICML): End-to-end TTS without separate vocoder, combining Variational Autoencoder + normalising flow + GAN in a single jointly trained model. MOS 4.43. Eliminates two-stage pipeline for speech synthesis.
• EnCodec (Meta, 2022): RVQ neural codec, 24/48 kHz, 1.5–12 kbps, 44.1 kHz open-source variant. Trained with multi-scale spectral discriminators. Real-time at 12x on single CPU core. Standard discrete token vocabulary for codec LMs.
• VALL-E (Microsoft, 2023): Codec LM (GPT-style AR decoder) over EnCodec tokens. Zero-shot voice cloning from 3-second prompt. MOS 4.58 for in-domain speakers. First system to beat human recordings in in-domain ABX tests on LibriSpeech.
• AudioLDM 2 (Liu et al., 2024, IEEE/ACM TASLP): Unified text-to-audio generation covering speech, music, and effects. Language of Audio (LOA) encoder + GPT-2 bridge + latent diffusion U-Net. FAD 1.74 on AudioCaps (SOTA at publication).
• Stable Audio 3.0 (Stability AI, 2025): Flow-matching DiT (1.3B params), VAE with 64x compression, Flan-T5 XL + CLAP conditioning. 44.1 kHz stereo, up to 6 minutes. Technical report: arXiv 2605.17991. Open weights (medium variant) on Hugging Face.
• Voicebox (Meta, 2023, NeurIPS): Flow-matching TTS/speech editing model, 6 languages, competitive with VALL-E on voice cloning, ~50x faster inference than diffusion models. Masked flow matching training objective enables inpainting and style transfer without re-training.
• ACE-Step v1.5 (2026, arXiv:2602.00744): Open-source music foundation model, 0.6-4B LM planner + DiT synthesiser + DCAE. Generates 10-min compositions in <2s on A100, <10s on RTX 3090. Supports lyrics, voice cloning, stem generation, remixing. Claims to outperform Suno/Udio on objective metrics.

Research & Literature

• 1. van den Oord, A., Dieleman, S., Zen, H., et al. (2016). WaveNet: A Generative Model for Raw Audio. arXiv:1609.03499. DeepMind.
• 2. Shen, J., Pang, R., Weiss, R.J., et al. (2018). Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictions. ICASSP 2018. (Tacotron 2)
• 3. Kong, J., Kim, J., Bae, J. (2020). HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis. NeurIPS 2020.
• 4. Défossez, A., Copet, J., Synnaeve, G., Adi, Y. (2022). High Fidelity Neural Audio Compression. arXiv:2210.13438. (EnCodec, Meta)
• 5. Kumar, R., Kumar, P., de Boissiere, T., et al. (2024). High-Fidelity Audio Compression with Improved RVQGAN. NeurIPS 2023. (Descript Audio Codec)
• 6. Wang, C., Chen, S., Wu, Y., et al. (2023). Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers. arXiv:2301.02111. (VALL-E, Microsoft)
• 7. Liu, H., Chen, Z., Yuan, Y., et al. (2023). AudioLDM: Text-to-Audio Generation with Latent Diffusion Models. ICML 2023.
• 8. Liu, H., Yuan, Y., Liu, X., et al. (2024). AudioLDM 2: Learning Holistic Audio Generation with Self-Supervised Pretraining. IEEE/ACM TASLP, 2024.
• 9. Copet, J., Kreuk, F., Gat, I., et al. (2023). Simple and Controllable Music Generation. NeurIPS 2023. (MusicGen, Meta)
• 10. Kreuk, F., Synnaeve, G., Défossez, A., et al. (2023). AudioGen: Textually Guided Audio Generation. ICLR 2023.
• 11. Vyas, A., Shi, B., Le, M., et al. (2023). Audiobox: Unified Audio Generation with Natural Language Prompts. arXiv:2312.15821. (Meta)
• 12. Evans, Z., Parker, J.D., Carr, C.J., Zukowski, Z. (2024). Stable Audio Open. arXiv:2407.14358. (Stability AI)
• 13. Evans, Z., Parker, J.D. (2024). Stable Audio 3.0 Technical Report. arXiv:2605.17991. (Stability AI)
• 14. Song, Y., Sohl-Dickstein, J., Kingma, D.P., et al. (2021). Score-Based Generative Modeling through Stochastic Differential Equations. ICLR 2021.
• 15. Borsos, Z., Marinier, R., Vincent, D., et al. (2023). AudioLM: a Language Modeling Approach to Audio Generation. IEEE/ACM TASLP.
• 16. Le, M., Vyas, A., Shi, B., et al. (2023). Voicebox: Text-Guided Multilingual Universal Speech Generation at Scale. NeurIPS 2023. (Meta)
• 17. Peng, S., Miao, Y., Zhang, J., et al. (2024). VoiceCraft: Zero-Shot Speech Editing and Text-to-Speech in the Wild. ACL 2024.
• 18. Wu, Y., Chen, K., Zhang, T., et al. (2023). Large-Scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentation. ICASSP 2023. (CLAP)
• 19. ACE-Step Authors (2025). ACE-Step: A Step Towards Music Generation Foundation Model. arXiv:2506.00045.
• 20. ACE-Step Authors (2026). ACE-Step 1.5: Pushing the Boundaries of Open-Source Music Generation. arXiv:2602.00744.
• 21. Popov, V., Vovk, I., Gogoryan, V., et al. (2021). Grad-TTS: A Diffusion Probabilistic Model for Text-to-Speech. ICML 2021.
• 22. Chen, S., Wu, Y., Wang, C., et al. (2023). SoundStorm: Efficient Parallel Audio Generation. arXiv:2305.09636. (Google DeepMind)
• 23. Roman, N., Rosenberg, A., Zhang, Y. (2024). Diff-SAGe: End-to-End Spatial Audio Generation Using Diffusion Models. arXiv:2410.11299.
• 24. Lacombe, Y., Muller, M., Adi, Y., et al. (2023). AudioSeal: Proactive Detection of Voice Cloning. arXiv:2308.14565. (Meta)
• 25. Kim, J., Kong, J., Son, J. (2021). Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speech. ICML 2021. (VITS)
• 26. Chen, G., Yi, Q., Chen, B., et al. (2026). ArtifactNet: Detecting AI-Generated Music via Forensic Residual Physics. arXiv:2604.16254.
• 27. Benetos, E., Dixon, S., Duan, Z., Ewert, S. (2019). Automatic Music Transcription: An Overview. IEEE Signal Processing Magazine, 36(1).

Provenance