Local text-to-audio generation for SwarmUI, as a native-feeling extension: sound effects, music loops, and full songs with vocals — using the exact same generate flow you already know (prompt, negative prompt, seed, steps, CFG, batches, presets, image history, metadata / reuse-params).
SwarmUI core natively supports one audio model family (ACE-Step 1.5). This extension adds three more families, a curated one-click download list, and the glue so they all behave like first-class citizens.
| Family | Architecture ID | Good for | Size | Notes |
|---|---|---|---|---|
| Stable Audio 3 Small (SFX / Music) | stable-audio-3 |
Sound effects, foley, instrumental loops | 2.3 GB | Turbo-distilled: 8 steps, CFG 1, fast even on 8 GB VRAM |
| Stable Audio 3 Medium | stable-audio-3 |
Higher quality SFX + music | 9.2 GB | Works on 8 GB via offloading, slower |
| Stable Audio Open 1.0 | stable-audio-open-1.0 |
SFX, samples, loops (≤ 47 s) | 4.9 GB | The classic; many community finetunes load too |
| ACE-Step v1 (3.5B) | ace-step-v1 |
Full songs with vocals + lyrics, up to ~4 min | 7.7 GB | Prompt = lyrics, Style = genre tags |
| ACE-Step 1.5 Turbo | ace-step-1_5 (core) |
Songs, fast | 10 GB | Supported by SwarmUI core; listed in our downloader for convenience |
All entries fit comfortably in 8 GB VRAM (ComfyUI auto-offloads the bigger ones). Text encoders that aren't packed into a checkpoint (t5-base for Stable Audio Open, t5gemma for Stable Audio 3) are auto-downloaded on first generation, exactly like core does for its own model families.
Clone the extension into SwarmUI's extension directory, then restart SwarmUI:
cd SwarmUI/src/Extensions
git clone https://github.com/tbbbtbbbtbbb/SwarmUI-AudioGen.git SwarmAudioGenThe launcher rebuilds automatically. No Python packages or custom ComfyUI nodes are needed—everything uses stock ComfyUI nodes (requires a reasonably current ComfyUI backend, 2026+).
Open Utilities → Model Downloader. A Featured Audio Models panel sits at the top: one click downloads with a pinned URL + SHA-256-verified file into your main models folder under Audio/, with title/description/usage metadata pre-filled. (Everything also registers into Swarm's internal known-models list, and you can of course download manually instead — the architecture is auto-detected from the file's tensor headers.)
- Select an audio model in the Models picker (they're in the
Audiofolder; the arch chip shows e.g.SA3/SAO1/AceV1). - The Text To Audio parameter group appears — enable it to set Duration (seconds) and, for ACE-Step, Style, BPM, etc.
- Prompt and generate as usual. Output is an mp3 in your normal output history (seeds, presets, reuse-params, star, etc. all work).
| Model | Prompting | Steps | CFG | Sampler / Scheduler |
|---|---|---|---|---|
| Stable Audio 3 (all) | Prompt = description ("rain on a tin roof, distant thunder") | 8 | 1 | lcm / simple |
| Stable Audio Open 1.0 | Prompt = description; keep Duration ≤ 47 | 50 | ~5 | dpmpp_3m_sde_gpu / exponential |
| ACE-Step v1 | Prompt = lyrics (or [instrumental]), Text2Audio Style = genre tags ("anime, j-pop, piano, fast, happy") |
50 | 5 | euler / simple |
Tip: save these as presets — audio params are ordinary registered params, so presets capture them like anything else.
- The main prompt box is the lyrics, with
[verse]/[chorus]/[instrumental]style section tags supported — this matches how SwarmUI core treats the prompt for ACE-Step 1.5. - Text2Audio Style carries the genre/instrument/mood tags.
- The negative prompt works (encoded as negative style tags); left empty, the official zeroed-negative is used.
- Text2Audio Lyrics Strength (advanced) controls lyric adherence (default 0.99, per the official workflow).
Sigma Shiftdefaults to 5 and Reinhard CFG tonemapping is applied, matching the official ComfyUI workflow. (DisableModel Specific Enhancementsto strip the tonemap.)
- Trained for clips up to 47.6 s — the extension caps Duration there as stupid-proofing. Disable
Model Specific Enhancementsif you really want to exceed it.
- Registers real
T2IModelClass/compat classes (withIsAudioModel), so architecture detection, the models browser, and metadata behave natively. - Swaps the empty-latent for
EmptyLatentAudio/EmptyAceStepLatentAudio(reserved node id 5), wires the checkpoint VAE as the audio VAE, and lets core's own audio-aware save path (VAEDecodeAudio→SaveAudioMP3) do the rest. - ACE-Step v1 text encoding routes through
TextEncodeAceStepAudio; the stable-audio families use the plain native conditioning path. - The featured downloads drive SwarmUI's own downloader (
ActiveModelDownload+DoModelDownloadWS), so you get the native progress cards, cancel, and metadata handling.
- Audio-to-audio (init audio + creativity, ACE repaint/cover) — planned; core has matching TODOs upstream.
- Output format is mp3 V0 (that's core's current audio save node; a quality/format param may come upstream).
- Image-oriented params (resolution, refiners, init image, etc.) are simply ignored by audio models — same behavior as core's ACE-Step 1.5.
- LoRAs for ACE-Step v1 exist but are untested here.
MIT