Voice Cloning: How It Works and the Consent Question

Voice cloning conditions a synthesis model on a representation of one speaker. The two dominant ways of building that representation behave quite differently, and neither copies as much of a person as the word “clone” suggests.

Two ways to clone a voice

Speaker embedding, borrowed from verification

The older and more interpretable route. A speaker encoder — the same kind of network used for diarisation, trained to tell speakers apart — maps a few seconds of audio to a fixed-length vector. That vector is fed to a text-to-speech model as an extra conditioning input alongside the text. Jia and colleagues at Google published the canonical version of this in 2018 under the title “Transfer Learning from Speaker Verification to Multispeaker Text-To-Speech Synthesis”, and the design is still everywhere.

The property that made it interesting is that the encoder is trained on thousands of speakers who never appear in the TTS training set, so it generalises to voices the synthesiser has never heard. The embedding is a small vector — a few hundred floats — which is also why it is easy to store, easy to compare, and, in most jurisdictions’ framing, easy to argue is biometric data.

Codec language models and in-context cloning

The current route. A neural audio codec compresses waveforms into sequences of discrete tokens; a language model is then trained to predict those tokens given text. Because it is a language model, it does in-context learning: give it a few seconds of a target speaker’s audio and the corresponding transcript as a prompt, and it continues in that voice. Microsoft’s VALL-E paper in 2023 is the reference description of this approach.

There is no speaker vector anywhere in that pipeline. The identity lives in the prompt tokens, which means the clone inherits far more than timbre — recording conditions, background noise, and the emotional register of the prompt all carry across, which is simultaneously why it sounds so much better and why it is harder to control.

How much audio it takes

Three regimes, and the distinction matters for both quality and risk assessment:

A practical note that saves a week: reference audio quality dominates reference audio quantity in the zero-shot regime. Thirty seconds of clean, close-miked, single-speaker audio with no music and no reverberation outperforms ten minutes of conference-call audio. Apply the same hygiene you would apply before recognition: correct sample rate, mono, no clipping, consistent loudness.

What a clone does not copy

The failure modes are the interesting part, and they are systematically under-described because demos are chosen to avoid them.

• Idiolect. A clone copies how a voice sounds, not how a person constructs sentences. Word choice, hesitation habits, the specific fillers somebody uses — none of that is in the acoustic conditioning, and the text is written by something else. Listeners who know the speaker well notice this before they notice anything acoustic.
• Register range. Conditioning on one calm reference gives a voice that is calm about everything. Shouting, laughing, whispering and crying are outside the range unless the reference contained them, and asking for them produces either a flat rendition or an artefact.
• Cross-lingual identity. Cloning a voice into a language the speaker does not speak works acoustically, but accent and phoneme inventory come from the model rather than the person, so the result frequently sounds like a different individual to native listeners of the target language.
• Consistency across a long piece. Zero-shot clones drift over minutes of continuous output as the model gets further from the prompt. Chunking and re-priming each chunk with the same reference is the usual mitigation, at the cost of audible seams.
• Singing and non-speech vocalisation. Almost universally out of scope for speech TTS, whatever a demo suggests.

Detection and watermarking

Two different questions, and they need different tools. “Is this audio synthetic?” is anti-spoofing, an active research area whose detectors generalise poorly to generators they were not trained on. “Did my system generate this audio?” is watermarking, and it is tractable because you control the encoder.

• Audio watermarking embeds an imperceptible signal that a paired detector recovers. Meta published AudioSeal in 2024 as an open localised watermarker for speech, designed to survive common transformations and to indicate which parts of a clip are generated. Several commercial voice providers watermark output by default. Robustness to re-encoding, resampling and telephony transcoding is the property to test, because that is what the audio will have been through by the time you need the detector.
• Provenance metadata — C2PA content credentials attached at generation time — is complementary and fails in the opposite direction: cryptographically strong, and stripped by any re-encode that does not preserve it.
• Speaker verification as a gate is the control that actually prevents misuse at the point of creation. Require the person being cloned to record a challenge phrase, in the same session, that you generated; verify it matches the reference audio they uploaded; keep both. This is worth more than any downstream detector.

Consent as a data model

Most guidance on this is a paragraph telling you to get consent. That is not implementable. What follows is the artefact — the record you need to be able to produce eighteen months later when somebody asks a question you did not anticipate.

voice_consent
  id                     uuid
  subject_name           text
  subject_contact        text            -- for withdrawal requests
  captured_at            timestamptz
  captured_by            text            -- your operator, or "self-serve"

  -- What they agreed to. Store the scope, not a boolean.
  permitted_uses         text[]          -- e.g. {ivr_greeting, marketing_video}
  prohibited_uses        text[]          -- explicit exclusions they asked for
  permitted_languages    text[]
  territory              text[]
  expires_at             timestamptz     -- consent with no end date is a
                                         -- liability, not a permission

  -- Proof. This is the part people omit and later need.
  consent_recording_uri  text            -- audio/video of them consenting
  consent_text_version   text            -- git sha of the wording shown
  challenge_phrase       text            -- generated by you, at capture time
  challenge_audio_uri    text            -- them reading it, same session
  verification_score     numeric         -- speaker-verification similarity
                                         -- between challenge and reference

  -- Lifecycle.
  reference_audio_uri    text
  model_artifact_uri     text            -- the embedding or adapter produced
  withdrawn_at           timestamptz
  deletion_completed_at  timestamptz     -- reference audio AND derived model

Every generation request writes a row referencing this id. Without
that link you cannot answer "what did we make with this voice", which
is the first question in any dispute.

Four properties of that schema are the ones that matter, and each corresponds to a way this goes wrong:

1. Scope, not a checkbox. “I consent to voice cloning” is not consent to a political advertisement. Enumerate uses, and make the generation path check them rather than trusting the caller.
2. An expiry. Perpetual, irrevocable consent is precisely the term that draws attention, and in several regimes withdrawal cannot be contracted away regardless of what was signed. A finite term with an explicit renewal is both safer and easier to defend.
3. A challenge phrase you generated. Consent audio the subject supplied could itself be synthetic or reused. A phrase your system produced at capture time, read back in the same session, is what makes the record evidence rather than a file.
4. Deletion that reaches the derived artefacts. Withdrawal has to delete the embedding and the adapter, not only the source recording. If your deletion job only clears the audio bucket, you can still generate the voice, and that is exactly the fact that will be discovered at the worst moment.

The questions to put to counsel

A voice is potentially several regulated things at once, and which ones depends entirely on where you and the subject are. These are the questions that decide the engineering, so ask them before you build:

• Is the voiceprint biometric data here? If a speaker embedding is treated as biometric identification data, a stricter consent standard, a stricter retention rule and a separate impact assessment may all attach to it — including to the vector, not just the audio.
• Whose permission is required, and in what form? Written, recorded, witnessed, revocable. And whether a performer’s union agreement or an employment contract governs the answer for staff voices.
• What are the publicity and likeness rights? Several jurisdictions now protect a person’s voice as a distinct attribute — Tennessee’s ELVIS Act in 2024 is the clearest example — and these rights can outlive the person and can be held by an estate.
• What must you disclose to listeners? Transparency obligations for synthetic media that impersonates a real person exist in the EU AI Act and are being legislated elsewhere. Disclosure is a product decision with a deadline attached.
• What does your provider’s contract require of you? Most voice vendors push the consent obligation to the customer, and several require you to be able to evidence it on request. Read that clause before you design the schema above; it may specify what it must contain.