KittenTTS module. by jsett · Pull Request #1107 · brailcom/speechd

jsett · 2026-08-12T21:18:20Z

KittenTTS Module.

This is a speech dispatch model for running Kitten TTS. Kitten TTS is a deep learning model which provides high quality natural sounding TTS generation using models ranging from 15M to 80M parameters. Due to its small size it is able to run in real-time on CPU. The goal of this project is to integrate Kitten TTS with speech dispatch while maintaining its near real-time speech generation, with special attention being place on reading of long text's such as ebooks. To achieve this the original python code was rewrote into c and tightly integrated into a speech dispatch model.

There is a lot here so let me give a quick summary of what all is going on here.

kitten_server.c:

This file is for handling the protocol and follows some what closely to the example modules for async servers with speech dispatch handling the audio. By design I made sure very little work is done in any function in this thread. Any long running code should be handed to an async queue and ran on one of the threads in the kitten_worker.c file.

kitten_worker.c:

This is where the handling of long running tasks is done. We create two different threads here one to one to handle passing audio back to the server(since the module_tts_output_server function can block and we want the generation to continue while we are outputting audio to the server). The other thread is dedicated to the generation of audio by the model. We synchronize all this using two GasyncQueue, one for handling incoming speak requests, and the other to handle the outputted audio from our model. we also keep track of how much audio we have generated and played so that long speak commands don't run the cpu unnecessarily hard (For example I have seen that Okular will in some cases send an entire book's text in a single speak command). There is also code here for parsing ssml text using libxml.

kitten_model.c:

The handles everything we need to do to generated output from onnx using our model. The most important functions here are init_voice_style to handle loading voice styles. reload_models_and_voices: to handle changing the voice style. And kitten_speak for generating audio.

kitten_downloader.c:

This handles downloading the model+voice styles if its not already on our computer. If it download it does verify the file against a sha256, but that verification is not strongly enforce and is more of a warning.

…issue when processing text over 400 char. Fix an issue when we get unmapped chars.

sthibaul

Thanks for the nice work!

In the future, we will probably want to integrate various onnx-based voices, so code will be useful to share between modules, but your writing seems already quite well structured so that it will be convenient to do.

There are just a few changes that need to happen before we can integrate this.

…dencies to the CI

unlimitedsola · 2026-08-17T09:15:26Z

I believe this is best to be handled by package managers instead?

As models update, we don't want to frequently update and recompile this as downstream packages might lag behind for years.

Also all the download links are outside of this project's control. Which means that we won't be able to fix a broken download until a new release is packaged by distributions and shipped to the end user (which as commented, might be months/years after).

I believe this is best to be handled by package managers

It already checks if the models/voices are in the path /usr/share/speech-dispatcher/models/kitten and uses that path first. If the package manger placed them there, then they will get used.

we won't be able to fix a broken download until a new release

I have added the ability to set the download configuration through the models dot conf file.

Here is an example dot conf on how to do that https://gist.github.com/jsett/6146eb2803f8780a1830ed816bdc94ef

I believe this is best to be handled by package managers

It already checks if the models/voices are in the path /usr/share/speech-dispatcher/models/kitten and uses that path first. If the package manger placed them there, then they will get used.

I'm afraid distribution would not organize their packages this way.

Like piper, kitten models are general purpose in the sense of the Text-to-Speech task. If we take example from piper, the AUR packages place the models into /usr/share/piper-voices/, because their use are not limited to speech-dispatcher, and the distributions would not want to package multiple copies of large model files just for different applications to use.

It is also likely that different distributions wouldn't come to agreement on where these models should be placed. So the distribution packagers would adjust the conf default to accommodate their packaging schemes.

I believe this is best to be handled by package managers instead?

Nevertheless, I personally wouldn't expect program like speech-dispatcher (as a daemon process) to make network requests (and I strongly believe that we should learn from the log4shell disaster where a logging library contains the code path to make network requests) . So adding this to me feels like a scope creep that I would like to push against, as indicated by the first time linking with libcurl.

/usr/share/speech-dispatcher was recommended in the comments above, but I really don't have any real issue changing it if that what is decided.

code path to make network requests

I'm not opposed to removing the downloader. But there is a trade off here. If you remove it you increase the complexity for users to get things working, if there distro did not include the models with the install. I would be interested in hearing what other people think.

Turning the downloader into a one-off script might not be a bad idea

That could be an idea yes. People installing speech-dispatcher by hand would use it, and distribution people would package the voices. People could still use the script to update the voices.

Again, if only upstream voices providers could manage these questions instead of just throwing .onnx files at people...

There could simply be a voice downloading manager, even independent from speech-dispatcher, that manages updates and whatnot, and speech-dispatcher would just happen to see the voice files show up, and use them.

upstream of these models could simply document where these are supposed to be shipped

Unfortunately I don't think hugging face models have a way of doing this currently.

Based off the above comments, I think this is what I am going to do.

With regards to the downloader.

Remove the downloader from the speechd module.

Create a separate script outside speechd code that has the downloader (this will likely be in python).

Add documentation on how to use the downloader.

With regards to the search path for the distro's.

Search if the models are in ~/.cache/speech-dispatcher/kitten (for user provided models).

Check if a path is provided in the module's .conf file. Something like a line for modelsPath "<path>" distro's then should be able to place the models where ever they desire and add a line in the conf file.

upstream of these models could simply document where these are supposed to be shipped

Unfortunately I don't think hugging face models have a way of doing this currently.

It would really just be a matter of documenting it in README or website or whatever, nothing fancy...

I made all those changes in the latest commit.

… file.

sthibaul · 2026-08-29T21:20:09Z

bleh, "ubuntu-latest" doesn't seem to really be "latest", the CI is taking 24.04 rather than 26.04... Let me see that

sthibaul · 2026-08-29T21:23:27Z

bleh, "ubuntu-latest" doesn't seem to really be "latest", the CI is taking 24.04 rather than 26.04... Let me see that

Could you rebase? I have bumped to 26.04 so onnxruntime is available.

sthibaul · 2026-08-31T20:23:02Z

You need to add kitten.h to EXTRA_DIST in Makefile.am so it gets shipped in the distributed tarball.

- The downloaded has been moved to a seperate package, documentation has been added on how to use the downloader. - modules dot conf file can now set the models+voices search path.

sthibaul · 2026-09-01T20:18:48Z

+      [kitten_libxml="yes"],
+      [AS_IF([test $with_kitten = "yes"],
+        [AC_MSG_FAILURE([libxml-2.0 is not available])])])
+   AS_IF([test "x$kitten_libonnxruntime" = "xyes" -a "x$kitten_libxml" = "xyes" -a "x$kitten_libcurl" = "xyes"],


You need to drop the libcurl condition, and you can drop it from the github workflow

I must have over looked that. Interesting that it did not cause my ./configure to fail.

./configure is meant to avoid failing unless the user explicitly requested --enable-kitten

sthibaul

we're getting close :)

sthibaul · 2026-09-01T20:21:31Z

@@ -0,0 +1,23 @@
+# Module conf file options.


Please rather put an example configuration file, otherwise it will be too hard for users to work out what it should look like. You can then stuff this information in the example configuration file.

Really, rather put the information of README.kitten.md in the configuration file example. That's the best way that users will see it and know how to configure it. Otherwise distributions will at worse not install the readme file, at best put it in /usr/share/doc/speech-dispatcher, where users won't find it, and be at a loss when trying to configure /etc/speech-dispatcher/config/kitten.conf

sthibaul · 2026-09-01T20:25:05Z

@@ -0,0 +1,23 @@
+# Module conf file options.
+
+The modules dot conf file can be used to configure the module using `AddVoiceFile` and `modelPath`. `AddVoiceFile` let you set models/voices in the case of updated models. `modelPath` let you set the modes/voices search path.


You can set the model in the example file to /usr/share/kitten, to give people an idea where the files should be put as a distribution (to be shared with other software using kitten models)

I updated the readme with examples.

sthibaul · 2026-09-01T20:46:59Z

+
+static const configoption_t options[] = {
+	{"AddVoiceFile", ARG_LIST, cb_voicefiles, NULL, CTX_ALL},
+	{"modelPath", ARG_STR, cb_modelPath, NULL, CTX_ALL},


Better add a Debug option, like the other modules, and make use of MSG and DBG instead of always printing to stderr, so people don't have a log file growly a lot, but can still easily add debugging output when something goes wrong and you need information from the execution on the user machine to investigate.

I'm confused on what we are talking about here. Are we talking about replacing the fprintf(stderr, functions(there in quite a few locations) with MSG/DBG. Or is this something that is added to the dot conf stuff?

yes, and add a Debug dot conf option that sets the Debug variable, that the MSG/DBG calls will use to determine whether they should actually print their content or not. That is very convenient to get users to provide debugging information easily

sthibaul · 2026-09-01T21:04:29Z

The CI said:

kitten_worker.c:416:57: warning: format ‘%d’ expects argument of type ‘int’, but argument 3 has type ‘size_t’ {aka ‘long unsigned int’} [-Wformat=]

This indeed should rather be a %zd, to properly print a size_t

sthibaul · 2026-09-01T21:05:41Z

The CI also said:

[8](https://github.com/brailcom/speechd/actions/runs/33558231873/job/100024779175?pr=1107#step:9:149)
kitten_worker.c: In function ‘parse_ssml_to_gqueue’:
kitten_worker.c:274:13: warning: ‘content’ is deprecated [-Wdeprecated-declarations]
  274 |             pl->text = g_string_new(buffer->content);
      |             ^~
In file included from /usr/include/libxml2/libxml/parser.h:20,
                 from kitten.h:28,
                 from kitten_worker.c:13:
/usr/include/libxml2/libxml/tree.h:115:14: note: declared here
  115 |     xmlChar *content XML_DEPRECATED_MEMBER;
      |              ^~~~~~~
kitten_worker.c:285:5: warning: ‘use’ is deprecated [-Wdeprecated-declarations]
  285 |     if (buffer->use > 0) {
      |     ^~
/usr/include/libxml2/libxml/tree.h:121:18: note: declared here
  121 |     unsigned int use XML_DEPRECATED_MEMBER;
      |                  ^~~
kitten_worker.c:289:9: warning: ‘content’ is deprecated [-Wdeprecated-declarations]
  289 |         pl->text = g_string_new(buffer->content);
      |         ^~
/usr/include/libxml2/libxml/tree.h:115:14: note: declared here
  115 |     xmlChar *content XML_DEPRECATED_MEMBER;
      |              ^~~~~~~

The xml header mentions that one should rather use xmlBufferContent and xmlBufferLength.

…to the logs - Fixed depr warning with libxml content and use - Fixed potental bug that could happen if the phonemizer failed to load

jsett added 9 commits August 5, 2026 20:01

kitten tts module for speechd

de1ef6c

Remove the buffering code. Everything runs fine without it. Fixed an …

8d7a4db

…issue when processing text over 400 char. Fix an issue when we get unmapped chars.

Added instructions into the readme on building/install

01cb7d1

small bug

2b2a6de

change stuff in model_change_voice to use Gstring's instead of char*'s.

a5ab64f

mark need to be a char*

3a611e6

updated readme.

9f5bf7e

updated readme

a284a5c

removed some old dependencies.

d188bf5

sthibaul requested changes Aug 13, 2026

View reviewed changes

jsett added 5 commits August 14, 2026 16:08

Updated the make and configure file for optional support. Added depen…

5a8baa9

…dencies to the CI

make sure that if any of our init's fail it get reported as a failure.

4a082cc

placed in license text

0249b33

added checks for a distro path.

b41356a

forgot to change this comment

acebf8e

unlimitedsola reviewed Aug 17, 2026

View reviewed changes

allow model/voice files to be configured throught the models dot conf…

b1ce2a2

… file.

jsett requested a review from sthibaul August 26, 2026 19:47

Merge branch 'brailcom:master' into kitten

9d015f4

jsett added 2 commits August 31, 2026 20:40

placed kitten.h into EXTRA_DIST

5198ca7

- Removed the downloader from the speechd module.

a8c493d

- The downloaded has been moved to a seperate package, documentation has been added on how to use the downloader. - modules dot conf file can now set the models+voices search path.

sthibaul reviewed Sep 1, 2026

View reviewed changes

sthibaul requested changes Sep 1, 2026

View reviewed changes

jsett added 3 commits September 1, 2026 20:26

libcurl var should have been removed

10bf356

added example configuration file

356abac

moved the example in the readme to be inline.

4272cc6

sthibaul reviewed Sep 1, 2026

View reviewed changes

Comment thread src/modules/README.kitten.md

sthibaul reviewed Sep 1, 2026

View reviewed changes

Comment thread .github/workflows/ci.yml Outdated

sthibaul reviewed Sep 1, 2026

View reviewed changes

removed curl from the ci

a794550

- Added a debug conf option that limits the amount of output printed …

b6db9f5

…to the logs - Fixed depr warning with libxml content and use - Fixed potental bug that could happen if the phonemizer failed to load

		@@ -0,0 +1,23 @@
		# Module conf file options.

		The modules dot conf file can be used to configure the module using `AddVoiceFile` and `modelPath`. `AddVoiceFile` let you set models/voices in the case of updated models. `modelPath` let you set the modes/voices search path.

Uh oh!

Conversation

jsett commented Aug 12, 2026

KittenTTS Module.

kitten_server.c:

kitten_worker.c:

kitten_model.c:

kitten_downloader.c:

Uh oh!

sthibaul left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

unlimitedsola Aug 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sthibaul commented Aug 29, 2026

Uh oh!

sthibaul commented Aug 29, 2026

Uh oh!

sthibaul commented Aug 31, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sthibaul left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sthibaul Sep 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

sthibaul Sep 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sthibaul Sep 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

unlimitedsola Aug 26, 2026 •

edited

Loading

sthibaul Sep 1, 2026 •

edited

Loading

sthibaul Sep 1, 2026 •

edited

Loading

sthibaul Sep 1, 2026 •

edited

Loading