How to search for long hashes and IDs with dict='keywords_32k'

[githubmanticore]

Originally published on the Manticore Search website on August 30, 2026

How to search for long hashes and IDs with dict='keywords_32k'

A practical guide to searching long hashes, event IDs, message IDs, and email addresses in Manticore Search: limits, exact matching, wildcard search, tokenization, migration, and limitations.

Full-text search usually works with ordinary words: product names, titles, comments, and descriptions. Such tokens are rarely longer than a few dozen characters.

Logs and technical data are different. A SHA-256 hash is 64 characters long, while message IDs, correlation keys, event identifiers, and some email addresses can be even longer. The value is often meaningful only as a whole: if its tail is lost, one ID can easily be confused with another.

Manticore Search provides dict='keywords_32k' for these cases.

keywords_32k is available starting with Manticore Search 27.1.1. Version 27.1.5 or newer is recommended when converting an existing table from keywords.

The problem with the regular dictionary

By default, Manticore uses dict='keywords'. Its maximum token length is 42 bytes after normalization.

The limit is measured in bytes, not characters. One ASCII character takes one byte, but a UTF-8 character can take several bytes.

If a token exceeds 42 bytes, Manticore truncates it:

• when indexing a document;
• when processing a search query.

As a result, querying the complete long value does not necessarily return zero results. Because the query is truncated too, the document may be found—but only by the first 42 bytes.

This creates a more serious problem: two different IDs with the same first 42 bytes become indistinguishable to full-text search. It is also impossible to find a value by a fragment located after the 42nd byte.

What keywords_32k changes

dict='keywords_32k' increases the maximum normalized token length to 32768 bytes, or 32 KB.

Behavior	dict='keywords'	dict='keywords_32k'
Maximum token length	42 bytes	32768 bytes
Token exceeding the limit	Truncated	Skipped with a warning
Prefix and infix search	Supported	Supported
Morphology for tokens longer than 42 bytes	Token is already truncated	Not applied
RT tables	Supported	Supported
Plain tables	Supported	Supported

The setting applies to the entire table:

CREATE TABLE events (
  message text,
  event_id text
)
dict='keywords_32k';

Regular short words in the same table continue to use the configured morphology. Tokens longer than 42 bytes are stored in their original normalized form, without stemming or lemmatization.

For machine identifiers, this is usually exactly what you need: a hash or event ID has no useful word stem.

When to use keywords_32k

Use it when both conditions are true:

1. The value can exceed 42 bytes after tokenization.
2. It must be searchable through MATCH(), by prefix, or by substring.

Typical examples include:

• SHA-256 and other long hashes;
• event IDs and message IDs;
• request IDs, trace IDs, and other technical identifiers;
• long record keys;
• email addresses with a long local or domain part;
• technical values from logs;
• long identifiers containing separators.

However, keywords_32k is not required for every ID lookup.

If you only need exact equality

If the application always receives the complete ID and only needs to check exact equality, a string attribute is sufficient:

CREATE TABLE events (
  message text,
  event_id string
);

Then use a regular filter:

SELECT id, message
FROM events
WHERE event_id =
  '9f86d081884c7d659a2feaa0c55ad015a3bf4f1b2b0b822cd15d6c15b0f00a08';

If you need both exact matching and full-text search

Use string attribute indexed:

CREATE TABLE events (
  message text,
  event_id string attribute indexed
)
dict='keywords_32k';

In this case, Manticore:

• stores the original value as a string attribute;
• lets you filter it with WHERE;
• also indexes it for MATCH() and wildcard search.

This is usually the most convenient schema for technical identifiers.

Searching by the complete token

Create a table and add a 64-character SHA-256 hash:

DROP TABLE IF EXISTS events;

CREATE TABLE events (
  message text,
  event_id string attribute indexed
)
dict='keywords_32k';

INSERT INTO events (id, message, event_id) VALUES
(
  1,
  'delivery accepted',
  '9f86d081884c7d659a2feaa0c55ad015a3bf4f1b2b0b822cd15d6c15b0f00a08'
);

Use the complete normalized token in a full-text search:

SELECT id, message
FROM events
WHERE MATCH(
  '@event_id 9f86d081884c7d659a2feaa0c55ad015a3bf4f1b2b0b822cd15d6c15b0f00a08'
);

The @event_id operator restricts the search to the required field. Without it, Manticore searches for the value in all full-text fields of the table.

Quotes are not needed around a single simple alphanumeric token here. Quotes denote a phrase search; they do not turn MATCH() into a byte-for-byte comparison of the original string.

A full-text match is not the same as exact equality

MATCH() operates on the result of tokenization and normalization. It can be affected by:

• charset_table;
• conversion to lowercase;
• blend_chars;
• ignore_chars;
• word forms and other text-processing settings.

For a strict comparison of the stored value, use a string attribute:

SET collation_connection='binary';

SELECT id, message
FROM events
WHERE event_id =
  '9f86d081884c7d659a2feaa0c55ad015a3bf4f1b2b0b822cd15d6c15b0f00a08';

binary enables byte-for-byte string comparisons in the current SQL session. This setting does not affect full-text search behavior.

The practical rule is simple:

• WHERE event_id = ... — a strict comparison of the stored string;
• MATCH('@event_id ...') — a normalized-token search;
• MATCH('@event_id prefix*') — a prefix search;
• MATCH('@event_id *fragment*') — a substring search.

How to inspect tokenization

Before loading a large volume of data, check that Manticore actually sees the value as a single token:

CALL KEYWORDS(
  '9f86d081884c7d659a2feaa0c55ad015a3bf4f1b2b0b822cd15d6c15b0f00a08',
  'events'
);

The normalized column should contain the complete 64-character hash.

CALL KEYWORDS is especially useful for values containing:

• periods;
• hyphens;
• the @ symbol;
• colons;
• slashes;
• characters from different writing systems.

This lets you inspect the actual token boundaries before indexing the data.

Prefix and substring search

To search inside a token, enable min_infix_len:

DROP TABLE IF EXISTS events_infix;

CREATE TABLE events_infix (
  message text,
  event_id string attribute indexed
)
dict='keywords_32k'
min_infix_len='4';

INSERT INTO events_infix (id, message, event_id) VALUES
(
  1,
  'delivery accepted',
  '9f86d081884c7d659a2feaa0c55ad015a3bf4f1b2b0b822cd15d6c15b0f00a08'
);

Prefix search:

SELECT id, message
FROM events_infix
WHERE MATCH('@event_id 9f86d081*');

Substring search:

SELECT id, message
FROM events_infix
WHERE MATCH('@event_id *b2b0b822*');

A positive min_infix_len also enables prefix search. This example uses 4 to prevent excessively short and broad patterns.

Why short patterns are dangerous

With dict='keywords_32k', as with regular dict='keywords', Manticore does not precompute every possible substring. Instead, at query time it expands a wildcard pattern into matching dictionary terms.

For example, the pattern *ab* may match a huge number of values. The more expansions it produces, and the more documents that contain each matching term, the more expensive the query becomes.

For production systems:

• do not allow users to search for excessively short fragments;
• choose min_infix_len based on real data;
• use expansion_limit to limit the number of expansions;
• test performance with a dictionary close to production size;
• restrict the search to a specific field with @field.

index_exact_words='1' is not required for wildcard search itself. You need it if you want to distinguish exact matches from wildcard matches when ranking, usually together with expand_keywords.

Email addresses and other values with separators

keywords_32k changes only the maximum token length. It does not determine where a token begins and ends.

By default, a period, @, hyphen, and other characters may split a value into multiple parts. If an email address or message ID must also be indexed as a whole, you can use blend_chars:

DROP TABLE IF EXISTS mail_events;

CREATE TABLE mail_events (
  sender string attribute indexed,
  subject text
)
dict='keywords_32k'
blend_chars='., @, -'
min_infix_len='4';

INSERT INTO mail_events (id, sender, subject) VALUES
(
  1,
  'alessandro.verylonggeneratedlocalpart@example-corporate-domain.test',
  'delivery accepted'
);

Blended characters are indexed in two ways:

• as part of the complete token;
• as separators between regular parts of the value.

This makes it possible to search both the complete email address and individual words within it.

Searching for the complete value:

SELECT id, subject
FROM mail_events
WHERE MATCH(
  '@sender "alessandro.verylonggeneratedlocalpart@example-corporate-domain.test"'
);

The quotes matter here because @ is also used in full-text query syntax. Inside a phrase, the parser can process it as a blended character.

Searching by a fragment:

SELECT id, subject
FROM mail_events
WHERE MATCH('@sender *generatedlocalpart*');

Inspect the tokenization result:

CALL KEYWORDS(
  '"alessandro.verylonggeneratedlocalpart@example-corporate-domain.test"',
  'mail_events'
);

In a real application, values passed to MATCH() must be escaped correctly. Simply appending user input to a query string can change the query's meaning because of @, -, |, !, ", *, and other operators.

How to convert an existing table

For an RT table, you can change the setting with ALTER TABLE:

ALTER TABLE events dict='keywords_32k';

However, this affects only documents added or replaced after the setting is changed.

Existing documents are not automatically tokenized again. Their long tokens remain in the old truncated form until the documents are reindexed.

Follow these steps:

1. Update dict.
2. Verify the setting with SHOW CREATE TABLE.
3. Reindex or reload the existing documents.
4. Check several long values with CALL KEYWORDS and MATCH().

For a plain table:

1. Change the configuration to dict = keywords_32k.
2. Apply the settings with ALTER TABLE ... RECONFIGURE if this fits your update workflow.
3. Rebuild the table completely from its data source.

Until the data is reindexed, the same table may contain both:

• old documents with truncated tokens;
• new documents with complete tokens.

This can produce different results for documents that appear identical.

Why dict='crc' does not solve this problem

dict='crc' stores keyword checksums instead of their original text. However, it does not increase the allowed token length.

The exception to the regular 42-byte limit is implemented specifically by dict='keywords_32k'.

The keywords and keywords_32k dictionaries also store term text, which allows Manticore to expand prefix and infix wildcard queries against the dictionary.

If you need to search long machine identifiers, crc is not a replacement for keywords_32k.

Current limitations

At the time of publication, dict='keywords_32k' has several limitations:

• CALL SUGGEST and CALL QSUGGEST are not supported;
• it cannot be used in percolate tables;
• tokens longer than 42 bytes are not highlighted in snippets or highlights;
• indextool --dumpdict cannot dump this type of dictionary;
• the full-text REGEX operator works with dict='keywords', but not with keywords_32k.

The last point should not be confused with the REGEX() function for filtering string attributes. If the field is declared as string attribute indexed, attribute filtering and full-text search remain two separate mechanisms.

Do not index secrets

Being able to search a long value does not mean that you should store it in a search index.

Do not index the following unless absolutely necessary:

• API keys;
• bearer tokens;
• session cookies;
• private keys;
• passwords and reset tokens;
• other data that grants access to the system.

keywords_32k solves the search problem, but does not protect the value from being read by users, backups, query logs, or system administrators.

If a secret must be matched by its exact value, it is safer to calculate a suitable hash in advance and store only the hash.

Quick checklist

Before enabling keywords_32k, check:

1. Is the normalized token actually longer than 42 bytes?
2. Do you need full-text or wildcard search rather than only WHERE value = ...?
3. Does CALL KEYWORDS see the entire value as one token?
4. Do you need blend_chars for periods, hyphens, @, and other separators?
5. Is min_infix_len large enough?
6. Is the number of wildcard expansions limited?
7. Have the old documents been reindexed?
8. Is the field free of secret data?
9. Does the application avoid relying on highlighting, SUGGEST, percolate, or full-text REGEX?

Summary

dict='keywords_32k' solves a specific problem: it allows a full-text index to store normalized tokens up to 32768 bytes long instead of the regular 42 bytes.

It is well suited to long hashes, event IDs, message IDs, email addresses, and other machine identifiers. Keep three points in mind:

• keywords_32k increases the token length but does not change tokenization rules;
• MATCH() on a complete token is not the same as a strict comparison of the original string;
• existing documents must be reindexed after the setting is changed.

If you only need exact equality, use a string attribute. If you need both exact matching and partial-value search, use string attribute indexed together with dict='keywords_32k'.

Documentation

• dict and keywords_32k limitations
• Token length limit
• Wildcard search settings
• blend_chars
• CALL KEYWORDS
• String attributes and indexed strings
• Updating full-text settings and reindexing
• Collations and string comparison
• Manticore Search 27.1.5