Repository navigation
Integration with embedding library #71
Description
Proposal:
We should add parsing of the parameters to enable connection with the new library for automatic embeddings.
Currently, I suggest starting with the following specification:
fields = "field1, field2": fields we should use as the source for text to generate embeddingsmodel_name = "openai/..."ormodel_name = "sentence-transformers/..."for local modelsapi_key = "...": for cases where we use a remote API for embeddings (we should be able to alter and change this one)use_gpu = 1|0: for cases where we use a local model and need to use GPU (if available), default is always CPU
Also we should consider to add new config:
embedding_cache_path = ...: path to the cache directory where we will store everything in case of local model usage
It may look like this:
CREATE TABLE test (
title TEXT,
image_vector FLOAT_VECTOR
KNN_TYPE='hnsw'
KNN_DIMS='4'
HNSW_SIMILARITY='l2'
MODEL_NAME = "..."
);Checklist:
To be completed by the assignee. Check off tasks that have been completed or are not applicable.
Details
- Implementation completed
- Tests developed
- Documentation updated
- Documentation reviewed
- Changelog updated
- OpenAPI YAML updated and issue created to rebuild clients
Activity
sanikolaev commentedon Oct 17, 2024
We may want to hide
api_keyinshow create tableoutput for security reasons.sanikolaev commentedon Oct 28, 2024
Dev call of Oct 28: ALTER should be supported for
api_keyglookka commentedon Oct 31, 2024
Adding new options is easy. However, it is not clear what should be done with these options
sanikolaev commentedon Nov 1, 2024
@donhardman pls advise.
donhardman commentedon Nov 4, 2024
We should parse it and store it somewhere to make it possible for the indexer stage to use it to generate embeddings with the lib as discussed before on call.
donhardman commentedon Nov 5, 2024
Here are points to consider:
model_name. This means we need a map of pointers to use in the future when indexing, where the key in this map ismodel_name.model_nameis specified, we should get dimensions from it by calling theget_hidden_sizemethod and not allow users to pass their ownknn_dimsas it would create a conflict. Therefore, whenmodel_nameis specified,knn_dimsis not required.model_nameparameter. If the specified model requires an API key and one is not provided when creating a pointer, an error will be thrown.The task primarily involves parsing rather than deep integration with the library.
59 remaining items
sanikolaev commentedon Jun 25, 2025
An empty vector means to generate it automatically.
glookka commentedon Jun 26, 2025
As of 7ef1e0d, if a vector is omitted from SQL insert (e.g.
insert into table (...) values (...)), embeddings are generated for such a vector. If an empty vector is inserted, it is excluded from KNN search.donhardman commentedon Aug 13, 2025
All tests were updated as required upon request, and new ones were added here: manticoresoftware/manticoresearch#3472
sanikolaev commentedon Aug 19, 2025
Some tests are still failing. Please update.
donhardman commentedon Sep 8, 2025
Closing as tests pass and old ones done and merged: manticoresoftware/manticoresearch#3472
github-actions commentedon Sep 8, 2025
Thanks for the update! However, the internal checklist is not yet complete. A Manticore team member will close this issue once everything is checked off. 🛠️
donhardman commentedon Sep 9, 2025
Clo