[Project] AI Standardization by akash9676 · Pull Request #239 · UKPLab/CARE

akash9676 · 2026-06-09T12:58:18Z

Summary

Introduces CARE’s AI stack (LiteLLM RPC, credentials/models/hooks/budgets/logs, admin dashboards) and an event-driven trigger system, plus prompt-template placeholders for AI hooks and study NLP workflows.

New User Features

Admin AI area: manage credentials, models, model sharing, hooks, budgets, and request logs from the dashboard
Configure AI hooks from prompt templates with ordered model fallbacks and output modes
Set spending limits on models, hooks, studies, and study steps; view token usage and cost in the AI log
Create triggers from predefined events and actions; view run history, cancel/retry/re-run queue jobs
Searchable provider select when creating AI credentials (LiteLLM providers)
Prompt templates support AI hook placeholders with preview/editing in the template UI
Study steps can use NLP request hooks with improved skill/input mapping in the workflow builder
Researcher docs for AI features and triggers (ai_features.rst, triggers.rst)

New Dev Features

LiteLLMRPC service + utils/rpcs/litellm Python bridge; docker-compose.yml / docker-dev.yml wiring
Modular AIService (chat, hook, request, runtime, helpers) exposed via serviceCommand
Database models and migrations for ai_* and trigger_* tables; nav/right migrations for new dashboard pages
TriggerSocket + triggerHandlers.js for trigger CRUD and queue management
MetaModel.foreignOwner / validateForeignUserId for AppSocket foreign-userId guard
templateResolver and placeholderTokens extended for prompt-template resolution
subscribeTable dashboards and $ai frontend plugin for AI service calls
Centralized triggerQueueStatus.js; budget cap tests in backend/tests/budget.caps.test.js
Socket, installation, and user-story documentation updated for AI and triggers

Improvements

Form Select gains searchable mode; Radio uses stable option-value keys
InputMap table-source rules aligned for Skills vs Hooks
Trigger log UI shows “Deleted Trigger” for missing trigger names
NLP service fallback/skill loading and fs.promises usage in nlp.js
Queue status labels/constants deduplicated; trigger transaction handling simplified
Settings nav group order adjusted (Triggers before Settings)

Bug Fixes

Select component value binding/emit logic corrected for form compatibility
AI budget default-false handling and column naming aligned with CARE patterns
Hook name removed from document data storage (uses workflow service name instead)
TriggerLogs display for deleted triggers

Known Limitations

apiKey is not yet excluded from broadcastTransactionChanges (only from sendTable) — credential saves may leak keys to the client store until fixed
PR still modifies tracked .env for RPC_LITELLM_*; vars should live in docs/.env.example only
abortChatCompletion does not yet verify the caller owns the ai_log request
studyNlpDocumentData.findAssessmentNlpService OR predicate does not match Assessment.vue AND check
API keys stored as plaintext at rest pending adoption of the cross-CARE encryption feature

Future Steps

Resolve the four blocking review items above before marking the PR ready
Run make doc and full CI on the branch
Adopt platform-wide credential encryption when available
Verify migration timestamps against dev before merge

… and update UI for testing functionality

…king and visualization of AI requests

…service response

…age handling in AI requests

…d model actions, enhancing UI clarity

…role, and study options, including UI components for selection and expiry date management

…components for improved modularity and maintainability

…data binding for user, role, and study selections

…service for detailed model insights

…nership validation with new utility methods

…nhance model validation logic

… functionality for AI model updates

…ons and improve model instance tracking

… for a credential and integrate with AIService and chat components

…error handling in model testing

…nd enhance model normalization logic

…ellation support

…l retrieval logic

…unexpected keyword arguments

…lution logic in chat and runtime services

…andling with streamlined functions in chat services

…label logic for improved clarity

…Completion and streamline request handling

…g messages for clarity

…amline model handling for improved clarity

…et default to null for improved flexibility

… related components for improved model loading functionality

…arity

…r user guidance

… improved soft-delete handling and transaction management

…ueue processing

…ing on unavailable hooks

…er resolution of unavailable hooks and improve documentation clarity

melolw · 2026-08-25T13:41:13Z

+    AiCredential.init({
+        userId: DataTypes.INTEGER,
+        name: DataTypes.STRING,
+        apiKey: DataTypes.TEXT,


apiKey is stored in plaintext, can't we do pgcrypto? or is it supposed to be that way?

Karim is working on the encryption so i can adapt it after he is done

melolw · 2026-08-25T13:46:44Z

      return this.$store.getters["table/configuration/getAll"] || [];
    },
+    // First validation configuration (type 1) — defines what files a submission must contain.
+    // TODO: support dynamicty here


is there a created issue for this TODO?

…ng logic to streamline server functionality

eyadmohamed01 · 2026-08-30T15:53:24Z

+          && Number.isFinite(costLimitValue)
+          && costLimitValue >= 0;
+        // Standard appDataUpdate chain: save the model, then update or create the ai_budget row.
+        if (wantsCap) {


(Similar problem as the ones in AIHookStepperModal.vue and AIModelShareStepper.vue) If an existing model has a cost cap and the user clears the "Cost limit" field in the frontend, the existing ai_budget row is left active because deleted is never set to true in the backend.

Could we detect that case and soft-delete the existing ai_budget row (deleted: true) so clearing the field in the frontend actually removes the cap in the backend? An else if (existing && !hasCostLimit) branch could handle this where const existing = this.findExistingCapRow(savedModelId); could be defined above if(wantsCap)

eyadmohamed01 · 2026-08-30T16:05:34Z

+
+        // Save / update the hook-level cost limit via appDataUpdate.
+        const costLimitValue = Number(this.hookForm.costLimit);
+        if (Number.isFinite(costLimitValue) && costLimitValue > 0) {


(Similar problem as the ones in AIModel.vue and AIModelShareStepper.vue) If an existing hook has a cost cap and the user clears the "Cost limit" field in the frontend, this condition becomes false and the ai_budget update is skipped, leaving the existing row active with deleted: false and the backend is not updated.

Could we detect when an existing cap has been cleared and soft-delete the corresponding ai_budget row (deleted: true) so the backend no longer enforces the old cap?

eyadmohamed01 · 2026-08-30T16:23:25Z

+        this.shareForm = {
+          mode: config.mode,
+          expiryDate: config.expiryDate ? this.toDateInputString(config.expiryDate) : "",
+          costLimit: null,


When editing an existing share, costLimit is initialized to null instead of being populated from the existing ai_budget cap. This can make the frontend show no limit even though an active cap still exists in the backend.

Could we pre-fill costLimit using findExistingShareCap() when opening an existing share?

eyadmohamed01 · 2026-08-30T16:30:58Z

+        // batch, via the standard appDataUpdate path.
+        const costLimitValue = Number(this.shareForm.costLimit);
+        const wantsCap = Number.isFinite(costLimitValue) && costLimitValue > 0;
+        if (wantsCap) {


(Similar problem as the ones in AIModel.vue and AIHookStepperModal.vue) If an existing share has a cost cap and the user clears the "Cost limit" field in the frontend, wantsCap becomes false and this block is skipped, leaving the existing ai_budget row active with deleted: false. This case is not handled here.

Could we detect when an existing cap has been cleared and soft-delete the corresponding ai_budget row (deleted: true) so the backend no longer enforces the old cap?

eyadmohamed01 · 2026-08-31T10:24:15Z


 -----

 **Extended form properties**


This PR also adds wrapperClass (e.g. in FormFields.vue) and labelButton (e.g. in Element.vue) as reusable field options, but they don't seem to be documented here yet.

Could we add them to the extended form properties so developers using BasicForm know these options are available?

eyadmohamed01 · 2026-08-31T10:27:20Z


 .. _Types:

 **Specific form properties**


This PR adds type: "button" (e.g. in FormFields.vue) as a new BasicForm field type and forwards its clicks through the new button-click event (e.g. Form.vue), but I don't see this documented in form.rst.

Could we add a Button subsection describing its options and the corresponding button-click event?

eyadmohamed01 · 2026-08-31T15:14:00Z

            await this.models["user_setting"].set(key, value, data.userId, { bypassSystemSettingCheck: true });
        } else {
            // Default: set for current user and refresh their settings
            console.log(`Setting ${key} for user ${this.userId} to ${value}`);


This looks like a leftover debug console.log in the backend. Could we remove it or use CARE's logger (e.g. this.logger.info(...)) instead to keep backend logging consistent?

dennis-zyska · 2026-08-31T11:21:30Z

+    if (!service) return [];
+
+    if (service.hookId) {
+        const keys = [service.name, service.type].filter(Boolean);


For hooks this returns the service name alone, but a trigger saves the result as nlpRequest_<hookName> (buildStudyHookKey, aiPreprocessing.js:135), so ~nlpAssessmentSuggestion~ never finds it. The step config already stores service.hookName and nothing reads it yet. Please build this key with buildStudyHookKey(service.name, service.hookName) and change the frontend twins (buildHookResultKey and NlpRequest.saveHookResult) to match, so all four sites use the same shape.

dennis-zyska · 2026-08-31T12:15:19Z

+                }
+            },
+        },
+        isAdmin: async () => true,


This stub always returns true, so the permission check in backgroundTask.js:104 (noPreprocessSubmissionPermission) passes on every trigger run, whatever rights the trigger owner currently has. Please check the real rights of trigger.userId here instead of hardcoding true.

dennis-zyska · 2026-08-31T16:01:43Z

+            server.db.models,
+            options
+        );
+        await server.sendMail(recipient.email, template.name, body, { isHtml: true });


server.sendMail hands the message to nodemailer with a callback and returns right away (Server.js:225), so a send error only reaches the logger and this await always resolves. The address is still pushed to sent and the queue item finishes as COMPLETED, so Trigger Logs reports a delivery that actually failed. Please make sendMail reject when the transport reports an error (drop the callback and await nodemailer's promise) and fail the run on that, while a disabled mail service keeps returning early without failing.

dennis-zyska · 2026-08-31T16:04:28Z

+        attributes: ["id", "userId", "provider", "apiKey", "apiBaseUrl", "apiVersion", "enabled"],
+    });
+    if (!credential) {
+        throw new Error("Credential not found");


These messages are plain English, but backend/webserver/services/ uses TranslatableError everywhere else (9 uses in dev, none here). createSocket only localizes a message that matches an i18n key, so these reach the client as raw English. Please throw TranslatableError with keys added to utils/modules/i18n/en/errors.json, here and for the other new throws in services/ai/.

eyadmohamed01 · 2026-08-31T22:57:24Z

+     * `configuration` carrying the action's collected data.
+     */
+    class Trigger extends MetaModel {
+        static autoTable = true;


Security issue / Blocking: Since trigger is writable through the generic appDataUpdate path, the isAdmin() checks in TriggerSocket can be bypassed. appDataUpdate has no equivalent admin check, and this model has no beforeCreate/beforeUpdate authorization hook, so a non-admin can create or modify trigger records directly.

Could we enforce the admin restriction at the model/data layer (and likewise for the related trigger tables, e.g. trigger_queue, trigger_action, trigger_event) so all write paths are protected?

eyadmohamed01 · 2026-09-01T00:17:00Z

@@ -85,7 +85,14 @@

        // check or set user information
        if ("userId" in data.data && !await this.checkUserAccess(data.data.userId)) {


Security issue / Blocking: validateForeignUserId() is only reached when checkUserAccess(userId) fails. If the caller shares another user's model to themselves (or omits userId and it later defaults to themselves) checkUserAccess() passes and the referenced model ownership check is skipped entirely. This allows a user to forge an ai_model_share for themselves, which _findActiveShare() later accepts as valid access.

Could we make the parent-ownership validation unavoidable for share writes, including update paths, ideally at the model layer, and apply the same fix to ai_hook_share?

eyadmohamed01 · 2026-09-01T01:11:07Z

+                throw new Error("Selected AI credential does not exist");
+            }
+
+            if (credential.userId !== aiModel.userId) {


Security issue / Blocking: This validates that the credential and model belong to the same user, but it does not verify that the authenticated requester (options.context.currentUserId) owns this ai_model. As a result, another authenticated user can update/disable/delete someone else's model as long as the existing credential-to-model ownership relationship is valid.

Could we also enforce aiModel.userId === options.context.currentUserId on updates?

eyadmohamed01 · 2026-09-01T02:12:16Z

+        allowNull: false,
+        defaultValue: Sequelize.fn('NOW'),
+      },
+    });


ai_log is queried frequently by combinations of userId, studySessionId, status, aiModelId, and later aiHookId, but this migration doesn't add indexes for those hot lookup paths. Since this table will grow with every AI request, these lookups and budget aggregations could become increasingly expensive.

Could we add indexes based on the actual query patterns here, similar to the indexes added for trigger / trigger_queue, and also review the share/budget tables for the same issue?

eyadmohamed01 · 2026-09-01T02:42:12Z

+        studyId, studySessionId, studyStepId, documentId,
+    } = request || {};
+
+    if (await _hasInflight(service, userId, studySessionId)) {


_hasInflight() and the subsequent ai_log.add() are separate DB operations, so two near-simultaneous requests can both observe no active row and then both create an in_progress log.

Could we make this single-flight check atomic (e.g. via a DB constraint/locking or another atomic claim mechanism) so duplicate concurrent requests cannot slip through?

eyadmohamed01 · 2026-09-01T04:22:10Z

+                priority > 1 &&
+                primaryRow &&
+                Number(primaryRow.id) !== currentId &&
+                Number(primaryRow.aiModelId) === aiModelId


Blocking: This validation also runs when an ai_hook_models row is being soft-deleted. For example, with primary A / fallback B, changing the primary to B commits that update first, then the subsequent { deleted: true } for the old fallback B is rejected here because it still appears as an active fallback matching the new primary during beforeUpdate. This leaves the model configuration partially updated.

Could we skip the priority/conflict validation for rows being deleted (after still verifying ownership), so cleanup deletes cannot be blocked by the state they're removing?

eyadmohamed01 · 2026-09-01T05:49:18Z

+      this.isLoadingProviders = true;
+      this.providerLookupError = "";
+      try {
+        const result = await this.emitAiServiceCommand("getProviders");


(Same problem as in AIModel.vue) This local socket wrapper has no client-side timeout. If the client disconnects after the request is sent but before the ack arrives, Socket.IO can abandon the callback, leaving the returned Promise pending and isLoadingProviders stuck until the modal is reset.

Could we route this through the existing $ai helper, or add equivalent client-side timeout handling here?

eyadmohamed01 · 2026-09-01T05:57:35Z

+      this.isLoadingModels = true;
+      this.modelLookupError = "";
+      try {
+        const result = await this.emitAiServiceCommand("getValidModels", {


(Same problem as in AICredential.vue) This local socket wrapper has no client-side timeout. If the client disconnects after the request is sent but before the ack arrives, Socket.IO can abandon the callback, leaving the returned Promise pending and isLoadingModels stuck until the modal is reset.

Could we route this through the existing $ai helper, or add equivalent client-side timeout handling here?

eyadmohamed01 · 2026-09-01T06:04:23Z

+      if (!this.$refs.form.validate()) return;
+
+      this.isTestingModel = true;
+      this.$socket.emit("serviceCommand", {


This call has no client-side timeout. If the socket disconnects after the request is sent but before the ack arrives, the callback may never run, so isTestingModel stays true and the test button remains stuck until the modal is reset.

Could we route this through the existing $ai helper or add equivalent timeout handling here?

eyadmohamed01 · 2026-09-01T07:07:57Z

+          const capData = existing
+            ? { id: existing.id, costLimit: costLimitValue }
+            : { aiModelId: Number(savedModelId), limitType: 0, costLimit: costLimitValue };
+          this.$socket.emit("appDataUpdate", { table: "ai_budget", data: capData }, (capResult) => {


The modal is closed and a success toast is shown immediately after sending the budget update, before capResult is known. If the budget write fails, the model has already been saved while the configured cost cap may not have been applied.

Could we wait for the budget acknowledgement before closing/showing success, and include client-side timeout handling so a disconnect cannot leave that wait pending indefinitely?

…d of names

akashgundapuneni@gmail.com added 30 commits May 4, 2026 14:54

feat(AIService): add testModel command for model usability validation…

6ae9059

… and update UI for testing functionality

feat(AI): enhance AI logging and dashboard components for better trac…

e882698

…king and visualization of AI requests

fix(AiLog): enable autoTable for AiLog model and update status in AI …

0aea285

…service response

feat(AIService): implement #extractInputText method for improved mess…

2e24800

…age handling in AI requests

refactor(AIModels): update button titles and styles for credential an…

c848888

…d model actions, enhancing UI clarity

feat(AIModelShare): implement model sharing functionality with user, …

803378f

…role, and study options, including UI components for selection and expiry date management

feat(AIModels): refactor AI credential and model forms into separate …

408789d

…components for improved modularity and maintainability

refactor(AIModelShareStepper): update selection handling and improve …

416adeb

…data binding for user, role, and study selections

feat(AIOverview): add model overview component and integrate with AI …

c217dc9

…service for detailed model insights

refactor(AIService): streamline command handling and enhance model ow…

36fa378

…nership validation with new utility methods

feat(AIService): implement cascading updates for AI credentials and e…

e0af6e9

…nhance model validation logic

feat(AIModelShareStepper): enhance selection handling and add refresh…

30c0e3b

… functionality for AI model updates

feat(AIService): add bulk update handling for AI credential transacti…

b048931

…ons and improve model instance tracking

feat(LiteLLMRPC): add getValidModels method to fetch available models…

6aa8946

… for a credential and integrate with AIService and chat components

fix(chat): update status from "test_failed" to "failed" for improved …

af3cabd

…error handling in model testing

feat(LiteLLMRPC): implement router functionality for model handling a…

5ccf4c3

…nd enhance model normalization logic

feat(LiteLLMRPC): enhance chatCompletion with async handling and canc…

59f554d

…ellation support

refactor(LiteLLMRPC): simplify credential management and enhance mode…

94fc5e5

…l retrieval logic

refactor(LiteLLMRPC): improve model retrieval robustness by handling …

4468743

…unexpected keyword arguments

refactor(AIService): streamline provider normalization and model reso…

cda15f0

…lution logic in chat and runtime services

refactor(AIService): replace response cost extraction and reasoning h…

7591a9c

…andling with streamlined functions in chat services

refactor(AIHelpers): simplify user display label and share recipient …

300e01e

…label logic for improved clarity

refactor(LiteLLMRPC, AIService): enforce __requestId presence in chat…

750f196

…Completion and streamline request handling

refactor(RPC, AI components): clean up unused code and improve loggin…

cc647e7

…g messages for clarity

refactor(AI components): remove unused credential properties and stre…

a48898c

…amline model handling for improved clarity

refactor(migration): update provider field to allow null values and s…

c2fe5f7

…et default to null for improved flexibility

refactor(AI components): add provider field to credentials and update…

b204048

… related components for improved model loading functionality

refactor(AI components): update credential label for improved user cl…

8c318f9

…arity

refactor(AI components): enhance input labels with tooltips for bette…

471d885

…r user guidance

refactor(AI components): enhance AiCredential and AiModel classes for…

ca5618f

… improved soft-delete handling and transaction management

akash9676 added 3 commits August 24, 2026 15:39

feat: add trigger management system with event context builders and q…

15abd9c

…ueue processing

fix: implement soft-skip logic for AI hook execution to prevent block…

d096220

…ing on unavailable hooks

refactor: enhance error handling in AI hook processing to ensure prop…

374d41e

…er resolution of unavailable hooks and improve documentation clarity

melolw reviewed Aug 25, 2026

View reviewed changes

refactor: remove TriggerManager class and associated trigger processi…

327f5fb

…ng logic to streamline server functionality

eyadmohamed01 reviewed Aug 30, 2026

View reviewed changes

mohammadsherif0 added 3 commits August 30, 2026 19:35

Merge branch 'dev' into project-1-ai-standardization

17b63d1

fix: check study access when a study step has no document

a68a54f

refactor: move template socket queries to model methods

1343106

eyadmohamed01 reviewed Aug 31, 2026

View reviewed changes

dennis-zyska requested changes Aug 31, 2026

View reviewed changes

eyadmohamed01 reviewed Aug 31, 2026

View reviewed changes

eyadmohamed01 reviewed Sep 1, 2026

View reviewed changes

UKPLab deleted a comment from akash9676 Sep 1, 2026

eyadmohamed01 reviewed Sep 1, 2026

View reviewed changes

akash9676 added 2 commits September 3, 2026 12:02

fix: enhance PDF text extraction by adding fallback for missing text

fd20504

refactor: standardize AI hook key generation by using hook IDs instea…

8078a7b

…d of names

bingobongomann force-pushed the project-1-ai-standardization branch from a60497c to 8078a7b Compare September 3, 2026 12:11

		@@ -85,7 +85,14 @@

		// check or set user information
		if ("userId" in data.data && !await this.checkUserAccess(data.data.userId)) {

Uh oh!

Conversation

akash9676 commented Jun 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

New User Features

New Dev Features

Improvements

Bug Fixes

Known Limitations

Future Steps

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

eyadmohamed01 Aug 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

eyadmohamed01 Aug 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

eyadmohamed01 Aug 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

eyadmohamed01 Aug 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

eyadmohamed01 Aug 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

eyadmohamed01 Aug 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

eyadmohamed01 Sep 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

eyadmohamed01 Sep 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

akash9676 commented Jun 9, 2026 •

edited

Loading

eyadmohamed01 Aug 30, 2026 •

edited

Loading

eyadmohamed01 Aug 30, 2026 •

edited

Loading

eyadmohamed01 Aug 30, 2026 •

edited

Loading

eyadmohamed01 Aug 31, 2026 •

edited

Loading

eyadmohamed01 Aug 31, 2026 •

edited

Loading

eyadmohamed01 Aug 31, 2026 •

edited

Loading

eyadmohamed01 Sep 1, 2026 •

edited

Loading

eyadmohamed01 Sep 1, 2026 •

edited

Loading