Feat 331 data export - #320
Conversation
| v-model:selectedWorkflowIds="selectedWorkflowIds" | ||
| v-model:includeEmptyStudies="includeEmptyStudies" | ||
| v-model:includeDocumentFiles="includeStudyDocumentFiles" | ||
| v-model:includeGrades="includeStudyGrades" |
There was a problem hiding this comment.
The parent binds v-model:includeGrades, but this child declares the prop and emit as includeScores, so the "Include scores" switch never reaches ExportModal. includeStudyGrades stays true, so includeGrades: true is always posted at line 505 and scores are exported even with the switch off. Change the binding here to v-model:includeScores="includeStudyGrades" and leave the POST key at line 505 as includeGrades, which is what the backend reads.
| const allDocIds = [doc.id, ...copies.map(c => c.id)]; | ||
|
|
||
| let [annotations, comments] = await Promise.all([ | ||
| server.db.models.annotation.findAll({ where: { documentId: allDocIds }, raw: true }), |
There was a problem hiding this comment.
These two queries are missing deleted: false, so soft-deleted annotations and comments end up in the export. The comment_vote query 12 lines below and the annotator-step queries in processStudyBasedExport both filter it, so this path is the odd one out. Add deleted: false here and to the copies lookup on line 423, which pulls deleted session copies for the same reason.
| ...annotations.map(a => a.userId), | ||
| ...comments.map(c => c.userId) | ||
| ].filter(Boolean))]; | ||
| const consentedUsers = await server.db.models.user.findAll({ |
There was a problem hiding this comment.
This is getConsentedUserIds written out by hand. The helper is already imported in this file and used in processDocumentForExport, so replace this block with const consentedIds = await getConsentedUserIds(server, allUserIds);. The same block is repeated again at line 741 for the editor steps.
| return { success: false, status: 400, message: "Unsupported export type." }; | ||
| } | ||
| workflowIds = typeof workflowIds === 'string' ? JSON.parse(workflowIds) : workflowIds; | ||
| if (!Array.isArray(workflowIds)) workflowIds = []; |
There was a problem hiding this comment.
An empty userIds is rejected with a 400 four lines below, but an empty workflowIds falls through. For a studies export that reaches processStudyBasedExport as workflowId: [], which Sequelize turns into IN (), so the user gets a 200 and an empty zip with no explanation. Reject it here the same way when exportType === "studies".
| * @param {Array<number>} documentTypes - List of document types to include (0=PDF, 1=HTML, 2=Modal, 4=ZIP). | ||
| * @returns {Promise<void>} | ||
| */ | ||
| async function processDocumentBasedExport(server, projectId, userIds, documentTypes, shouldExcludeNonConsentingEdits, shouldExcludeNonConsentingAnnotations, baseFolderName, archive) { |
There was a problem hiding this comment.
The split into exportGrades.js helped, but this route file is now 945 lines, up from 796 on dev, because the four new processors landed here rather than in a helper. processDocumentForExport, processDocumentBasedExport, processStudyBasedExport and processUserBehaviourExport are pure functions over server and archive, so they can move to utils/helper/ next to the others and leave the route as the request handler plus a switch.
|
|
||
| <script> | ||
| import BasicForm from "@/basic/Form.vue"; | ||
| import BasicButton from "@/basic/Button.vue"; |
There was a problem hiding this comment.
BasicButton is imported and registered but never used in the template.
| * @param {boolean} shouldGenerateAliases - Whether student names should be anonymized. | ||
| * @param {boolean} hasPrivateInfoRight - Whether the requester may export real names. | ||
| * @param {Object} userMapping - Map of user IDs to generated aliases. | ||
| * @returns {Promise<{records: Array<Object>, criteriaReferencesByConfigId: Map<number, Object>}>} |
There was a problem hiding this comment.
There are two docblocks stacked here and only the second one attaches to the function, so the @param and @returns tags in the first are dropped by JSDoc and by IDE tooltips. Merge them into one block: keep the options.sessionIds explanation from the second and the tag list from the first.
| return sorted; | ||
| } | ||
|
|
||
| async function processStudyBasedExport(server, projectId, userIds, users, shouldGenerateAliases, hasPrivateInfoRight, userMapping, workflowIds, shouldIncludeEmptyStudies, shouldExcludeNonConsentingEdits, shouldExcludeNonConsentingAnnotations, shouldIncludeDocumentFiles, shouldIncludeGrades, shouldIncludeAiScores, baseFolderName, archive) { |
There was a problem hiding this comment.
This function has 16 positional parameters and no JSDoc, while every other new function in this file has one. Six of them are booleans in a row (shouldIncludeEmptyStudies through shouldIncludeAiScores), so a call site is easy to get wrong and hard to read. Group the flags into one options object and add a docblock describing it.
| } | ||
| } | ||
|
|
||
| async function attachTagNames(server, annotations) { |
There was a problem hiding this comment.
attachTagNames, resolveIsAdmin (line 297) and createCsvRowsStream (line 359) have no JSDoc, while the other functions in this file all do. createJsonArrayStream has a partial one that documents fetchPage but not mapRow, pageSize, or the return value. Worth a pass over the four before merge.
| } | ||
| } | ||
|
|
||
| function sortSteps(items, prevKey) { |
There was a problem hiding this comment.
This is the same walk as workflow_step.getSortedWorkflowSteps (backend/db/models/workflow_step.js:61), generalised over the key name. CARE keeps that one as a static method on the model, so the matching home for this is study_step.getSortedStudySteps(studyId), which also makes it reusable outside the export.
2 participants
Summary
Adds a new data-export flow to the project dashboard, streaming zip archives directly from the backend instead of building them client-side. Extends the existing submissions/grades exports with new documents, studies and user behaviour export type, and reworks the shared export UI (user selection, per-type options, confirmation) to support it alongside the existing types.
Added Features
New export type: User Behaviour
statistictable rows (site usage/interaction logs) for selected users into the export archive, admin-gated (resolveIsAdmin).StepOptionsUserBehaviour.vue).dataobject) or CSV (flatteneddatafield) — same underlyingaction, data, timestamp, user, username, userId, sessionshape either way.id-based, notOFFSET) directly into the archive using customReadablestreams (createJsonArrayStream/createCsvRowsStreaminutils/helper/export.js), so memory use stays bounded regardless of table size — the naive full-table-dump approach (mirroring the existingstatsGetsocket handler) hitRangeError: Invalid string lengthon a 2M-row table during testing.user.acceptStats) instead of a count.Studies export
nlpRequestservice, split into a separatescores_ai.jsonper session (alongsidescores.jsonfor non-AI-assisted grades) — optional, default on.scores.jsoninstead ofgrades.json) for consistency with the new AI-scores split.Documents export
Shared user-selection table (
StepSelectUsers.vue, formerlyStepSelectStudents.vue)getOrCreateRow/forEachUserRowhelper, removing per-type duplication.user_role/user_role_matching) across all export types, following the same pattern as the grades "Assessment Configuration(s)" column: compact numeric IDs in the cell, filter dropdown resolves them to readable"id: name"labels (fixes filtering being broken for multi-value cells, which previously matched on a single concatenated string like"Multiple (A, B)").ExportModal.vue's newresetOptions()), fixing stale/mismatched selection counts (e.g. "174/173 selected") when switching between export types with different eligible-user sets.Grades export
tagNamefield (resolved fromtagId) alongside the existing raw ID, inserted immediately aftertagIdin each record.