-
Notifications
You must be signed in to change notification settings - Fork 4
feat: add project settings migration script #30
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,26 @@ | ||
| # Copyright The Linux Foundation and each contributor to LFX. | ||
| # SPDX-License-Identifier: MIT | ||
|
|
||
| .PHONY: help run build clean test | ||
|
|
||
| help: ## Display available commands | ||
| @awk 'BEGIN {FS = ":.*?## "} /^[a-zA-Z_-]+:.*?## / {printf "\033[36m%-20s\033[0m %s\n", $$1, $$2}' $(MAKEFILE_LIST) | ||
|
|
||
| run: ## Run the migration script (requires PROJECT_UID) | ||
| @if [ -z "$(PROJECT_UID)" ]; then \ | ||
| echo "Usage: make run PROJECT_UID=<project-uid>"; \ | ||
| echo "Example: make run PROJECT_UID=7cad5a8d-19d0-41a4-81a6-043453daf9ee"; \ | ||
| exit 1; \ | ||
| fi | ||
| go run main.go $(PROJECT_UID) | ||
|
|
||
| build: ## Build the migration script | ||
| go build -o bin/migrate-project-settings main.go | ||
|
|
||
| clean: ## Clean built binaries | ||
| rm -rf bin/ | ||
|
|
||
| test: ## Run tests | ||
| go test -v ./... | ||
|
|
||
| .DEFAULT_GOAL := help |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,58 @@ | ||
| # Project Settings Migration Script | ||
|
|
||
| This script migrates project settings from the old format (where writers and auditors are arrays of strings) to the new format (where they are arrays of UserInfo objects with name, username, email, and avatar fields). | ||
|
|
||
| ## Usage | ||
|
|
||
| ```bash | ||
| cd scripts/migrate-project-settings | ||
| go run main.go <project-uid> | ||
| ``` | ||
|
|
||
| ## Environment Variables | ||
|
|
||
| - `NATS_URL`: NATS server URL (defaults to `nats://localhost:4222`) | ||
|
|
||
| ## What it does | ||
|
|
||
| 1. Connects to NATS and retrieves the project settings from the `project-settings` KV store | ||
| 2. Checks if the settings are already in the new format | ||
| 3. If in old format, prompts for user details (name, username, email, avatar) for each writer and auditor | ||
| 4. Updates the settings in the NATS KV store with the new format | ||
| 5. Sends an indexer sync message to update the search index | ||
|
|
||
| ## Example | ||
|
|
||
| ```bash | ||
| # Set NATS URL if different from default | ||
| export NATS_URL=nats://localhost:4222 | ||
|
|
||
| # Run migration for a specific project | ||
| go run main.go 7cad5a8d-19d0-41a4-81a6-043453daf9ee | ||
| ``` | ||
|
|
||
| The script will prompt you for each user: | ||
|
|
||
| ``` | ||
| Migrating writer: johndoe | ||
| Enter details for user 'johndoe': | ||
| Name: John Doe | ||
| Username [johndoe]: | ||
| Email: john.doe@example.com | ||
| Avatar URL (optional): https://example.com/avatar.jpg | ||
|
|
||
| Migrating auditor: janesmith | ||
| Enter details for user 'janesmith': | ||
| Name: Jane Smith | ||
| Username [janesmith]: | ||
| Email: jane.smith@example.com | ||
| Avatar URL (optional): | ||
| ``` | ||
|
|
||
| ## Notes | ||
|
|
||
| - The script preserves all existing project settings data | ||
| - Only writers and auditors fields are migrated from string arrays to UserInfo arrays | ||
| - The script updates the `updated_at` timestamp | ||
| - An indexer sync message is sent after successful migration | ||
| - If the project is already in the new format, the script will exit without making changes |
| Original file line number | Diff line number | Diff line change | ||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| @@ -0,0 +1,242 @@ | ||||||||||||||||||||||||||||||||
| // Copyright The Linux Foundation and each contributor to LFX. | ||||||||||||||||||||||||||||||||
| // SPDX-License-Identifier: MIT | ||||||||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||||||||
| package main | ||||||||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||||||||
| import ( | ||||||||||||||||||||||||||||||||
| "bufio" | ||||||||||||||||||||||||||||||||
| "context" | ||||||||||||||||||||||||||||||||
| "encoding/json" | ||||||||||||||||||||||||||||||||
| "fmt" | ||||||||||||||||||||||||||||||||
| "log/slog" | ||||||||||||||||||||||||||||||||
| "os" | ||||||||||||||||||||||||||||||||
| "strings" | ||||||||||||||||||||||||||||||||
| "time" | ||||||||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||||||||
| "github.com/linuxfoundation/lfx-v2-project-service/internal/domain/models" | ||||||||||||||||||||||||||||||||
| natsmsg "github.com/linuxfoundation/lfx-v2-project-service/internal/infrastructure/nats" | ||||||||||||||||||||||||||||||||
| "github.com/linuxfoundation/lfx-v2-project-service/pkg/constants" | ||||||||||||||||||||||||||||||||
| "github.com/nats-io/nats.go" | ||||||||||||||||||||||||||||||||
| "github.com/nats-io/nats.go/jetstream" | ||||||||||||||||||||||||||||||||
| ) | ||||||||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||||||||
| // OldProjectSettings represents the old format with string arrays | ||||||||||||||||||||||||||||||||
| type OldProjectSettings struct { | ||||||||||||||||||||||||||||||||
| UID string `json:"uid"` | ||||||||||||||||||||||||||||||||
| MissionStatement string `json:"mission_statement"` | ||||||||||||||||||||||||||||||||
| AnnouncementDate *time.Time `json:"announcement_date"` | ||||||||||||||||||||||||||||||||
| Auditors []string `json:"auditors"` // Old format: array of strings | ||||||||||||||||||||||||||||||||
| Writers []string `json:"writers"` // Old format: array of strings | ||||||||||||||||||||||||||||||||
| MeetingCoordinators []string `json:"meeting_coordinators"` | ||||||||||||||||||||||||||||||||
| CreatedAt *time.Time `json:"created_at"` | ||||||||||||||||||||||||||||||||
| UpdatedAt *time.Time `json:"updated_at"` | ||||||||||||||||||||||||||||||||
| } | ||||||||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||||||||
| func main() { | ||||||||||||||||||||||||||||||||
| if len(os.Args) != 2 { | ||||||||||||||||||||||||||||||||
| fmt.Fprintf(os.Stderr, "Usage: %s <project-uid>\n", os.Args[0]) | ||||||||||||||||||||||||||||||||
| os.Exit(1) | ||||||||||||||||||||||||||||||||
| } | ||||||||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||||||||
| projectUID := os.Args[1] | ||||||||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||||||||
| // Get NATS URL from environment or use default | ||||||||||||||||||||||||||||||||
| natsURL := os.Getenv("NATS_URL") | ||||||||||||||||||||||||||||||||
| if natsURL == "" { | ||||||||||||||||||||||||||||||||
| natsURL = "nats://localhost:4222" | ||||||||||||||||||||||||||||||||
| } | ||||||||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||||||||
| // Connect to NATS | ||||||||||||||||||||||||||||||||
| nc, err := nats.Connect(natsURL) | ||||||||||||||||||||||||||||||||
| if err != nil { | ||||||||||||||||||||||||||||||||
| slog.Error("Failed to connect to NATS", "error", err) | ||||||||||||||||||||||||||||||||
| os.Exit(1) | ||||||||||||||||||||||||||||||||
| } | ||||||||||||||||||||||||||||||||
| defer nc.Close() | ||||||||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||||||||
| // Create JetStream context | ||||||||||||||||||||||||||||||||
| js, err := jetstream.New(nc) | ||||||||||||||||||||||||||||||||
| if err != nil { | ||||||||||||||||||||||||||||||||
| slog.Error("Failed to create JetStream context", "error", err) | ||||||||||||||||||||||||||||||||
| os.Exit(1) | ||||||||||||||||||||||||||||||||
| } | ||||||||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||||||||
| // Get the project-settings KV store | ||||||||||||||||||||||||||||||||
| kv, err := js.KeyValue(context.Background(), constants.KVStoreNameProjectSettings) | ||||||||||||||||||||||||||||||||
| if err != nil { | ||||||||||||||||||||||||||||||||
| slog.Error("Failed to get project-settings KV store", "error", err) | ||||||||||||||||||||||||||||||||
| os.Exit(1) | ||||||||||||||||||||||||||||||||
| } | ||||||||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||||||||
| // Get current project settings | ||||||||||||||||||||||||||||||||
| entry, err := kv.Get(context.Background(), projectUID) | ||||||||||||||||||||||||||||||||
| if err != nil { | ||||||||||||||||||||||||||||||||
| slog.Error("Failed to get project settings", "uid", projectUID, "error", err) | ||||||||||||||||||||||||||||||||
| os.Exit(1) | ||||||||||||||||||||||||||||||||
| } | ||||||||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||||||||
| // Try to unmarshal as new format first | ||||||||||||||||||||||||||||||||
| var newSettings models.ProjectSettings | ||||||||||||||||||||||||||||||||
| if err := json.Unmarshal(entry.Value(), &newSettings); err == nil { | ||||||||||||||||||||||||||||||||
| // Check if it's already in new format (has UserInfo structs) | ||||||||||||||||||||||||||||||||
| if len(newSettings.Auditors) > 0 || len(newSettings.Writers) > 0 { | ||||||||||||||||||||||||||||||||
| // Check if first auditor/writer has the UserInfo structure | ||||||||||||||||||||||||||||||||
| if len(newSettings.Auditors) > 0 && newSettings.Auditors[0].Name != "" { | ||||||||||||||||||||||||||||||||
| fmt.Printf("Project %s is already in new format\n", projectUID) | ||||||||||||||||||||||||||||||||
| return | ||||||||||||||||||||||||||||||||
| } | ||||||||||||||||||||||||||||||||
| if len(newSettings.Writers) > 0 && newSettings.Writers[0].Name != "" { | ||||||||||||||||||||||||||||||||
| fmt.Printf("Project %s is already in new format\n", projectUID) | ||||||||||||||||||||||||||||||||
| return | ||||||||||||||||||||||||||||||||
| } | ||||||||||||||||||||||||||||||||
|
Comment on lines
+84
to
+91
|
||||||||||||||||||||||||||||||||
| } | ||||||||||||||||||||||||||||||||
|
Comment on lines
+81
to
+92
|
||||||||||||||||||||||||||||||||
| // Check if it's already in new format (has UserInfo structs) | |
| if len(newSettings.Auditors) > 0 || len(newSettings.Writers) > 0 { | |
| // Check if first auditor/writer has the UserInfo structure | |
| if len(newSettings.Auditors) > 0 && newSettings.Auditors[0].Name != "" { | |
| fmt.Printf("Project %s is already in new format\n", projectUID) | |
| return | |
| } | |
| if len(newSettings.Writers) > 0 && newSettings.Writers[0].Name != "" { | |
| fmt.Printf("Project %s is already in new format\n", projectUID) | |
| return | |
| } | |
| } | |
| // If unmarshaling into new format succeeds, treat as new format regardless of array contents | |
| fmt.Printf("Project %s is already in new format\n", projectUID) | |
| return |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Fix new‑format detection to avoid false negatives and hard exits
If JSON is valid for models.ProjectSettings, treat it as “already migrated.” Current Name checks can fail when Name is empty but arrays are objects, leading to an old‑format unmarshal error and exit.
Apply:
- // Try to unmarshal as new format first
+ // Try to unmarshal as new format first
var newSettings models.ProjectSettings
- if err := json.Unmarshal(entry.Value(), &newSettings); err == nil {
- // Check if it's already in new format (has UserInfo structs)
- if len(newSettings.Auditors) > 0 || len(newSettings.Writers) > 0 {
- // Check if first auditor/writer has the UserInfo structure
- if len(newSettings.Auditors) > 0 && newSettings.Auditors[0].Name != "" {
- fmt.Printf("Project %s is already in new format\n", projectUID)
- return
- }
- if len(newSettings.Writers) > 0 && newSettings.Writers[0].Name != "" {
- fmt.Printf("Project %s is already in new format\n", projectUID)
- return
- }
- }
- }
+ if err := json.Unmarshal(entry.Value(), &newSettings); err == nil {
+ fmt.Printf("Project %s is already in new format\n", projectUID)
+ return
+ }🤖 Prompt for AI Agents
In scripts/migrate-project-settings/main.go around lines 78 to 93, the code
attempts to detect the "new" ProjectSettings format by unmarshaling into
models.ProjectSettings but then rejects it if Name fields are empty, causing
false negatives and exiting; change the logic so that any successful
json.Unmarshal into models.ProjectSettings is treated as "already migrated" —
remove the additional checks on Auditors/Writers' Name fields and simply log
that the project is already in the new format and return when unmarshaling
succeeds, ensuring we don't perform further old-format processing or exit
erroneously.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🛠️ Refactor suggestion | 🟠 Major
Use optimistic concurrency (KV.Update) to avoid overwriting concurrent edits
Replace Put with Update using the fetched entry revision.
- _, err = kv.Put(context.Background(), projectUID, newSettingsBytes)
+ _, err = kv.Update(context.Background(), projectUID, newSettingsBytes, entry.Revision())
if err != nil {
slog.Error("Failed to update project settings in KV store", "error", err)
os.Exit(1)
}📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| // Update in NATS KV store | |
| _, err = kv.Put(context.Background(), projectUID, newSettingsBytes) | |
| if err != nil { | |
| slog.Error("Failed to update project settings in KV store", "error", err) | |
| os.Exit(1) | |
| } | |
| // Update in NATS KV store | |
| _, err = kv.Update(context.Background(), projectUID, newSettingsBytes, entry.Revision()) | |
| if err != nil { | |
| slog.Error("Failed to update project settings in KV store", "error", err) | |
| os.Exit(1) | |
| } |
🤖 Prompt for AI Agents
scripts/migrate-project-settings/main.go around lines 153-158: the code
currently calls kv.Put which can clobber concurrent updates; replace it with
KV.Update using the entry revision fetched from the store (call kv.Get or use
the previously fetched entry) and pass entry.Revision as the expected revision
to kv.Update, handle the possible conditional update errors (e.g., key not found
or revision mismatch) by logging and retrying or exiting as appropriate, and
ensure the context and error handling remain consistent with the surrounding
code.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Handle missing KV keys gracefully
If the project key is absent, exit with a clear message instead of a generic error.
Note: add imports if needed.
📝 Committable suggestion
🤖 Prompt for AI Agents