feat: add Postgres storage with Scalasql and Hikari; cleanup deprecated code and configs#1164
Open
warcholjakub wants to merge 10 commits intoscalacenter:mainfrom
Open
feat: add Postgres storage with Scalasql and Hikari; cleanup deprecated code and configs#1164warcholjakub wants to merge 10 commits intoscalacenter:mainfrom
warcholjakub wants to merge 10 commits intoscalacenter:mainfrom
Conversation
…ed code and configs - Introduced new Postgres storage implementation using Scalasql and HikariCP connection pool - Removed deprecated components such as UsersContainer - Updated and cleaned up MongoDB config keys - Added configuration files and keys for Postgres support - Removed deprecated storage tests - Integrated Flyway for database migrations and added SQL migration scripts - Added custom encoders/decoders for existing types where needed
warcholjakub
force-pushed
the
database-change
branch
from
eba4c38 to
ad45919
Compare
Contributor
There was a problem hiding this comment.
Pull request overview
This PR introduces a comprehensive Postgres storage implementation for Scastie using Scalasql and HikariCP, while cleaning up deprecated code related to user privacy policy management that was scheduled for removal.
Changes:
- Added a complete Postgres storage backend with Scalasql ORM, HikariCP connection pooling, and Flyway database migrations
- Removed deprecated
UsersContainertrait and all privacy policy-related user management code across the codebase - Renamed
Problem.linefield toProblem.startLinefor consistency and clarity across all usage sites - Updated MongoDB configuration keys from
scastie.mongodbtoorg.scastie.mongodbfor namespace consistency
Reviewed changes
Copilot reviewed 49 out of 50 changed files in this pull request and generated 12 comments.
Show a summary per file
| File | Description |
|---|---|
| storage/src/main/scala/org/scastie/storage/postgres/* | New Postgres storage implementation with container, converters, and table definitions |
| storage/src/main/resources/db/migration/*.sql | Flyway migration scripts defining the Postgres database schema |
| storage/src/main/scala/org/scastie/storage/UsersContainer.scala | Removed deprecated UsersContainer trait |
| storage/src/main/scala/org/scastie/storage/*/Filesystem/MongoDB/InMemoryUsersContainer.scala | Removed all UsersContainer implementations |
| storage/src/main/scala/org/scastie/storage/SnippetsContainer.scala | Removed readOldSnippet method |
| storage/src/main/scala/org/scastie/storage/*/FilesystemSnippetsContainer.scala | Removed readOldSnippet implementation |
| api/src/main/scala/org/scastie/api/CompilerInfo.scala | Renamed Problem.line to Problem.startLine |
| api/src/main/scala/org/scastie/api/SnippetId.scala | Added fromString method for parsing snippet IDs from URL strings |
| api/src/main/scala/org/scastie/api/ScalaTarget.scala | Changed encoder/decoder to use tagged union format for Postgres storage |
| runtime-api/src/main/scala/org/scastie/runtime/api/* | Added JSON string parsing methods for RuntimeError, Instrumentation, Position, and Render |
| server/src/main/scala/org/scastie/server/RestApiServer.scala | Removed deprecated privacy policy API methods |
| server/src/main/scala/org/scastie/server/routes/ApiRoutes.scala | Removed privacy policy API routes |
| client/src/main/scala/org/scastie/client/* | Removed PrivacyPolicyPrompt component and related state/backend methods |
| balancer/src/main/scala/org/scastie/balancer/DispatchActor.scala | Added Postgres container support; removed privacy policy message handlers |
| storage/src/test/scala/org/scastie/storage/ContainerTest.scala | Updated to support Postgres container testing; removed UsersContainer tests |
| sbt-runner/src/test/scala/org/scastie/sbt/SbtActorTest.scala | Updated test assertions for startLine field rename |
| scala-cli-runner/src/test/scala/org/scastie/scalacli/ScalaCliRunnerTest.scala | Updated test assertions for startLine field rename |
| sbt-runner/src/main/scala/org/scastie/sbt/* | Updated code for startLine field rename |
| instrumentation/src/main/scala/org/scastie/instrumentation/InstrumentedInputs.scala | Updated for startLine field rename |
| client/src/main/scala/org/scastie/client/components/editor/CodeEditor.scala | Updated editor diagnostics for startLine field |
| server/src/main/resources/reference.conf | Updated MongoDB config key and added Postgres configuration |
| deployment/*.conf | Updated to use Postgres storage; added Postgres config template |
| project/Deployment.scala | Added Postgres configuration validation in deployment script |
| scripts/PostgresMigrationScript.scala | Added MongoDB to Postgres migration script |
| build.sbt | Added dependencies for Scalasql, HikariCP, Flyway, and Chimney |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+13
to
+40
| object Render { | ||
| /* Convert JSON string to Render */ | ||
| /* Example: {"Value":{"v":"some value","className":"some class"}} -> Value("some value", "some class") */ | ||
| def fromJsonString(json: String): Option[Render] = { | ||
| if (json.startsWith("""{"Value":""")) { | ||
| val vPattern = """"v":"(.*?)"""".r | ||
| val classNamePattern = """"className":"(.*?)"""".r | ||
| for { | ||
| v <- vPattern.findFirstMatchIn(json).map(_.group(1)) | ||
| className <- classNamePattern.findFirstMatchIn(json).map(_.group(1)) | ||
| } yield Value(v, className) | ||
| } else if (json.startsWith("""{"Html":""")) { | ||
| val aPattern = """"a":"(.*?)"""".r | ||
| val foldedPattern = """"folded":(true|false)""".r | ||
| for { | ||
| a <- aPattern.findFirstMatchIn(json).map(_.group(1)) | ||
| folded <- foldedPattern.findFirstMatchIn(json).map(_.group(1).toBoolean) | ||
| } yield Html(a, folded) | ||
| } else if (json.startsWith("""{"AttachedDom":""")) { | ||
| val uuidPattern = """"uuid":"(.*?)"""".r | ||
| val foldedPattern = """"folded":(true|false)""".r | ||
| for { | ||
| uuid <- uuidPattern.findFirstMatchIn(json).map(_.group(1)) | ||
| folded <- foldedPattern.findFirstMatchIn(json).map(_.group(1).toBoolean) | ||
| } yield AttachedDom(uuid, folded) | ||
| } else None | ||
| } | ||
| } |
There was a problem hiding this comment.
The regex pattern (.*?) used to extract JSON values doesn't handle escaped quotes or other special characters correctly. If the JSON values contain escaped quotes (e.g., "value with \" quote"), the regex will incorrectly match up to the first escaped quote. Consider using a proper JSON parser (like Circe which is already a dependency) instead of regex-based parsing for more robust and correct deserialization.
Comment on lines
+18
to
+34
| def fromJsonString(json: String): Option[RuntimeError] = { | ||
| val msgPattern = """"message":"(.*?)"""".r | ||
| val linePattern = """"line":(null|\d+)""".r | ||
| val stackPattern = """"fullStack":"(.*?)"""".r | ||
|
|
||
| for { | ||
| msgMatch <- msgPattern.findFirstMatchIn(json) | ||
| stackMatch <- stackPattern.findFirstMatchIn(json) | ||
| lineMatch <- linePattern.findFirstMatchIn(json) | ||
| msg = msgMatch.group(1) | ||
| stack = stackMatch.group(1) | ||
| line = lineMatch.group(1) match { | ||
| case "null" => None | ||
| case numStr => Some(numStr.toInt) | ||
| } | ||
| } yield RuntimeError(msg, line, stack) | ||
| } |
There was a problem hiding this comment.
Similar to the Render.fromJsonString, this regex-based JSON parsing doesn't handle escaped quotes or nested JSON structures correctly. The pattern (.*?) will fail on complex JSON strings. Using a proper JSON parser would be more robust and correct.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR introduces a new Postgres storage implementation using Scalasql and HikariCP for connection pooling.
Key changes include
UsersContainerand related legacy storage tests.