Path to Scala 3

[felixbr]

This is an issue for discussing and tracking necessary step to prepare for a Dotty/Scala3 cross-build.

Inspired by the awesome "Scala Love" conference, I attempted to cross-build util-core with Dotty 0.23.0-RC1. So far it went smoother than expected. In the end there were about 4 compile-errors that I couldn't directly address.

In any case there's some work that has to be done before a real cross-build can be achieved. So far I've identified the following but I will try and update this list as we learn more.

Preparations:

• Update sbt to recent version (e.g. 1.3.10; needed by sbt-dotty)
• Update Scalatest to 3.1.1 (which is cross-published for Dotty)
• Update twitter/util to Scalatest version available for latest Dotty (Dependency on Twitter codebase) | @mosesn
• Remove deprecated Config code | @mosesn
• Wait for Dotty to fix varargs override issue (see Overriding an @varargs method fails under separate compilation scala/scala3#9463)
• Wait for Scalatest 3.2.0 to be published for Dotty version which includes the varargs fix
• Refactor code which is not supported in Scala 3 (e.g. usage of Manifest/TypeTag reflection)
• Add Scala 3 to the CI matrix to prevent regressions in already supported modules
• Cross-build and cross-publish modules which are needed by Finagle and twitter-server (e.g. util-test and util-tunable)

Resources for Dotty migration/cross-building:

• https://github.com/scalacenter/scala-3-migration-guide
• https://github.com/lampepfl/dotty-example-project#getting-your-project-to-compile-with-dotty

I hope you don't mind this type of issue. This is not meant to push anyone, just as a place to discuss and organize things.

Cheers
~ Felix

[mosesn]

@felixbr yep, we're perfectly happy to have those kinds of conversations here. If it turns into a big effort, we'll probably set up a ticket like twitter/finagle#771 for doing it. We're going to upgrade scalatest soon to support twitter/finatra#527.

I can file an internal ticket around upgrading our sbt version, but we're going through some turbulence with our builds, so there's a good chance that we'll have to hold off on that until we've worked out some of the details.

[mosesn]

Oh, also, as we find things that are challenging to migrate, that's probably good feedback for the scala compiler team. Especially if there are things that scalafix handles poorly. Please let us know what your experience in trying it out is like! Thanks for taking this on, I'm excited to see what you find =)

[cacoco]

@felixbr FYI, we've updated to sbt 1.3.10 in 6ee7b49. Scalatest upgrade may take a bit longer. Thanks.

[mosesn]

@felixbr sorry for the delay, scalatest was a beast. It's finally updated db63a6d

[felixbr]

Oh, that's great news! :)

[mosesn]

@felixbr what do you think are the next steps to supporting scala 3? We've tackled the two you suggested before.

[felixbr]

The biggest problem is that Dotty/Scala 3 is not stable yet, so (if I understand correctly) for any given Dotty version we want to use for a cross-build, the dependencies have to be available. It's a moving target and that makes it difficult at the moment.

I just looked at maven-central and it seems that for Dotty 0.23 a Scalatest 3.1.2 version and for Dotty 0.24 a Scalatest 3.2.0 version exists. The latest Dotty version is 0.25 (released 5 days ago).

My guess is that moving to Scalatest 3.2.0 could be a lot of work again, so for the moment we could attempt the cross-build with Dotty 0.23 but I have no idea how outdated this is compared to 0.25.

Maybe that's something we should clarify with the Dotty-Team before we attempt cross-building with an older version.

[mosesn]

@felixbr I'm inclined to just start with 0.23, and we can move to newer versions after we upgrade our scalatest. I think scalatest 3.1.x's scalafix rewrite will be easier to use than the 3.0.8 one, so I'm cautiously optimistic that the 3.2.0 upgrade won't take that long.

@SethTisue what do you think is the right way forward here?

[mosesn]

Poking around a bit, it looks like @martijnhoekstra also tried porting util to dotty, and ran into some issues: https://contributors.scala-lang.org/t/scala-3-migration-guide/4237/6

What is the usual strategy that people take? Do they keep two different sets of code, or do they use the subset of scala that is supported by both scala 2 and scala 3? I think one approach I would be happy with would be to keep developing in scala 2, and then have a tag that continuously tracks "develop but scala-3-ified"

[martijnhoekstra]

@mosesn to me it made sense at the time to report the encountered issues to dotty and not bother with workarounds yet, since introducing (sometimes ugly) workarounds for bugs that probably won't be in scala 3.0 release didn't make sense to me.

IIRC the biggest issue was the dropped support for macros which is used in IIRC the deprecated Configuration stuff (with existing notes about breakages). Apart from that, cross-building with dotty seems plausible on a unified codebase without any scala-2/scala-3 specific stuff.

Until scala 3.1 rolls along that is, but it seems too early to worry excessively about that.

[mosesn]

Cool! I'm glad to hear it. Which Configuration stuff? Util doesn't have any macros afaik

[martijnhoekstra]

I took another look @mosesn

It wasn't macros, but it was a dependency on scala-reflect in Config (filtering out synthetics).

Apart from that, for now the dealbreakers look like the same as mentioned on the contributors thread: no @varargs support just yet (so no java-compatible varargs forwarders and no overriding of java-compatible varargs, needed in InMemoryStatCounter) and no good story for cross-compiling dynamic member selection (on unsafe in jvm/HotSpot)

[martijnhoekstra]

develop...martijnhoekstra:dotty-compat is a quick port -- in this state all main projects compile (but test crash-and-burns)

[mosesn]

Got it. What do you think would be the right way to interact with the dotty team to get them to prioritize this stuff? It would be really neat for us to be able to add util to the dotty community build.

[felixbr]

@martijnhoekstra I found pretty much the same blockers when I attempted the cross-build. I didn't really understand what exactly scala-reflect was used for. Do you think we can just remove it?

Maybe we should gather the missing features and then talk to the Dotty team, which of them can be expected to be solved by them and which need a separate workaround anyway.

• Dependency on scala-reflect for "synthetic term names" (?)
• Java-compatible varargs support
• Dependency on scalactic.source.Position macro for logging
• "dynamic member selection" (?)

Are these all blockers right now (apart from tests, of course)?

In general I'm in favor of trying a unified codebase first even though writing code that works on 2.11 through 3.0 will be interesting 😄