Add support for VFS-pointed module maps (was: can't find module dependency)

[keith]

While trying to setup the Lyft project to use xcbuild, I ran into some issues with dependencies. We use CocoaPods to integrated 3rd party, and some first party development dependencies. The structure looks something like this:

Main Project
  -> Alamofire
  -> Internal Library
       -> zipzap
  ...

In this simplified example, the issue we're running into is that zipzap cannot be found. This leads to errors in the Internal Library, and stops us from getting anywhere else.

I've attempted to reproduce the same issue in a sample project:

test-xcbuild4.zip

In this project if you run (with or without specifying -executor ninja):

$ xcbuild -executor ninja -workspace test-xcbuild4 -scheme test-xcbuild4

You will see that it fails to build with the error:

{
  "kind": "finished",
  "name": "compile",
  "pid": 48974,
  "output": "blah blah/AppDelegate.swift:10:8: error: no such module 'zipzap'\nimport zipzap\n       ^\n",
  "exit-status": 1
}

(here's the entire build log)

But if you build from within Xcode, everything compiles correctly.

I would love some advice for where to start when I'm facing errors like this, as I'd love to be able to set our project up to benefit from the speed of Ninja! ⚡

Thanks!

[grp]

Thanks for including a sample project!

My first step is usually to compare the command lines used for swiftc between xcbuild and xcodebuild. I usually paste them into text files, replace spaces with newlines, and use diff to see how they compare.

In this case, I see a number of differences:

• xcbuild passes an extra -parse-as-library, which I think is the default, so shouldn't change behavior.
• xcodebuild uses -ivfsoverlay instead of header maps. My understanding is that this should be roughly equivalent, but since this is an import issue, it definitely seems suspicious. (xcbuild doesn't support VFS overlays for header maps right now.)
• xcodebuild passes -g, -Onone, --module-cache-path, and -j4 instead of -j8 (the last one is machine dependent). Will need to investigate the first three, but don't seem immediately relevant.

Interestingly, after running xcodebuild on the project, the build succeeds even with xcbuild! That points me towards something missing in the derived data after building previous components, rather than something being wrong in the Swift invocation itself. Starting to look into that now.

[grp]

I looked a little more into the header map differences. Seems like that's not the issue:

• swift-overrides.hmap (passed by xcodebuild) is trying to limit header accessibility, not find anything, so doesn't seem critical.
• The VFS overlay that xcodebuild passes is empty.

[grp]

It looks like the issue is that xcbuild doesn't copy (Ditto) in $(CONFIGURATION_BUILD_DIR)/zipzap.framework/Modules/module.modulemap for the zipzap target.

CocoaPods appears to be generating that and specifying it with MODULEMAP_FILE, but a quick grep shows that xcbuild doesn't use that build setting at all. Will have to look into how that setting is supposed to work.

[grp]

Started implementing MODULEMAP_FILE, hit a few minor issues. Putting my notes here so I can take another look tomorrow:

• The module map file is copied as an auxiliary file (pbxbuild::Tool::Invocation::AuxiliaryFile). This is unusual, since usually auxiliary files are generated in code, and there's not a good system setup to handle invalidation for copying one from the filesystem, since they're expected not to change.
• If a module map isn't specified, one is supposed to be generated pointing to the umbrella header. Right now the list of header files (the pbxproj::PBX::Target's build phases) isn't available where the module map is built, needed to find the umbrella header.
• When copying the module map from TARGET_TEMP_DIR to TARGET_BUILD_DIR, this should use the built-in copy utility — but right now, that requires a pbxproj::PBX::BuildFile (in a pbxbuild::Phase::File), not just a path. Not immediately clear how to cleanly refactor that to work with either type.
• It's unclear how module maps should work in non-framework targets, since there may not be a good Modules folder (or a CONTENTS_FOLDER_PATH at all) to put them in.

[grp]

OK, @keith, the grp/modulemap-file branch has a WIP implementation that's enough to build the sample project. Would be great if you could let me know if that helps with your internal project too! I'll work on cleaning it up to land.

[keith]

Oh man thanks for diving into this and getting somewhere with it! I'll definitely compare build logs next time to try and come up with some more info. Let me try this branch now!

[keith]

It looks like on our internal project we're on to a new error where we were relying on implicit imports coming from somewhere. I'll add the rest and see what happens next!

[keith]

Ok so as far as I can tell, based on your branch, the original issue here is fixed. If this next issue doesn't sound related to you, feel free to close this one and I can open a different one as needed!

The next issue I've ran into, which I can't yet tell if it's related, is that using xcodebuild it seems like (for better or worse) many imports are implicitly added. For example we have files like this:

$ cat Foo.swift
extension UIView {
   ...
}

Where we have no explicit import UIKit, yet these work somehow. I think some of this might have to do with other files that are importing 3rd party dependencies, and maybe those imports are polluting the entire modules namespace. Another similar case we have is:

$ cat Foo.swift
import UIKit

... NSBundle ...

Which will also fail to compile because of a missing import to Foundation. Although it seems like in other places, import UIKit will implicitly make the Foundation import. Again I'm not a fan of this behavior, but it has apparently been something we're relying on. I'm currently adding all the missing imports by running xcbuild until it fails, copying the command that fails, running it, fixing the imports, repeat. Also as a note, these don't get surfaced in xcpretty, which would probably make this easier, instead I've been grepping the log for exit-status": 1

Let me know if you think this is at all related.

[keith]

Ok so after finding all these missing imports, it seems like that issue only applies to the non-app dependencies of our project. I'm not sure exactly what the difference is there, but I only had to modify imports in our first party dependencies, some from CocoaPods and some just custom frameworks in our main project, but none of our main project files. There could be a difference here since the main project is the only place with a bridging header, which might be a slightly more explicit way of polluting the global namespace 😬

[grp]

Hm, very interesting! It would be useful to see a comparison of the swiftc invocations between xcbuild and xcodebuild for a target hitting the issue. Hard to imagine what could cause a difference like that, though!

[grp]

(I do think it's separate from this issue, though, so I'll make a new issue for it.)