8382428: libsyslookup.so is not needed for Linux

[YaSuenag]

SystemLookup in FFM uses libsyslookup to find symbols from the system. But it is not needed because RTLD_DEFAULT can be used for this purpose on Linux.
Removing unneeded library improves security.

I've sent email to core-libs-dev, but I've not yet received any comments, so I created this PR.

syslookup.dll for Windows is needed because some functions might not be lookup'ed. OTOH on Linux, dlsym can lookup symbols from library dependencies. In SystemLookup, handle of libsyslookup would be passed to dlsym eventually, but it is better to pass RTLD_DEFAULT in this case.
It works when the handle of libsyslookup is passed, but RTLD_DEFAULT is better because Javadoc of Linker::defaultLookup says it returns a set of commonly used libraries.

In addition, I guess we can apply this change to all of POSIX platforms because dlsym is defined in POSIX, but I'm not sure we can do (especially AIX - it has own syslookup.c in JDK source tree).

This change passed all of jdk_foreign tests.

---

• I confirm that I make this contribution in accordance with the OpenJDK Interim AI Policy.

---

Progress

• Change must be properly reviewed (1 review required, with at least 1 Reviewer)
• Change must not contain extraneous whitespace
• Commit message must refer to an issue

Issue

• JDK-8382428: libsyslookup.so is not needed for Linux (Enhancement - P4)

Reviewing

Using git

Checkout this PR locally:
$ git fetch https://git.openjdk.org/jdk.git pull/30794/head:pull/30794
$ git checkout pull/30794

Update a local copy of the PR:
$ git checkout pull/30794
$ git pull https://git.openjdk.org/jdk.git pull/30794/head

Using Skara CLI tools

Checkout this PR locally:
$ git pr checkout 30794

View PR using the GUI difftool:
$ git pr show -t 30794

Using diff file

Download this PR as a diff file:
https://git.openjdk.org/jdk/pull/30794.diff

Using Webrev

Link to Webrev Comment

[bridgekeeper]

👋 Welcome back ysuenaga! A progress list of the required criteria for merging this PR into master will be added to the body of your pull request. There are additional pull request commands available for use with this pull request.

[openjdk]

❗ This change is not yet ready to be integrated.
See the Progress checklist in the description for automated requirements.

[openjdk]

@YaSuenag The following labels will be automatically applied to this pull request:

• build
• core-libs

When this pull request is ready to be reviewed, an "RFR" email will be sent to the corresponding mailing lists. If you would like to change these labels, use the /label pull request command.

[mlbridge]

Webrevs

• 00: Full (99543dc5)

[JornVernee]

Why do you think RTLD_DEFAULT is better? I don't think this will necessarily expose the same set of symbols. For example, if a symbols is in a library not already loaded by the process. Conversely, this could also find extra VM-internal symbols from libraries that the VM itself links against.

[YaSuenag]

Why do you think RTLD_DEFAULT is better?

For security.
Unneeded libraries might be help to attack by malicious user. (e.g. DLL hijack)

For example, if a symbols is in a library not already loaded by the process.

I could not understand this case.
If the symbol what the user want to use attempts to be resolved, the library which provides it need to be loaded before. It is same with current semantics on FFM. Exception should be thrown if the symbol was not found in loaded libraries. (SymbolLookup::findOrThrow)

Conversely, this could also find extra VM-internal symbols from libraries that the VM itself links against.

Indeed, but this concern already exists because the user can issue dlsym() with RTLD_DEFAULT from both native code and FFM.

[JornVernee]

If the symbol what the user want to use attempts to be resolved, the library which provides it need to be loaded before.

Right, and the library we load for that is libsyslookup. The default lookup should not expose some arbitrary set of symbols, but something we have some control over. That's the whole reason why we use libsyslookup in the first place. We have no control (or at least much less) over the symbols returned by RTLD_DEFAULT.

[YaSuenag]

I think this comes down to the question of which option is "risky" in security POV.

I understand it is better to control library dependencies, but it could be "uncontrollable" eventually if the user tried to use FFM. Thus I think it make sence to remove libsyslookup to prevent attacks relates to library like DLL hijack.

[JornVernee]

I understand it is better to control library dependencies, but it could be "uncontrollable" eventually if the user tried to use FFM.

Not sure what you mean here. We control libsyslookup.

Thus I think it make sence to remove libsyslookup to prevent attacks relates to library like DLL hijack.

Sorry, but I'm not convinced by this argument at all. The JVM uses dozens of other native libraries. Heck, even the JVM itself is a library. I see no reason why having another library would be a problem.

[YaSuenag]

@JornVernee
I found a discussion in #4316.

I understood shim library (libsyslookup) was approved to ensure it does not include more symbols than necessary.
However I wonder why you said being it was a bug to be able to find any symbols from the JVM. Javadoc for defaultLookup recognizes to implementor to choose libraries that are widely recommends as useful on the OS and processor combination. It does not specify precise set of symbols to be exposed. I do not agree with the symbols in JVM is "useful", but I feel there are no significant reason not to prevent to access them. It could be like "Unsafe" API - it might be the reason what you want to prevent to access...