VUFIND-1210: Use Solr JSON APIs by maccabeelevine · Pull Request #4991 · vufind-org/vufind

maccabeelevine · 2025-12-31T22:27:21Z

Solr has a "new" (a decade old?) JSON-based query APIs, with JSON in the request body instead of a long series of query parameters. This new API in turn supports more complex faceting.

TODO

Lots of cleanup
Marked TODOs in the code
Convert facet params to new facet API, so it actually works again
Fix all the tests
Resolve VUFIND-1210 when merging
Note breaking change to DefaultParameters syntax. There is an error for the old syntax but no backwards compatibility.
Note breaking change that "facet_matches_by_field" (regex) support in facets.ini is removed, though the similar facet_prefix_by_field capability is still supported.

maccabeelevine · 2025-12-31T22:37:56Z

     * @var array
     */
-    protected $params = [];
+    protected $items = [];


All Solr queries will now have a very large array params within the ParamBag. Having the variable also called $params makes debugging a nightmare. Could be called $contents or whatever else.

I guess we need to weigh debug readability against backward-compatibility; this change will break any custom subclasses of the ParamBag that might exist. Most likely, that's not really a problem, since I can't imagine a scenario where somebody would be using such a subclass -- but we should note this in the changelog just in case. As for what to call it, I think I might slightly prefer contents to items because it seems to fit better with the bag metaphor... but I don't really have strong feelings.

maccabeelevine · 2025-12-31T22:43:17Z

I considered implementing this with a flag to revert to the old method (query parameters) as needed. But the code changes are so significant that it would have been a real pain to support both.

maccabeelevine · 2026-01-02T19:21:00Z

Basic functionality including basic faceting now works with the JSON APIs. Lots more to go. @demiankatz @sturkel89 I will eventually need a lot of testing help on this, there are a lot of VuFind features that use Solr that I've haven't used personally before so I can change the API but I don't really know what to expect, unless the integration tests are 100%. It's not ready yet though ... lots of TODOs in the code. I'm off on vacation until Jan 12th. Happy new year!

dmj · 2026-01-07T09:37:55Z

+    public function json(): string
+    {
+        $jsonObject = $this->jsonObject($this->items);
+        return json_encode($jsonObject);


Should deal with JSON errors

Thanks. To start, I changed this to throw the error, which seems to be the usual practice in our codebase. Open to more specific handling if folks thing there is a useful fallback.

maccabeelevine · 2026-01-16T15:04:21Z

                );
                $fields = $this->getFields($shards);
                $specs  = $this->getSearchSpecs($fields);
                $this->backend->getQueryBuilder()->setSpecs($specs);


The logic below is just a first draft, I don't like how it breaks the ParamBag abstraction completely in order to use array_filter. Either have to build some sort of filtering into ParamBag itself or just create a new ParamBag on the fly instead.

Also, I will need assistance actually testing this with shards.

You should be able to test it with any two instances of Solr running the same schema. Let me know if you need more specific help on setting up a test environment; I haven't done it in a few years, but I don't recall it being too difficult.

Ok thanks...will try pointing it at our test and prod environments...

That should work as long as they contain different record IDs (otherwise it will be hard to tell if the sharding is working).

I have (mostly) overlapping record IDs, so the sharding doesn't really work. But the faceting logic that I'm implementing does seem to -- I'm able to remove a facet group with [StripFields]. I suppose I can create a few fresh schemas and do partial indexes into each, but if someone already has sharding set up that will be a plus.

One simple approach could be to spin up a standard test instance using phing startup in a virtual machine, and then shard that with your institutional test instance.

(If you can't easily do this, I certainly can -- just have to find time, since I remain buried under a large review backlog right now).

Sorry I'm still not having any luck with testing the shards config...if/when you have time, I'd appreciate it.

… Facet API.

maccabeelevine · 2026-02-13T17:12:28Z

     * @return ParamBag
     */
-    public function build($id)
+    public function build(string $id, ?ParamBag $params = null): ParamBag


I've backported these SimilarBuilder changes to #5069. Waiting approval/merge there to finalize here.

This has been merged. Is any further action needed before we close this conversation thread?

maccabeelevine · 2026-02-13T22:18:37Z

        $result = $clone->getFacetList();
-        $filteredCounts = $clone->getFilteredFacetCounts();
+
+        // Apply "facet contains"


This logic used to be in Solr/Params. But the JSON Facet API doesn't support it. (It supports prefix filtering, but I don't think that's sufficient.) So I've changed it to post-search filtering. A few thoughts:

Will performance take a hit? It seems minor.

It seems silly that the same filtering is being applied to all facet fields, even though in the UI there was only filtering on one. But that's the same as the previous behavior, and no harm in practice.

If we're ok with this, then in theory it means I could also built back the regex matches logic, doing that as a post-search filter as well, as it's practically the same. And that regex matches logic actually is field-specific. @dmj ?

Yes, it is field specific.

demiankatz

Thanks for the progress here, @maccabeelevine. Looks like there are some conflicts to resolve. Do you mind taking a look at those? Once they are fixed, I'll see if I can find time to set up a sharding scenario for testing. I suspect the quickest way to do this may be to run two Solr instances locally on different ports using SOLR_ADDITIONAL_START_OPTIONS=-Dsolr.disable.allowUrls=true ./solr.sh start to force-enable sharding.

VUFIND-1210: Use Solr JSON request API

48a8396

maccabeelevine commented Dec 31, 2025

View reviewed changes

Comment thread module/VuFindSearch/src/VuFindSearch/Backend/Solr/Backend.php Outdated

maccabeelevine commented Dec 31, 2025

View reviewed changes

Comment thread module/VuFindSearch/src/VuFindSearch/Backend/Solr/Backend.php Outdated

maccabeelevine commented Dec 31, 2025

View reviewed changes

Comment thread module/VuFindSearch/src/VuFindSearch/ParamBagBag.php Outdated

demiankatz added this to the 12.0 milestone Jan 1, 2026

demiankatz added the architecture pull requests that involve significant refactoring / architectural changes label Jan 1, 2026

maccabeelevine added 5 commits January 2, 2026 14:33

Fix spelling listener

caae5eb

First pass at several listeners

8e80568

First version of JSON Facets api, working at basic level

e82a8e9

Fix styles

27c2e6d

Mark some TODOs

2a56ef9

dmj reviewed Jan 7, 2026

View reviewed changes

Comment thread module/VuFindSearch/src/VuFindSearch/ParamBagBag.php Outdated

dmj reviewed Jan 7, 2026

View reviewed changes

maccabeelevine mentioned this pull request Jan 15, 2026

Fix DefaultParametersListener and add Mink test #5013

Merged

maccabeelevine commented Jan 15, 2026

View reviewed changes

Comment thread module/VuFind/src/VuFind/Search/Solr/DefaultParametersListener.php Outdated

maccabeelevine mentioned this pull request Jan 16, 2026

Add default for ShardPreferences->showCheckboxes #5016

Closed

First attempt at MultiIndexListener

f6d550a

maccabeelevine commented Jan 16, 2026

View reviewed changes

Comment thread module/VuFind/src/VuFind/Search/Solr/Params.php Outdated

maccabeelevine added 7 commits January 16, 2026 17:19

Support facet prefix

e35bfe1

Support OR facets

6206ef4

Simply creating a ParamBagBag from null

5db7650

Handle multiple values with the same name, i.e. filters.

09be7e1

Implement JsonSerializable

2f2d089

Throw any JSON encoding error

d943442

Support TreeDataSource, maybe

9451ac2

maccabeelevine commented Jan 16, 2026

View reviewed changes

Comment thread module/VuFind/src/VuFind/Hierarchy/TreeDataSource/Solr.php

maccabeelevine commented Jan 17, 2026

View reviewed changes

Comment thread module/VuFind/src/VuFind/Search/Solr/Params.php

maccabeelevine added 9 commits January 19, 2026 18:01

Merge branch 'dev' into solr-json-search

3a51fa7

Remove fields not needed for Explanation

af4ee52

Merge branch 'dev' into solr-json-search

5cc2f9f

Break up the tests a bit

990c9ed

Merge branch 'dev' into solr-json-search

1faec2b

Revert accidental change in merge

cf8e76d

Implement DefaultParametersListener

9d45b06

Fix json_decode call

d3f0dde

Start building ParamBagBagTest

7d2d26b

maccabeelevine commented Jan 30, 2026

View reviewed changes

Comment thread module/VuFindSearch/tests/unit-tests/src/VuFindTest/ParamBagBagTest.php Outdated

maccabeelevine added 8 commits February 2, 2026 17:27

Add the rest of the ParamBagBag unit tests

6e84cf5

Fix setNested calls after stronger typing

4ff1d6e

Simplify addMultiNested

e285203

Fix a hundred or so tests, more to go

9828993

Fix Blender behavior and tests

db3abd5

Merge branch 'dev' into solr-json-search

31e30c8

Rename ParamBagBag to NestingParamBag

c08ae52

Remove facet_matches_by_field config. It is not supported by new JSON…

c16edfc

… Facet API.

maccabeelevine mentioned this pull request Feb 13, 2026

Fix duplicate 'rows' param from SimilarBuilder #5069

Merged

maccabeelevine added 3 commits February 13, 2026 16:53

Fix SimilarBuilder behavior

0e44e67

Fix TreeDataSource

b34c2ee

Fix styles

5cea405

maccabeelevine commented Feb 13, 2026

View reviewed changes

maccabeelevine added 4 commits February 13, 2026 17:14

Fix SimilarItemsTest

7398011

Throw better exceptions in NestingParamBag

e2cb768

Fix add()

e73f2ef

Implement "facet contains" after the search

67ba9b5

maccabeelevine commented Feb 13, 2026

View reviewed changes

demiankatz requested changes Mar 18, 2026

View reviewed changes

Conversation

maccabeelevine commented Dec 31, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

maccabeelevine commented Dec 31, 2025

Uh oh!

maccabeelevine commented Jan 2, 2026

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

demiankatz Jan 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

demiankatz left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

maccabeelevine commented Dec 31, 2025 •

edited

Loading

demiankatz Jan 21, 2026 •

edited

Loading