Reducing requests using https protocol

[zlmarshall]

Hello,

When opening ROOT files through the https protocol, as in
https://github.com/atlas-outreach-data-tools/notebooks-collection-opendata/blob/master/13-TeV-examples/uproot_python/HZZAnalysisNew.ipynb

We get a:

HTTPError: 429 Client Error: Too Many Requests for url: https://opendata.cern.ch//eos/opendata/atlas/rucio/opendata/ODEO_FEB2025_v0_exactly4lep_mc_364242.Sherpa_222_NNPDF30NNLO_WWW_3l3v_EW6.exactly4lep.root

Because uproot seems to generate https requests for every (?) interaction with a TTree, there's too much traffic on the http endpoint, and we get blocked. We can work around that by downloading the files and accessing them locally:

        # Download the file, use a local copy
        local_filename = fileString.split("/")[-1]
        print(f"Downloading {fileString} to {local_filename} ...")
        with requests.get(fileString, stream=True) as r:
            r.raise_for_status()
            with open(local_filename, "wb") as f:
                for chunk in r.iter_content(chunk_size=8192):
                    f.write(chunk)
        tree = uproot.open(local_filename + ":analysis")

That's not terribly nice, if the aim here is streaming. Is there a way to be more greedy with the https requests from uproot, to have fewer requests and more data per request?

Thanks,
Zach

[ariostas]

Hi @zlmarshall

You can add simplecache:: before the url and Uproot will download the whole file at once instead of submitting a bunch of http requests.

So something like this

with uproot.open("simplecache::https://opendata.cern.ch//eos/opendata/atlas/rucio/opendata/ODEO_FEB2025_v0_exactly4lep_mc_364242.Sherpa_222_NNPDF30NNLO_WWW_3l3v_EW6.exactly4lep.root") as f:
    # your code here

[ariostas]

It should still work for object handles. At least for me this is working fine.
tree = uproot.open("simplecache::https://opendata.cern.ch//eos/opendata/atlas/rucio/opendata/ODEO_FEB2025_v0_exactly4lep_mc_364242.Sherpa_222_NNPDF30NNLO_WWW_3l3v_EW6.exactly4lep.root:analysis")

[zlmarshall]

Update: no, that doesn't seem to be the explanation either. I still get crashes from the same problem using both methods, where doing a simple/stupid file copy doesn't. The fsspec (https://filesystem-spec.readthedocs.io/en/latest/features.html#caching-files-locally) is really extremely light on this feature. An increased block_size also didn't do the job.

There's something I'm not following in the errors here. If I use for example:

        # Open file - use heavy caching to reduce http server request load
        my_file = uproot.open("simplecache::"+fileString,block_size=8192*8)
        tree = my_file["analysis"]

and then loop:

        # Loop over data in the tree
        for data in tree.iterate(variables + weight_variables + ["sum_of_weights", "lep_n"], 
                                 library="ak", 
                                 entry_stop=tree.num_entries*fraction):

The crash I get is during the iteration, and it looks like this:

...
[~/.local/lib/python3.11/site-packages/uproot/models/TBasket.py](https://r3arnb19n3j-496ff2e9c6d22116-0-colab.googleusercontent.com/outputframe.html?vrz=colab_20250613-060040_RC00_771035453#) in read_members(self, chunk, cursor, context, file)
    225             self._members["fKeylen"],
    226             self._members["fCycle"],
--> 227         ) = cursor.fields(chunk, _tbasket_format1, context)
    228 
    229         # skip the class name, name, and title

[~/.local/lib/python3.11/site-packages/uproot/source/cursor.py](https://r3arnb19n3j-496ff2e9c6d22116-0-colab.googleusercontent.com/outputframe.html?vrz=colab_20250613-060040_RC00_771035453#) in fields(self, chunk, format, context, move)
    199         if move:
    200             self._index = stop
--> 201         return format.unpack(chunk.get(start, stop, self, context))
    202 
    203     def field(

[~/.local/lib/python3.11/site-packages/uproot/source/chunk.py](https://r3arnb19n3j-496ff2e9c6d22116-0-colab.googleusercontent.com/outputframe.html?vrz=colab_20250613-060040_RC00_771035453#) in get(self, start, stop, cursor, context)
    444         """
    445         if (start, stop) in self:
--> 446             self.wait(insist=stop)
    447             local_start = start - self._start
    448             local_stop = stop - self._start

[~/.local/lib/python3.11/site-packages/uproot/source/chunk.py](https://r3arnb19n3j-496ff2e9c6d22116-0-colab.googleusercontent.com/outputframe.html?vrz=colab_20250613-060040_RC00_771035453#) in wait(self, insist)
    386         """
    387         if self._raw_data is None:
--> 388             self._raw_data = numpy.frombuffer(self._future.result(), dtype=self._dtype)
    389             if insist is True:
    390                 requirement = len(self._raw_data) == self._stop - self._start

[~/.local/lib/python3.11/site-packages/uproot/source/coalesce.py](https://r3arnb19n3j-496ff2e9c6d22116-0-colab.googleusercontent.com/outputframe.html?vrz=colab_20250613-060040_RC00_771035453#) in result(self, timeout)
     37             # Pyodide futures don't support timeout
     38             return self._parent.result()[self._s]
---> 39         return self._parent.result(timeout=timeout)[self._s]
     40 
     41 

TypeError: 'ClientResponseError' object is not subscriptable

The last one in particular is my issue: ClientResponseError suggests there is still an https interaction happening here, that it is not being correctly taken from the file. So maybe this just isn't working? Is there any way for me to check further, or version of uproot / fsspec from which this is supported?

Thanks,
Zach

[ariostas]

Hmm okay, I guess I was wrong about simplecache:: downloading the whole file at once. I'll think if there's a better alternative other than manually downloading it first.

[zlmarshall]

The documentation suggests that you're right ("the option exists to copy files locally when you first access them, and thereafter to use the local data"). There is a logger, so I could add:

import logging
fslogc = logging.getLogger("fsspec.cached")
fslogh = logging.getLogger("fsspec.http")
fslogc.setLevel(10)
fslogh.setLevel(10)

I can't quite tell whether I'm having ordering issues, but for a file that "works" I get this kind of output from uproot+fsspec:

DEBUG:fsspec.cached:Opening local copy of https://opendata.cern.ch//eos/opendata/atlas/rucio/opendata/ODEO_FEB2025_v0_exactly4lep_data15_periodD.exactly4lep.root
DEBUG:fsspec.cached:Opening local copy of https://opendata.cern.ch//eos/opendata/atlas/rucio/opendata/ODEO_FEB2025_v0_exactly4lep_data15_periodD.exactly4lep.root
DEBUG:fsspec.cached:Opening local copy of https://opendata.cern.ch//eos/opendata/atlas/rucio/opendata/ODEO_FEB2025_v0_exactly4lep_data15_periodD.exactly4lep.root
DEBUG:fsspec.cached:Opening local copy of https://opendata.cern.ch//eos/opendata/atlas/rucio/opendata/ODEO_FEB2025_v0_exactly4lep_data15_periodD.exactly4lep.root
Processing Data samples
	https://opendata.cern.ch//eos/opendata/atlas/rucio/opendata/ODEO_FEB2025_v0_exactly4lep_data15_periodD.exactly4lep.root:
DEBUG:fsspec.http:https://opendata.cern.ch//eos/opendata/atlas/rucio/opendata/ODEO_FEB2025_v0_exactly4lep_data15_periodD.exactly4lep.root

or:

DEBUG:fsspec.cached:Opening local copy of https://opendata.cern.ch//eos/opendata/atlas/rucio/opendata/ODEO_FEB2025_v0_exactly4lep_data15_periodE.exactly4lep.root
DEBUG:fsspec.cached:Opening local copy of https://opendata.cern.ch//eos/opendata/atlas/rucio/opendata/ODEO_FEB2025_v0_exactly4lep_data15_periodE.exactly4lep.root
DEBUG:fsspec.cached:Opening local copy of https://opendata.cern.ch//eos/opendata/atlas/rucio/opendata/ODEO_FEB2025_v0_exactly4lep_data15_periodE.exactly4lep.root
DEBUG:fsspec.cached:Opening local copy of https://opendata.cern.ch//eos/opendata/atlas/rucio/opendata/ODEO_FEB2025_v0_exactly4lep_data15_periodE.exactly4lep.root
DEBUG:fsspec.http:https://opendata.cern.ch//eos/opendata/atlas/rucio/opendata/ODEO_FEB2025_v0_exactly4lep_data15_periodE.exactly4lep.root
DEBUG:fsspec.http:https://opendata.cern.ch//eos/opendata/atlas/rucio/opendata/ODEO_FEB2025_v0_exactly4lep_data15_periodE.exactly4lep.root
DEBUG:fsspec.http:https://opendata.cern.ch//eos/opendata/atlas/rucio/opendata/ODEO_FEB2025_v0_exactly4lep_data15_periodE.exactly4lep.root
DEBUG:fsspec.http:https://opendata.cern.ch//eos/opendata/atlas/rucio/opendata/ODEO_FEB2025_v0_exactly4lep_data15_periodE.exactly4lep.root
DEBUG:fsspec.http:https://opendata.cern.ch//eos/opendata/atlas/rucio/opendata/ODEO_FEB2025_v0_exactly4lep_data15_periodE.exactly4lep.root
DEBUG:fsspec.http:https://opendata.cern.ch//eos/opendata/atlas/rucio/opendata/ODEO_FEB2025_v0_exactly4lep_data15_periodE.exactly4lep.root
DEBUG:fsspec.http:https://opendata.cern.ch//eos/opendata/atlas/rucio/opendata/ODEO_FEB2025_v0_exactly4lep_data15_periodE.exactly4lep.root
		 nIn: 3424,	 nOut: 	2	 in 2.1s
	https://opendata.cern.ch//eos/opendata/atlas/rucio/opendata/ODEO_FEB2025_v0_exactly4lep_data15_periodE.exactly4lep.root:

For one that doesn't work, here's what it looks like instead:

DEBUG:fsspec.http:https://opendata.cern.ch//eos/opendata/atlas/rucio/opendata/ODEO_FEB2025_v0_exactly4lep_data16_periodB.exactly4lep.root
DEBUG:fsspec.http:https://opendata.cern.ch//eos/opendata/atlas/rucio/opendata/ODEO_FEB2025_v0_exactly4lep_data16_periodB.exactly4lep.root
DEBUG:fsspec.http:https://opendata.cern.ch//eos/opendata/atlas/rucio/opendata/ODEO_FEB2025_v0_exactly4lep_data16_periodB.exactly4lep.root

Those are repeated 263 times before we get the error reported above. In that case, tons of http logging, and nothing from the cache. I'm still digging around a bit, but in case that makes anybody thing of something I should try (longer timeout? something else?) please let me know!

[zlmarshall]

One more update, in case anybody has ideas. It looks like ~all these calls are coming from http.HTTPFileSystem::_cat_file

https://github.com/fsspec/filesystem_spec/blob/8463a6a587f722f279e987b3e8508f2c9bba50d2/fsspec/implementations/http.py#L221

I can't see why this would suddenly generate a load of requests. I've tried passing block_size and chunk_size parameters to uproot's open function (the extras seem to get passed along, but let me know if I've misunderstood). There's no indication that those have any effect. I don't see a good way to tell what's generating these (if it's failing requests that snowball, so that putting a 'wait' somewhere would solve it; or if it's small requests, such that we need a bigger block size and I just haven't found the right parameter yet; or if it's something entirely different).

All suggestions or requests for other things to try are very welcome.

[zlmarshall]

Hi all,

Ok, I think I've made a little progress in understanding this issue. This is only my understanding from tinkering, adding some printing, and reading some code, so it's entirely possible that I got things wrong — please let me know if that's the case.

Here's a simple reproducer for the issue:

import uproot
tree = uproot.open('simplecache::https://opendata.cern.ch//eos/opendata/atlas/rucio/opendata/ODEO_FEB2025_v0_exactly4lep_data16_periodB.exactly4lep.root:analysis')
for data in tree.iterate(['lep_pt', 'lep_eta', 'lep_phi', 'lep_e', 'lep_charge', 'lep_type', 'trigE', 'trigM', 'lep_isTrigMatched', 'lep_isLooseID', 'lep_isMediumID', 'lep_isLooseIso', 'lep_type', 'filteff', 'kfac', 'xsec', 'mcWeight', 'ScaleFactor_PILEUP', 'ScaleFactor_ELE', 'ScaleFactor_MUON', 'ScaleFactor_LepTRIGGER', 'sum_of_weights', 'lep_n'],library='ak’,entry_stop=72544):
    # Number of events in this batch
    nIn = len(data)
    print(nIn)

• simplecache:: here is meant to allow caching in fsspec. It means that fsspec will copy the file locally upon first access. So far so good.
• Within uproot.reading.open, the FSSpecSource class is used to read the first chunk of the file. The file path is set using fsspec.core.url_to_fs, which strips simplecache:: from self._file_path. self._fs returns a fsspec.implementations.cached.SimpleCacheFileSystem, which seems correct to me.
• The call to FSSpecSource.chunk directly goes through self._fs.cat_file and uses self._file_path, so that simplecache:: is long gone at this point. That means our first access here, which is a great opportunity, doesn't hit the cache + local copy system.

Then later on:

• tree.iterate fires a load of asynchronous calls (async with) to get file chunks from the address without simplecache. These of course all hit the server at the same time, and it seems there is one async with per branch that's being accessed. That makes some sense, but again none of it hits the caching infrastructure.

My sense is that the solution is to make sure the first chunk gotten in uproot.reading.open hits the caching infrastructure. That's the opportunity to copy the file back. After that, the subsequent branch gets should be fine as async with, since they'll be local file accesses and won't hit the upstream server.

I'm going to keep tinkering a bit, but I wanted to post this diagnosis in case somebody knows how to solve the problem right away.

Thanks,
Zach

[ariostas]

Oh interesting, thanks for looking into this @zlmarshall!

[zlmarshall]

One more update, with the fsspec issue here:

fsspec/filesystem_spec#1869

It does look like my guess was correct, that _cat_file in the tree.iterate function are not hitting the cache, because they're using a different access system (a different file system class). One of the folks there offered to look into it. I hope we'll have this solved reasonably soon.

Best,
Zach