Unhandled OpenSSL.SSL.WantReadError and WantWriteError raised by PyOpenSSL

[Rohitbels]

❓ I'm submitting a ...

• 🐞 bug report
• 🐣 feature request
• ❓ question about the decisions made in the repository

🐞 Describe the bug. What is the current behavior?
This issue is reproducible against huge post request like multiple lines of textarea in form.
We are upgrading from python 2.7 to 3.7 and hence also upgraded the cheroot 8.2.0.
Cherrypy : 18+

📋 Details

Traceback (most recent call last):
  File "/usr/local/pyenv/versions/3.7.0/lib/python3.7/site-packages/cherrypy/_cprequest.py", line 628, in respond
    self._do_respond(path_info)
  File "/usr/local/pyenv/versions/3.7.0/lib/python3.7/site-packages/cherrypy/_cprequest.py", line 680, in _do_respond
    self.body.process()
  File "/usr/local/pyenv/versions/3.7.0/lib/python3.7/site-packages/cherrypy/_cpreqbody.py", line 982, in process
    super(RequestBody, self).process()
  File "/usr/local/pyenv/versions/3.7.0/lib/python3.7/site-packages/cherrypy/_cpreqbody.py", line 559, in process
    proc(self)
  File "/usr/local/pyenv/versions/3.7.0/lib/python3.7/site-packages/cherrypy/_cpreqbody.py", line 145, in process_urlencoded
    qs = entity.fp.read()
  File "/usr/local/pyenv/versions/3.7.0/lib/python3.7/site-packages/cherrypy/_cpreqbody.py", line 817, in read
    data = self.fp.read(chunksize)
  File "/usr/local/pyenv/versions/3.7.0/lib/python3.7/site-packages/cheroot/server.py", line 383, in read
    data = self.rfile.read(size)
  File "/usr/local/pyenv/versions/3.7.0/lib/python3.7/site-packages/cheroot/makefile.py", line 420, in read
    val = super().read(*args, **kwargs)
  File "/usr/local/pyenv/versions/3.7.0/lib/python3.7/_pyio.py", line 1014, in read
    return self._read_unlocked(size)
  File "/usr/local/pyenv/versions/3.7.0/lib/python3.7/_pyio.py", line 1054, in _read_unlocked
    chunk = self.raw.read(wanted)
  File "/usr/local/pyenv/versions/3.7.0/lib/python3.7/socket.py", line 589, in readinto
    return self._sock.recv_into(b)
  File "/usr/local/pyenv/versions/3.7.0/lib/python3.7/site-packages/OpenSSL/SSL.py", line 1335, in recv_into
    self._raise_ssl_error(self._ssl, result)
  File "/usr/local/pyenv/versions/3.7.0/lib/python3.7/site-packages/OpenSSL/SSL.py", line 1149, in _raise_ssl_error
    raise WantReadError()
OpenSSL.SSL.WantReadError

📋 Environment

• Cheroot version: 8.2.0
• CherryPy version: 18.1.1 (if applicable)
• Python version: 3.6.7
• OS: linux
• Browser: [all | Chrome XX | Firefox XX | IE XX | Safari XX | Mobile Chrome XX | Android X.X Web Browser | iOS XX Safari | iOS XX UIWebView | iOS XX WKWebView ]

📋 Additional context
I have went through this #113
but still was unable to figure out the solution

[webknjaz]

Try using the builtin TLS adapter instead.

I think, we may need to implement some retry logic based on:

• Setting socket timeouts breaks handshake pyca/pyopenssl#168 (comment)
• https://www.pyopenssl.org/en/stable/internals.html#exceptions
• https://www.pyopenssl.org/en/stable/api/ssl.html#OpenSSL.SSL.WantReadError

[jpeckham]

I was getting WantWriteError on very large js files and we did this monkey patch

def _flush_unlocked(self):
    self._checkClosed('flush of closed file')
    while self._write_buf:
        try:
            # ssl sockets only except 'bytes', not bytearrays
            # so perhaps we should conditionally wrap this for perf?
            n = self.raw.write(self._write_buf)
        except io.BlockingIOError as e:
            n = e.characters_written
        except (OpenSSL.SSL.WantReadError,OpenSSL.SSL.WantWriteError, OpenSSL.SSL.WantX509LookupError) as e:
            n = 0
        del self._write_buf[:n]

BufferedWriter._flush_unlocked = _flush_unlocked

[Rohitbels]

@jpeckham Thanks a lot for this!

@webknjaz and for any one else:

I was able to fix this by replacing to read call at

File "/cheroot/makefile.py", line 420, in read val = super().read(*args, **kwargs)

with

val = self._safe_call(True, super().read, *args, **kwargs)

safe_call is specifically defined in pyopenssl.py for this reason.

@webknjaz This fix looks good?

[jussike]

Works for me.

Is that Rohitbels's workaround safe for fixing the issue @webknjaz ?

[vashek]

I suggest changing the description of this issue to "OpenSSL.SSL.WantReadError and WantWriteError", because both errors occur for similar reasons. (I encountered the one with WantWriteError, which is probably the more likely of the two because you're most likely to send big data than receive, but I'm only guessing.)

Thanks a lot @jpeckham, your workaround helped me.

If anyone can't reproduce the issue, this indicates that the problem might be OS-dependent.

And, by the way, pyca/pyopenssl#176 says that sendall() should not be used at all until fixed because the same exception can occur and will cause loss of state.

@webknjaz, I was originally using the BuiltinSSLAdapter, but the server then failed some tlsfuzzer tests (specifically I think it was test-fuzzed-ciphertext.py) and eventually stopped responding after these fuzzing tests or after a Qualys test. That's why I switched to pyOpenSSLAdapter, which works better in this regard but has the issue with these exceptions, which seems much easier to fix than the fuzzing failures. (Of course, if anyone happens to know why that happens or how to fix it, I'd welcome that information.)

[webknjaz]

If anyone can't reproduce the issue, this indicates that the problem might be OS-dependent.

I'd guess, it's more of an OpenSSL backend dependent. It may be pyOpenSSL is a wrapper around cryptography and cryptography is a wrapper around OpenSSL. So it may be compiled against many different OpenSSL versions that may behave differently, or just have different compilation settings in general.

@webknjaz, I was originally using the BuiltinSSLAdapter, but the server then failed some tlsfuzzer tests (specifically I think it was test-fuzzed-ciphertext.py) and eventually stopped responding after these fuzzing tests or after a Qualys test. That's why I switched to pyOpenSSLAdapter, which works better in this regard but has the issue with these exceptions, which seems much easier to fix than the fuzzing failures. (Of course, if anyone happens to know why that happens or how to fix it, I'd welcome that information.)

Interesting. We may want to integrate a tool like that into our test suite one day...

As for why WantReadError and WantWriteError happen: it's usually because some buffers get full and the expectation is that the calling code will do retries.

[vodnokoki]

Faced with above issue, builtin module is working perfectly, but when audit suggest to use only tlsv1.2 and tls1.3, i didn't now how to implement with builtin, so chanhed logic with pyopenssl and using context allowing only tlsv1.2 and tls1.3. Unhandled exceptions raised when server response big js files, architecture is react+Django.

After applying changes from @vashek everything works perfectly and there is no error on server side. When do you plan to accept above change from @vashek and create release.