[16.0][FIX] account_statement_import_ofx: handle mislabeled OFX charsets

[renatonlima]

Summary

Fix OFX imports failing with a misleading "file format is not supported" error when the file's declared charset does not match its real encoding.

Problem

ofxparse reads the charset declared in the OFX SGML header and decodes the body with a strict cp1252 codec. Several banks emit OFX files whose header declares CHARSET:1252 while the body is actually UTF-8. Decoding then fails on bytes that are undefined in cp1252 (e.g. 0x81):

import io
from ofxparse import OfxParser

with open("/path/to/Extrato_3246_020550_18-06-2026.ofx", "rb") as file:
    data = file.read()
    ofx = OfxParser.parse(io.BytesIO(data))
​

output

Traceback (most recent call last):
  File "<stdin>", line 4, in <module>
  File "/Users/renato/DEV/odoo16/odoo/lib/python3.10/site-packages/ofxparse/ofxparse.py", line 396, in parse
    ofx_file = OfxPreprocessedFile(file_handle)
  File "/Users/renato/DEV/odoo16/odoo/lib/python3.10/site-packages/ofxparse/ofxparse.py", line 160, in __init__
    ofx_string = self.fh.read()
  File "/Users/renato/.pyenv/versions/3.10.13/lib/python3.10/codecs.py", line 504, in read
    newchars, decodedbytes = self.decode(data, self.errors)
  File "/Users/renato/.pyenv/versions/3.10.13/lib/python3.10/encodings/cp1252.py", line 15, in decode
    return codecs.charmap_decode(input,errors,decoding_table)
UnicodeDecodeError: 'charmap' codec can't decode byte 0x81 in position 906: character maps to <undefined>

_check_ofx catches this exception and returns False, so the user only sees the generic dialog:

In the affected file the accented name ROMÁRIO is stored in UTF-8 (Á = 0xC3 0x81), so the lone 0x81 byte breaks the cp1252 decoder even though the header claims CHARSET:1252.

Solution

Decode the raw OFX bytes ourselves before handing them to ofxparse:

1. UTF-8 first — it is self-validating, so it rarely matches non-UTF-8 data by accident.

2. Then the charset declared in the header (cp1252 / iso-8859-1).

3. Finally latin-1, which maps every byte value and never raises. The decoded text is re-encoded to UTF-8 and the SGML ENCODING header is normalized to UTF-8, so ofxparse decodes the body consistently regardless of the original (possibly wrong) declaration. Passing bytes rather than a text stream also prevents ofxparse from re-running its own cp1252 detection on the content.

This works across locales — Western Europe, Latin America and Asia — without corrupting characters outside latin-1 (the previous behaviour silently turned a UTF-8 € or CJK text into mojibake).

Tests

Added regression tests covering four encoding scenarios, each asserting that the special characters survive the round-trip:

Test	Scenario
test_ofx_encoding_latin	UTF-8 body mislabeled as CHARSET:1252 (the reported bug; contains byte 0x81)
test_ofx_encoding_cp1252	Genuine cp1252 body with the Euro sign and Western accents
test_ofx_encoding_utf8_euro	UTF-8 body with characters outside latin-1 (Euro sign, em dash)
test_ofx_encoding_asian	UTF-8 body with CJK characters

[OCA-git-bot]

Hi @alexis-via,
some modules you are maintaining are being modified, check this out!

[renatonlima]

There's an error in another module; I believe the cause is the version of the Python library ofxtools. It's likely the newest version of this library isn't compatible. I've changed the status of this PR to draft while I investigate the error in CI.