Skip to content

Commit 97a9152

Browse files
committed
update test.py: better reading of 4CAT responses
1 parent 7f5fe57 commit 97a9152

3 files changed

Lines changed: 260 additions & 3 deletions

File tree

‎tests/README.md‎

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -39,6 +39,32 @@ but you can also pass one with the `--profiledir` argument. Log in to the
3939
various sites (Instagram, etc) in your 'normal' Firefox, and the tests then
4040
borrow that login.
4141

42+
`tests/blank_firefox_profile` is a profile kept for exactly this, with logins
43+
to the platforms that need one. It is the profile to pass to `--profiledir`.
44+
45+
**Adding or refreshing a login in it.** The run works on a *copy* of the
46+
profile (`.temp-profile`) and deletes that copy when it finishes, so logging in
47+
during a run is thrown away. Log in to the profile itself instead: close the
48+
tests, start Firefox on it directly, log in, and close the window so the
49+
cookies are written to disk.
50+
51+
```powershell
52+
& 'C:/Program Files/Mozilla Firefox/firefox.exe' -no-remote -profile 'C:/path/to/zeeschuimer/tests/blank_firefox_profile'
53+
```
54+
55+
`-no-remote` matters: without it Firefox hands the request to an instance that
56+
is already running and uses that instance's profile instead, and nothing is
57+
saved where the tests will look. `about:profiles` in the new window shows which
58+
profile is actually in use.
59+
60+
Logins expire. If a platform's tests start failing in a way that looks like
61+
being logged out, refresh its login the same way. The stored cookies and their
62+
expiry dates can be read from `cookies.sqlite` in that folder.
63+
64+
The same applies to the 4CAT server used by `--fourcat`: uploads authenticate
65+
with the browser session, so log in to it in this profile too and the tests
66+
will stop asking.
67+
4268
RedNote (Xiaohongshu) is the exception: an account needs a Chinese phone
4369
number, and no current developer has one, so all RedNote development and
4470
testing is done logged out. Logged out, search redirects to the login page,

‎tests/test.py‎

Lines changed: 21 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -376,6 +376,7 @@ def export_platform(platform):
376376
("platform not accepted by this 4CAT", "does not accept"),
377377
("could not connect", "Could not connect to 4CAT server"),
378378
("malformed response", "malformed response from 4CAT server"),
379+
("4CAT could not read the data", "Processor error"),
379380
("upload error", "during upload"),
380381
]
381382

@@ -410,20 +411,37 @@ def upload_platform(platform):
410411
driver.execute_script("document.querySelector(arguments[0]).click();", button)
411412

412413
deadline = time.time() + 600
414+
reported = None
413415
while time.time() < deadline:
414416
state = driver.execute_script("""
415417
const status = document.getElementById('upload-status');
416-
const link = status ? status.querySelector('a[href]') : null;
417-
return {text: status ? status.innerText : '', url: link ? link.href : null};
418+
if (!status) return {text: '', url: null};
419+
const link = status.querySelector('a[href]');
420+
// While uploading, the status area also holds a Cancel button; its
421+
// label is a control, not part of the message.
422+
const text = [...status.childNodes]
423+
.filter(node => !(node.nodeType === Node.ELEMENT_NODE && node.tagName === 'BUTTON'))
424+
.map(node => node.textContent)
425+
.join(' ');
426+
return {text: text, url: link ? link.href : null};
418427
""")
419428
if state["url"]:
420429
return state["url"]
421430
for label, needle in UPLOAD_FAILURES:
422431
if needle in state["text"]:
423432
return ("error", label, " ".join(state["text"].split()))
433+
434+
# Importing takes as long as it takes, and 4CAT names the stage it is
435+
# on. Echo each new stage so a long wait shows its progress instead of
436+
# looking like a hung script.
437+
progress = " ".join(state["text"].split())
438+
if progress and progress != reported:
439+
print(f"{platform_indent(platform)} 4CAT: {progress}")
440+
reported = progress
424441
time.sleep(1)
425442

426-
return ("error", "timed out", "4CAT did not finish processing within 600 seconds")
443+
return ("error", "timed out",
444+
f"4CAT did not finish processing within 600 seconds; last reported: {reported or 'nothing'}")
427445

428446

429447
def upload_platform_supervised(platform):

‎tests/upload-poll.test.js‎

Lines changed: 213 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,213 @@
1+
/**
2+
* Tests for the 4CAT upload progress poll in popup/interface.js.
3+
*
4+
* `upload_poll.init` asks 4CAT whether an uploaded dataset has finished
5+
* processing, and on completion writes the dataset link into the interface and
6+
* records the upload. Everything it does happens inside an
7+
* `onreadystatechange` handler, so which readyState the handler acts on
8+
* decides whether any of it runs.
9+
*
10+
* An XMLHttpRequest reports four readyStates, and only the last one
11+
* (`DONE`, 4) is guaranteed to carry the whole response body:
12+
*
13+
* 2 HEADERS_RECEIVED status is known, responseText is still empty
14+
* 3 LOADING responseText holds whatever has arrived so far
15+
* 4 DONE responseText is complete
16+
*
17+
* The tests below drive those states explicitly, including the case where the
18+
* body arrives across two of them, which is what tells a handler reading state
19+
* 4 apart from one reading state 3.
20+
*/
21+
22+
import { readFileSync } from 'node:fs';
23+
import { dirname, join } from 'node:path';
24+
import { fileURLToPath } from 'node:url';
25+
import { TextEncoder as NodeTextEncoder } from 'node:util';
26+
27+
const __dirname = dirname(fileURLToPath(import.meta.url));
28+
const INTERFACE_SOURCE = readFileSync(join(__dirname, '..', 'popup', 'interface.js'), 'utf8');
29+
30+
if (typeof globalThis.TextEncoder === 'undefined') {
31+
globalThis.TextEncoder = NodeTextEncoder;
32+
}
33+
34+
/** A finished check-query response, as 4CAT sends it. */
35+
const COMPLETED = JSON.stringify({
36+
done: true,
37+
status: 'Dataset completed.',
38+
url: 'http://localhost/results/f7b31c256ae30dc451d2fe0d5e37bd4b/',
39+
rows: 310,
40+
});
41+
42+
/**
43+
* An XMLHttpRequest whose readyStates are driven by the test.
44+
*
45+
* `steps` is replayed in order, each entry setting readyState, status and
46+
* responseText before the handler is called, so a test can say exactly how
47+
* much of the body had arrived at each state.
48+
*/
49+
function install_fake_xhr(steps) {
50+
const seen = [];
51+
const requests = [];
52+
function FakeXHR() {
53+
requests.push(this);
54+
this.DONE = 4;
55+
this.readyState = 0;
56+
this.status = 0;
57+
this.responseText = '';
58+
}
59+
FakeXHR.prototype.open = function (method, url) { this.url = url; };
60+
FakeXHR.prototype.send = function () {
61+
this.replay = (async () => {
62+
for (const step of steps) {
63+
this.readyState = step.readyState;
64+
this.status = step.status ?? 200;
65+
this.responseText = step.responseText ?? '';
66+
if (this.onreadystatechange) {
67+
await this.onreadystatechange();
68+
}
69+
// what the interface was showing after this state was handled
70+
seen.push({
71+
readyState: step.readyState,
72+
message: document.getElementById('upload-status').innerText,
73+
link: document.querySelector('#upload-status a')?.getAttribute('href') ?? null,
74+
});
75+
}
76+
})();
77+
};
78+
globalThis.XMLHttpRequest = FakeXHR;
79+
80+
/**
81+
* Wait for every request raised so far to finish replaying.
82+
*
83+
* init() awaits the stored 4CAT URL before it opens the request, and the
84+
* completion branch awaits the upload being recorded, so the work outlives
85+
* the call that started it.
86+
*/
87+
const settled = async () => {
88+
for (let i = 0; i < 50 && !requests.length; i++) await Promise.resolve();
89+
await Promise.all(requests.map(r => r.replay));
90+
for (let i = 0; i < 10; i++) await Promise.resolve();
91+
};
92+
93+
return { seen, settled };
94+
}
95+
96+
/**
97+
* Run popup/interface.js the way the browser runs it, as a classic script, and
98+
* take references out of its own scope.
99+
*
100+
* The file is evaluated unchanged; only a tail is appended to hand back the
101+
* things these tests need, since `const`/`var` at the top of a script are not
102+
* reachable from outside when the script is evaluated as a function body.
103+
*/
104+
function load_interface() {
105+
const uploads = [];
106+
globalThis.browser = {
107+
extension: {
108+
getBackgroundPage: () => ({
109+
browser: {
110+
storage: { local: {
111+
get: async () => ({ '4cat-url': 'http://localhost' }),
112+
set: async () => {},
113+
} },
114+
},
115+
db: { uploads: { add: async (row) => { uploads.push(row); } } },
116+
zeeschuimer: { modules: {} },
117+
}),
118+
},
119+
downloads: { onChanged: { addListener() {} } },
120+
storage: { local: { get: async () => ({}), set: async () => {} } },
121+
};
122+
123+
const handle = new Function(
124+
INTERFACE_SOURCE + '\n;return {upload_poll, is_uploading: () => is_uploading};'
125+
)();
126+
return { ...handle, uploads };
127+
}
128+
129+
/** The bits of the popup that upload_poll reads and writes. */
130+
function build_dom() {
131+
document.body.innerHTML = `
132+
<p id="upload-status"></p>
133+
<button class="upload-to-4cat" data-platform="douyin.com" disabled></button>
134+
`;
135+
}
136+
137+
const buttons_disabled = () =>
138+
[...document.querySelectorAll('.upload-to-4cat')].every(b => b.hasAttribute('disabled'));
139+
140+
describe('upload_poll', () => {
141+
beforeEach(build_dom);
142+
143+
test('reports completion when the whole body arrives at once', async () => {
144+
const { settled } = install_fake_xhr([
145+
{ readyState: 2, responseText: '' },
146+
{ readyState: 3, responseText: COMPLETED },
147+
{ readyState: 4, responseText: COMPLETED },
148+
]);
149+
const { upload_poll, uploads } = load_interface();
150+
151+
await upload_poll.init({ key: 'f7b31c25' }, 'douyin.com');
152+
await settled();
153+
154+
expect(document.querySelector('#upload-status a').getAttribute('href'))
155+
.toBe('http://localhost/results/f7b31c256ae30dc451d2fe0d5e37bd4b/');
156+
expect(uploads).toEqual([expect.objectContaining({
157+
platform: 'douyin.com', items: 310,
158+
url: 'http://localhost/results/f7b31c256ae30dc451d2fe0d5e37bd4b/',
159+
})]);
160+
expect(buttons_disabled()).toBe(false);
161+
});
162+
163+
test('reports completion when the body arrives across two readyStates', async () => {
164+
// State 3 carries a fragment: valid JSON cannot be parsed from it yet.
165+
// This is the case a handler acting on state 3 cannot serve.
166+
const half = COMPLETED.slice(0, 40);
167+
const { seen, settled } = install_fake_xhr([
168+
{ readyState: 2, responseText: '' },
169+
{ readyState: 3, responseText: half },
170+
{ readyState: 4, responseText: COMPLETED },
171+
]);
172+
const { upload_poll, uploads } = load_interface();
173+
174+
await upload_poll.init({ key: 'f7b31c25' }, 'douyin.com');
175+
await settled();
176+
177+
expect(document.querySelector('#upload-status a')?.getAttribute('href'))
178+
.toBe('http://localhost/results/f7b31c256ae30dc451d2fe0d5e37bd4b/');
179+
expect(uploads).toHaveLength(1);
180+
expect(seen.map(s => s.readyState)).toEqual([2, 3, 4]);
181+
});
182+
183+
test('shows no error while the response is still arriving', async () => {
184+
const { seen, settled } = install_fake_xhr([
185+
{ readyState: 2, responseText: '' },
186+
{ readyState: 3, responseText: COMPLETED },
187+
{ readyState: 4, responseText: COMPLETED },
188+
]);
189+
const { upload_poll } = load_interface();
190+
191+
await upload_poll.init({ key: 'f7b31c25' }, 'douyin.com');
192+
await settled();
193+
194+
const complained = seen.filter(s => (s.message ?? '').includes('malformed'));
195+
expect(complained).toEqual([]);
196+
});
197+
198+
test('a poll that cannot be read gives the upload buttons back', async () => {
199+
const { settled } = install_fake_xhr([
200+
{ readyState: 2, responseText: '' },
201+
{ readyState: 4, status: 500, responseText: 'Internal Server Error' },
202+
]);
203+
const { upload_poll } = load_interface();
204+
205+
await upload_poll.init({ key: 'f7b31c25' }, 'douyin.com');
206+
await settled();
207+
208+
expect(document.getElementById('upload-status').innerText)
209+
.toContain('Error while checking for upload status');
210+
// Without this the popup can never start another upload.
211+
expect(buttons_disabled()).toBe(false);
212+
});
213+
});

0 commit comments

Comments
 (0)