VXI-11, HiSLIP,.. support wrap up: loose ends

[hb020]

Done in PR #645 :

• investigate gaps remaining:
  • events (check VPP-4.3)
  • locking on serial/socket (check VPP-4.3)
  • mandatory attributes (VPP-4.3 page 5-66 .. 5-73)
• document gaps remaining:
  • VXI-11+HiSLIP:
    • shared locking and nested locking,
    • authentication
  • secure channel on HiSLIP
• document additions:
  • VXI-11 docmd
  • prologix with howto (see Handle Prologix USB to GPIB pyvisa#112)
  • trigger on prologix
• doCMD example

See also PRs #651, #653, #655, #657, #659

TODO: split the below up in several PRs

(This list is made with the help from berg's tool; some of it is already documented in compliance.rst, and some may be mentioned multiple times)

VXI-11:

• add viTerminate/abort
• add viFlush
• add shared and nested locking (provokes many failures with berg's script, excluded from the list below)
• VI_ATTR_MAX_QUEUE_LENGTH is writeable before viEnableEvent: setting VI_ATTR_MAX_QUEUE_LENGTH on a fresh session returned <StatusCode.error_nonsupported_attribute_state: -1073807330>; 3.2.5 makes it writeable until viEnableEvent is called [VPP-4.3 3.2.5]
• VI_ATTR_MAX_QUEUE_LENGTH is read-only after viEnableEvent: the write was refused with <StatusCode.error_nonsupported_attribute_state: -1073807330> but the value changed to 50 anyway [VPP-4.3 3.2.6]

HiSLIP:

like VXI-11:

• add viFlush
• add shared and nested locking (provokes many failures with berg's script, excluded from the list below)
• VI_ATTR_MAX_QUEUE_LENGTH is writeable before viEnableEvent: setting VI_ATTR_MAX_QUEUE_LENGTH on a fresh session returned <StatusCode.error_nonsupported_attribute_state: -1073807330>; 3.2.5 makes it writeable until viEnableEvent is called [VPP-4.3 3.2.5]
• VI_ATTR_MAX_QUEUE_LENGTH is read-only after viEnableEvent: the write was refused with <StatusCode.error_nonsupported_attribute_state: -1073807330> but the value changed to 50 anyway [VPP-4.3 3.2.6]

Others:

• repeated terminate/recover cycles all succeed: did not return within 30s [VPP-4.3 §3.5.1.1]
• clear discards an uncollected response: viClear raised instead of returning a status: RuntimeError: expected message type 'DeviceClearAcknowledge', received 'DataEnd: bytearray(b'stale-marker\n')' [VPP-4.3 3.2.3]
• clear resyncs mid-message: viClear raised instead of returning a status: RuntimeError: protocol synchronization error [VPP-4.3 3.2.3]
• large reads still intact after clears: unexpected exception
• viClear discards an uncollected response: unexpected exception
• a connection lost mid-reply is reported, not hung: expected a connection-lost or timeout error, got RuntimeError: Connection was dropped by server. [VPP-4.3 §6.1.1]

For both:

Cannot be solved in pyvisa-py:

• every component of the resource name is case-insensitive: 4.3.17 requires a case-insensitive compare, but lowercasing the resource class suffix was refused with VI_ERROR_INV_RSRC_NAME (-1073807342), the the whole name was refused with VI_ERROR_INV_RSRC_NAME (-1073807342) [VPP-4.3 4.3.17]

  This is not good, some instruments require specific case. I might however improve case insensitivity on the "TCPIP" and "INSTR" comparisons, but that is inside pyvisa/rname

[hb020]

Working on documentation in https://github.com/hb020/pyvisa-py/blob/unknown_attribute/docs/source/compliance.rst

[berg]

hey @hb020 - if it is helpful, i ported the internal thing i'd been using over to using github actions:
https://berg.github.io/testgear-network-stress/

it basically compares pyvisa-py's behavior against a few commercial libs (connecting to a hislip/vxi-11 test harness) and hopefully cites specs along the way for the tests it runs.

[hb020]

Nice! Will test that. And if I see any tests that can be added/adapted I'll let you know.

[MatthieuDartiailh]

@berg Could this be adapted to finally have some better covering of pyvisa and pyvisa-py ? I have the project to make a reusable test framework for this kind of things but if you already have it no need to reinvent the wheel.

[berg]

@MatthieuDartiailh definitely! happy to talk through any ideas you have there (maybe we can move to another issue)?

I originally threw this together to improve conformance testing of https://github.com/berg/ugpibd (and it uses part of the core server from that codebase as a result), which means it was easiest to just make it GPLv3 which isn't my first preference generally, but happy to navigate that

[MatthieuDartiailh]

Interesting I was planning to use https://github.com/Atmelfan/lxi-rs to create a fake instrument.

What I would like to achieve is a framework living in pyvisa making it easy to test a subset of a backend capabilities based on the capabilities of the instrument available for testing. This way people with regular access to some hardware could have their custom tests and help test PyVISA projects.

Could you also share the testsuite you have for me to have a look at how you organized it ?

[berg]

Sure thing - the test suite is here: https://github.com/berg/testgear-network-stress/tree/main/checks

It is admittedly a bit messy due to its history as independent scripts, but we can certainly clean that up

[berg]

(Also I think https://github.com/berg/testgear-network-stress/blob/main/checks/14_vxi11_flags.py is in the format I most prefer, I just didnt port all the checks over to it after coming up with a better API)

[hb020]

watch out: testing against 'perfect' mock servers is one thing. Another is working with the quirks of real instruments. Examples:

• KS 34465A hard crashes when you send a VXI-11 DEVICE_ENABLE_SRQ with an empty handle. (even when you disable the SRQ, there where you would not need to provide a handle). This is not covered by any VXI-11 rule or observation.
• Keithley DMM6500 disregards VXI-11 rules B.4.5, B.6.6 and B.6.7. When you do 2 subsequent CREATE_LINK calls with lockDevice=True, the last one will time out. If then, you close the core channels, the first lock gets released, but not the second one that suddenly activates because the first one is unlocked. And you even cannot remove that lock anymore because the sessions are gone. And because of lack of B.4.5, the lock is maintained and all subsequent CREATE_LINK calls fail. You need to reboot the device. grrrr. NI-VISA circumvents this by not doing CREATE_LINK(lockDevice=True) when it is requested. What it does is is separate that into a CREATE_LINK and a DEVICE_LOCK.

Meaning: sometimes there is a reason for not being fully compliant because many real world instruments are not compliant themselves.

[berg]

For sure! So what you're saying is... all that stuff I dream about buying on eBay is worth it...

[berg]

(FWIW, the mock server that's there does try to misbehave in certain useful ways, so it is possible to encode some of that behavior into it and test against regressions at least in an automated fashion.)

[hb020]

For sure! So what you're saying is... all that stuff I dream about buying on eBay is worth it...

early 488.1 or pre-488.1 stuff is even worse.... But how many boat anchors do you want/can you store?

[berg]

oh yes. I've got far more of the early stuff than modern gear that speaks the network protocols natively! ugpibd was partly inspired to justify keeping those anchors around just in case I ever get around to owning a boat...