Need Schematron test suite and automated testing

[tgraham-antenna]

One stumbling block (other than the people) for making changes to the Schematron code, such as #48, is the absence of a comprehensive test suite for making sure that changes don't unintentionally break things.

Presumably the GitHub way to do it would be to use Travis. Possibly the best way to run a test suite would be by using @vincentml's Schematron support that was recently added to xspec/xspec#105.

Does anyone know enough about Travis to be able to set this up?

[vincentml]

It does seem like XSpec would be a natural fit for testing the Schematron XSLTs.

I shared this question with Sandro Cirulli (@cirulls) and asked about the automated testing CI that is set up for XSpec, which works very well using Travis and AppVeyor. @cirulls offered to help out, and shared these links:

How to guide for integrating XSpec with Travis: https://github.com/xspec/xspec/wiki/Integration-with-Travis

Configuration for Travis to test XSpec on Linux-like systems: https://github.com/xspec/xspec/blob/master/.travis.yml

Configuration for AppVeyor to test XSpec on Windows systems: https://github.com/xspec/xspec/blob/master/appveyor.yml

[cirulls]

Yes, I'll be happy to help setting this up for Schematron. Do you already have some tests for Schematron?

[tgraham-antenna]

The only official tests that I know of are at https://github.com/Schematron/schematron/tree/master/trunk/schematron/test

Notionally, there would ideally be tests for every testable statement in the spec. Tests for statements in the spec would ideally be applicable to every Schematron implementation, but for that to be true, the tests would have to avoid being dependent on the XSLT binding. How would XSpec Schematron tests work with a Schematron that is not implemented in XSLT?

Another question is whether or not it's necessary to start by also planning to test every aspect of the stylesheets in the XSLT pipeline stages or whether it's sufficient to add tests for those only as issues arise for individual XSLT stylesheets.

[vincentml]

A test suite for Schematron would probably need two sets of tests, with each set having multiple XSpec tests. The two sets of tests:

1. There could be a set of Schematrons with accompanying sample XML that should produce specific expected results (thrown asserts, etc.). These Schematrons and sample XML could be organized with a set of XSpec tests. These XSpec tests could be used to test Schematron XSLT using xspec.bat/xspec.sh. These XSpec tests could potentially be transformed from XML into documentation which might be useful for building tests of other Schematron implementations, and use the label elements of XSpec to hold descriptions of each test scenario. It might even be possible to transform these XSpec tests from XML into runnable code in other languages (i.e. JUnit for Java).

2. There could also be a set of XSpec tests to test the Schematron XSLTs at a more granular level. For example testing the xsl:template's in iso_abstract_expand.xsl, etc. This set of XSpec tests would verify that the Schematron XSLTs work correctly to compile Schematron into XSLT.

The set of XSpec tests described above first is probably more important because of the potential to test other Schematron implementations. The set of granular XSpec tests on the Schematron XSLTs might not need to do as much if we have the other set of tests, but could still be useful to have when developing on the Schematron XSLTs.

[cirulls]

If you start writing some XSpec tests for Schematron (even some trivial ones as a proof-of-concept), I can give it a go to plug these XSpec tests into Travis for the Schematron repository. I may need some extended permissions on the Schematron repository to do that (write access is probably sufficient). Once the Travis configuration is plugged in, the test suite will run automatically once a new branch or a pull request is submitted to Schematron. You can then extend the test suite by adding more tests into a specific folder.

[tgraham-antenna]

I have added some XSpec tests for the existing tests that were in trunk/schematron/test. I've now also moved the test directory to the new 'Schematron/schematron-test' repository in the hope that the tests will also be usable by other Schematron implementations.

To begin with, testing a Schematron version should show that the Schematron version has the same results as the '2017-02-09' release. At such time as the 'Schematron/schematron' Schematron makes an incompatible change, we'll have to change the tests so they all pass (not a problem if code was changed to match changed tests in good BDD style) and make a new release.

I think this means that the 'Schematron/schematron-test' repository would have to make releases whenever 'Schematron/schematron' makes releases. 'Schematron/schematron' will always use specific versions of the 'Schematron/schematron-test' submodule, but making a release might be necessary so that other implementations know which 'Schematron/schematron-test' version goes with which 'Schematron/schematron' release.

@cirulls, is this enough so that you can make a fork, get Travis working, and submit a PR?

[tgraham-antenna]

If we go to parallel XSLT 1.0 and XSLT 2.0 code bases (a la #55) (and possibly also XSLT 3.0), the regression testing needs to test every flavour. This implies that Travis cannot just use the latest Saxon-HE when testing the XSLT 1.0 flavour because the latest Saxon-HE will refuse to process XSLT 1.0 code (a la #55).

[cirulls]

@tgraham-antenna: that should be enough to get me started. I can't guarantee when this will be done but I'll try to get started during this month.

Regarding the multiple versions of XSLT, we can run tests with different versions of Saxon HE so that it is still possible to test XSLT 1.0. We implemented this in XSpec where we run tests in parallel with three different versions of Saxon (see the configuration in the Travis file).

[tgraham-antenna]

@cirulls: thanks. I know that XSpec does multiple Saxon versions, since I'd already read the Travis file. My comment was made to get the requirement on the record for this project.