XProc Test Suite markup

Table of Contents

1. Introduction

The XProc Test Suite consists of a set of test suite documents and assorted ancillary files. The test suite documents are defined by the schema described in this document.

The test suite schema is normatively defined in RELAX NG Compact Syntax, but it isn’t necessary for an implementation to use this schema for validation.

Each individual test is defined by a test element. Tests can be grouped together with div elements and/or collected together in a test-suite. A test suite document must begin with one of these elements. All of the test suite elements are in the test suite namespace, see Namespaces. 

start = test | test-suite | test-div

2. Metadata

All tests, test suites, and divisions begin with a required metadata element named info. Metadata consists of a title, a simple string with inline markup, a revision-history that documents changes to the test, and optional specref elements. 

info =
    element t:info {
	title
	& revision-history
	& specref*
    }

2.1. Revision history

A revision history is a collection of one or more revisions. 

revision-history =
    element t:revision-history {
	revision+
    }

Revisions must be listed in reverse chronological order, so the most recent revision appears first. Each revision consists of a date, an author, and a description of the revision (see Description). 

revision =
    element t:revision {
	attribute initials { text }?,
	date,
	author?,
	description
    }

The date must be an ISO 8601 date or dateTime. An author has a name and an optional email address and uri. The description should describe what has been changed and why. 

author =
    element t:author {
	attribute initials { text }?
      & name?
      & email?
      & uri?
    }

2.1.1. Author initials

Repeating all of the author information in each revision can be tedious. The initials attribute can simplify the markup: it acts as an ID/IDREF link. The first time an author is mentioned, provide as much information as possible and mark that author with a (unique) set of initials. In other revisions, specify the author initials but leave the author element otherwise empty. For example, both of these revisions have the same authorship: 

<t:revision>
  <t:date>2020—01-04</t:date>
  <t:author initials="jd"/>
  <t:description xmlns="http://www.w3.org/1999/xhtml">
    <p>Corrected typo in test.</p>
  </t:description>
</t:revision>

<t:revision>
  <t:date>2019-12—31</t:date>
  <t:author initials="jd">
    <t:name>Jane Doe</t:name>
    <t:email>jane@example.com</t:email>
    <t:uri>https://example.com/jane/</t:uri>
  </t:author>
  <t:description xmlns="http://www.w3.org/1999/xhtml">
    <p>Initial checkin.</p>
  </t:description>
</t:revision>

2.2. Specref

If a test, or set of tests, relates to a particular aspect of a particular specification, a link can be provided. The link consists of a spec attribute that identifies the specification and a linkend attribute that must contain the ID value in the specification with which this test is associated. 

specref =
    element t:specref {
	attribute spec { "xproc" | "steps" }?
      & attribute linkend { xsd:NMTOKEN }
      & empty
    }

(It’s not clear if this turned out to be as useful in practice as it was in theory.)

3. Description

The description can contain any mixture of text and HTML elements. HTML elements must be in the HTML namespace: http://www.w3.org/1999/xhtml.

4. Test suites and divisions

The test-suite and div elements serve to group tests together. 

test-suite =
    element t:test-suite {
	testattr,
	(info, description?, property*, test*, test-div*)
    }

Divisions may be nested. 

test-div =
    element t:div {
	(info, description?, property*, test*, test-div*)
    }

5. Common attributes

Each test (and test-suite) has attributes to identify the features it requires and the circumstances under which it should be run. 

testattr = attribute features {
	       list { xsd:token+ }
	   } ?
	   & attribute when { text }?
	   & attribute src { text }?
	   & attribute version { text }?

The attributes should be interpreted as follows:

  • features is a set of one or more feature tokens. Feature tokens identify features that the implementation must support in order for this test to pass. The test should be skipped if it identifies any unsupported features.

    The set of feature tokens is a bit ad hoc. Here’s a summary of the feature tokens at the time this document was written:

    • archive-order, ???
    • dtd-id-ref-warning, a RELAX NG validator capable of reporting DTD-based ID/IDREF warnings is required.
    • lazy-eval, lazy evaluation of variable bindings is required.
    • p-validate-with-relax-ng, a RELAX NG validator is required.
    • p-validate-with-schematron, a Schematron validator is required.
    • p-validate-with-xsd, a W3C XML Schema validator is required.
    • webaccess, access to the web is required (this test cannot be run offline).
    • xquery_1_0, an XQuery 1.0 processor is required.
    • xquery_3_0, an XQuery 3.0 processor is required.
    • xquery_3_1, an XQuery 3.1 processor is required.
    • xslt-1, an XSLT 1.0 processor is required.
    • xslt-1-output-base-uri, ???
    • xslt-2, an XSLT 2.0 processor is required.
    • xslt-3, an XSLT 3.0 processor is required.
    • xslt-serialization, ???
  • when specifies an XPath expression that can be evaluated statically by the test processor. If the effective boolean value of the result is false(), the test is ignored.
  • src specifies a URI where the test (or test-suite) is located. If this attribute is specified, the element on which it occurs must be empty.
  • version specifies the version of the XProc processor required for this test.

6. Tests

If a test specifies a particular platform, it is only run on that platform.

There are two kinds of tests, passing tests and failing tests. 

test = passingTest | failingTest

Passing and failing tests are mostly the same. Tests have common attributes, metadata, and a description. The body of the test in each case consists of property, filetest–, ~input, output, and pipeline elements.

A passing test has an expected attribute (that must have the value be pass) and may have schematron tests to verify the results. 

passingTest =
    element t:test {
	testattr
      & attribute expected { "pass" }
      & attribute platform { ("Linux" | "MacOS" | "Windows") }?
      & (info, description?,
	 (property* & file-environment? & input* & option* & pipeline? & schematron?))
    }

A failing test has an expected attribute (that must have the value be fail) and must specify one or more expected error conditions. 

failingTest =
    element t:test {
	testattr
      & attribute expected { "fail" }
      & attribute code { xsd:NMTOKENS }
      & attribute platform { ("Linux" | "MacOS" | "Windows") }?
      & (info, description?,
	 (property* & file-environment? & input* & option* & pipeline?))
    }

Broadly speaking, the test harness sets the specified properties and runs the specified pipeline with the specified inputs and options. A passing test must complete without error, or it must be recorded as a failure. If it completes, its output is tested against any Schematron rules provided. No assertions must be raised.

If a failing test raises one of the specified error codes, it is a “pass” from the perspective of the test suite, otherwise it must be recorded as a failure.

Like tests and test-suites, inputs, pipelines, and Schematron rules can be specified inline or with a src attribute that points to their content.

6.1. Properties

A property is a name/value pair. These must be made available to the implementation, but the precise mechanism is necessarily implementation defined. 

property =
    element t:property {
	attribute name { text }
      & attribute value { text }
    }

On the Java platform, these are system properties.

6.2. File environment

The optional file-environment element defines a test environment in the local filesystem to run tests for steps from the file steps specification. The file element specifies that a file with the the given path (and the requested properties) should be created. The folder element specifies the same for folders. All pathes must be relative. The test harness has to resolve them agains a folder "testfolder". This folder is located in the parent folder of the test element. The test harness is required to create "testfolder" when a file-environment element is present and delete it after the test is finished. 

file-environment =
    element t:file-environment{
	(file* & folder*)
    }
file = 
    element t:file{
	filetest-attr
	& filetest-content?
    }
filetest-content = text
folder = 
    element t:folder{
	filetest-attr
    }
filetest-attr = 
    attribute path {xsd:anyURI} 
    & attribute last-modified {xsd:dateTime}?
    & attribute readable {xsd:boolean}?
    & attribute writable {xsd:boolean}?
    & attribute hidden {xsd:boolean}?

6.3. Inputs

Each input element identifies the input for a source port on the pipeline. XML and text documents can be placed inline; other media types must be stored externally and identified by URI.

If several inputs identify the same port, then the sequence of documents identified appears on that port, in document order of the input elements in the test. 

input =
    element t:input {
	attribute port { text }
      & attribute src { text }?
      & any*
    }

6.4. Options

Each option element identifies the value for a pipeline option. If a select attribute is provided, it must be an XPath expression. Evaluating that expression gives the value of the option. If the select attribute is not provided, the contents of the element is the value of the option.

If static is specified, and true, the option is a static option and must be provided to the processor at compile time. Options which do not specify a value for static, or that specify the value false, are dynamic options passed to the processor at runtime with the pipeline. 

option =
    element t:option {
	attribute name { text }
      & attribute static { xsd:boolean }?
      & attribute select { text }?
      & any*
    }

6.5. Pipeline

The pipeline element, which can only occur once, provides the pipeline to be tested. The pipeline must be written so that it has only a single, primary output port named result. 

pipeline =
    element t:pipeline {
	attribute src { text }?
      & any*
    }

6.6. Schematron

The schematron element contains Schematron rules. Assertions in the Schematron are tested against the results of the pipeline. 

schematron =
    element t:schematron {
	attribute src { text }?
      & any*
    }

7. Namespaces

The following namespaces are defined in this schema. 

namespace rng  = "http://relaxng.org/ns/structure/1.0"
namespace t = "http://xproc.org/ns/testsuite/3.0"
namespace h = "http://www.w3.org/1999/xhtml"
default namespace = "http://xproc.org/ns/testsuite/3.0"

8. Relax NG Schema

These fragments are stitched together (with a few other fragments) into a complete schema. 

namespace rng  = "http://relaxng.org/ns/structure/1.0"
namespace t = "http://xproc.org/ns/testsuite/3.0"
namespace h = "http://www.w3.org/1999/xhtml"
default namespace = "http://xproc.org/ns/testsuite/3.0"

start = test | test-suite | test-div

testattr = attribute features {
	       list { xsd:token+ }
	   } ?
	   & attribute when { text }?
	   & attribute src { text }?
	   & attribute version { text }?

test-suite =
    element t:test-suite {
	testattr,
	(info, description?, property*, test*, test-div*)
    }

passingTest =
    element t:test {
	testattr
      & attribute expected { "pass" }
      & attribute platform { ("Linux" | "MacOS" | "Windows") }?
      & (info, description?,
	 (property* & file-environment? & input* & option* & pipeline? & schematron?))
    }

failingTest =
    element t:test {
	testattr
      & attribute expected { "fail" }
      & attribute code { xsd:NMTOKENS }
      & attribute platform { ("Linux" | "MacOS" | "Windows") }?
      & (info, description?,
	 (property* & file-environment? & input* & option* & pipeline?))
    }

file-environment =
    element t:file-environment{
	(file* & folder*)
    }
file = 
    element t:file{
	filetest-attr
	& filetest-content?
    }
filetest-content = text
folder = 
    element t:folder{
	filetest-attr
    }
filetest-attr = 
    attribute path {xsd:anyURI} 
    & attribute last-modified {xsd:dateTime}?
    & attribute readable {xsd:boolean}?
    & attribute writable {xsd:boolean}?
    & attribute hidden {xsd:boolean}?

test-div =
    element t:div {
	(info, description?, property*, test*, test-div*)
    }

info =
    element t:info {
	title
	& revision-history
	& specref*
    }

author =
    element t:author {
	attribute initials { text }?
      & name?
      & email?
      & uri?
    }

name = element t:name { text }
email = element t:email { text }
uri = element t:uri { xsd:anyURI }

revision-history =
    element t:revision-history {
	revision+
    }

revision =
    element t:revision {
	attribute initials { text }?,
	date,
	author?,
	description
    }

title =
    element t:title { text }

date =
    element t:date { xsd:date|xsd:dateTime }

specref =
    element t:specref {
	attribute spec { "xproc" | "steps" }?
      & attribute linkend { xsd:NMTOKEN }
      & empty
    }

test = passingTest | failingTest

pipeline =
    element t:pipeline {
	attribute src { text }?
      & any*
    }

schematron =
    element t:schematron {
	attribute src { text }?
      & any*
    }

input =
    element t:input {
	attribute port { text }
      & attribute src { text }?
      & any*
    }

option =
    element t:option {
	attribute name { text }
      & attribute static { xsd:boolean }?
      & attribute select { text }?
      & any*
    }

any =
    element * {
	attribute * { text }*
      & (text | any)*
    }

anyhtml =
    element h:* {
	attribute * { text }*
      & (text | anyhtml)*
    }

description =
    element t:description {
	anyhtml+
    }

property =
    element t:property {
	attribute name { text }
      & attribute value { text }
    }