Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions docs/adminguide/src/site/_config.yml
Original file line number Diff line number Diff line change
Expand Up @@ -81,3 +81,4 @@ netcdf-java_docset_version: 5.9
tomcat_version: 10.1
java_version: 17
servlet_spec: 6.0
inv_catalog_schema_version: 1.3
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,6 @@ Or, you can simply use the [THREDDS Catalog Validation service](https://thredds.
This service already knows where the schemas are located, so it's not necessary to add that information to the catalog; you only need it if you want to do your own validation.

{%include note.html content="
For more information, you can look at the [schema](https://www.unidata.ucar.edu/schemas/thredds/InvCatalog.1.0.7.xsd){:target='_blank'} referenced in the above example.
For more information, you can look at the [schema](https://schemas.unidata.ucar.edu/thredds/InvCatalog.1.0.7.xsd){:target='_blank'} referenced in the above example.
However, you'll probably want to study the [catalog specification](client_side_catalog_specification.html) instead, as it is much more digestible.
" %}
Original file line number Diff line number Diff line change
Expand Up @@ -105,7 +105,7 @@ And, this is the result:

~~~xml
<?xml version="1.0" encoding="ISO-8859-1"?>
<catalog xmlns="http://www.unidata.ucar.edu/namespaces/thredds/InvCatalog/v1.0" version="1.2" name="FNL-2010-01" xmlns:xlink="http://www.w3.org/1999/xlink">
<catalog xmlns="http://www.unidata.ucar.edu/namespaces/thredds/InvCatalog/v1.0" version="{{ site.inv_catalog_schema_version }}" name="FNL-2010-01" xmlns:xlink="http://www.w3.org/1999/xlink">
<service name="VirtualServices" base="" serviceType="Compound">
<service name="ncdods" base="/thredds/dodsC/" serviceType="OPENDAP"/>
<service name="wcs" base="/thredds/wcs/" serviceType="WCS"/>
Expand Down
25 changes: 20 additions & 5 deletions docs/adminguide/src/site/pages/thredds/DatasetScanRef.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ Here is a minimal catalog containing a `datasetScan` element:

~~~xml
<?xml version="1.0" encoding="UTF-8"?>
<catalog name="Unidata Workshop 2006 - NCEP Model Data" version="1.0.1"
<catalog name="Unidata Workshop 2006 - NCEP Model Data" version="{{ site.inv_catalog_schema_version }}"
xmlns="http://www.unidata.ucar.edu/namespaces/thredds/InvCatalog/v1.0"
xmlns:xlink="http://www.w3.org/1999/xlink">
<service name="myserver" serviceType="OpenDAP" base="/thredds/dodsC/" />
Expand All @@ -40,7 +40,7 @@ The main points are:
In the catalog that the TDS server sends to a client, the `datasetScan` element is shown as a catalog reference:

~~~xml
<catalog name="Unidata Workshop 2006 - NCEP Model Data" version="1.0.1"
<catalog name="Unidata Workshop 2006 - NCEP Model Data" version="{{ site.inv_catalog_schema_version }}"
xmlns="http://www.unidata.ucar.edu/namespaces/thredds/InvCatalog/v1.0"
xmlns:xlink="http://www.w3.org/1999/xlink">

Expand Down Expand Up @@ -124,7 +124,7 @@ The `datasetScan` element is an extension of a `dataset` element, and it can con
Typically, you want all of its contained datasets to inherit the `metadata`, so add an inherited `metadata` element contained in the `datasetScan` element, for example:

~~~xml
<catalog name="Unidata Workshop 2006 - NCEP Model Data" version="1.0.1"
<catalog name="Unidata Workshop 2006 - NCEP Model Data" version="{{ site.inv_catalog_schema_version }}"
xmlns="http://www.unidata.ucar.edu/namespaces/thredds/InvCatalog/v1.0">

<service name="myserver" serviceType="OpenDAP" base="/thredds/dodsC/" />
Expand Down Expand Up @@ -176,7 +176,7 @@ Its a good idea to always use a filter element with explicit includes, so if str

~~~xml
<?xml version="1.0" encoding="UTF-8"?>
<catalog name="Unidata Workshop 2006 - NCEP Model Data" version="1.0.1"
<catalog name="Unidata Workshop 2006 - NCEP Model Data" version="{{ site.inv_catalog_schema_version }}"
xmlns="http://www.unidata.ucar.edu/namespaces/thredds/InvCatalog/v1.0"
xmlns:xlink="http://www.w3.org/1999/xlink">

Expand Down Expand Up @@ -209,7 +209,7 @@ A few gotchas to remember:

~~~xml
<?xml version="1.0" encoding="UTF-8"?>
<catalog name="Unidata Workshop 2006 - NCEP Model Data" version="1.0.1"
<catalog name="Unidata Workshop 2006 - NCEP Model Data" version="{{ site.inv_catalog_schema_version }}"
xmlns="http://www.unidata.ucar.edu/namespaces/thredds/InvCatalog/v1.0"
xmlns:xlink="http://www.w3.org/1999/xlink">

Expand Down Expand Up @@ -357,3 +357,18 @@ So, when cataloged, this dataset would end up as something like this:
<dataset name="NCEP GFS 191km Alaska 2005-10-11 00:00:00 GMT"
urlPath="models/NCEP/GFS/Alaska_191km/GFS_Alaska_191km_20051011_0000.grib1"/>
~~~

## Skipping Empty Directories

By default, `datasetScan` will add a `catalogRef` for each directory that it finds, including directories that are empty.
Starting with version 5.9 of the TDS, you can skip empty directories by adding the `excludeEmptyDirs="true"` attribute to the `datasetScan` element.
For example:

~~~xml
<datasetScan name="Dataset Name" path="unique/path"
location="/data/directory/to/scan"
excludeEmptyDirs="true">
~~~

This will cause `datasetScan` to skip adding `catalogRef`s for empty directories.
To preserve backwards compatibility, the `excludeEmptyDirs` attribute defaults to `false`.
4 changes: 2 additions & 2 deletions docs/adminguide/src/site/pages/thredds/GribConfigRef.md
Original file line number Diff line number Diff line change
Expand Up @@ -345,9 +345,9 @@ Sort the files [lexicographically](https://en.wikipedia.org/wiki/Lexicographical

## `gribConfig` XML Schema

The `gribConfig` schema definition, version 1.2.
The `gribConfig` schema definition, version 1.3.

see: [https://www.unidata.ucar.edu/schemas/thredds/InvCatalog.1.2.xsd](https://www.unidata.ucar.edu/schemas/thredds/InvCatalog.1.2.xsd){:target="_blank"}
see: [https://schemas.unidata.ucar.edu/thredds/InvCatalog.{{ site.inv_catalog_schema_version }}.xsd](https://schemas.unidata.ucar.edu/thredds/InvCatalog.{{ site.inv_catalog_schema_version }}.xsd){:target="_blank"}

~~~xsd
<xsd:complexType name="gribConfigType">
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ Related resources:

* [Client Catalog Primer](basic_client_catalog.html)
* [Client Catalog Example](https://thredds.ucar.edu/thredds/idd/forecastModels.xml){:target="_blank"}
* [Catalog XML Schema](https://www.unidata.ucar.edu/schemas/thredds/InvCatalog.1.2.xsd){:target="_blank"}
* [Catalog XML Schema](https://schemas.unidata.ucar.edu/thredds/InvCatalog.{{ site.inv_catalog_schema_version }}.xsd){:target="_blank"}
* [Server-Side Catalog Specification](server_side_catalog_specification.html)

## Base Catalog Elements
Expand Down
14 changes: 8 additions & 6 deletions docs/adminguide/src/site/pages/thredds/InvCatalogServerSpec.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ Also, see:

* [DatasetScan](tds_dataset_scan_ref.html) reference
* [FeatureCollection](feature_collections_ref.html) reference
* [Catalog XML Schema](https://www.unidata.ucar.edu/schemas/thredds/InvCatalog.1.2.xsd){:target="_blank"}
* [Catalog XML Schema](https://schemas.unidata.ucar.edu/thredds/InvCatalog.{{ site.inv_catalog_schema_version }}.xsd){:target="_blank"}
* [Client-Side Catalog Specification](client_side_catalog_specification.html)

## Base Catalog Elements
Expand Down Expand Up @@ -154,18 +154,20 @@ A `datasetScan` can be used wherever a `dataset` element is allowed.
<xsd:attribute name="path" type="xsd:string" use="required"/>
<xsd:attribute name="location" type="xsd:string"/>
<xsd:attribute name="addLatest" type="xsd:boolean"/>
<xsd:attribute name="excludeEmptyDirs" type="xsd:boolean"/>
</xsd:extension>
</xsd:complexContent>
</xsd:complexType>
</xsd:element>
~~~

* The `datasetScan` element generates nested THREDDS catalogs by scanning the directory named in the `location` attribute, and creating a `dataset` for each file found, and a `catalogRef` for each subdirectory.
* The location must be an absolute path.
* The `path` attribute is used to create the URL for these files and catalogs.
* The path must be globally unique over all paths for the TDS.
* Do not put leading or trailing slashes on the path.
* The `datasetScan` element generates nested THREDDS catalogs by scanning the directory named in the `location` attribute, and creating a `dataset` for each file found, and a `catalogRef` for each subdirectory.
The location must be an absolute path.
* The `path` attribute is used to create the URL for these files and catalogs.
The path must be globally unique over all paths for the TDS.
Do not put leading or trailing slashes on the path.
* The `addLatest` attribute adds a *latest resolver service* to the `datasetScan`.
* The `excludeEmptyDirs` configures the `datasetScan` to skip adding `catalogRef`s for empty directories.

A `datasetScan` element is in the `dataset substitutionGroup`, so it can be used wherever a `dataset` element can be used.
It is an extension of a `DatasetType`, so any of the `dataset` element nested elements and attributes can be used in it.
Expand Down
1 change: 1 addition & 0 deletions docs/quickstart/src/site/_config.yml
Original file line number Diff line number Diff line change
Expand Up @@ -81,3 +81,4 @@ netcdf-java_docset_version: 5.9
tomcat_version: 10.1
java_version: 17
servlet_spec: 6.0
inv_catalog_schema_version: 1.3
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,6 @@ Or, you can simply use the [THREDDS Catalog Validation service](https://thredds.
This service already knows where the schemas are located, so it's not necessary to add that information to the catalog; you only need it if you want to do your own validation.

{%include note.html content="
For more information, you can look at the [schema](https://www.unidata.ucar.edu/schemas/thredds/InvCatalog.1.0.7.xsd){:target='_blank'} referenced in the above example.
For more information, you can look at the [schema](https://schemas.unidata.ucar.edu/thredds/InvCatalog.1.0.7.xsd){:target='_blank'} referenced in the above example.
However, you'll probably want to study the [catalog specification](client_side_catalog_specification.html) instead, as it is much more digestible.
" %}
Original file line number Diff line number Diff line change
Expand Up @@ -105,7 +105,7 @@ And, this is the result:

~~~xml
<?xml version="1.0" encoding="ISO-8859-1"?>
<catalog xmlns="http://www.unidata.ucar.edu/namespaces/thredds/InvCatalog/v1.0" version="1.2" name="FNL-2010-01" xmlns:xlink="http://www.w3.org/1999/xlink">
<catalog xmlns="http://www.unidata.ucar.edu/namespaces/thredds/InvCatalog/v1.0" version="{{ site.inv_catalog_schema_version }}" name="FNL-2010-01" xmlns:xlink="http://www.w3.org/1999/xlink">
<service name="VirtualServices" base="" serviceType="Compound">
<service name="ncdods" base="/thredds/dodsC/" serviceType="OPENDAP"/>
<service name="wcs" base="/thredds/wcs/" serviceType="WCS"/>
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -323,7 +323,7 @@ Here are some additional, optional changes you can make to increase maintainabil

~~~xml
<?xml version='1.0' encoding='UTF-8'?>
<catalog name="ESGF Master Catalog" version="1.2"
<catalog name="ESGF Master Catalog" version="{{ site.inv_catalog_schema_version }}"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xlink="http://www.w3.org/1999/xlink"
xmlns="http://www.unidata.ucar.edu/namespaces/thredds/InvCatalog/v1.0"
xsi:schemaLocation="http://www.unidata.ucar.edu/namespaces/thredds/InvCatalog/v1.0 https://schemas.unidata.ucar.edu/thredds/InvCatalog.1.2.xsd">
Expand Down
25 changes: 20 additions & 5 deletions docs/quickstart/src/site/pages/thredds/DatasetScanRef.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ Here is a minimal catalog containing a `datasetScan` element:

~~~xml
<?xml version="1.0" encoding="UTF-8"?>
<catalog name="Unidata Workshop 2006 - NCEP Model Data" version="1.0.1"
<catalog name="Unidata Workshop 2006 - NCEP Model Data" version="{{ site.inv_catalog_schema_version }}"
xmlns="http://www.unidata.ucar.edu/namespaces/thredds/InvCatalog/v1.0"
xmlns:xlink="http://www.w3.org/1999/xlink">
<service name="myserver" serviceType="OpenDAP" base="/thredds/dodsC/" />
Expand All @@ -40,7 +40,7 @@ The main points are:
In the catalog that the TDS server sends to a client, the `datasetScan` element is shown as a catalog reference:

~~~xml
<catalog name="Unidata Workshop 2006 - NCEP Model Data" version="1.0.1"
<catalog name="Unidata Workshop 2006 - NCEP Model Data" version="{{ site.inv_catalog_schema_version }}"
xmlns="http://www.unidata.ucar.edu/namespaces/thredds/InvCatalog/v1.0"
xmlns:xlink="http://www.w3.org/1999/xlink">

Expand Down Expand Up @@ -124,7 +124,7 @@ The `datasetScan` element is an extension of a `dataset` element, and it can con
Typically, you want all of its contained datasets to inherit the `metadata`, so add an inherited `metadata` element contained in the `datasetScan` element, for example:

~~~xml
<catalog name="Unidata Workshop 2006 - NCEP Model Data" version="1.0.1"
<catalog name="Unidata Workshop 2006 - NCEP Model Data" version="{{ site.inv_catalog_schema_version }}"
xmlns="http://www.unidata.ucar.edu/namespaces/thredds/InvCatalog/v1.0">

<service name="myserver" serviceType="OpenDAP" base="/thredds/dodsC/" />
Expand Down Expand Up @@ -176,7 +176,7 @@ Its a good idea to always use a filter element with explicit includes, so if str

~~~xml
<?xml version="1.0" encoding="UTF-8"?>
<catalog name="Unidata Workshop 2006 - NCEP Model Data" version="1.0.1"
<catalog name="Unidata Workshop 2006 - NCEP Model Data" version="{{ site.inv_catalog_schema_version }}"
xmlns="http://www.unidata.ucar.edu/namespaces/thredds/InvCatalog/v1.0"
xmlns:xlink="http://www.w3.org/1999/xlink">

Expand Down Expand Up @@ -209,7 +209,7 @@ A few gotchas to remember:

~~~xml
<?xml version="1.0" encoding="UTF-8"?>
<catalog name="Unidata Workshop 2006 - NCEP Model Data" version="1.0.1"
<catalog name="Unidata Workshop 2006 - NCEP Model Data" version="{{ site.inv_catalog_schema_version }}"
xmlns="http://www.unidata.ucar.edu/namespaces/thredds/InvCatalog/v1.0"
xmlns:xlink="http://www.w3.org/1999/xlink">

Expand Down Expand Up @@ -357,3 +357,18 @@ So, when cataloged, this dataset would end up as something like this:
<dataset name="NCEP GFS 191km Alaska 2005-10-11 00:00:00 GMT"
urlPath="models/NCEP/GFS/Alaska_191km/GFS_Alaska_191km_20051011_0000.grib1"/>
~~~

## Skipping Empty Directories

By default, `datasetScan` will add a `catalogRef` for each directory that it finds, including directories that are empty.
Starting with version 5.9 of the TDS, you can skip empty directories by adding the `excludeEmptyDirs="true"` attribute to the `datasetScan` element.
For example:

~~~xml
<datasetScan name="Dataset Name" path="unique/path"
location="/data/directory/to/scan"
excludeEmptyDirs="true">
~~~

This will cause `datasetScan` to skip adding `catalogRef`s for empty directories.
To preserve backwards compatibility, the `excludeEmptyDirs` attribute defaults to `false`.
2 changes: 1 addition & 1 deletion docs/quickstart/src/site/pages/thredds/GribConfigRef.md
Original file line number Diff line number Diff line change
Expand Up @@ -347,7 +347,7 @@ Sort the files [lexicographically](https://en.wikipedia.org/wiki/Lexicographical

The `gribConfig` schema definition, version 1.2.

see: [https://www.unidata.ucar.edu/schemas/thredds/InvCatalog.1.2.xsd](https://www.unidata.ucar.edu/schemas/thredds/InvCatalog.1.2.xsd){:target="_blank"}
see: [https://schemas.unidata.ucar.edu/thredds/InvCatalog.{{ site.inv_catalog_schema_version }}.xsd](https://schemas.unidata.ucar.edu/thredds/InvCatalog.{{ site.inv_catalog_schema_version }}.xsd){:target="_blank"}

~~~xsd
<xsd:complexType name="gribConfigType">
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ Related resources:

* [Client Catalog Primer](basic_client_catalog.html)
* [Client Catalog Example](https://thredds.ucar.edu/thredds/idd/forecastModels.xml){:target="_blank"}
* [Catalog XML Schema](https://www.unidata.ucar.edu/schemas/thredds/InvCatalog.1.2.xsd){:target="_blank"}
* [Catalog XML Schema](https://schemas.unidata.ucar.edu/thredds/InvCatalog.{{ site.inv_catalog_schema_version }}.xsd){:target="_blank"}
* [Server-Side Catalog Specification](server_side_catalog_specification.html)

## Base Catalog Elements
Expand Down
14 changes: 8 additions & 6 deletions docs/quickstart/src/site/pages/thredds/InvCatalogServerSpec.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ Also, see:

* [DatasetScan](tds_dataset_scan_ref.html) reference
* [FeatureCollection](feature_collections_ref.html) reference
* [Catalog XML Schema](https://www.unidata.ucar.edu/schemas/thredds/InvCatalog.1.2.xsd){:target="_blank"}
* [Catalog XML Schema](https://schemas.unidata.ucar.edu/thredds/InvCatalog.{{ site.inv_catalog_schema_version }}.xsd){:target="_blank"}
* [Client-Side Catalog Specification](client_side_catalog_specification.html)

## Base Catalog Elements
Expand Down Expand Up @@ -154,18 +154,20 @@ A `datasetScan` can be used wherever a `dataset` element is allowed.
<xsd:attribute name="path" type="xsd:string" use="required"/>
<xsd:attribute name="location" type="xsd:string"/>
<xsd:attribute name="addLatest" type="xsd:boolean"/>
<xsd:attribute name="excludeEmptyDirs" type="xsd:boolean"/>
</xsd:extension>
</xsd:complexContent>
</xsd:complexType>
</xsd:element>
~~~

* The `datasetScan` element generates nested THREDDS catalogs by scanning the directory named in the `location` attribute, and creating a `dataset` for each file found, and a `catalogRef` for each subdirectory.
* The location must be an absolute path.
* The `path` attribute is used to create the URL for these files and catalogs.
* The path must be globally unique over all paths for the TDS.
* Do not put leading or trailing slashes on the path.
* The `datasetScan` element generates nested THREDDS catalogs by scanning the directory named in the `location` attribute, and creating a `dataset` for each file found, and a `catalogRef` for each subdirectory.
The location must be an absolute path.
* The `path` attribute is used to create the URL for these files and catalogs.
The path must be globally unique over all paths for the TDS.
Do not put leading or trailing slashes on the path.
* The `addLatest` attribute adds a *latest resolver service* to the `datasetScan`.
* The `excludeEmptyDirs` configures the `datasetScan` to skip adding `catalogRef`s for empty directories.

A `datasetScan` element is in the `dataset substitutionGroup`, so it can be used wherever a `dataset` element can be used.
It is an extension of a `DatasetType`, so any of the `dataset` element nested elements and attributes can be used in it.
Expand Down
1 change: 1 addition & 0 deletions docs/userguide/src/site/_config.yml
Original file line number Diff line number Diff line change
Expand Up @@ -80,3 +80,4 @@ netcdf-java_docset_version: 5.9
tomcat_version: 10.1
java_version: 17
servlet_spec: 6.0
inv_catalog_schema_version: 1.3
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,6 @@ Or, you can simply use the [THREDDS Catalog Validation service](https://thredds.
This service already knows where the schemas are located, so it's not necessary to add that information to the catalog; you only need it if you want to do your own validation.

{%include note.html content="
For more information, you can look at the [schema](https://www.unidata.ucar.edu/schemas/thredds/InvCatalog.1.0.7.xsd){:target='_blank'} referenced in the above example.
For more information, you can look at the [schema](https://schemas.unidata.ucar.edu/thredds/InvCatalog.1.0.7.xsd){:target='_blank'} referenced in the above example.
However, you'll probably want to study the [catalog specification](client_side_catalog_specification.html) instead, as it is much more digestible.
" %}
Original file line number Diff line number Diff line change
Expand Up @@ -105,7 +105,7 @@ And, this is the result:

~~~xml
<?xml version="1.0" encoding="ISO-8859-1"?>
<catalog xmlns="http://www.unidata.ucar.edu/namespaces/thredds/InvCatalog/v1.0" version="1.2" name="FNL-2010-01" xmlns:xlink="http://www.w3.org/1999/xlink">
<catalog xmlns="http://www.unidata.ucar.edu/namespaces/thredds/InvCatalog/v1.0" version="{{ site.inv_catalog_schema_version }}" name="FNL-2010-01" xmlns:xlink="http://www.w3.org/1999/xlink">
<service name="VirtualServices" base="" serviceType="Compound">
<service name="ncdods" base="/thredds/dodsC/" serviceType="OPENDAP"/>
<service name="wcs" base="/thredds/wcs/" serviceType="WCS"/>
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -322,7 +322,7 @@ Here are some additional, optional changes you can make to increase maintainabil

~~~xml
<?xml version='1.0' encoding='UTF-8'?>
<catalog name="ESGF Master Catalog" version="1.2"
<catalog name="ESGF Master Catalog" version="{{ site.inv_catalog_schema_version }}"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xlink="http://www.w3.org/1999/xlink"
xmlns="http://www.unidata.ucar.edu/namespaces/thredds/InvCatalog/v1.0"
xsi:schemaLocation="http://www.unidata.ucar.edu/namespaces/thredds/InvCatalog/v1.0 https://schemas.unidata.ucar.edu/thredds/InvCatalog.1.2.xsd">
Expand Down
Loading