Skip to content

README.md

Latest commit

 

History

History
81 lines (63 loc) · 2.95 KB

README.md

File metadata and controls

81 lines (63 loc) · 2.95 KB
Raw

validate-ddf

validate-ddf is a DDF dataset validator written in PureScript. It reads a directory of CSV files structured according to the DDF (Data Description Format) naming conventions, parses and validates them in layers (concepts → entities → datapoints → synonyms → datapackage), and reports structured errors and warnings.

CLI usage

  1. install
    • npm install -g @gapminder/validate-ddf
  2. run validate-ddf on target path
$ validate-ddf -h
validate-ddf - DDF dataset validator

Usage: validate-ddf [--no-warning] [PATH] [-m|--mode ARG]
                       [-p|--generate-datapackage] [-f|--fix]
  validate DDF dataset at PATH (default to current working dir)

Available options:
  --no-warning             don't show warnings
  PATH                     The dataset path to validate
  -m,--mode ARG            configure how validator find files (filenames or
                           datapackage, default filenames)
  -p,--generate-datapackage
                           whether to generate a datapackage.json after
                           validation. (default false)
  -f,--fix                 auto-fix CSV format issues (BOM, CRLF) and datapackage path separators
  -h,--help                Show this help text

There are 2 modes: filenames and datapackage. In filenames mode, the validator will enumerate all files in a given folder and try to find all the filenames that match ddf standard. In datapackage mode, the validator will read all files in the resources list in datapackage.json.

Resource paths in datapackage.json must use forward slashes (/), including packages created on Windows. The validator reports E_DATAPACKAGE_PATH_SEPARATOR with counts of backslashes and affected paths. Run validate-ddf --fix to replace backslashes with / in resource paths and validate the result. Other metadata is preserved. This works in both filenames and datapackage modes.

In JSON source, "folder\\file.csv" encodes a Windows-style path with one backslash. Malformed JSON must be corrected before automatic path fixing.

JS usage in another app

npm install @gapminder/validate-ddf

import { validate } from "@gapminder/validate-ddf";

const { success, errors } = await validate("/path/to/dataset", {
  onlyErrors: false,
  generateDP: false,
  onProgress: (msg) => console.log(msg),
  // e.g. "checking format: 12/245"
  // e.g. "validating datapoints: 7/30 indicator groups"
});

how to build

  1. install purescript and spago
    • npm install -g purescript
    • npm install -g spago
  2. install dependencies in the project folder
    • npm install
    • spago install
  3. run npm run build (which will run spago build)
  4. build and create bundles npm run bundle

run test

  • run npm test

create new version

  1. update version in package.json
  2. update version in src/Main.purs, look for validatorVersion
  3. build the app and module bundles: npm run bundle, see if tests look good npm test
  4. npm publish