-
-
Notifications
You must be signed in to change notification settings - Fork 185
feat: add metascraper-citation #873
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
6 commits
Select commit
Hold shift + click to select a range
45b8cff
feat: add metascraper-citation
Kikobeats 026a1fb
test(citation): add PLOS ONE HTML fixture
Kikobeats 1d3928d
test(citation): snapshot publisher HTML fixtures
Kikobeats 2b60439
test(citation): add five publisher HTML fixtures
Kikobeats d17d74a
feat(citation): read Dublin Core meta tags
Kikobeats 3345449
fix(citation): prefer citation_date over online_date
Kikobeats File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,4 @@ | ||
| # Change Log | ||
|
|
||
| All notable changes to this project will be documented in this file. | ||
| See [Conventional Commits](https://conventionalcommits.org) for commit guidelines. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,59 @@ | ||
| <div align="center"> | ||
| <br> | ||
| <img style="width: 500px; margin:3rem 0 1.5rem;" src="https://metascraper.js.org/static/logo-banner.png" alt="metascraper"> | ||
| <br> | ||
| <br> | ||
| <p align="center"><strong>metascraper-citation</strong>: Get title, author, date, and publisher from Highwire Press <code>citation_*</code> and Dublin Core <code>dc.*</code> / <code>dcterms.*</code> meta tags.</p> | ||
| <p align="center">See our <a href="https://metascraper.js.org" target='_blank' rel='noopener noreferrer'>website</a> for more information.</p> | ||
| <br> | ||
| </div> | ||
|
|
||
| ## Install | ||
|
|
||
| ```bash | ||
| $ npm install metascraper-citation --save | ||
| ``` | ||
|
|
||
| Load it **before** the generic property packages so these values win over Open Graph (e.g. a clean `citation_title` instead of `og:title` with a site suffix). | ||
|
|
||
| ```js | ||
| const metascraper = require('metascraper')([ | ||
| require('metascraper-citation')(), | ||
| require('metascraper-author')(), | ||
| require('metascraper-date')(), | ||
| require('metascraper-publisher')(), | ||
| require('metascraper-title')() | ||
| ]) | ||
| ``` | ||
|
|
||
| ## Supported tags | ||
|
|
||
| The bundle is a no-op unless [`test`](#test) finds at least one Highwire or Dublin Core tag. Highwire selectors run first; Dublin Core fills gaps (and covers repositories that only emit DC). Missing fields fall through to the generic rules. | ||
|
|
||
| Only the four properties below are set. Bibliographic extras (`citation_doi`, `dc.identifier`, ISSN, volume, issue, pages, PDF URL, language, abstract) are ignored. | ||
|
|
||
| | Property | Tags (first match wins) | | ||
| | --- | --- | | ||
| | **title** | `citation_title`, `dc.title`, `dcterms.title` | | ||
| | **author** | `citation_author`, `dc.creator`, `dcterms.creator`, `dc.contributor`, `dcterms.contributor` (first author only) | | ||
| | **date** | `citation_publication_date`, `citation_date`, `citation_online_date`, `dcterms.issued`, `dcterms.created`, `dc.date`, `dcterms.date` | | ||
| | **publisher** | `citation_publisher`, `citation_journal_title`, `dc.publisher`, `dcterms.publisher` | | ||
|
|
||
| Names are matched case-insensitively (`DC.Title` and `dc.title` are the same tag). | ||
|
|
||
| ## API | ||
|
|
||
| ### test | ||
|
|
||
| ```js | ||
| const { test } = require('metascraper-citation') | ||
| ``` | ||
|
|
||
| Returns `true` when the Cheerio document has at least one `meta[name]` starting with `citation_`, `dc.`, or `dcterms.` (case-insensitive). Used as `rules.test` so the bundle skips pages with no scholarly markup. | ||
|
coderabbitai[bot] marked this conversation as resolved.
|
||
|
|
||
| ## License | ||
|
|
||
| **metascraper-citation** © [Microlink](https://microlink.io), released under the [MIT](https://github.com/microlinkhq/metascraper/blob/master/LICENSE.md) License.<br> | ||
| Authored and maintained by [Microlink](https://microlink.io) with help from [contributors](https://github.com/microlinkhq/metascraper/contributors). | ||
|
|
||
| > [microlink.io](https://microlink.io) · GitHub [microlinkhq](https://github.com/microlinkhq) · X [@microlinkhq](https://x.com/microlinkhq) | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,50 @@ | ||
| { | ||
| "name": "metascraper-citation", | ||
| "description": "Metascraper rules for Highwire Press citation_* and Dublin Core dc.* / dcterms.* meta tags.", | ||
| "homepage": "https://github.com/microlinkhq/metascraper/packages/metascraper-citation", | ||
|
Kikobeats marked this conversation as resolved.
|
||
| "version": "5.55.2", | ||
| "types": "src/index.d.ts", | ||
| "main": "src/index.js", | ||
| "author": { | ||
| "email": "hello@microlink.io", | ||
| "name": "microlink.io", | ||
| "url": "https://microlink.io" | ||
| }, | ||
| "repository": { | ||
| "directory": "packages/metascraper-citation", | ||
| "type": "git", | ||
| "url": "git+https://github.com/microlinkhq/metascraper.git" | ||
| }, | ||
| "bugs": { | ||
| "url": "https://github.com/microlinkhq/metascraper/issues" | ||
| }, | ||
| "keywords": [ | ||
| "citation", | ||
| "dublin-core", | ||
| "extract", | ||
| "google-scholar", | ||
| "highwire", | ||
| "html", | ||
| "metadata", | ||
| "metascraper", | ||
| "scraper", | ||
| "scholar" | ||
| ], | ||
| "dependencies": { | ||
| "@metascraper/helpers": "workspace:*" | ||
| }, | ||
| "devDependencies": { | ||
| "ava": "8", | ||
| "cheerio": "latest" | ||
| }, | ||
| "engines": { | ||
| "node": ">= 22" | ||
| }, | ||
| "files": [ | ||
| "src" | ||
| ], | ||
| "scripts": { | ||
| "test": "NODE_PATH=.. TZ=UTC ava --timeout 15s" | ||
| }, | ||
| "license": "MIT" | ||
| } | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,11 @@ | ||
| declare function rules(): import('metascraper').Rules; | ||
|
|
||
| declare namespace rules { | ||
| /** | ||
| * `true` when the document has at least one Highwire (`citation_*`) | ||
| * or Dublin Core (`dc.*` / `dcterms.*`) meta tag. | ||
| */ | ||
| function test(htmlDom: import('cheerio').CheerioAPI): boolean; | ||
| } | ||
|
|
||
| export = rules; |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,64 @@ | ||
| 'use strict' | ||
|
|
||
| const { | ||
| author, | ||
| date, | ||
| memoizeOne, | ||
| publisher, | ||
| title, | ||
| toRule | ||
| } = require('@metascraper/helpers') | ||
|
|
||
| const toAuthor = toRule(author) | ||
| const toDate = toRule(date) | ||
| const toPublisher = toRule(publisher) | ||
| const toTitle = toRule(title) | ||
|
|
||
| const meta = name => $ => $(`meta[name="${name}" i]`).attr('content') | ||
|
|
||
| const MARKUP = | ||
| 'meta[name^="citation_" i], meta[name^="dc." i], meta[name^="dcterms." i]' | ||
|
|
||
| const test = memoizeOne( | ||
| $ => $(MARKUP).length > 0, | ||
| memoizeOne.EqualityFirstArgument | ||
| ) | ||
|
|
||
| module.exports = () => { | ||
| const rules = { | ||
| title: [ | ||
| toTitle(meta('citation_title')), | ||
| toTitle(meta('dc.title')), | ||
| toTitle(meta('dcterms.title')) | ||
| ], | ||
| author: [ | ||
| toAuthor(meta('citation_author')), | ||
| toAuthor(meta('dc.creator')), | ||
| toAuthor(meta('dcterms.creator')), | ||
| toAuthor(meta('dc.contributor')), | ||
| toAuthor(meta('dcterms.contributor')) | ||
| ], | ||
| date: [ | ||
| toDate(meta('citation_publication_date')), | ||
| toDate(meta('citation_date')), | ||
| toDate(meta('citation_online_date')), | ||
| toDate(meta('dcterms.issued')), | ||
| toDate(meta('dcterms.created')), | ||
| toDate(meta('dc.date')), | ||
| toDate(meta('dcterms.date')) | ||
| ], | ||
| publisher: [ | ||
| toPublisher(meta('citation_publisher')), | ||
| toPublisher(meta('citation_journal_title')), | ||
| toPublisher(meta('dc.publisher')), | ||
| toPublisher(meta('dcterms.publisher')) | ||
| ] | ||
| } | ||
|
|
||
| rules.test = ({ htmlDom }) => test(htmlDom) | ||
| rules.pkgName = 'metascraper-citation' | ||
|
|
||
| return rules | ||
| } | ||
|
|
||
| module.exports.test = test |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.