Skip to content

Optimize processing of sitemap with Tika #5206

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
stweil wants to merge 1 commit into
base: dev
Choose a base branch
from
Open

Optimize processing of sitemap with Tika #5206

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
62 changes: 50 additions & 12 deletions module/VuFind/src/VuFind/XSLT/Import/VuFindSitemap.php
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -99,6 +99,44 @@ protected static function getApertureFields($htmlFile)
];
}

/**
* Load JSON data about an HTML document using Tika.
*
* @param string $url URL or local file containing HTML.
*
* @return array
*/
protected static function getTikaData($url)
{
// Extract and decode the full text from the XML:
$json = json_decode(static::harvestWithTika($url, '--jsonRecursive --text'), true);
$doc = $json[0];

$title = $doc['dc:title'] ?? $doc['title'] ?? '';
$description = $doc['dc:description'] ?? $doc['description'] ?? '';
$fulltext = trim($title . ' ' . ($doc['X-TIKA:content'] ?? ''));
$keywords = [];
if (!empty($doc['keywords'])) {
// keywords may come back as a string or an array
$raw = is_array($doc['keywords'])
? $doc['keywords']
: [$doc['keywords']];
foreach ($raw as $current) {
$keywords[] = html_entity_decode($current, ENT_QUOTES, 'UTF-8');
}
}

//print("doc = "); var_dump($doc);

// Send back the extracted fields:
return [
'title' => $title,
'keywords' => $keywords,
'description' => $description,
'fulltext' => $title . ' ' . $fulltext,
];
}

/**
* Load metadata about an HTML document using Tika.
*
Expand Down Expand Up @@ -247,29 +285,29 @@ protected static function getDocumentFieldArray($url)
return [];
}

// Grab the HTML and write it to disk:
$htmlFile = tempnam('/tmp', 'htm');
$html = file_get_contents($url);
file_put_contents($htmlFile, $html);

// Use the appropriate full text parser:
switch ($parser) {
case 'Aperture':
// Grab the HTML and write it to disk:
$htmlFile = tempnam('/tmp', 'htm');
$html = file_get_contents($url);
file_put_contents($htmlFile, $html);

$fields = static::getApertureFields($htmlFile);

// Add data loaded directly from HTML:
$fields += static::getHtmlFields($html);

// Clean up HTML file:
@unlink($htmlFile);
break;
case 'Tika':
$fields = static::getTikaFields($htmlFile);
$fields = static::getTikaData($url);
Copy link
Copy Markdown
Contributor Author

@stweil stweil Apr 12, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here getTikaFields is no longer called. Do we need it? I think its functionality could be better implemented in getTikaData which has all relevant information.

break;
default:
throw new \Exception('Unexpected parser: ' . $parser);
}

// Clean up HTML file:
@unlink($htmlFile);

// Add data loaded directly from HTML:
$fields += static::getHtmlFields($html);

// Clean up/normalize full text:
$fields['fulltext'] = trim(
preg_replace(
Expand Down
Loading