Skip to content

About

No description, website, or topics provided.

Resources

Stars

4 stars

Watchers

5 watching

Forks

Repository files navigation

scrapy-feedexporter-google-drive

PyPI Version Supported Python Versions License

scrapy-feedexporter-google-drive is a Scrapy Feed Exporter Storage Backend that allows you to export Scrapy items to Google Drive.

Requirements

  • Python 3.9+

Installation

pip install scrapy-feedexporter-google-drive

Usage

Add this storage backend to the FEED_STORAGES Scrapy setting:

FEED_STORAGES = {
    "gdrive": "scrapy_gdrive_exporter.gdrive_exporter.GoogleDriveFeedStorage"
}

Configure authentication with a service account, setting GDRIVE_SERVICE_ACCOUNT_CREDENTIALS_JSON to the contents of its JSON credentials file:

GDRIVE_SERVICE_ACCOUNT_CREDENTIALS_JSON = '{"type": "service_account", "project_id": "…", "private_key_id": "…", "private_key": "…", "client_email": "…", "client_id": "…", "auth_uri": "…", "token_uri": "…", "auth_provider_x509_cert_url": "…", "client_x509_cert_url": "…"}'

Give the service account access to the folder where you want to export the file, by sharing that folder with the service account's email (client_email in the credentials).

Configure in the FEEDS Scrapy setting the Google Drive URI where the feed needs to be exported:

FEEDS = {
    "gdrive://drive.google.com/<folder_id>/<file_name.extension>": {
        "format": "json",
    },
}

You can get the folder_id from the address bar while inside the folder: https://drive.google.com/drive/folders/<folder_id>.

Feed options

The overwrite feed option is not supported: every export is stored as a new file, and existing files are not overwritten. If overwrite is set to True or False, a warning is logged.

About

No description, website, or topics provided.

Resources

Stars

4 stars

Watchers

5 watching

Forks

Releases

Packages

Used by

Contributors

Languages