scrapy-feedexporter-google-drive is a Scrapy Feed Exporter Storage Backend that allows you to export Scrapy items to Google Drive.
- Python 3.9+
pip install scrapy-feedexporter-google-driveAdd this storage backend to the FEED_STORAGES Scrapy setting:
FEED_STORAGES = {
"gdrive": "scrapy_gdrive_exporter.gdrive_exporter.GoogleDriveFeedStorage"
}Configure authentication
with a service account, setting GDRIVE_SERVICE_ACCOUNT_CREDENTIALS_JSON to
the contents of its JSON credentials file:
GDRIVE_SERVICE_ACCOUNT_CREDENTIALS_JSON = '{"type": "service_account", "project_id": "…", "private_key_id": "…", "private_key": "…", "client_email": "…", "client_id": "…", "auth_uri": "…", "token_uri": "…", "auth_provider_x509_cert_url": "…", "client_x509_cert_url": "…"}'Give the service account access to the folder where you want to export the
file, by sharing that folder with the service account's email (client_email
in the credentials).
Configure in the FEEDS Scrapy setting the Google Drive URI where the feed needs to be exported:
FEEDS = {
"gdrive://drive.google.com/<folder_id>/<file_name.extension>": {
"format": "json",
},
}You can get the folder_id from the address bar while inside the folder:
https://drive.google.com/drive/folders/<folder_id>.
The overwrite feed option is
not supported: every export is stored as a new file, and existing files are not
overwritten. If overwrite is set to True or False, a warning is
logged.