Skip to content

1240 uploader look into processing h5ad in chunks#1291

Draft
adkinsrs wants to merge 5 commits into
develfrom
1240-uploader---look-into-processing-h5ad-in-chunks
Draft

1240 uploader look into processing h5ad in chunks#1291
adkinsrs wants to merge 5 commits into
develfrom
1240-uploader---look-into-processing-h5ad-in-chunks

Conversation

@adkinsrs

@adkinsrs adkinsrs commented Jul 1, 2026

Copy link
Copy Markdown
Member

(Still a WIP... need to test the rabbitMQ consumer)

This pull request removes legacy Python code related to dataset uploading and AnnData file handling, and adds documentation and service configuration for a new AnnData upload consumer. The changes help to streamline the codebase, reduce redundancy, and introduce a new, containerized RabbitMQ consumer for AnnData uploads.

Removal of legacy dataset upload code:

  • Deleted lib/gear/dataarchive.py, which previously handled detection, parsing, and writing of various dataset formats (MEX, 3tab) into AnnData objects.
  • Deleted lib/gear/datasetuploader.py, which included the dataset uploader factory and logic for determining file types and handling tarball contents.
  • Deleted lib/gear/exceluploader.py, which implemented Excel file parsing and conversion to AnnData, including validation and statistics/coloring calculations.

Introduction of new AnnData upload consumer:

  • Added a new anndata_upload_consumer service to the Docker Compose template (docker/docker-compose.yml.template), which builds from a dedicated Dockerfile and is responsible for handling AnnData and related file uploads via RabbitMQ.

Documentation updates:

  • Updated the RabbitMQ consumers documentation to describe the new AnnData upload consumer, including its listener script, queue name, and systemd service details.

@adkinsrs adkinsrs linked an issue Jul 1, 2026 that may be closed by this pull request
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Uploader - Look into processing H5AD in chunks

1 participant