get rid of pip and migrate to uv

.github/workflows/pre-commit.yml

@@ -13,10 +13,14 @@ jobs:
       uses: actions/setup-python@v6
       with:
         python-version-file: '.python-version'
-        cache: 'pip'
+
+    - name: Install uv
+      uses: astral-sh/setup-uv@v7
+      with:
+        enable-cache: true
 
     - name: Install dependencies
-      run: pip install -r requirements.txt
+      run: uv sync
 
     - name: pre-commit
       uses: pre-commit/action@v3.0.1

CONTRIBUTING.md

@@ -18,7 +18,7 @@ this code. Please report unacceptable behavior to admin@ihr.live
 1. Make sure that your code is formatted and passed linting according to the project
    requirements. This can easily be achieved by installing the `pre-commit` command as
    described below.
-1. Ensure any new dependencies are added to the `requirements.txt` file.
+1. Ensure any new dependencies are added to the `pyproject.toml` file.
 1. Add only relevant files to the commit and ignore the rest to keep the repo clean.
     - If you add a new dataset / crawler, include a README.md describing the crawler
        and the nodes / relationships it will push to the database.
@@ -42,9 +42,9 @@ git fetch upstream
 git merge upstream/main
 
 ## Step 3: Setup Virtual Environment and Install Dependencies
-python3 -m venv --upgrade-deps .venv
+# Requires uv: https://docs.astral.sh/uv/getting-started/installation/
+uv sync
 source .venv/bin/activate
-pip install -r requirements.txt
 
 ## Step 4: Setup pre-commit
 pre-commit install

documentation/crawler-best-practices.md

@@ -28,7 +28,8 @@ If you update the ontology (i.e., add a new node and/or relationship type):
 
 If you add a new Python dependency:
 
-- [ ] Update [requirements.txt](../requirements.txt)
+- [ ] Update [pyproject.toml](../pyproject.toml) and run `uv lock` to update
+  [uv.lock](../uv.lock)
 
 ## Code considerations
 

documentation/database-management.md

@@ -54,12 +54,12 @@ git clone https://github.com/InternetHealthReport/internet-yellow-pages.git
 cd internet-yellow-pages
 ```
 
-Create Python environment and install Python libraries:
+Create the Python environment and install the dependencies. This requires
+[uv](https://docs.astral.sh/uv/getting-started/installation/):
 
 ```bash
-python3 -m venv --upgrade-deps .venv
+uv sync
 source .venv/bin/activate
-pip install -r requirements.txt
 ```
 
 Create a configuration file from the example file and add API keys. **Note that some

pyproject.toml

@@ -0,0 +1,43 @@
+[project]
+name = "internet-yellow-pages"
+version = "4.4.0"
+description = "The Internet Yellow Pages (IYP) is a knowledge database that gathers information about Internet resources."
+readme = "README.md"
+license = "GPL-3.0-only"
+requires-python = ">=3.12"
+dependencies = [
+    "neo4j",
+    "iso3166",
+    "progressbar2",
+    "arrow",
+    "tldextract",
+    "py-radix",
+    "requests",
+    "urllib3",
+    "chardet",
+    "bs4",
+    "requests-cache",
+    "lz4",
+    "frozendict",
+    "docker",
+    "boto3",
+    "botocore",
+    "pandas",
+    "flatdict",
+    "requests_futures",
+    "autopep8",
+    "flake8",
+    "pre-commit",
+    "PyGithub",
+    "clickhouse_driver",
+    "paramiko",
+    "scp",
+    "pyarrow",
+    "ijson",
+]
+
+# This repository is a collection of scripts (crawlers) rather than an installable
+# library, so tell uv not to try to build/install it as a package.
+[tool.uv]
+package = false
+