Add GCP Compute Engine backend

docs/source/compute_backends.rst

@@ -44,3 +44,4 @@ Compute Backends
    compute_config/ibm_vpc.md
    compute_config/aws_ec2.md
    compute_config/azure_vms.md
+   compute_config/gcp_compute_engie.md

docs/source/compute_config/aws_ec2.md

@@ -130,6 +130,8 @@ In summary, you can use one of the following settings:
 |aws_ec2 | soft_dismantle_timeout | 300 |no| Time in seconds to stop the VM instance after a job **completed** its execution |
 |aws_ec2 | hard_dismantle_timeout | 3600 | no | Time in seconds to stop the VM instance after a job **started** its execution |
 |aws_ec2 | exec_mode | reuse | no | One of: **consume**, **create** or **reuse**. If set to  **create**, Lithops will automatically create new VMs for each map() call based on the number of elements in iterdata. If set to **reuse** will try to reuse running workers if exist |
+|aws_ec2 | extra_apt_packages | [] | no | Extra Debian/Ubuntu packages on master/worker VMs during setup (YAML list or space-separated string) |
+|aws_ec2 | extra_python_packages | [] | no | Extra pip packages on master/worker VMs after Lithops (YAML list or space-separated string) |
 
 
 ## Additional configuration

docs/source/compute_config/azure_vms.md

@@ -97,6 +97,8 @@ Edit your lithops config and add the relevant keys:
 |azure_vms | soft_dismantle_timeout | 300 |no| Time in seconds to stop the VM instance after a job **completed** its execution |
 |azure_vms | hard_dismantle_timeout | 3600 | no | Time in seconds to stop the VM instance after a job **started** its execution |
 |azure_vms | exec_mode | reuse | no | One of: **consume**, **create** or **reuse**. If set to **create**, Lithops will automatically create new VMs for each `map()` call based on the number of elements in `iterdata`. If set to **reuse**, Lithops will try to reuse running workers if they exist |
+|azure_vms | extra_apt_packages | [] | no | Extra Debian/Ubuntu packages on master/worker VMs during setup (YAML list or space-separated string) |
+|azure_vms | extra_python_packages | [] | no | Extra pip packages on master/worker VMs after Lithops (YAML list or space-separated string) |
 
 
 ## Consume mode

docs/source/compute_config/gcp_compute_engie.md

@@ -0,0 +1,210 @@
+# Google Compute Engine (GCE)
+
+The GCP Compute Engine backend of Lithops can provide a serverless user experience on top of GCE where Lithops creates new Virtual Machines (VMs) dynamically at runtime and scales Lithops jobs against them (create and reuse modes). Alternatively Lithops can start and stop an existing VM instance (consume mode).
+
+The backend key is `gcp_compute_engie` (matches the Lithops module name).
+
+## Choose an operating system image for the VM
+
+Any VM needs an operating system image. By default Lithops uses Ubuntu 24.04 (`ubuntu-2404-lts-amd64`). Lithops installs required dependencies on the VM on first use (this can take a few minutes).
+
+To list available images:
+
+```bash
+lithops image list -b gcp_compute_engie
+```
+
+Use the **Image ID** column as `source_image` in your config.
+
+## Installation
+
+1. Install GCP backend dependencies:
+
+```bash
+python3 -m pip install lithops[gcp]
+```
+
+## Create and reuse modes
+
+In the `create` mode, Lithops automatically creates new worker VM instances at runtime, runs the job on them, and deletes the workers when the job completes (unless configured otherwise).
+
+In the `reuse` mode, Lithops keeps the master and worker VMs stopped between jobs and starts them again when needed. It reuses running workers when possible and only creates new workers if necessary.
+
+### Configuration
+
+1. Enable the Compute Engine API:
+
+```bash
+gcloud services enable compute.googleapis.com --project <PROJECT_ID>
+```
+
+2. Create a service account (or use an existing one) and grant these roles on the project:
+
+```bash
+gcloud projects add-iam-policy-binding <PROJECT_ID> \
+  --member="serviceAccount:<SERVICE_ACCOUNT_EMAIL>" \
+  --role="roles/compute.admin"
+
+gcloud projects add-iam-policy-binding <PROJECT_ID> \
+  --member="serviceAccount:<SERVICE_ACCOUNT_EMAIL>" \
+  --role="roles/storage.objectAdmin"
+```
+
+3. Set the Lithops config file:
+
+```yaml
+lithops:
+  backend: gcp_compute_engie
+
+gcp:
+  credentials_path: <FULL_PATH_TO_SERVICE_ACCOUNT_JSON>
+
+gcp_compute_engie:
+  project_name: <GCP_PROJECT_ID>
+  zone: <ZONE>
+  exec_mode: reuse
+```
+
+Lithops attaches the service account from `credentials_path` to master and worker VMs so they can access GCS via the metadata service. You can set `service_account: <EMAIL>` explicitly if needed.
+
+### Summary of configuration keys for GCP
+
+|Group|Key|Default|Mandatory|Additional info|
+|---|---|---|---|---|
+|gcp | credentials_path | |no | Service account JSON used by Lithops on your machine and to select the VM service account |
+|gcp | region | |no | GCP region. Derived from `zone` if omitted |
+
+### GCE - Create and Reuse Modes
+
+|Group|Key|Default|Mandatory|Additional info|
+|---|---|---|---|---|
+|gcp_compute_engie | project_name | |yes | GCP project ID |
+|gcp_compute_engie | zone | |yes | Compute Engine zone, for example `us-east1-b` |
+|gcp_compute_engie | region | derived from zone |no | Region used for subnet and NAT |
+|gcp_compute_engie | service_account | |no | Service account email attached to VMs. Default: `client_email` from `credentials_path` |
+|gcp_compute_engie | network_name | |no | Existing VPC name. If not provided, Lithops creates a new network |
+|gcp_compute_engie | subnet_name | |no | Existing subnet name when using a custom VPC |
+|gcp_compute_engie | source_image | ubuntu-2404-lts-amd64 |no | Boot image reference |
+|gcp_compute_engie | master_instance_type | e2-small |no | Master VM machine type |
+|gcp_compute_engie | worker_instance_type | e2-standard-2 |no | Worker VM machine type |
+|gcp_compute_engie | ssh_username | ubuntu |no | Username to access the VM |
+|gcp_compute_engie | ssh_password | |no | Password for worker VMs. If not provided, it is created randomly |
+|gcp_compute_engie | ssh_key_filename | ~/.ssh/id_rsa |no | SSH private key for the master VM. If not provided, Lithops creates one |
+|gcp_compute_engie | request_spot_instances | False |no | Use Spot VMs for workers |
+|gcp_compute_engie | delete_on_dismantle | True |no | Delete worker VMs when stopped. Master VM is never deleted when stopped |
+|gcp_compute_engie | max_workers | 100 |no | Max number of workers per `FunctionExecutor()` |
+|gcp_compute_engie | worker_processes | AUTO |no | Parallel Lithops processes per worker. Default: CPUs of `worker_instance_type` |
+|gcp_compute_engie | runtime | python3 |no | Runtime name. Default: python3 on the VM |
+|gcp_compute_engie | auto_dismantle | True |no | If False, VMs are not stopped automatically |
+|gcp_compute_engie | soft_dismantle_timeout | 300 |no | Seconds to stop the VM after a job **completed** |
+|gcp_compute_engie | hard_dismantle_timeout | 3600 |no | Seconds to stop the VM after a job **started** |
+|gcp_compute_engie | exec_mode | reuse |no | One of: **consume**, **create** or **reuse** |
+|gcp_compute_engie | extra_apt_packages | [] |no | Extra apt packages on master/worker VMs during setup |
+|gcp_compute_engie | extra_python_packages | [] |no | Extra pip packages on master/worker VMs after Lithops |
+
+## Consume mode
+
+In this mode, Lithops uses an existing VM. The VM must be reachable by SSH and have a service account with GCS access.
+
+### Configuration
+
+```yaml
+lithops:
+  backend: gcp_compute_engie
+
+gcp:
+  credentials_path: <FULL_PATH_TO_SERVICE_ACCOUNT_JSON>
+
+gcp_compute_engie:
+  exec_mode: consume
+  project_name: <GCP_PROJECT_ID>
+  zone: <ZONE>
+  instance_name: <EXISTING_VM_NAME>
+```
+
+### Summary of configuration keys for the consume mode
+
+|Group|Key|Default|Mandatory|Additional info|
+|---|---|---|---|---|
+|gcp_compute_engie | instance_name | |yes | Existing VM instance name |
+|gcp_compute_engie | project_name | |yes | GCP project ID |
+|gcp_compute_engie | zone | |yes | Compute Engine zone |
+|gcp_compute_engie | ssh_username | ubuntu |no | Username to access the VM |
+|gcp_compute_engie | ssh_key_filename | ~/.ssh/id_rsa |no | Path to the SSH private key |
+|gcp_compute_engie | worker_processes | AUTO |no | Parallel Lithops processes per worker |
+
+## Test Lithops
+
+Once you have your compute and storage backends configured, you can run a hello world function with:
+
+```bash
+lithops hello -b gcp_compute_engie -s gcp_storage
+```
+
+## Viewing the execution logs
+
+You can view the function execution logs on your local machine using the Lithops client:
+
+```bash
+lithops logs poll
+```
+
+## VM Management
+
+Lithops for GCE follows a master-worker architecture (1:N).
+
+All VMs, including the master, are automatically stopped after a configurable timeout (see hard/soft dismantle timeouts). Stopped master and worker VMs are started again on the next job in reuse mode.
+
+You can open an SSH session to the master VM with:
+
+```bash
+lithops attach -b gcp_compute_engie
+```
+
+The master and worker VMs store Lithops service logs in `/tmp/lithops-root/*-service.log`.
+
+To list available workers:
+
+```bash
+lithops worker list -b gcp_compute_engie
+```
+
+To list submitted jobs:
+
+```bash
+lithops job list -b gcp_compute_engie
+```
+
+To delete workers only:
+
+```bash
+lithops clean -b gcp_compute_engie -s gcp_storage
+```
+
+To delete workers, the master VM, and Lithops-created network resources:
+
+```bash
+lithops clean -b gcp_compute_engie -s gcp_storage --all
+```
+
+## Architecture diagram
+
+```mermaid
+flowchart TB
+  subgraph gcp [GCP project / region]
+    NET["VPC lithops-net-XXXXXX"]
+    SUB["Subnet lithops-net-XXXXXX-subnet"]
+    FW1["Firewall SSH :22 from internet"]
+    FW2["Firewall internal :8080/8081/6379/22"]
+    NAT["Cloud Router + NAT lithops-net-XXXXXX-router"]
+    M["Master VM lithops-master-XXXXXX\n+ ephemeral external IP"]
+    W["Worker VMs lithops-worker-*\nprivate IP only"]
+  end
+  LAPTOP["Your laptop"] -->|SSH :22| M
+  M -->|private network| W
+  W -->|egress| NAT
+  NAT --> INTERNET[Internet apt/docker/pip]
+  M --> SUB
+  W --> SUB
+  SUB --> NET
+```

docs/source/compute_config/ibm_vpc.md

@@ -109,6 +109,8 @@ ibm_vpc:
 |ibm_vpc | exec_mode | reuse | no | One of: **consume**, **create** or **reuse**. If set to **create**, Lithops will automatically create new VMs for each `map()` call based on the number of elements in `iterdata`. If set to **reuse**, Lithops will try to reuse running workers if they exist |
 |ibm_vpc | singlesocket | False | no | Try to allocate workers with a single-socket CPU. If they end up running on multiple sockets, a warning message is printed to the user. If **True**, the standalone **workers_policy** must be set to **strict** to track worker states |
 |ibm_vpc | gpu | False | no | If `True`, Docker is started with GPU support. Requires the host to have the necessary hardware and software pre-configured, and a Docker image runtime with GPU support specified |
+|ibm_vpc | extra_apt_packages | [] | no | Extra Debian/Ubuntu packages on master/worker VMs during setup (YAML list or space-separated string) |
+|ibm_vpc | extra_python_packages | [] | no | Extra pip packages on master/worker VMs after Lithops (YAML list or space-separated string) |
 
 ## Consume mode
 

docs/source/compute_config/vm.md

@@ -54,6 +54,8 @@ In this backend, you can use any docker image that contains all the required dep
 |vm | ssh_key_filename | | no | Path to SSH key |
 |vm | runtime |  python3  |no | `python3` or a docker image name |
 |vm | worker_processes | 1 | no | Number of Lithops processes within the VM. This can be used to parallelize function activations within the VM. It is recommended to set it to the same number of CPUs as the VM |
+|vm | extra_apt_packages | [] | no | Extra Debian/Ubuntu packages during Lithops setup on the VM (YAML list or space-separated string) |
+|vm | extra_python_packages | [] | no | Extra pip packages after Lithops on the VM (YAML list or space-separated string) |
 
 ## Test Lithops
 

docs/source/configuration.rst

@@ -37,6 +37,7 @@ Choose your compute and storage engines from the table below:
 || `IBM Virtual Private Cloud <compute_config/ibm_vpc.html>`_        ||                                                                   |
 || `AWS Elastic Compute Cloud (EC2) <compute_config/aws_ec2.html>`_  ||                                                                   |
 || `Azure Virtual Machines <compute_config/azure_vms.html>`_         ||                                                                   |
+|| `Google Compute Engine <compute_config/gcp_compute_engie.html>`_  ||                                                                   |
 +--------------------------------------------------------------------+--------------------------------------------------------------------+
 
 Configuration File

docs/source/execution_modes.rst

@@ -83,4 +83,4 @@ underlying infrastructure.
 
     fexec = lithops.StandaloneExecutor()
 
-- Available backends: `IBM Virtual Private Cloud <compute_config/ibm_vpc.html>`_, `AWS Elastic Compute Cloud (EC2) <compute_config/aws_ec2.html>`_, `Azure Virtual Machines <compute_config/azure_vms.html>`_, `Virtual Machine <compute_config/vm.html>`_
+- Available backends: `IBM Virtual Private Cloud <compute_config/ibm_vpc.html>`_, `AWS Elastic Compute Cloud (EC2) <compute_config/aws_ec2.html>`_, `Azure Virtual Machines <compute_config/azure_vms.html>`_, `Google Compute Engine <compute_config/gcp_compute_engie.html>`_, `Virtual Machine <compute_config/vm.html>`_

docs/source/supported_clouds.rst

@@ -52,6 +52,8 @@ Currently, Lithops for Google Cloud Platform supports these backends:
      - `Google Cloud Storage <https://cloud.google.com/storage/docs>`_
    * - `Google Cloud Run <https://cloud.google.com/run/docs>`_
      -
+   * - `Google Compute Engine (GCE) <https://cloud.google.com/compute/docs>`_
+     -
 
 Microsoft Azure
 ---------------

lithops/config.py

@@ -256,6 +256,7 @@ def extract_standalone_config(config):
     sa_config = config[c.STANDALONE].copy()
     backend = config['lithops']['backend']
     sa_config['backend'] = backend
+    sa_config['storage'] = config['lithops'].get('storage')
     sa_config[backend] = config[backend] if backend in config and config[backend] else {}
     sa_config[backend]['user_agent'] = f'lithops/{__version__}'
 

lithops/constants.py

@@ -93,7 +93,9 @@
     'start_timeout': 300,
     'auto_dismantle': True,
     'soft_dismantle_timeout': 300,
-    'hard_dismantle_timeout': 3600
+    'hard_dismantle_timeout': 3600,
+    'extra_apt_packages': [],
+    'extra_python_packages': [],
 }
 
 SERVERLESS_BACKENDS = [
@@ -118,5 +120,6 @@
     'ibm_vpc',
     'aws_ec2',
     'azure_vms',
+    'gcp_compute_engie',
     'vm'
 ]

lithops/scripts/cli.py

@@ -907,7 +907,7 @@ def list_images(config, backend, region, debug):
     compute_config = extract_standalone_config(config)
     compute_handler = StandaloneHandler(compute_config)
 
-    logger.info('Listing all Ubuntu Linux 22.04 VM Images')
+    logger.info('Listing all Ubuntu VM images')
     vm_images = compute_handler.list_images()
 
     headers = ['Image Name', 'Image ID', 'Creation Date']

lithops/standalone/backends/aws_ec2/aws_ec2.py

@@ -27,7 +27,7 @@
 from concurrent.futures import ThreadPoolExecutor
 
 from lithops.version import __version__
-from lithops.util.ssh_client import SSHClient
+from lithops.util.ssh_client import SSHClient, ssh_boot_status_message
 from lithops.constants import COMPUTE_CLI_MSG, CACHE_DIR
 from lithops.config import load_yaml_config, dump_yaml_config
 from lithops.standalone.utils import CLOUD_CONFIG_WORKER, CLOUD_CONFIG_WORKER_PK, StandaloneMode, get_host_setup_script
@@ -647,7 +647,7 @@ def build_image(self, image_name, script_file, overwrite, include, extra_args=[]
 
         logger.debug(f"Uploading installation script to {build_vm}")
         remote_script = "/tmp/install_lithops.sh"
-        script = get_host_setup_script()
+        script = get_host_setup_script(lithops_pip_spec='lithops[aws,redis]')
         build_vm.get_ssh_client().upload_data_to_file(script, remote_script)
         logger.debug("Executing Lithops installation script. Be patient, this process can take up to 3 minutes")
         build_vm.get_ssh_client().run_remote_command(f"chmod 777 {remote_script}; sudo {remote_script}; rm {remote_script};")
@@ -1099,14 +1099,13 @@ def is_ready(self):
         """
         Checks if the VM instance is ready to receive ssh connections
         """
-        login_type = 'password' if 'password' in self.ssh_credentials and \
-            not self.public else 'publickey'
         try:
             self.get_ssh_client().run_remote_command('id')
         except LithopsValidationError as err:
             raise err
         except Exception as err:
-            logger.debug(f'SSH to {self.public_ip if self.public else self.private_ip} failed ({login_type}): {err}')
+            ip = self.public_ip if self.public else self.private_ip
+            logger.debug(f'SSH to {ip}: {ssh_boot_status_message(err)}')
             self.del_ssh_client()
             return False
         return True

lithops/standalone/backends/azure_vms/azure_vms.py

@@ -27,7 +27,7 @@
 from azure.core.exceptions import ResourceNotFoundError
 
 from lithops.version import __version__
-from lithops.util.ssh_client import SSHClient
+from lithops.util.ssh_client import SSHClient, ssh_boot_status_message
 from lithops.constants import COMPUTE_CLI_MSG, CACHE_DIR, SA_CONFIG_FILE
 from lithops.config import load_yaml_config, dump_yaml_config
 from lithops.standalone.utils import StandaloneMode
@@ -752,14 +752,13 @@ def is_ready(self):
         """
         Checks if the VM instance is ready to receive ssh connections
         """
-        login_type = 'password' if 'password' in self.ssh_credentials and \
-            not self.public else 'publickey'
         try:
             self.get_ssh_client().run_remote_command('id')
         except LithopsValidationError as err:
             raise err
         except Exception as err:
-            logger.debug(f'SSH to {self.public_ip if self.public else self.private_ip} failed ({login_type}): {err}')
+            ip = self.public_ip if self.public else self.private_ip
+            logger.debug(f'SSH to {ip}: {ssh_boot_status_message(err)}')
             self.del_ssh_client()
             return False
         return True

lithops/standalone/backends/azure_vms/config.py

@@ -25,7 +25,7 @@
     'ssh_username': 'ubuntu',
     'ssh_password': str(uuid.uuid4()),
     'request_spot_instances': True,
-    'delete_on_dismantle': False,
+    'delete_on_dismantle': True,
     'max_workers': 100,
     'worker_processes': 'AUTO'
 }

lithops/standalone/backends/gcp_compute_engie/__init__.py

@@ -0,0 +1,3 @@
+from .gcp_compute_engie import GCPComputeEngieBackend as StandaloneBackend
+
+__all__ = ['StandaloneBackend']