Skip to content

google_cloud_storage.md

Latest commit

 

History

History
97 lines (62 loc) · 5.19 KB

google_cloud_storage.md

File metadata and controls

97 lines (62 loc) · 5.19 KB
title Google Cloud Storage Destination
disable_toc false
products
name icon url
Logs
logs
/observability_pipelines/configuration/?tab=logs#pipeline-types

{{< product-availability >}}

Overview

For Worker versions 2.7 and later, the Google Cloud destination supports uniform bucket-level access. Google recommends using uniform bucket-level access.
For Worker version older than 2.7, only Access Control Lists is supported.

Use the Google Cloud Storage destination to send your logs to a Google Cloud Storage bucket. If you want to send logs to Google Cloud Storage for archiving and rehydration, you must configure Log Archives. If you do not want to rehydrate logs in Datadog, skip to Set up the destination for your pipeline.

The Observability Pipelines Worker uses standard Google authentication methods. See Authentication methods at Google for more information about choosing the authentication method for your use case.

Configure Log Archives

This step is only required if you want to send logs to Google Cloud Storage for archiving and rehydration, and you don't already have a Datadog Log Archive configured for Observability Pipelines. If you already have a Datadog Log Archive configured or do not want to rehydrate your logs in Datadog, skip to Set up the destination for your pipeline.

If you already have a Datadog Log Archive configured for Observability Pipelines, skip to Set up the destination for your pipeline.

You need to have Datadog's Google Cloud Platform integration installed to set up Datadog Log Archives.

{{% observability_pipelines/configure_log_archive/google_cloud_storage/instructions %}}

Set up the destination for your pipeline {#set-up-the-destinations}

Set up the Google Cloud Storage destination and its environment variables when you set up an Archive Logs pipeline. The information below is configured in the pipelines UI.

  1. Enter the name of your Google Cloud storage bucket. If you configured Log Archives, it's the bucket you created earlier.
  2. If you have a credentials JSON file, enter the path to your credentials JSON file. If you configured Log Archives it's the credentials you downloaded earlier. The credentials file must be placed under DD_OP_DATA_DIR/config. Alternatively, you can use the GOOGLE_APPLICATION_CREDENTIALS environment variable to provide the credential path.
  3. Select the storage class for the created objects.
  4. Select the access level of the created objects.

Optional settings

Prefix to apply to all key objects

Enter a prefix that you want to apply to all key objects.

  • Prefixes are useful for partitioning objects. For example, you can use a prefix as an object key to store objects under a particular directory. If using a prefix for this purpose, it must end in / to act as a directory path; a trailing / is not automatically added.
  • See template syntax if you want to route logs to different object keys based on specific fields in your logs.
    • Note: Datadog recommends that you start your prefixes with the directory name and without a lead slash (/). For example, app-logs/ or service-logs/.
Metadata
  1. Click Add Header to add metadata.
  2. Enter values for the header name and value.
Buffering

{{% observability_pipelines/destination_buffer %}}

Set secrets

{{% observability_pipelines/set_secrets_intro %}}

{{< tabs >}} {{% tab "Secrets Management" %}}

There are no secret identifiers to configure.

{{% /tab %}}

{{% tab "Environment Variables" %}}

{{% observability_pipelines/configure_existing_pipelines/destination_env_vars/datadog_archives_google_cloud_storage %}}

{{% /tab %}} {{< /tabs >}}

How the destination works

Event batching

A batch of events is flushed when one of these parameters is met. See event batching for more information.

Maximum Events Maximum Size (MB) Timeout (seconds)
None 100 900