[kube-up][scale-up][mizar]: event-exporter CrashLoopBackOff with error: Failed to initialize sink

[sonyafenge]

What happened:
start scale-up with mizar using kube-up.sh, event-exporter crashloopbackoff with error:

{"log":"F0224 00:29:21.335208       1 main.go:75] Failed to initialize sink: failed to build sink config: not running on GCE, which is not supported for Stackdriver sink\n","stream":"stderr","time":"2022-02-24T00:29:21.33556028Z"}

$ kubectl get pods -AT | grep event-exporter
system   kube-system   event-exporter-v0.2.5-868dff6494-h4hw4               4772125109102601041   0/1     CrashLoopBackOff   26         118m

What you expected to happen:
event-exporter start successfully
How to reproduce it (as minimally and precisely as possible):

$ export KUBEMARK_NUM_NODES=100 NUM_NODES=2 SCALEOUT_CLUSTER=false SCALEOUT_TP_COUNT=1 SCALEOUT_RP_COUNT=1 RUN_PREFIX=sonyaperf1-022322
$ export MASTER_DISK_SIZE=500GB MASTER_ROOT_DISK_SIZE=500GB KUBE_GCE_ZONE=us-west2-b MASTER_SIZE=n1-highmem-32 NODE_SIZE=n1-highmem-16 NODE_DISK_SIZE=500GB GOPATH=$HOME/go KUBE_GCE_ENABLE_IP_ALIASES=true KUBE_GCE_PRIVATE_CLUSTER=true CREATE_CUSTOM_NETWORK=true KUBE_GCE_INSTANCE_PREFIX=${RUN_PREFIX} KUBE_GCE_NETWORK=${RUN_PREFIX} ENABLE_KCM_LEADER_ELECT=false ENABLE_SCHEDULER_LEADER_ELECT=false ETCD_QUOTA_BACKEND_BYTES=8589934592 SHARE_PARTITIONSERVER=false LOGROTATE_FILES_MAX_COUNT=200 LOGROTATE_MAX_SIZE=200M KUBE_ENABLE_APISERVER_INSECURE_PORT=true KUBE_ENABLE_PROMETHEUS_DEBUG=true KUBE_ENABLE_PPROF_DEBUG=true TEST_CLUSTER_LOG_LEVEL=--v=2 HOLLOW_KUBELET_TEST_LOG_LEVEL=--v=2 GCE_REGION=us-west2-b NETWORK_PROVIDER=mizar
$ ./cluster/kube-up.sh

Anything else we need to know?:

Environment:

• Arktos version (use kubectl version):

https://github.com/CentaurusInfra/arktos/commit/e26207736cff2828906c6f96845c9b677960b7a9

• Cloud provider or hardware configuration:
• OS (e.g: cat /etc/os-release):
• Kernel (e.g. uname -a):
• Install tools:
• Network plugin and version (if this is a network-related bug):
• Others:

[Sindica]

May not be 130 release blocker

[sonyafenge]

Debug this issue and found:

1. ubuntu is missing cloud logging agent, cloud monitor agent. these need be installed to support event-exporter;
2. Event-exporter requests metadata.onGCE to be true. debug to metadata.onGCE and found this is trying to LookupHost(ctx, "metadata.google.internal"), support for internet access in pods is still in progress based on issue [kube-up][scale-out] pod unable to access internet IP address #1373