1

I'm cloning multiple repositories (using git clone) inside my Google Cloud Run job and one of them makes Google Cloud killing the container with exit 1. See the log:

    {
      insertId: "xxx",
      labels: {
        instanceId: "xxx"
        run.googleapis.com/execution_name: "xxx",
        run.googleapis.com/task_attempt: "0",
        run.googleapis.com/task_index: "0"
      },
      logName: "xxx/run.googleapis.com%2Fstderr",
      receiveTimestamp: "2024-03-13T09:05:28.686787232Z",
      resource: {
        labels: {
          job_name: "xxx",
          location: "xxx",
          project_id: "xxx"
        },
        type: "cloud_run_job"
      },
      textPayload: "Killed",
      timestamp: "2024-03-13T09:05:28.685133Z"
    }

The repository as cloned has 10.828 files and its size is 1.8 GB. The memory and CPU usage didn't exceed 50 %. It happens during git clone [email protected]:usr/bad_repo.git.

I assume it's related to a limit but I didn't find anything.

Steps to reproduce:

  1. Create a Docker container with a Bash script that clones a repository like this:
git clone -q --depth 1 --no-tags --filter=blob:limit=100k [email protected]:supabase/supabase.git
  1. Run this Docker container as Cloud Run job
  2. The container will exit with status code 1 triggered by Cloud Run

All logs:

enter image description here

Improve this question
4
  • Can you share minimal replicating steps and additional logs, if found?
    – Roopa M
    Commented Mar 14 at 10:47
  • @RoopaM I added steps to reproduce and all logs
    – Julian
    Commented Mar 14 at 18:34
  • 1
    I tried with this Dockerfile and script.sh following this document and got same issue. After increasing memory and CPU, its working fine. I can also able to successfully run with alphine image with 2gb memory and 1 CPU. Can you try to increase memory and CPU? Also share your Dockerfile.
    – Roopa M
    Commented Mar 15 at 15:14
  • Increasing the RAM helped. Makes sense now, the repository is 1.8 GB, RAM was at 2.0 GB. Likely, it didn't fit
    – Julian
    Commented Mar 15 at 17:31

2 Answers 2

Reset to default
3

As proposed by Roopa M in the comments: Increasing the RAM helped, even though there is no spike in usage.

Improve this answer
1

Yes, increasing RAM will help but when you see:

textPayload: "Killed",

it's good to check K8s events with kubectl get events, and you should see something like:

20m Warning OOMKilling node/gke-pool-3-fc1704-8fd05719-gohy Memory cgroup out of memory: Killed process 707119 (service) total-vm:6709308kB, anon-rss:4181156kB, file-rss:99392kB, shmem-rss:0kB, UID:0 pgtables:8732kB oom_score_adj:969

For me, it always meant it was a huge but short memory spike, so I couldn't see it in the metrics, and then the only whey is to check events.

Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .