Uploaded image for project: 'Bitbucket Data Center'
  1. Bitbucket Data Center
  2. BSERV-19503

Viewing pull requests that contain large, binary files in a forking workflow can leave behind tmp_pack files

XMLWordPrintable

      Issue Summary

      Viewing pull requests that contain large, binary files in a forking workflow can leave behind tmp_pack_* files.

      The issue occurs when the backend git fetch command for retrieving the PR's effective diff takes longer than a pre-configured limit of 60 seconds.

       

      This is reproducible on Data Center: yes

      Steps to Reproduce

      1. Use forking workflow
      2. Create a PR from the fork repo to the target repo
      3. Add several commits that include non-LFS tracked large, binary files to the source repo
        Note: there should be enough differences between the source and target branches to make the backend git fetch command called by the getEffectiveDiff method take more than 60 seconds and cause a timeout.
      4. View the pull request's Diff tab

      Expected Results

      The PR Diff tab can be viewed.

      Actual Results

      Viewing the Diff tab returns a "Server Unreachable" error in the web UI.

      This action can leave behind large tmp_pack_* files in the <$BITBUCKET_HOME>/shared/data/repositories/<REPO_ID>/objects/pack directory.

      In severe cases, the tmp_pack_* files can lead to all disk space being consumed and cause an outage.

       

      The following error is written to the atlassian-bitbucket.log file:

      2024-06-17 15:48:24,439 ERROR [http-nio-127.0.0.1-13997-exec-8 url: /projects/PROJECT1/repos/repo1/pull-requests/468/commits/b09c8db02dc08c8356ce24d8a6600ca05035cd71, /mvc/projects/PROJECT1/repos/repo1/pull-requests/468/commits/b09c8db02dc08c8356ce24d8a6600ca05035cd71; user: user1] user1 @1TYRSEUx948x20867902x6 1pvupif 10.195.72.232,127.0.0.1 "GET /projects/PROJECT1/repos/repo1/pull-requests/468/commits/b09c8db02dc08c8356ce24d8a6600ca05035cd71 HTTP/1.1" c.a.s.i.w.HandlerExceptionResolverImpl 'git fetch --negotiation-tip=696292162c592cee7d4622e92ee8475ab9b1743f --negotiation-tip=85093196d772c56cbede3f2b1dfa30ec9aec5e47 --negotiation-tip=85093196d772c56cbede3f2b1dfa30ec9aec5e47 /var/atlassian/application-data/bitbucket/shared/data/repositories/285513 b09c8db02dc08c8356ce24d8a6600ca05035cd71:' timed out; exited with code 15
com.atlassian.bitbucket.scm.CommandTimeoutException: 'git fetch --negotiation-tip=696292162c592cee7d4622e92ee8475ab9b1743f --negotiation-tip=85093196d772c56cbede3f2b1dfa30ec9aec5e47 --negotiation-tip=85093196d772c56cbede3f2b1dfa30ec9aec5e47 /var/atlassian/application-data/bitbucket/shared/data/repositories/285513 b09c8db02dc08c8356ce24d8a6600ca05035cd71:' timed out; exited with code 15

      It shows that the git fetch command has timed out.

      The Mesh sidecar profiler logs show that the git fetch command took longer than 60 seconds:

      2024-06-17 15:48:24,435 | grpc-server:thread-2693 | 6LKQF02Rx947x4383288x4
[61648.2ms] - void com.atlassian.bitbucket.mesh.git.GrpcPullRequestService.getEffectiveDiff(RpcGetPullRequestEffectiveDiffRequest,StreamObserver)
 [61648.2ms] - PullRequestEffectiveDiff com.atlassian.bitbucket.mesh.git.pull.DefaultPullRequestManager.getEffectiveDiff(GetPullRequestEffectiveDiffParameters)
 [61646.6ms] - git fetch --negotiation-tip=696292162c592cee7d4622e92ee8475ab9b1743f --negotiation-tip=85093196d772c56cbede3f2b1dfa30ec9aec5e47 --negotiation-tip=85093196d772c56cbede3f2b1dfa30ec9aec5e47 /var/atlassian/application-data/bitbucket/shared/data/repositories/285513 b09c8db02dc08c8356ce24d8a6600ca05035cd71:

      Workaround

      Possible workarounds

      • Open PR's with smaller changes
      • Change gc.prune.expire.default=24 in <$BITBUCKET_HOME>/mesh/mesh.properties for the Sidecar or in <$MESH_HOME>/mesh.properties for standalone Mesh nodes

       

      Long term recommendations:

      • Use Git LFS for tracking large files.

       

              95e5dd71de92 David Zhu (Inactive)
              jmariano@atlassian.com JP Mariano
              Votes:
              1 Vote for this issue
              Watchers:
              12 Start watching this issue

                Created:
                Updated:
                Resolved: