Issue Summary

In Bitbucket Data Center version 9.2.0, when nodes are started simultaneously, they may generate different encryption keys due to a bug in the file locking mechanism. This results in decryption failures when accessing secrets.

Steps to Reproduce

1. Upgrade to Bitbucket 9.2.0 on a multi-node Data Center setup.
2. Start all nodes simultaneously.

Expected Results

Only one encryption key should be generated and all nodes should use the same key to decrypt secrets from the database.

Actual Results

Some nodes generate a different encryption key, leading to decryption failures. The below exception is thrown in the logs.

com.atlassian.secrets.api.SecretServiceException: This secret cannot be decrypted with the configured encryption key
        at com.atlassian.secrets.service.aes.AESEncryptionBackend.unseal(AESEncryptionBackend.java:121)
        at com.atlassian.secrets.service.DefaultSecretService.get(DefaultSecretService.java:75)
        at com.atlassian.stash.internal.secrets.BitbucketSecretService$1.load(BitbucketSecretService.java:54)
        at com.atlassian.stash.internal.secrets.BitbucketSecretService$1.load(BitbucketSecretService.java:49)

Workaround

Follow these steps to allow Bitbucket DC to generate a new key to be used across all nodes.

• Make a backup of your database and shared home
• Shutdown all nodes in the cluster
• Manually delete the $BITBUCKET_HOME/shared/secrets-config.yaml file (this will allow a new key to be regenerated on startup)
• Delete the files mentioned in Rotate the system GPG key (don't restart yet)
• Delete all rows from the bb_sealed_secret table
• Start up only one node and wait until it has fully started
• Confirm that a new $BITBUCKET_HOME/shared/secrets-config.yaml file has been generated
• Start all remaining nodes

Once the cluster has fully restarted, you may need to manually re-encrypt affected passwords.

• Mail server password - Follow these steps to re-enter the mail server password in the Admin UI
• Search server password
  • If the search password was specified in bitbucket.properties, no action is needed.
  • For bundled search, see this page for instructions on resetting the search password
  • For external search using basic authentication, see this page for setting the password via the Admin UI.
• OAuth 1.0 application links - This is only applicable if there are any existing OAuth 1.0 application links. Delete and recreate any OAuth application links via the UI. See this page for more details on creating application links.