GcsStorageClient throws non-retryable exception of error code 504

[renjiezh]

Describe the bug
When writing blob into GCS, we received an StorageException of message "Unexpected Error code 504 trying to get security access token from Compute Engine metadata for the default service account: Gateway Timeout". Common-jvm checks whether it is retriable here. But this one was not retryable thus it thrown the error. However, the code 504 should indicate it is retryable.

It happened 5 times within 3 minutes then recovered.

A Google internal bug ticket has been created by @renjiezh.

Steps to reproduce
Run stress test and there is a chance to reproduce.

Component(s) affected
QA env.

Version
0.5.7-rc2

Environment
QA env

Additional context
Exception detail:

Unexpected Error code 504 trying to get security access token from Compute Engine metadata for the default service account: Gateway Timeout

com.google.cloud.storage.StorageException: Unexpected Error code 504 trying to get security access token from Compute Engine metadata for the default service account: Gateway Timeout

	at com.google.cloud.storage.StorageException.translate(StorageException.java:170)
	at com.google.cloud.storage.spi.v1.HttpStorageRpc.translate(HttpStorageRpc.java:329)
	at com.google.cloud.storage.spi.v1.HttpStorageRpc.get(HttpStorageRpc.java:554)
	at com.google.cloud.storage.StorageImpl.lambda$internalGetBlob$63(StorageImpl.java:1585)
	at com.google.api.gax.retrying.DirectRetryingExecutor.submit(DirectRetryingExecutor.java:103)
	at com.google.cloud.RetryHelper.run(RetryHelper.java:76)
	at com.google.cloud.RetryHelper.runWithRetries(RetryHelper.java:50)
	at com.google.cloud.storage.Retrying.run(Retrying.java:65)
	at com.google.cloud.storage.StorageImpl.run(StorageImpl.java:1524)
	at com.google.cloud.storage.StorageImpl.internalGetBlob(StorageImpl.java:1583)
	at com.google.cloud.storage.StorageImpl.get(StorageImpl.java:328)
	at com.google.cloud.storage.StorageImpl.get(StorageImpl.java:321)
	at org.wfanet.measurement.gcloud.gcs.GcsStorageClient$getBlob$blob$1.invokeSuspend(GcsStorageClient.kt:114)
	at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
	at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:108)
	at kotlinx.coroutines.internal.LimitedDispatcher$Worker.run(LimitedDispatcher.kt:115)
	at kotlinx.coroutines.scheduling.TaskImpl.run(Tasks.kt:103)
	at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:584)
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:793)
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:697)
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:684)
Caused by: java.io.IOException: Unexpected Error code 504 trying to get security access token from Compute Engine metadata for the default service account: Gateway Timeout

	at com.google.auth.oauth2.ComputeEngineCredentials.refreshAccessToken(ComputeEngineCredentials.java:214)
	at com.google.auth.oauth2.OAuth2Credentials$1.call(OAuth2Credentials.java:270)
	at com.google.auth.oauth2.OAuth2Credentials$1.call(OAuth2Credentials.java:267)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at com.google.auth.oauth2.OAuth2Credentials$RefreshTask.run(OAuth2Credentials.java:634)
	at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:31)
	at com.google.auth.oauth2.OAuth2Credentials$AsyncRefreshResult.executeIfNew(OAuth2Credentials.java:582)
	at com.google.auth.oauth2.OAuth2Credentials.asyncFetch(OAuth2Credentials.java:233)
	at com.google.auth.oauth2.OAuth2Credentials.getRequestMetadata(OAuth2Credentials.java:183)
	at com.google.auth.http.HttpCredentialsAdapter.initialize(HttpCredentialsAdapter.java:96)
	at com.google.cloud.http.HttpTransportOptions$1.initialize(HttpTransportOptions.java:159)
	at com.google.cloud.http.CensusHttpModule$CensusHttpRequestInitializer.initialize(CensusHttpModule.java:109)
	at com.google.cloud.storage.spi.v1.HttpStorageRpc$InvocationIdInitializer.initialize(HttpStorageRpc.java:168)
	at com.google.api.client.http.HttpRequestFactory.buildRequest(HttpRequestFactory.java:91)
	at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.buildHttpRequest(AbstractGoogleClientRequest.java:442)
	at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:552)
	at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:493)
	at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:603)
	at com.google.cloud.storage.spi.v1.HttpStorageRpc.get(HttpStorageRpc.java:551)
	... 18 more