Hyperparameter tuning failed on Google Cloud

[pzet]

I'm trying to follow the Keras tutorial (https://tensorflow.rstudio.com/tools/cloudml/tuning/) for hyperparameter tuning with few modifications in the configuration YAML file (which I introduced because of runtimeVersion error). After submitting the job with CloudML library, I see in the logs that the model is being trained (epochs are running), but after going through all the epochs all the HyperTune trials show status "Failed".

Below is the model from mnist_mlp.R I'm trying to hypertune:

library(keras)

FLAGS <- flags(
  flag_integer("dense_units1", 128),
  flag_numeric("dropout1", 0.4),
  flag_integer("dense_units2", 128),
  flag_numeric("dropout2", 0.3)
)

mnist <- dataset_mnist()
x_train <- mnist$train$x
y_train <- mnist$train$y
x_test <- mnist$test$x
y_test <- mnist$test$y

x_train <- array_reshape(x_train, c(nrow(x_train), 784))
x_test <- array_reshape(x_test, c(nrow(x_test), 784))
x_train <- x_train / 255
x_test <- x_test / 255

y_train <- to_categorical(y_train, 10)
y_test <- to_categorical(y_test, 10)

model <- keras_model_sequential() %>%
  layer_dense(units = FLAGS$dense_units1, activation = 'relu',
              input_shape = c(784)) %>%
  layer_dropout(rate = FLAGS$dropout1) %>%
  layer_dense(units = FLAGS$dense_units2, activation = 'relu') %>%
  layer_dropout(rate = FLAGS$dropout2) %>%
  layer_dense(units = 10, activation = 'softmax')


model %>% compile(
  loss = 'categorical_crossentropy',
  optimizer = optimizer_rmsprop(),
  metrics = c('accuracy')
)

model %>% fit(
  x_train, y_train,
  epochs = 20, batch_size = 128,
  validation_split = 0.2
)

and here's the code for my configuration file:

trainingInput:
  scaleTier: CUSTOM
  masterType: "n1-standard-4"
  runtimeVersion: "1.15"
  hyperparameters:
    goal: MAXIMIZE
    hyperparameterMetricTag: "acc"
    maxTrials: 10
    maxParallelTrials: 2
    params:
      - parameterName: "dropout1"
        type: DOUBLE
        minValue: 0.2
        maxValue: 0.6
        scaleType: UNIT_LINEAR_SCALE
      - parameterName: "dropout2"
        type: DOUBLE
        minValue: 0.1
        maxValue: 0.5
        scaleType: UNIT_LINEAR_SCALE

The JSON output of an exemplary trial looks like this:

{
      "trialId": "8",
      "hyperparameters": {
        "dropout1": "0.59895355546833817",
        "dropout2": "0.10417636518800295"
      },
      "startTime": "2020-04-28T15:13:30.645077193Z",
      "endTime": "2020-04-28T15:22:47Z",
      "state": "FAILED"
    }
  ],
  "consumedMLUnits": 0.52,
  "isHyperparameterTuningJob": true,
  "hyperparameterMetricTag": "acc"
}

As in the tutorial, I use R CloudML package to submit the job to Google Cloud Platform. Is there any reason why hypertuning is not returning accuracy properly?

[aginpatrick]

Same for me. Did you find an explanation to the failing?

[pzet]

Hi, unfortunately not. I've got a few emails from people facing the same issue, but no one resolved it yet.