Spot provisioning partition

[noahharrison64]

Hi,

I've had support from the team here at hpc-toolkit on setting up an autoscaling cluster and now it's all up and running I've been thinking about how to minimise costs.

Is it possible to use spot instances in the sort of auto-scaling setup detailed here? It would be fantastic if it were possible to have 2 partitions, the first being on-demand instances and the other being spot instances. I've had a look this repo and saw that spot provisioning was added as a feature but I haven't found any documentation in the examples folder or elsewhere detailing how to go about achieving this.

If this feature is possible any advice on how to implement would be really appreciated.

Thanks,
Noah

[nick-stroud]

Hey @noahharrison64, I believe the link titled detailed here is mistaken as it points to the "Start a new discussion" page. What blueprint would be most similar to what you are running? What scheduler are you using?

Many of our schedulers do support spot including Slurm V5, GKE, HTCondor, and PBS Pro.

For example, with Slurm v5 you could create two partitions, one with spot and the other on-demand, with the following blueprint:

---
blueprint_name: slurm-gcp-v5-spot

vars:
  project_id: # Project ID
  deployment_name: spot-slurm
  region: us-central1
  zone: us-central1-c

deployment_groups:
- group: primary
  modules:

  - id: network1
    source: modules/network/vpc

  - id: node_group_spot
    source: community/modules/compute/schedmd-slurm-gcp-v5-node-group
    settings:
      node_count_dynamic_max: 4
      machine_type: n2d-standard-2
      enable_spot_vm: true
      preemptible: true

  - id: partition_spot
    source: community/modules/compute/schedmd-slurm-gcp-v5-partition
    use:
    - network1
    - node_group_spot
    settings:
      partition_name: spot
      is_default: true


  - id: node_group_on_demand
    source: community/modules/compute/schedmd-slurm-gcp-v5-node-group
    settings:
      node_count_dynamic_max: 4
      machine_type: n2d-standard-2

  - id: partition_on_demand
    source: community/modules/compute/schedmd-slurm-gcp-v5-partition
    use:
    - network1
    - node_group_on_demand
    settings:
      partition_name: demand

  - id: slurm_controller
    source: community/modules/scheduler/schedmd-slurm-gcp-v5-controller
    use:
    - network1
    - partition_on_demand
    - partition_spot
    settings:
      machine_type: e2-standard-2

As you can see the spot behavior is configured on the schedmd-slurm-gcp-v5-node-group. The settings for that module are documented here.