cloudify

How to properly auto-scale groups of VMs in Cloudify?


I'm using cloudify community version 19.01.24.

Trying to figure out how to auto-scale a group of two VMs. Here's what I came up with so far (skipped irrelevant parts):

  monitored_vm1_port:
    type: cloudify.openstack.nodes.Port
    properties:
      openstack_config: *openstack_config
    relationships:
      - type: cloudify.relationships.contained_in
        target: proxy_server_network

  monitored_vm2_port:
    type: cloudify.openstack.nodes.Port
    properties:
      openstack_config: *openstack_config
    relationships:
      - type: cloudify.relationships.contained_in
        target: proxy_server_network

  monitored_vm1_host:
    type: cloudify.openstack.nodes.Server
    properties:
      image: { get_input: image }
      flavor: { get_input: flavor }
      resource_id: { concat: ['monitored_vm1-', { get_input: client_name }] }
      agent_config:
        user: { get_input: agent_user }
        key: { get_property: [ keypair, private_key_path ] }
    interfaces:
      cloudify.interfaces.monitoring_agent:
        install:
          implementation: diamond.diamond_agent.tasks.install
          inputs:
            diamond_config:
              interval: 10
        start: diamond.diamond_agent.tasks.start
        stop: diamond.diamond_agent.tasks.stop
        uninstall: diamond.diamond_agent.tasks.uninstall
      cloudify.interfaces.monitoring:
        start:
          implementation: diamond.diamond_agent.tasks.add_collectors
          inputs:
            collectors_config:
              NetworkCollector: {}
    relationships:
     - type: cloudify.openstack.server_connected_to_port
       target: monitored_vm1_port
     - type: cloudify.openstack.server_connected_to_keypair
       target: keypair

  monitored_vm2_host:
    type: cloudify.openstack.nodes.Server
    properties:
      image: { get_input: image }
      flavor: { get_input: flavor }
      resource_id: { concat: ['monitored_vm2-', { get_input: client_name }] }
      agent_config:
        user: { get_input: agent_user }
        key: { get_property: [ keypair, private_key_path ] }
    interfaces:
      cloudify.interfaces.monitoring_agent:
        install:
          implementation: diamond.diamond_agent.tasks.install
          inputs:
            diamond_config:
              interval: 10
        start: diamond.diamond_agent.tasks.start
        stop: diamond.diamond_agent.tasks.stop
        uninstall: diamond.diamond_agent.tasks.uninstall
      cloudify.interfaces.monitoring:
        start:
          implementation: diamond.diamond_agent.tasks.add_collectors
          inputs:
            collectors_config:
              NetworkCollector: {}
    relationships:
     - type: cloudify.openstack.server_connected_to_port
       target: monitored_vm2_port
     - type: cloudify.openstack.server_connected_to_keypair
       target: keypair

groups:
  vm_group:
    members: [monitored_vm1_host, monitored_vm2_host]

  scale_up_group:
    members: [monitored_vm1_host, monitored_vm2_host]
    policies:
      auto_scale_up:
        type: scale_policy_type
        properties:
          policy_operates_on_group: true
          scale_limit: 2 # max additional instances
          scale_direction: '<'
          scale_threshold: 31457280
          service_selector: .*monitored_vm1_host.*network.eth0.rx.bit
          cooldown_time: 60
        triggers:
          execute_scale_workflow:
            type: cloudify.policies.triggers.execute_workflow
            parameters:
              workflow: scale
              workflow_parameters:
                delta: 1
                scalable_entity_name: vm_group
                scale_compute: true

policies:
  vm_group_scale_policy:
    type: cloudify.policies.scaling
    properties:
      default_instances: 1
    targets: [vm_group]

So the blueprint gets deployed correctly, and scale workflow is being triggered according to specified condition (traffic on the VM's interface), but it fails during creation of the new VMs instances with following errors:

2019-11-18 14:54:46,591:ERROR: Task nova_plugin.server.create[f736f81c-7f8c-4f82-a280-8352c1d01bff] raised:
Traceback (most recent call last):
  (...)
NonRecoverableError: Port 3b727b5e-a2ec-47cc-b711-37cb80a7b4e5 is still in use. [status_code=409]

Looks like Cloudify is trying to spawn new instances using existing ports, weird. So I thought, maybe I should explicitly place VM's ports in the scaling group as well, to replicate them along with VMs. Tried it like this:

  vm_group:
    members: [monitored_vm1_host, monitored_vm1_port, monitored_vm2_host, monitored_vm2_port]

But in that case I get an error regarding some missing objects relationships, already at blueprint verification stage:

Invalid blueprint - Node 'monitored_vm1_host' and 'monitored_vm1_port' belong to some shared group but they are not contained in any shared node, nor is any ancestor node of theirs.
  in: /opt/manager/resources/blueprint-with-scaling-d79fed3d-0b3b-4459-a851-fedd9ecf50c6/blueprint-with-scaling.yaml

I've gone through documentation and any examples I could find (there are not many), but they're unclear to me.

How can I get it scale correctly?


Solution

  • You got the first error as you said because Cloudify tried to scale the VM and connect it to the port that was already bound to the first VM.

    The second error means that you cannot scale the port if it does not depend on a node that is also scaled, this is to avoid the scale of resources that can't be scaled.

    The solution for this would be, to have a node of type cloudify.nodes.Root and connect to it by relationship the port, if the port depends on this node and this node will be part of the scale group you will be able to scale.

    Your blueprint will have something like this:

     my_relationship_node:
        type:  cloudify.nodes.Root
    
     port:
        type: cloudify.openstack.nodes.Port
        properties:
          openstack_config: *openstack_config
        relationships:
        - type: cloudify.relationships.connected_to
          target: public_network
        - type: cloudify.relationships.depends_on
          target: public_subnet
        - type: cloudify.openstack.port_connected_to_security_group
          target: security_group
        - type: cloudify.openstack.port_connected_to_floating_ip  
          target: ip
        - type: cloudify.relationships.contained_in
          target: my_relationship_node
    

    I hope it helps.