Deleting Stuck Executions

You are here:
< Back
  • Executions in pending state, even though there are sufficient available Celery workers to handle them
  • Executions stuck in cancelling or force_cancelling state

Stuck executions can prevent the deleting or modifying a deployment. Even worse, stuck system workflow can bring all workflows on a Cloudify Manager grinding to a halt.
To recover from that, the trick is to update the execution status manually to a “completed” status (such as terminated, cancelled or failed).

Method 1: Using PostgreSQL

  • Log in to PostgreSQL as follows:

    • Log into Cloudify Manager with SSH.
      • If you work in an HA environment, SSH into the active manager.
  • If you have sudo access, execute the following command:

    sudo -u postgres psql cloudify_db

  • If you don’t have sudo access, execute the following command:

    psql -U cloudify -W -d cloudify_db -h localhost

  • Query the database to ensure that you know what you’re doing:

    select status from executions where id='execution-id';

    (Replace execution-id with the bad execution ID. Keep the apostrophes.)

  • Update the execution:

    update executions set status='cancelled' where id='execution-id';

    That will update the execution to have a cancelled status.

Method 2: Using the REST API

You can send an HTTP PATCH request as follows:

4.x
 Expand source
curl --request PATCH -H "Content-Type: application/json" -H "Tenant: <tenant-name>" -u <username>:<password> -d '{"status": "cancelled"}' https://<manager-ip>/api/v3.1/executions/<execution-id>

 

3.4.x
 Expand source
curl --request PATCH -H "Content-Type: application/json" -d '{"status": "cancelled"} 'https://<manager-ip>/api/v2.1/executions/<execution-id>

 

  • Replace username and password with a username and password who can be authenticated against the REST service.
  • Replace manager-ip with the IP / hostname of the manager.
  • Replace https with http if your REST service is not SSL-protected (which should never really happen).
  • Replace execution-id with the execution’s ID.
  • Replace tenant-name with the name of the tenant to which the execution belongs.


Founded in 2012, Cloudify has robust financial backing from Intel Capital, VMware, BRM Group, Claridge and other leading strategic and financial investors. Cloudify has headquarters in Herzliya, Israel, and holds offices across the US and Europe.