Introduction

In some cases, you need to automate a complete graph of integration workflows which depend on each other and can sometimes run in parallel or after each other.Although cmemc is not a workflow orchestration tool, you can easily use it do some basic workflow orchestration. This page describes how you can execute and orchestrate workflows together.For simplicity all given examples do not select a specific connection (--connection your-cmem). We simply assume you selected your instance via an environment variable (export CMEMC_CONNECTION=your-cmem).

Simple Execution

cmemc allows for execution of workflows with the workflow execute command. To start a workflow, simply use this command:

$ cmemc workflow execute cmem:my-workflow
cmem:my-workflow ... Started
BASH

Workflow identifier can be added with command-line completion on the command line but you can also get a list of workflows with the workflow list command:

$ cmemc workflow list
cmem:my-workflow
cmem:second-workflow
BASH

The default workflow execute command starts a workflow without waiting for a response. In order to wait until the workflow is finished you need to use the --wait option:

$ cmemc workflow execute cmem:my-workflow --wait
cmem:my-workflow ... Started ... Finished (Finished in 32.931s, just now)
BASH

For a reference on the workflow execute command, have a look at the Command Reference or look at the command specific help text (cmemc workflow execute --help).

Retrieve Status Information

At any time, you can retrieve status information of a workflow with the workflow status command:

$ cmemc workflow status cmem:my-workflow
cmem:my-workflow ... Finished (Finished in 32.931s, 4 minutes ago)
BASH

In addition to that, you can retrieve raw JSON data about a workflow, which can easily used used for post-processing:

$ cmemc -workflow status cmem:my-workflow --raw
{
  "activity": "ExecuteLocalWorkflow",
  "runtime": 32931,
  "project": "cmem",
  "failed": false,
  "message": "Finished in 32.931s",
  "task": "my-workflow",
  "isRunning": false,
  "statusName": "Finished",
  "progress": 100,
  "cancelled": false,
  "startTime": 1593679211989,
  "exceptionMessage": null,
  "lastUpdateTime": 1593679244920
}
BASH

For a reference on the workflow status command, have a look at the Command Reference or look at the command specific help text (cmemc workflow status --help).

Serial Execution

The workflow execute command is able to start multiple workflows in a chain, waiting for each of the workflow and exit if there is an error with one of the workflows.

To do this, use the --wait option and simply add more than one workflow identifier the command:

$ cmemc workflow execute cmem:my-workflow cmem:second-workflow --wait
cmem:my-workflow ... Started ... Finished (Finished in 30.984s, just now)
cmem:second-workflow ... Started ... Finished (Finished in 50.579s, just now)
BASH

Starting these workflows in this way means that cmemc exits with an error code 1 in the moment the first workflow throws an error. All later workflows will not be executed.

Parallel Execution

Sometimes you want to execute workflows in parallel, because they do not depend on each other and it can fasten up the overall runtime.

To do this, there is a little bit of extra scripting needed at the moment. The main idea is, to start the parallel workflows without waiting and then poll the status information until they are not running anymore.

Here is an example script which does exactly this.

#!/usr/bin/env bash
# @(#) Example: execute two workflows in parallel and wait for the results (exit 1 on failure)
# Use the unofficial bash strict mode: http://redsymbol.net/articles/unofficial-bash-strict-mode/
set -euo pipefail; export FS=$'\n\t'

# setup the used instance
export CMEMC_CONNECTION=your-cmem

# check SSL config
if [ "$(cmemc config get SSL_VERIFY 2>&1)" == True ]
then
    NUM=0
else
    NUM=1
fi

# start the given set of workflows
WORKFLOW_IDS="cmem:my-workflow cmem:second-workflow"
cmemc workflow execute $WORKFLOW_IDS

# loop until they are not running anymore
RUNNING=-1
until [ $RUNNING == $NUM ]
do
    # wait 5 seconds - polling time
    sleep 5
    # use the the filter option to show only running workflows
    RUNNING=$(cmemc workflow status $WORKFLOW_IDS --filter running 2>&1 | wc -l)
    if [ $RUNNING != $NUM ]; then
        echo "We still have $RUNNING running workflows ..."
    fi
done

# look for failed workflows
FAILED=$(cmemc workflow status $WORKFLOW_IDS --filter failed 2>&1 | wc -l)
if [ $FAILED != $NUM ]; then
    echo "Some workflows failed :-("
    exit 1
else
    echo "All workflows finished successfully :-)"
    exit 0
fi
BASH