AWS Step Function Tips and Tricks

AWS Step Functions in action

Having worked with AWS step functions for a little while now, I've had to jump a few hurdles to get what I wanted, and I figured other people could benefit from the hours of hair-pulling. Sharing is caring, or so they say.

1. Lifting a single value into an Array

Lifting a parameter into an array

Occasionally you'll want to convert a value to an array of that value. This can often be the case if you want to map over a single value or use a dynamic parameter with an ECS task invocation as a Command argument (because it only accepts arrays, not a string 🤷‍♂).

Interestingly the Pass type allows you to perform some interesting parameter movements. Both InputPath and ResultPath are valid (and the sneaky Parameters option is also available if you want to create bigger objects).

{
  "StartAt": "Lift to Array",
  "States": {
    "Lift to Array": {
      "Type": "Pass",
      "InputPath": "$.param",
      "ResultPath": "$.paramList[0]",
      "Next": "Replace"
    },
    "Replace": {
      "Type": "Pass",
      "InputPath": "$.paramList",
      "ResultPath": "$.param",
      "End": true
    }
  }
}

As you can see from the above, we lift the $.param into $.paramList[0] i.e. position 0 of an array, then we just replace the $.param in the subsequent step - $.param is now an array.

2. Concatenating items into a single array

As with the previous example, but one step further. When you need a multi-value array (again — ECS task Command arguments or Map with ItemsPath), and you want to construct it from several keys in your state, e.g., turning {k1:1, k2:2, k3:3} into {concat: [1,2,3]}.

{
  "StartAt": "Concat",
  "States": {
    "Concat": {
      "Type": "Parallel",
      "InputPath": "$",
      "ResultPath": "$.concat",
      "Branches": [
        {
          "StartAt": "Concat Key1",
          "States": {
            "Concat Key1": {
              "InputPath": "$.k1",
              "Type": "Pass",
              "End": true
            }
          }
        },
        {
          "StartAt": "Concat Key2",
          "States": {
            "Concat Key2": {
              "Result": "two",
              "Type": "Pass",
              "End": true
            }
          }
        },
        {
          "StartAt": "Concat Key3",
          "States": {
            "Concat Key3": {
              "InputPath": "$.k3",
              "Type": "Pass",
              "End": true
            }
          }
        }
      ],
      "End": true
    }
  }
}

Notice in the above example I substituted the second value for a concrete "two” by using Result instead of InputPath.

The resultant output now being
{k1:1, k2:2, k3:3, concat: ["1","two","3"]}

3. Avoid lambda::invoke

If you use the UI and/or look at the AWS documentation, the current recommended route for invoking a lambda is to set your Resource key to arn:aws:states:::lambda:invoke . The issue here is that the response you get back from the Lambda includes all the SDK metadata in the result of the invocation. This leaves you with a lot of tidy up to do to clean up and augment your state. Normally you're only interested in the response that your lambda returned, thus we need to find a better way.

DO NOT do this!

"THE BAD WAY": {
  "Type": "Task",
  "Resource": "arn:aws:states:::lambda:invoke",
  "ResultPath": "$.theOutput",
  "Parameters": {
    "FunctionName": "<THE_LAMBDA_ARN_HERE>",
    "Payload": {
      "param1.$": "$.k1"
    }
  },
  "End": true
}

Instead of doing the above, you can invoke the resource directly, i.e. NOT via the recommended lambda:invoke methodology, as follows:

DO THIS INSTEAD

"THE GOOD WAY": {
  "Type": "Task",
  "Resource": "<THE_LAMBDA_ARN_HERE>",
  "ResultPath": "$.theOutput",
  "Parameters": {
    "param1.$": "$.k1"
  },
  "End": true
}

Now we should get only the return message that our lambda produces at the $.theOutput path in our step function state. 👍

4. Handling missing values (optional values)

AWS's usage of JSONPath does not cater for optional/missing keys, and thus there is no way to ignore or default these missing values.

Two example payloads might look like this:
{"always": "here"} and {"always": "here", "sometimes": "here"}

You will usually encounter this when you want to do something like invoke a workflow with a date in one scenario, but NOT a date in another (because you want the workflow step to assume the time is now() ). In reality, the only option to solve this situation is to have a surrogate key that you always utilise to split your workflow based on whether you have the key or not.

The main downside here is that you'll end up copy-pasting your step invocation in 2 places.

{
  "Comment": "A conditional parameter workflow",
  "StartAt": "Has Optional Parameter",
  "States": {
    "Has Optional Parameter": {
      "Type": "Choice",
      "InputPath": "$",
      "Choices":[
        {
          "Variable": "$.hasTheConditional",
          "BooleanEquals": true,
          "Next": "With Optional"
        },
        {
          "Variable": "$.hasTheConditional",
          "BooleanEquals": false,
          "Next": "Without Optional"
        }
      ],
      "Default": "Without Optional"
    },
    "With Optional": {
      "Type": "Pass",
      "End": true
    },
    "Without Optional": {
      "Type": "Pass",
      "Result": "$.conditionalParameter",
      "End": true
    }
  }
}

Conclusion

Thanks for reading. If you liked my stories then go ahead and give it a clap or leave a comment if there's something you want me to cover next time. If you disliked it, I hope I get the opportunity to please you some other time 🤷‍♂.

In the meantime, go ahead and use what you've learned to make your step functions just that little bit easier to handle. Remember, you don't always need a lambda to solve your problem with step functions.