Photo by JJ Ying on Unsplash

In November 2023, AWS announced support for HTTPS endpoints directly in AWS Step Functions. This allows you to call (and test!) third-party API endpoints directly from your State Machine.

AWS Cloud Development Kit (CDK) is a framework that allows engineers to define their infrastructure as code (IaC) in a familiar programming language instead of having to manage their infrastructure manually or write raw CloudFormation.

Background

In one of my projects, I use AWS Step Functions State Machines to manage a data pipeline.

This state machine is kicked off by an EventBridge Scheduler every evening, and it manages a series of Lambda functions and Fargate Tasks to procure, process, and analyze data, based on a set of nightly criteria. If some step of the state machine fails, SNS sends me an email letting me know that the pipeline hit a failure.

However, on a couple of occasions, I’ve found that I have missed the “Failure” email, so I wanted to make sure that it was more visible and that the rest of the team could easily see it as well. To do this, I wanted to create a new support ticket in our ticketing system (we use Outseta) that everyone could see (this also has its own integration to ping our Slack).

This is a very simple implementation of this functionality, as it happens at the end of my state machine failing, and therefore I’m not expecting any data back from this endpoint, nor am I passing it anything custom at this point.

HTTP Endpoint Task

Adding an HTTP Endpoint to your State Machine is easy in the AWS Console: Step Functions has a drag-and-drop workflow builder that lets you choose the type of Action you want, and an interface to fill out all the information you need to provide.

Example of information you provide when building an HTTP Endpoint in the Step Functions UI:

The UI is great for playing around and trying things out — also for testing — but when it comes to the production version of my state machine, I use AWS Cloud Development Kit (CDK) to manage my infrastructure as code (IaC). In my project, I’m using CDK in TypeScript, but it’s also available in other languages like Python and Java.

However, at the time of writing (December 2023), I couldn’t find anything directly related to these new third-party endpoints in the Step Functions Tasks module of CDK. This isn’t surprising, as CDK typically lags behind the official releases a bit — but it means it can be a little more work to implement new features using CDK!

Note: There are currently constructs for Step Functions tasks that call HTTP/REST endpoints through API Gateway. These are different than calling a third-party endpoint!

The best place to start was the AWS Step Functions documentation for calling third-party APIs. Here AWS steps through the requirements needed to run an HTTP Task, starting with an EventBridge Connection.

EventBridge Connection

EventBridge Connections are required to manage the authentication of an HTTP Task. For my use case, I’m using an API Key for authentication. EventBridge Connections create and store their own secrets in AWS Secrets Manager with the encrypted parameters needed to make a secure connection to the API endpoint.

// EventBridge Connection to Outseta
const outsetaConnection = new events.Connection(this, "outsetaConnection", {
  description: "Outseta Connection Via API Key",
  authorization: events.Authorization.apiKey(
    "Authorization",
    cdk.SecretValue.secretsManager("outsetaKey")
  ),
});

Note: This was my first time using an EventBridge connection, and when I first implemented it I was a bit surprised at the need for 2 secrets for one connection — secrets are typically $0.40 (plus fees for reading) per secret per month! However, it turns out that the secret the connection makes is included in the charge for using an API destination.

IAM Policy

The next step was to make sure that I had the proper IAM Policy so that my state machine had the proper permissions to run the Task. There were 3 statements involved in this policy:

1. The ability to POST to the given HTTP endpoint via the states:InvokeHTTPEndpoint action. (See more on this below)
2. The action events:RetrieveConnectionCredentials to the EventBridge Connection resource defined above.
3. The ability to GetSecretValue and DescribeSecret on Secrets Manager secrets created by EventBridge Connections.

const httpTaskPolicy = new iam.Policy(this, "httpTaskPolicy", {
  document: new iam.PolicyDocument({
    statements: [
      new iam.PolicyStatement({
        actions: ["states:InvokeHTTPEndpoint"],
        resources: ["*"],
        effect: iam.Effect.ALLOW,
        conditions: {
          StringEquals: {
            "states:HTTPMethod": "POST",
          },
          StringLike: {
            "states:HTTPEndpoint":
              "https://myoutseta.outseta.com/api/v1/support/*",
          },
        },
      }),
      new iam.PolicyStatement({
        actions: ["events:RetrieveConnectionCredentials"],
        resources: [outsetaConnection.connectionArn],
        effect: iam.Effect.ALLOW,
      }),
      new iam.PolicyStatement({
        actions: [
          "secretsmanager:GetSecretValue",
          "secretsmanager:DescribeSecret",
        ],
        resources: [
          "arn:aws:secretsmanager:*:*:secret:events!connection/*",
        ],
        effect: iam.Effect.ALLOW,
      }),
    ],
  }),
});

// Attach the policy to the state machine''s execution role
myStateMachine.role.attachInlinePolicy(httpTaskPolicy);

Possible Issues (December 2023) I heavily borrowed from the JSON policy in the documentation, but at the time of writing, I ran into 2 possible issues:

1. The provided example is invalid JSON, so be careful if you copy and paste.
2. The example shows the states:InvokeHTTPEndpoint action is assigned to the state machine as the resource. It’s possible that I messed something up here, but I repeatedly ran into States.Http.AccessDenied here until I opened up the resources on that statement to be * — this is something I plan to debug more later.

Custom State

Finally, we need to provide the CDK for the HTTP state itself. As I mentioned above, there doesn’t appear to be a unique construct available for the HTTP Endpoint (yet!) but luckily the Step Functions CDK includes the ability to define a CustomState .

const callOutsetaJSON = {
  Type: "Task",
  Resource: "arn:aws:states:::http:invoke",
  Parameters: {
    ApiEndpoint: "https://myoutseta.outseta.com/api/v1/support/cases",
    Authentication: {
      ConnectionArn: outsetaConnection.connectionArn,
    },
    Method: "POST",
    Headers: {
      "Content-Type": "application/json",
    },
    QueryParameters: {
      sentAutoResponder: false,
    },
    RequestBody: {
      FromPerson: { Uid: "MY_BOT_USER_ID" },
      Subject: "An error occured in the data pipeline!",
      Body: "There was an error during the data pipeline. Go check it out!",
      Source: 2,
    },
  },
};

const callOutsetaState = new stepfunctions.CustomState(
  this,
  "callOutseta",
  { stateJson: callOutsetaJSON }
);

In the CustomState ‘s stateJSON , I was able to define the HTTP Task’s definition using Amazon States Language (ASL).

Finally, the last step was to include this state in the failure path for the rest of my state machine:

const failure = new stepfunctions.Fail(this, "Data Pipeline Failed!");
const failurePath = callOutsetaState.next(failure);

<tasksThatMightFail>.addCatch(failurePath, {
  errors: ["States.ALL"],
  resultPath: "$.Payload.error",
});

Testing the Task

I did come back to the Console to test. My entire pipeline takes over an hour to run, and this path only gets taken in the event of a failure, so it’s hopefully not something that I see very often!

Luckily, AWS offers an easy way to test the endpoint that you’re adding, allowing you to call the endpoint directly while providing it with any custom input you might need.

Example of the test state interface showing an Access Denied error:

This did help me debug a couple of IAM permissions I had wrong (like the one pictured above) and helped me ensure that I got the endpoint working correctly.

Next Steps

This was a quick way for me to play around with the new functionality in Step Functions (and to help solve an issue in the process), so I recognize that I could make this better:

1. Narrow the IAM policy — as I mentioned above, I had trouble getting the states:InvokeHTTPEndpoint to work as the documentation indicated. It’s entirely possible (and probable) that this was a bug or typo on my part, and I should revisit my policies.
2. Custom Error Message — right now I’m passing a generic message to the support ticket to go look at the State Machine. I should probably include the error in the support ticket.
3. Replace other pieces of my state machine with HTTP endpoints — Other pieces of this pipeline currently consist of a lambda calling a third-party endpoint. These could be simplified using this new functionality!

Other CDK Resources

Here are a few other resources that may come in handy when working with AWS CDK:

• The official AWS Reference contains documentation on all the constructs
• Construct Hub contains other open-source constructs
• Also, please check other articles I wrote on CDK: a. Getting Started with AWS Cloud Development Kit (CDK) b. Managing Secrets With AWS Cloud Development Kit (CDK) c. Creating an Amazon Eventbridge Scheduler with CDK

I enjoy writing about software development, project management, and my journey in the AWS Cloud. If you’d like to read more, please consider following me here on Medium, Twitter, or LinkedIn.