AWS Step Functions with Lambda: Scheduling and Passing Data

A Lambda that calls another Lambda that calls a third one works fine right up until the middle one fails. Then you are reading three separate log groups trying to work out which invocation died and what it was holding at the time. That is usually the week a team moves to AWS Step Functions.

We run this pattern for scheduled meeting recordings: a workflow that starts a recorder at a timestamp the user picked, waits, then stops it and uploads. Simple on paper. The two things that actually cost us time were getting the Lambda return value out of the response wrapper, and deciding where the scheduling should live. Both are covered below, with the state machine JSON that works.

What AWS Step Functions with Lambda actually changes

Step Functions is a state machine service. You describe your workflow as JSON in Amazon States Language, and each step is a state: a Task that calls something, a Choice that branches, a Wait that pauses, a Map that fans out. Lambda is just one of the things a Task can call, but it is the one most people start with.

Three things you get that a chain of Lambdas does not give you:

• Execution history you can actually read. Every state transition, its input, and its output are recorded. When a workflow fails you open one execution and see exactly which state broke and what it was handed.
• Retries and catches as configuration. A Retry block with backoff on a Task state replaces the try/catch/sleep code you would otherwise write in every function.
• Waiting that costs nothing. A Lambda that sleeps for an hour bills you for an hour and cannot do it anyway, because the function timeout maxes out at 15 minutes. A Wait state in a Standard workflow can sit for a year and costs one state transition.

That last point is the whole reason the scheduling pattern in this article works.

How to invoke Lambda from a state machine

There are two ways to point a Task state at a Lambda function, and picking the wrong one is where the return value confusion starts.

Optimized integration (what you almost always want)

Set the Resource to the Step Functions Lambda integration ARN and name your function in the arguments:

{
  "StartRecorder": {
    "Type": "Task",
    "QueryLanguage": "JSONata",
    "Resource": "arn:aws:states:::lambda:invoke",
    "Arguments": {
      "FunctionName": "arn:aws:lambda:us-east-1:123456789012:function:startRecorder",
      "Payload": {
        "meetingId": "{% $states.input.meetingId %}",
        "quality": "720p"
      }
    },
    "Next": "WaitForMeetingEnd"
  }
}

If you are still on JSONPath, the same state uses Parameters instead of Arguments, and fields ending in .$ to pull values from the input:

{
  "StartRecorder": {
    "Type": "Task",
    "Resource": "arn:aws:states:::lambda:invoke",
    "Parameters": {
      "FunctionName": "arn:aws:lambda:us-east-1:123456789012:function:startRecorder",
      "Payload": {
        "meetingId.$": "$.meetingId",
        "quality": "720p"
      }
    },
    "Next": "WaitForMeetingEnd"
  }
}

The optimized integration is what unlocks InvocationType set to Event for fire-and-forget calls, the Qualifier field for a specific version or alias, and the .waitForTaskToken callback pattern where the workflow pauses until something external calls SendTaskSuccess.

Direct function ARN (the quiet alternative)

You can also put the function ARN straight in the Resource field:

{
  "StartRecorder": {
    "Type": "Task",
    "Resource": "arn:aws:lambda:us-east-1:123456789012:function:startRecorder",
    "Next": "WaitForMeetingEnd"
  }
}

The task result here is only your function's output, with no metadata wrapper. Cleaner, and honestly tempting. But you cannot use .waitForTaskToken, you cannot set InvocationType, and the whole state input is passed as the event with no shaping. I use it for small internal utility steps and the optimized form everywhere else.

The AWS Step Function Lambda return value trap

This is the single most searched problem with this pairing, and the cause is one line in the docs. With the optimized integration, the task result is the full Lambda Invoke API response, not your function's return value:

{
  "ExecutedVersion": "$LATEST",
  "Payload": {
    "recordingId": "rec-8841",
    "startedAt": "2026-07-29T10:00:03Z"
  },
  "SdkHttpMetadata": {
    "HttpHeaders": { "Content-Type": "application/json" },
    "HttpStatusCode": 200
  },
  "SdkResponseMetadata": {
    "RequestId": "6b3bebdb-9251-453a-ae45-512d9e2bf4d3"
  },
  "StatusCode": 200
}

So the recordingId your function returned is not at $.recordingId. It is at $.Payload.recordingId. Every "my Choice state never matches" thread comes down to this.

Three ways to fix it, depending on what you want downstream.

OutputPath: throw the metadata away

The one-liner. Everything except the function output is discarded:

{
  "Type": "Task",
  "Resource": "arn:aws:states:::lambda:invoke",
  "Parameters": { "FunctionName": "startRecorder" },
  "OutputPath": "$.Payload",
  "Next": "WaitForMeetingEnd"
}

ResultSelector: reshape before anything else runs

ResultSelector rewrites the raw result into a shape you choose, and it runs before ResultPath. Useful when you want the payload plus one piece of metadata:

{
  "Type": "Task",
  "Resource": "arn:aws:states:::lambda:invoke",
  "Parameters": { "FunctionName": "startRecorder" },
  "ResultSelector": {
    "recording.$": "$.Payload",
    "httpStatus.$": "$.SdkHttpMetadata.HttpStatusCode"
  },
  "ResultPath": "$.recorder",
  "Next": "WaitForMeetingEnd"
}

JSONata Output: read the result directly

In a JSONata state, the raw task result is available as $states.result, so you say what you want and skip the path juggling entirely:

{
  "Type": "Task",
  "QueryLanguage": "JSONata",
  "Resource": "arn:aws:states:::lambda:invoke",
  "Arguments": { "FunctionName": "startRecorder" },
  "Output": "{% $states.result.Payload %}",
  "Next": "WaitForMeetingEnd"
}

Passing data between Lambdas without the payload growing

By default a task's output completely replaces the state input. So if Lambda A returns a recording ID and Lambda B needs both that ID and the original meeting ID from the execution input, B only sees the recording ID. This surprises people once, then they learn ResultPath.

The JSONPath way: merge with ResultPath

ResultPath says where in the existing input to graft the result, instead of replacing it:

{
  "StartRecorder": {
    "Type": "Task",
    "Resource": "arn:aws:states:::lambda:invoke",
    "Parameters": { "FunctionName": "startRecorder" },
    "ResultSelector": { "recordingId.$": "$.Payload.recordingId" },
    "ResultPath": "$.recorder",
    "Next": "StopRecorder"
  }
}

The next state now receives the original input with a recorder object added to it. Set ResultPath to null and the task result is discarded entirely, which is the clean way to call a notification Lambda mid-workflow without it clobbering your payload.

The catch is that the object gets fatter with every state, and you end up threading data through five steps that do not care about it. Which is exactly the problem variables solve.

The JSONata way: assign once, read anywhere

AWS added variables and JSONata to Step Functions in late 2024, and for workflows longer than about four states it is a real improvement. Any state except Succeed and Fail can carry an Assign block that stores values, and any later state can read them with a $ prefix:

{
  "StoreRequest": {
    "Type": "Pass",
    "QueryLanguage": "JSONata",
    "Assign": {
      "meetingId": "{% $states.input.meetingId %}",
      "endsAt": "{% $states.input.endsAt %}"
    },
    "Next": "StartRecorder"
  },
  "StartRecorder": {
    "Type": "Task",
    "QueryLanguage": "JSONata",
    "Resource": "arn:aws:states:::lambda:invoke",
    "Arguments": {
      "FunctionName": "startRecorder",
      "Payload": { "meetingId": "{% $meetingId %}" }
    },
    "Assign": {
      "recordingId": "{% $states.result.Payload.recordingId %}"
    },
    "Next": "WaitForMeetingEnd"
  }
}

Five states later, $recordingId is still there. No state in between had to carry it.

Two details that catch people out. Assignments are evaluated all at once using the values as they were on state entry, so referencing a variable you are also assigning in the same block gives you the old value, not the new one. And variables are workflow-local: states inside a Map or Parallel can read outer-scope variables but get their own scope for writing, and when the branch ends those inner variables are gone. Use the Output field to get data back out.

You do not have to migrate anything to start using this. QueryLanguage can be set per state, so a state machine can hold JSONPath states written in 2022 next to JSONata states written today.

Scheduling: Wait state or EventBridge Scheduler

"Scheduling a Lambda with Step Functions" covers two genuinely different problems, and people pick the wrong tool because they sound the same.

Wait state: pausing inside an execution

Use this when the workflow already started and needs to hold until a time it was told about at runtime. Four fields, and you pick exactly one:

{
  "WaitForMeetingEnd": {
    "Type": "Wait",
    "TimestampPath": "$.endsAt",
    "Next": "StopRecorder"
  },
  "WaitTenSeconds": {
    "Type": "Wait",
    "Seconds": 10,
    "Next": "PollStatus"
  }
}

Seconds and Timestamp are literals. SecondsPath and TimestampPath read from the input and are JSONPath-only. In a JSONata state you drop the Path variants and put an expression in Seconds or Timestamp instead.

Timestamps must be RFC3339 with a capital T separator and a capital Z, like 2026-07-29T11:00:00Z. JavaScript's toISOString() gives you a valid one; the milliseconds are simply truncated. A lowercase z or a numeric offset like +05:30 is rejected at execution time, not at deploy time, which is a fun way to find out in production.

EventBridge Scheduler: recurring runs

For "every night at 2am Colombo time", the schedule belongs outside the workflow. EventBridge Scheduler has a templated target for Step Functions StartExecution, so no glue Lambda is needed:

aws scheduler create-schedule \
  --name nightly-recording-cleanup \
  --schedule-expression "cron(0 2 * * ? *)" \
  --schedule-expression-timezone "Asia/Colombo" \
  --flexible-time-window '{"Mode":"OFF"}' \
  --target '{
    "Arn":"arn:aws:scheduler:::aws-sdk:sfn:startExecution",
    "RoleArn":"arn:aws:iam::123456789012:role/SchedulerInvokeSfnRole",
    "Input":"{\"job\":\"cleanup\"}",
    "SfnParameters":{"StateMachineArn":"arn:aws:states:us-east-1:123456789012:stateMachine:RecordingWorkflow"}
  }'

Scheduler beats the old CloudWatch Events rule on three counts: cron expressions get a time zone, one-time schedules are first-class instead of a rule you have to remember to delete, and flexible time windows let you spread thousands of invocations over a window so you do not slam a downstream API at exactly 02:00:00. That last one is off by default; turn it on if you are scheduling at any volume.

If the scheduled workflow spins up EC2 capacity, say a fleet of recorder or transcoder instances, a fresh AWS account will hit the vCPU service quota long before it hits any Step Functions limit. Our guide to increasing the AWS vCPU quota covers the request. The same nightly-job shape works for database maintenance too, which is roughly what our automated MongoDB backup on AWS walkthrough sets up.

A full working example

Here is the recording workflow, trimmed to the parts that matter. The execution input carries a meeting ID and an end timestamp. The workflow starts a recorder, waits until the meeting is scheduled to end, stops it, and stores the result.

{
  "Comment": "Start a recorder, wait until the meeting ends, stop it",
  "QueryLanguage": "JSONata",
  "StartAt": "StoreRequest",
  "States": {
    "StoreRequest": {
      "Type": "Pass",
      "Assign": {
        "meetingId": "{% $states.input.meetingId %}",
        "endsAt": "{% $states.input.endsAt %}"
      },
      "Next": "StartRecorder"
    },
    "StartRecorder": {
      "Type": "Task",
      "Resource": "arn:aws:states:::lambda:invoke",
      "Arguments": {
        "FunctionName": "arn:aws:lambda:us-east-1:123456789012:function:startRecorder",
        "Payload": { "meetingId": "{% $meetingId %}" }
      },
      "Assign": { "recordingId": "{% $states.result.Payload.recordingId %}" },
      "Retry": [
        {
          "ErrorEquals": ["Lambda.ServiceException", "Lambda.TooManyRequestsException"],
          "IntervalSeconds": 2,
          "MaxAttempts": 4,
          "BackoffRate": 2
        }
      ],
      "Next": "WaitForMeetingEnd"
    },
    "WaitForMeetingEnd": {
      "Type": "Wait",
      "Timestamp": "{% $endsAt %}",
      "Next": "StopRecorder"
    },
    "StopRecorder": {
      "Type": "Task",
      "Resource": "arn:aws:states:::lambda:invoke",
      "Arguments": {
        "FunctionName": "arn:aws:lambda:us-east-1:123456789012:function:stopRecorder",
        "Payload": {
          "meetingId": "{% $meetingId %}",
          "recordingId": "{% $recordingId %}"
        }
      },
      "Catch": [
        {
          "ErrorEquals": ["States.ALL"],
          "Next": "AlertOnFailure"
        }
      ],
      "Output": "{% $states.result.Payload %}",
      "End": true
    },
    "AlertOnFailure": {
      "Type": "Task",
      "Resource": "arn:aws:states:::lambda:invoke",
      "Arguments": {
        "FunctionName": "arn:aws:lambda:us-east-1:123456789012:function:alertOps",
        "Payload": {
          "recordingId": "{% $recordingId %}",
          "error": "{% $states.errorOutput %}"
        }
      },
      "End": true
    }
  }
}

Note $states.errorOutput in the catch handler. It is only available inside a Catch, and it is how you get the actual error message into your alert instead of a generic "workflow failed" page at 2am.

The Lambda side is unremarkable, which is the point. Node.js 22 runtime:

export const handler = async (event) => {
  const { meetingId } = event;

  const recordingId = await startRecording(meetingId);

  // Whatever you return here arrives at $states.result.Payload
  return { recordingId, startedAt: new Date().toISOString() };
};

And starting an execution from your API, using AWS SDK v3:

import { SFNClient, StartExecutionCommand } from "@aws-sdk/client-sfn";

const sfn = new SFNClient({ region: "us-east-1" });

await sfn.send(new StartExecutionCommand({
  stateMachineArn: process.env.STATE_MACHINE_ARN,
  name: `recording-${meetingId}`,
  input: JSON.stringify({
    meetingId,
    endsAt: "2026-07-29T11:00:00Z"
  })
}));

That name is doing more work than it looks. Execution names must be unique within 90 days on a Standard workflow, so deriving it from the meeting ID gives you free idempotency: a duplicate API call throws ExecutionAlreadyExists instead of starting a second recorder. Catch that specific error and treat it as success. We learned this after a retrying client kicked off two recorders for the same meeting and both wrote to the same S3 prefix.

One more thing about the recording case specifically: the recorder is a headless browser that joins the call like any other participant, so it needs the same TURN relay as everyone else. If you are self-hosting that piece, our Coturn developer guide covers the deployment.

Limits and costs that bite

The numbers worth knowing before you design around this, all current as of July 2026 and taken from the AWS quotas and pricing pages:

• 256 KB payload between states. Standard and Express both. If a Lambda returns a big result set, write it to S3 and pass the object key instead. The pattern has a name, claim check, and you will end up using it eventually.
• Variables: 256 KiB for a single variable, 256 KiB combined per Assign block, and 10 MiB total across an execution. Variable names cap at 80 characters.
• Wait: one year on Standard, five minutes on Express. This alone rules Express out for most scheduling work.
• Standard pricing: $0.000025 per state transition in us-east-1, with 4,000 free per month. A six-state workflow run 10,000 times a month is 60,000 transitions, so about $1.40. Step Functions is rarely the expensive part of a serverless bill.
• Express pricing: $1.00 per million requests plus $0.00001667 per GB-second. Much cheaper at high volume, but no five-minute-plus waits and weaker execution history.

If you are starting from scratch today, write new states in JSONata, keep the optimized Lambda integration, set OutputPath or an Output expression on every Lambda task so downstream states see a clean object, and put recurring schedules in EventBridge Scheduler rather than a workflow that loops forever. That combination has held up well for us, and it is a much shorter list of things to remember than the JSONPath-only version was.

Related Articles

How to increase AWS vCPU quota - Developer Guide

Welcome to our comprehensive guide on maximizing your AWS experience by increasing quota limits. If you've ever felt the frustration of hitting service limits in your AWS account, you're alone. But why do these limits exist in the first place? They serve a crucial purpose: preventing inadvertent resource over-provisioning and safeguarding services from potential abuse by limiting API request rates.

By Hiruna Kumara • Meetrix.io

How Easily Backup - Restore MongoDB in AWS as 1,2,3: Automated daily backups.

A comprehensive guide to easily backup and restore MongoDB databases in AWS, including automated daily backups for robust data protection.

By Kelum Sampath • Meetrix.io

Meetrix Coturn - Developer Guide

Welcome to the Meetrix Coturn Developer Guide! This guide is designed to assist you in seamlessly integrating Coturn into your AWS environment. Whether you're new to AWS or an experienced developer, you'll discover step-by-step instructions, configuration details, and troubleshooting tips to ensure a smooth experience.

By Dinesh Chathuranga • Meetrix.io