This content originally appeared on Bits and Pieces - Medium and was authored by Ashan Fernando
How Composable Architecture Will Help AWS Lambda Development at Scale
AWS Lambda stands out as a great tool for building scalable serverless applications. Event-based scaling, usage-based pricing, and the least management overhead make it popular among developers.
However, one of the main challenges with AWS Lambda is its development ecosystem. Though AWS Lambda provides a user interface to upload the Lambda artefacts, it's hardly used in day-to-day development. Developers need ways to source control Lambda code and their configuration and iterate fast on writing to testing code.
Over the last several years, tools and frameworks like the Serverless framework, AWS SAM, CDK, and Pulumi have improved the developer experience somewhat.
Elegance in Modern Infrastructure as Code
I will use Pulumi as an example to illustrate this. Check the following code, which creates an S3 bucket and a Lambda function triggered when a file is uploaded to the bucket.
import * as aws from "@pulumi/aws";
// Create an S3 bucket.
const docsBucket = new aws.s3.Bucket("docs");
// Create an AWS Lambda event handler on the bucket using a magic function.
docsBucket.onObjectCreated("docsHandler", (event: aws.s3.BucketEvent) => {
// Your Lambda code here.
});
Once you run pulumi up command ensures the resources are created and the code is uploaded to the function. Behind the scenes, it will ensure that the relevant IAM roles are attached to the function and register the trigger for the Lambda.
Let’s look at another example of creating an API Gateway and a Lambda function trigger.
import * as apigateway from "@pulumi/aws-apigateway";
import { APIGatewayProxyEvent, Context } from "aws-lambda";
const api = new apigateway.RestAPI("api", {
routes: [
{
path: "/api",
eventHandler: async (event: APIGatewayProxyEvent, context: Context) => {
// Your Lambda code here.
},
},
],
});
These examples are quite elegant and easy to understand. Yet they pose multiple challenges when developing Lambda practice.
- The magic functions in Pulumi make isolating code for unit testing difficult.
- Uploads the entire node_module directory with each Lambda function code, increasing its size over time.
- Reusing code between Lambda functions is difficult, and once shared code is modified, we need to test each dependent function in isolation that doesn’t scale well.
- Creating a modular code within Lambda functions isn’t empowered by the IaC framework.
This is where the need for a build system arises. Using Bit, we can break the IaC monolith into independent components and provide explicit instructions on how each component needs to be built using Envs.
This is ideal for Lambda functions, where we can bundle the Lambda code before deploying it. Besides, we can also structure the Lambda code into more granular components that can be reused across functions and projects.
Making Lambda Composable
Making a Lambda function composable is straightforward. To fulfil this, we need to follow two principles.
1. Separate Handler Code from Lambda Configuration
The first step to make Lambda composable is to separate its infrastructure configuration from its code. To facilitate this with Pulumi, switch from magic functions to Lambda functions as resources.
// Pulumi example of using function as resources
const docsHandlerFunc = new aws.lambda.Function("dateHandler", {
runtime: "nodejs18.x",
role: docsHandlerRole.arn,
handler: "index.handler",
// Upload the code for the Lambda from the "./date-lambda" directory.
code: new pulumi.asset.AssetArchive({
".": new pulumi.asset.FileArchive("./date-lambda")
})
});
As you can see, here we define a directory path for the code instead of directly writing the handler logic inline. The lambda code inside the count-lambda looks as follows.
import { APIGatewayProxyHandler } from 'aws-lambda';
import { dateUtil } from '@bit-pulumi-lambda/demo.utils.date-util'
export const handler: APIGatewayProxyHandler = async () => {
const dateType = process.env.DATE_TYPE || "";
const message = `Hey!, ${dateType} is ${dateUtil(dateType)}`;
return {
statusCode: 200,
body: JSON.stringify({ message }),
};
};2. Dependencies of Lambda Should be Packages or Bit Components
As the next step, each dependency of the Lambda code should be an NPM package or Bit Component. This makes the Lambda code composable and reusable across other functions.
If you look at the date-lambda, you can see that it uses date-util component to format the date.
// date-lambda functin
import { dateUtil } from '@bit-pulumi-lambda/demo.utils.date-util'
If we look at the dependency graph of date-lambda in bit.cloud, this becomes clearly visible.
This way, date-utils become composable and can be used by other Lambda functions.
Understanding the Big Picture
So far, we have looked at the properties of a composable Lambda function. Let’s look at an example of building an API with Lambda functions to understand how different components work together.
- Bit Remote Scope: https://bit.cloud/bit-pulumi-lambda
- GitHub Source: https://github.com/AshanFernando/bit-pulumi-lambda
Pulumi App Component
In the Bit ecosystem, we have a special app component. These type of components have a unique pipeline configuration to run and deploy the applications. Bit team have already created several sets of app types for both frontend and backend frameworks and libraries.
You can also create any custom app type if needed. We have created a new Pulumi app type, which can be used to create a new app component for your Pulumi project.
Following is an example of an app created using this app type.
import { PulumiApp } from '@bitpulumi/development.app-types.pulumi';
import * as pulumi from "@pulumi/pulumi";
import { apiRoutes } from "@bit-pulumi-lambda/demo.api-gateway";
import { countTable } from "@bit-pulumi-lambda/demo.dynamodb.count-table"
const API_NAME = "api";
const apiRouteInstance = apiRoutes(API_NAME);
export const apiUrl = apiRouteInstance.url.apply(
(url) => `${url}api`
) as pulumi.Output<string>;
export const countTableName = countTable.name;
/**
* load the application to Bit.
*/
export default new PulumiApp();This app type handles executing Pulumi commands in the background to build and deploy the app both locally and from CI.
You can define your entry points and outputs for Pulumi resources here. Pulumi.yaml can also be found inside the app component project configuration.
API Gateway Component
API Gateway component is responsible for creating API Gateway resources and connecting them with the respective Lambda functions. You can pass an array of Lambda resources as routes.
import * as apigateway from "@pulumi/aws-apigateway";
import bitpulumi from "@bit-pulumi-lambda/demo.awsx.lambda";
import { countTable } from "@bit-pulumi-lambda/demo.dynamodb.count-table";
export function apiRoutes(endpointName: string) {
const api = new apigateway.RestAPI(endpointName, {
routes: [
{
path: "/api/date",
method: "GET",
eventHandler: new bitpulumi.awsx.Lambda(
"date-lambda",
require.resolve("@bit-pulumi-lambda/demo.lambdas.date-lambda"),
{
environment: {
variables: { DATE_TYPE: "today" }, // Optional environment variables
},
}
),
},
// Define more API Routes
],
});
return api;
}
You can find a complete set of routes created in this API Gateway component.
Lambda Component
Let’s look at how we can define a Lambda function component and its configuration.
1. Function Configuration
As seen in the API Gateway component, the Lambda configuration (e.g environment) is defined with the API Gateway Lambda mapping object.
eventHandler: new bitpulumi.awsx.Lambda(
"date-lambda",
require.resolve("@bit-pulumi-lambda/demo.lambdas.date-lambda"),
{
environment: {
variables: { DATE_TYPE: "today" }, // Optional environment variables
},
}
),
In addition to that, you can define several configurations including runtime, memorySize, layers etc.
2. Function Handler
This component contains your Lambda code. If you use TypeScript to write Lambda code, it's like a standalone TypeScript project. The main difference is that Bit manages the component's dependencies and is exported to bit.cloud. You can also write unit tests for each Lambda function by using its spec file.
You can pass values to the Lambda function using its event parameters and environment variables. In the following example, the API Gateway component passes the DynamoDB table name created in AWS to the Lambda function using environment variables.
The beauty of this architecture is that you can also create a separate component for the Count DynamoDB table, where its actual name is found after it is created in AWS. Pulumi manages the order of resource creation by looking at the dependencies, first creating a DynamoDB table and then creating the Lambda function by setting the table name in its Environment variables.
// @component: DynamoDB Table
import * as pulumi from '@pulumi/pulumi';
import * as aws from '@pulumi/aws';
export const countTable = new aws.dynamodb.Table('count-table', {
billingMode: 'PAY_PER_REQUEST',
attributes: [
{
name: 'sessionId',
type: 'S',
},
{
name: 'count',
type: 'N',
},
],
tableClass: 'STANDARD',
hashKey: 'sessionId',
globalSecondaryIndexes: [
{
name: 'count-index',
hashKey: 'sessionId',
rangeKey: 'count',
projectionType: 'ALL',
},
],
});
// @component: API Gateway
import * as apigateway from "@pulumi/aws-apigateway";
import bitpulumi from "@bit-pulumi-lambda/demo.awsx.lambda";
import { countTable } from "@bit-pulumi-lambda/demo.dynamodb.count-table";
export function apiRoutes(endpointName: string) {
const api = new apigateway.RestAPI(endpointName, {
routes: [
{
path: "/api/count",
method: "GET",
eventHandler: new bitpulumi.awsx.Lambda(
"count-lambda",
require.resolve("@bit-pulumi-lambda/demo.lambdas.count-lambda"),
{
environment: {
variables: { COUNT_TABLE: countTable.name }, // Optional environment variables
},
}
),
},
// ...
If we pass any HTTP parameters for the GET method, they will be available in the events object.
3. Lambda Env (Bundler)
In the Bit ecosystem, we can create a development environment that can be assigned to different components. We can define the component’s build pipeline and configurations using an Env. We have created a new Env for Lambda functions.
This Env uses Esbuild to bundle its source, reducing the code size uploaded to Lambda. This allows the Lambda function to benefit from faster loading times. If you plan to customize the bundle, you can fork the Env into your workspace and modify it as required. By default, it uses the following configuration.
{
"entryPoints": ["index.ts"],
"bundle": true,
"platform": "node",
"target": "node18",
"external": [],
"outfile": "../dist/index.js",
"minifyWhitespace": true,
"minifyIdentifiers": false,
"minifySyntax": true,
"sourcemap": false
}4. Function Wrapper
In the API Gateway configuration, you have seen a special class named bitpulumi.awsx.Lambda. This wrapper component makes it easier to map Lambda function code with a minimal set of parameters.
{
path: "/api/date",
method: "GET",
eventHandler: new bitpulumi.awsx.Lambda(
"date-lambda",
require.resolve("@bit-pulumi-lambda/demo.lambdas.date-lambda")
),
}Internally, it maps the bundler output from the Lambda Env into the Pulumi file archive.
code: new pulumi.asset.AssetArchive({
".": new pulumi.asset.FileArchive("./date-lambda")
})You can find the Function Wrapper component here.
5. Lambda Generator
Generators are types of components you can use to define a template to create new components. The Lambda generator serves the exact same purpose. It helps you create new Lambda components by running a simple command in your workspace.
Node Module Components
Another type of component in the example project is the utility component. These are nothing more than plain JavaScript modules that can be imported into Lambda functions.
The main benefit of this component is that you can reuse them across different Lambda functions.
In a real-world project, you can define these components' authentication, validation, and reusable business logic.
Creating your Lambda Composable Project
Let's look at how you can create your first Lambda composable project structure.
1. Create a Remote Scope
First, you can create a new remote scope by logging into bit.cloud. Let’s assume its my-pulumi-project.demo.
2. Create a New Workspace
You can create a new workspace by using the following Bit command.
bit init --default-scope my-pulumi-project.demo
This will create a workspace.jsonc file and a package.json in your workspace.
3. Fork the App Component
After that, you can fork (take a copy) of the app component into your workspace by running:
bit fork bit-pulumi-lambda.demo/pulumi-app
You must set the following environment variables for your App component to function.
export PULUMI_STACK_NAME=dev
export PULUMI_ACCESS_TOKEN=<my-token>
export AWS_ACCESS_KEY_ID=<my-key>
export AWS_SECRET_ACCESS_KEY=<my-secret>
Note: To get a PULUMI_ACCESS_TOKEN, you need to create a Pulumi account. You can update the stack name by modifying the Pulumi.<stack>.yaml available inside your app component.
To test whether everything works together, use the command line to the app component path in your workspace and run the following command.
bit compile
pulumi up
Note: You can run any Pulumi command by navigating to the app component path.
4. Fork the API Gateway Component
Next, you can fork the API Gateway component into your workspace.
bit fork bit-pulumi-lambda.demo/api-gateway
After that, you need to update the app component references in the API Gateway component to point to the newly forked component.
// @file pulumi-component/pulumi-app.bit-app.ts
import { apiRoutes } from "@bit-pulumi-lambda/demo.api-gateway" // Old (Need to Modify)
import { apiRoutes } from "@my-pulumi-project/demo.api-gateway"; // New Updated
The original demo contains multiple Lambda functions. Now, you can modify the API Gateway component and fork the required Lambda functions.
5. Generating New Lambda Functions
When you want to create new Lambda functions, you can use the Lambda component generator.
To do that, first update your workspace.jsonc file with the following.
"teambit.generator/generator": {
"envs": [
"bitpulumi.development/envs/lambda-env"
]
},After doing this, run the following commands.
bit install
bit templates
This should show you aws-lambda template is available in your workspace.
By running the following command, let’s create a new Lambda function component using the generator.
bit create aws-lambda my-lambda-function
This will create a new my-lambda-function directory with files that you can use as a starting point.
If all is good, you can go back to the Pulumi App directory and test pulumi up and proceed with the deployment.
6. Configuring Ripple CI
Ripple CI is a CI/CD solution built specifically to build, test, and deploy components. To use it, you need to add the environment variables required by the Pulumi App component to Ripple CI secrets.
PULUMI_STACK_NAME
PULUMI_ACCESS_TOKEN
AWS_ACCESS_KEY_ID
AWS_SECRET_ACCESS_KEY
After that, once you tag the components and export them into bit.cloud, Ripple will build the components and deploy the component changes into AWS. To do so, run the following commands.
bit compile
bit tag -m "Initial set of components"
bit export
Ripple CI can detect component dependencies and build only the modified ones and their dependencies, making the entire build efficient. It can also continue the build from where it failed, reducing the overall build time.
7. Watch for Logs
When debugging Lambda functions, we usually check for the Cloud Watch logs to find any traces. Instead of logging into the AWS Console and looking for the function logs, you can open a new terminal and watch real-time logs by running the following command.
pulumi logs -f
This increases the developer's productivity, and you can deploy and test lambda functions in seconds.
Learn More
- Composable Software Architectures are Trending: Here’s Why
- Moving Beyond IaC: Infrastructure into Components
- Composable Microservices: Deploying into K8 Cluster
AWS Lambda Development at Scale: Using Composable Architecture was originally published in Bits and Pieces on Medium, where people are continuing the conversation by highlighting and responding to this story.
This content originally appeared on Bits and Pieces - Medium and was authored by Ashan Fernando
Ashan Fernando | Sciencx (2024-06-14T13:38:06+00:00) AWS Lambda Development at Scale: Using Composable Architecture. Retrieved from https://www.scien.cx/2024/06/14/aws-lambda-development-at-scale-using-composable-architecture/
Please log in to upload a file.
There are no updates yet.
Click the Upload button above to add an update.