Invoking AWS Lambda with gRPC Protobuf

This article explains how to use AWS Lambda with gRPC Protobufs. Before diving deep into the implementation details, here are brief descriptions of AWS Lambda and gRPC.

AWS Lambda — Lambda is also known as Function as a Service(FAAS) or simply “Serverless”. This service by AWS lets users define a sequence of computations as a function. Lambda saves a lot of time for developers, as they only need to write the core logic, and everything else is taken care of by the Cloud provider. This logic will automatically execute whenever a trigger event happens.

gRPC — Google's implementation of RPC protocol. Unlike traditional RPC(Remote Procedure Call), gRPC is language-independent. To attain language independence, gRPC uses Protocol Buffers or Protobufs. Protobufs takes care of translating the data to programming language-specific data types. Also, gRPC is multi-folds faster than REST API. The important reason is gRPC does not have to deal with JSON packaging and parsing during communication, which skips a significant overhead time especially in micro-service architecture settings.

Is it possible to trigger a Lambda function using gRPC?

On an abstract level, both gRPC functions and Lambda are “Remote Methods”. End-user/Client tries to invoke a method defined in a remote computer. But there are differences in the way gRPC and Lambda are invoked. To proceed further into this question, it's vital to understand the internals of gRPC and Lambda.

gRPC implementation is Client-Server architecture. If a Computer-B wants to execute a method on Computer A, here gRPC-Client(Computer-B) requests the gRPC-Server(Computer-A) over HTTP/2 to execute the remote method. So, the server will always be running waiting for requests from clients to be served.

Whereas, Lambda is based on Serverless architecture. There will be no active servers listening for requests. Instead, we can configure Lambda to execute on certain events. On such events, AWS will spawn up a lightweight Virtual Machine, load and execute the user-defined Lambda function, return the computed response, and kill the VM.

While gRPC needs an active server, Lambda is a serverless function. It is possible to make Lambda a gRPC client, but not as a gRPC server. So currently at the time of this writing, it is not possible to invoke a Lambda function using gRPC by default.

Workarounds

The major challenge here was to deal with the design differences. gRPC requires a Server, while Lambda is designed to be serverless.

It is still possible to write a Lambda function that can act as a gRPC server listening to the requests from clients(using some workarounds obviously). Once when the Lambda is triggered, the gRPC server defined in the lambda starts and it keeps on running. This may seem like a valid workaround but is not a viable solution.

The cost of AWS Lambda is computed based on the memory it consumes and the execution time. gRPC server on Lambda function essentially means it keeps on executing without termination. It will eventually be very expensive compared to setting up a gRPC server on a standalone EC2 machine itself.

Other people also tried to combine gRPC and Lambda, and there is no direct way of achieving this. This article from Coinbase Blog is well documented and shares different approaches they tried to achieve this. I'm skipping my failed attempts, as the Coinbase article itself is very extensive.

What if we write gRPC server code on Lambda, but we make sure it only runs when an event is triggered and shuts it down once the execution is complete.

Combining the Pros of gRPC and Lambda:

gRPC's main advantage is the Protobuffs. Protobuffs provide gRPC the language independence, so the client and server can be written in different programming languages, but still, they can communicate with each other. Protobufs provide an internal representation for the methods and data types. It acts as a translator and helps translate different data types across programming languages.

What if we make use of these Protobufs to communicate with the Lambda function. It doesn't have a practical use case, but it's just fun to experiment with! So go ahead with this approach.

Implementation design:

Client:

My Client is built with Scala.

1. Receive input from the User

2. Validate the input. On failure, prompt the user to provide input again.

3. Create a Protobuf message object using the user-defined inputs.

4. Convert the object to a string, then encode to bytes, and send it over to AWS Lambda API Gateway endpoint.

5. Display the response from Lambda to the user.

Lambda Server:

1. Setup API Gateway endpoint as a trigger function of Lambda invocation.

2. On invocation, read the request body from the client.

3. Decode the encoded request body, and use the resulting string to recreate the Protobuf message object.

4. Using the information from the Protobuf message object, perform computation, and return the result to the client.

We don't use the full functionality of gRPC, but only the Protobuf. This basic idea can be extended to work with any programming language or application. But we need to be careful of overhead time to encode and decode Protobuf messages.

Example Source code:

Find the GitHub repo for the project that I built to invoke the Lambda function using gRPC. The client requests a timestamp, and Lambda checks the log files for the existence of logs for that timestamp. GitHub - laxmena/AWS-Lambda-with-gRPC: AWS Lambda Implementation in Scala with gRPC and REST. *This project uses AWS Lambda to implement gRPC based request using Protobuf, built with Scala and sbt. Scala 2.13.4 SBT…*github.com

More readings:

• How to write and deploy AWS Lambda Function (Will be published soon)

• How to install python dependent libraries in AWS Lambda Environment (Will be published soon)

• Getting started with gRPC in Scala

There you have it. Thank you for reading.

Find me on Lakshmanan Meiyappan Lakshmanan Meiyappan - Graduate Research Assistant Software Developer - UIC Breathe Chicago Center©… laxmena - Overview