Sanjay Siddhanti

Serverless Web Apps in Python

Sun, 05 Jul 2020 00:00:00 +0000

Serverless technologies have a lot of people excited - in theory, they offer infinite scaling with no maintenance. After mostly seeing “hello-world on Lambda” tutorials, I was curious how it would work for a real application with a frontend and a database.

This article reviews my experience using AWS Lambda for full-stack web applications. If you’re familiar with the theory and want to know more about how it works in practice, you’re in the right place.

Why serverless

Serverless could be a good choice for anyone who doesn’t want to manage infrastructure. Specifically, it’s a great fit for bursty workloads that need to scale in short bursts (why pay for a server to idle all day?), and for functions that need to run every time an item enters a queue.

A typical web app is none of the above, yet that’s what I cover in this post. I wanted to try deploying a web app with zero maintenance. For a side project, Heroku wasn’t worth the money and monitoring & maintaining a server wasn’t worth my time. So I gave Lambda a shot.

I built this project using Flask, ReactJS, and Postgres, and if you squint hard enough, it looks like a production system. While I can only share my experience, I believe it can be generalized to any Python backend and perhaps beyond.

Zappa

Zappa is a library that makes it easy to deploy Python applications to Lambda. Lambda expects a zip archive of your code and your Python dependencies. With a couple commands, Zappa creates the archive and much more, including the Lambda function, IAM roles, and API gateway.

Zappa hooks into your virtualenv in order to zip up the Python dependencies. Many popular libraries, such as numpy, include compiled C code for performance. When you pip install numpy, you get compiled binaries for your current environment - MacOS in my case. These binaries likely won’t work on Lambda, which runs on a Linux environment. So it’s clear that naively packaging a virtualenv won’t work. To remediate this, Zappa has a sister repository of precompiled binaries for popular Python packages and it swaps these in when building the archive. If you use a dependency with compiled binaries that isn’t on this list, it might not work on Lambda - depends on your development environment. In this case I recommend using Docker to create the archive (see below).

Zappa addresses cold starts by periodically invoking the Lambda function to keep a server warm. This fixes one of the most common complaints with serverless functions wherein requests are slow after the app hasn’t been used in a while.

Here I’ll go over how to handle a few common use cases with Zappa:

Deploy application code

More or less just follow the docs.

pip install zappa in your virtualenv (if you don’t use virtualenv, keep reading)
Run zappa init to get a zappa_settings.json file. Mine looks like this for a Flask app:

{
    "dev": {
        "app_function": "src.app.app",
        "profile_name": null,
        "project_name": "my-project",
        "runtime": "python3.7",
        "s3_bucket": "my-bucket",
        "aws_region": "us-east-2",
        "exclude": ["node_modules", "package*.json"]
    }
}

I recommend using exclude to keep the size of your archive down. Zappa supports large projects but don’t be tricked into thinking you need this, when in reality your archive is full of non-Python code.

Store secrets

Most web apps don’t exist in isolation; they need to talk to other services, such as a database, and need to store secrets to do this. If you have a small number of secrets that don’t change often, the most straightforward approach is to set them in the Lambda console.

This makes the values available in the Lambda runtime environment, so you can read them into your application like normal:

import os

DATABASE_URI = os.environ.get(
    "DATABASE_URI", "postgresql://postgres:postgres@postgres:5432"
)

If you have too many secrets and prefer to manage them programmatically, you can also write them to a file in S3 that your service reads and extracts the values from at startup.

Init / migrate the database

Now that we have the DB secrets available, we want to initialize and migrate the database. I used Amazon RDS for my DB. While it’s possible to spin up an EC2 instance and run the migration commands there, that defeats the purpose of not having to manage servers. Zappa allows you to run any function in your app using zappa invoke, so let’s use that.

First you need the function to execute in the right VPC and with the appropriate security groups that allow it to connect to the DB. I had to add this the "dev" section of my zappa_settings.json:

"vpc_config" : {
    "SubnetIds": [ "my-subnet" ], 
    "SecurityGroupIds": [ "my-security-group" ]
}

Next, run the script. I started with zappa invoke dev 'db_utils.create_schema_and_tables' to initialize the schema and tables, and zappa invoke dev 'db_utils.seed_db' to seed the DB with some testing data. You can run a similar command to migrate the DB. Make sure those functions connect to the DB using the environment variables mentioned above.

For Django projects, Zappa supports execution of management commands.

Serving static files

A common setup for web apps is to serve the frontend and backend on the same server. There’s a reverse proxy web server, such as nginx, that sits in front of the WSGI / ASGI Python server. The reverse proxy, which is very fast, will serve the static files itself and forward other requests to the Python server.

Since requests for static files never hit the Python code, these files could be served from some other location.

With Lambda, you don’t have access to the underlying web server and therefore can’t give special treatment to static files. This leaves you with two options:

Serve the static files through Python code, which may be slow
Serve the static files from a separate service, like S3 or CloudFront

Either should work, but I went with option 2 because it’s clean, easy, and performant.

There are many guides on the internet for static web hosting using S3, so I won’t repeat them here. However, I’ll zoom into a couple issues that are often left out.

Server side rendering

Single page applications (SPAs) typically send the client a Javascript bundle that creates the HTML for the app. Since not all web crawlers can execute Javascript code, the crawlers only see a barebones HTML page that says “load App.js”. This means websites that are rendered exclusively on the client will struggle with SEO, since search engines cannot index all of their content.

To improve this people use server-side rendering (SSR), where some or all of the HTML is rendered on the server. This makes the site SEO-friendly and faster for the initial page load.

Server-side rendering usually means you need a Node.js backend that can execute JavaScript code. While it looks possible to do SSR with a Python backend, I’m not sure if Lambda gives you enough control to set that up. You should investigate more before committing down this path.

Client side routing

SPAs use client-side routing to prevent full page reloads as the user navigates through the site. When a user clicks on a link to get a “new page”, the page itself remains the same but JS replaces the contents of it.

With static hosting in S3, client-side routing works mostly as expected…until you refresh the page when you aren’t on the home page. The browser sees a URL like /foo and looks for foo.html in the S3 bucket. This file doesn’t exist since the contents of the page are created by Javascript running in the browser.

There are a couple solutions: set the error document to index.html or configure the routing rules.

CORS

Since your frontend and backend live on different servers, you’ll need to enable cross-origin resource sharing on the backend server. This is straightforward with existing plugins like Flask-Cors.

Debugging

Logs are available in CloudWatch, but I recommend running zappa tail and keeping it open while developing to get the latest logs. You can also configure an exception handler function that will send unhandled exceptions to your application monitoring tool.

One word of caution: Zappa’s default is to time out if a function doesn’t return output within 30 seconds. The timeout messages are generic and easy to miss. So if you start seeing weird behavior, it’s possible something timed out. For example, I saw a post about how someone’s migration script takes >30 seconds to run, so Zappa timed it out and it didn’t finish migrating the DB. Look out for this and configure the setting accordingly.

Docker users

I like developing in Docker containers. They’re portable and I have the assurance that my development environment is exactly the same as the production environment. Related to this, I have two gripes about Lambda / Zappa:

If you aren’t developing on Linux, your development environment will be different than the runtime environment.
Although Lambda allows building the package from Docker, Zappa only works with virtualenvs. This adds an extra step as I have to create a virtualenv, whereas normally I pip install directly into the container.

As mentioned above, Zappa maintains a list of precompiled binary extensions and swaps out your binaries for Lambda-compatible equivalents. This is usually fine. However, you might find yourself using a dependency that ships with binary extensions but isn’t popular enough to make the list. Then you’ll wish for a better development environment.

Docker to the rescue! There’s a Zappa Docker Image, which is based off of the LambCI Lambda Docker Image that provides a Lambda-like environment in Docker. You can use it as your base image.

In the future, I hope Lambda will accept an arbitary Docker image. Other clouds already do this.

Conclusion

This article was a brain dump of what I learned using Zappa, and I hope it was useful for you. We talked about when to use Zappa and how to migrate a DB, store secrets, serve a frontend, and get a reproducible dev environment.

I recommend Zappa for individuals or small teams that want to focus on application code instead of infrastructure. It was great for deploying a side project to the internet and not having to monitor it. If nobody pings my web app until a year from now, it should still be there and I’ll have paid almost nothing for compute. At scale, cost becomes a consideration and managing a cluster still makes sense.

If you have any comments, message me on Twitter! I’d love to hear about your experience using Lambda and Zappa.

3 ways to test S3 in Python

Wed, 08 Apr 2020 00:00:00 +0000

At some point, every engineer has to decide whether to write tests for something or just ship the feature and move on.

Under a time crunch, I’ll often write tests for the easy things (e.g. pure functions) or write the tests that provide the biggest bang for their buck (e.g. an end-to-end integration test for the service).

Testing code that interacts with external systems, like a database or S3, requires a bit more effort. However, important business logic often happens in this code and recently I’ve become more interested in testing it.

In this post, I’ll explore three ways of testing S3 in Python.

Setup

Let’s consider a simple CRUD app for recipes, backed by S3.

from dataclasses import dataclass
import json

import boto3

S3_BUCKET = "recipes"


def get_s3():
    return boto3.client("s3")


@dataclass
class Recipe:
    name: str
    instructions: str

    @classmethod
    def get_by_name(cls, name: str):
        """Looks up a Recipe by name
        
        Args:
            name (str): Recipe name
        
        Returns a Recipe object
        """
        response = get_s3().get_object(Bucket=S3_BUCKET, Key=name)
        response = json.loads(response["Body"].read())
        return cls(response["name"], response["instructions"])

    @classmethod
    def update_instructions(cls, name: str, new_instructions: str):
        """Updates the instructions for a recipe
        
        Args:
            name (str): Name of the recipe to update
            new_instructions (str): New instructions
        """
        recipe = cls.get_by_name(name)
        recipe.instructions = new_instructions
        return recipe

    @classmethod
    def delete(cls, name: str):
        """Deletes a recipe
        
        Args:
            name (str): Name of the recipe to delete
        """
        get_s3().delete_object(Bucket=S3_BUCKET, Key=name)

    def to_json(self):
        """Serialize the recipe to json
        
        Returns:
            str: JSON representation of the Recipe
        """
        return json.dumps({"name": self.name, "instructions": self.instructions})

    def save(self):
        """Persists a recipe to S3
        """
        serialized_recipe = self.to_json().encode("utf-8")
        get_s3().put_object(Bucket=S3_BUCKET, Key=self.name, Body=serialized_recipe)

All tests below use pytest. All code is runnable and available on Github.

Option 1: moto

Moto is a Python library that makes it easy to mock out AWS services in tests. Let’s use it to test our app.

First, create a pytest a fixture that creates our S3 bucket. All S3 interactions within the mock_s3 context manager will be directed at moto’s virtual AWS account.

import boto3
from moto import mock_s3
import pytest


from recipe import Recipe, S3_BUCKET


@pytest.fixture
def s3():
    """Pytest fixture that creates the recipes bucket in 
    the fake moto AWS account
    
    Yields a fake boto3 s3 client
    """
    with mock_s3():
        s3 = boto3.client("s3")
        s3.create_bucket(Bucket=S3_BUCKET)
        yield s3

Next, we can test creating a new Recipe and fetching it.

def test_create_and_get(s3):
    Recipe(name="nachos", instructions="Melt cheese on chips").save()

    recipe = Recipe.get_by_name("nachos")
    assert recipe.name == "nachos"
    assert recipe.instructions == "Melt cheese on chips"

If we try to fetch a Recipe that doesn’t exist, an exception should be raised. This test covers that scenario.

def test_get_does_not_exist(s3):
    with pytest.raises(s3.exceptions.NoSuchKey):
        recipe = Recipe.get_by_name("sandwich")

We can also update a Recipe. This test confirms that the data is updated after save() is called.

def test_update(s3):
    old_instructions = "Melt cheese on chips"
    new_instructions = "Microwave a plate full of tortilla chips and cheese"

    Recipe(name="nachos", instructions=old_instructions).save()

    new_recipe = Recipe.update_instructions(
        name="nachos", new_instructions=new_instructions
    )

    # Nothing changes until you call save()
    recipe = Recipe.get_by_name("nachos")
    assert recipe.instructions == old_instructions

    new_recipe.save()

    # Recipe updates after saving
    recipe = Recipe.get_by_name("nachos")
    assert recipe.instructions == new_instructions

Finally, we can delete a recipe and confirm that the data in S3 disappears.

def test_delete(s3):
    Recipe(name="nachos", instructions="Melt cheese on chips").save()

    response = s3.list_objects_v2(Bucket=S3_BUCKET)
    assert len(response["Contents"]) == 1
    assert response["Contents"][0]["Key"] == "nachos"

    Recipe.delete("nachos")

    # Data in S3 is gone after deleting the recipe
    response = s3.list_objects_v2(Bucket=S3_BUCKET)
    assert "Contents" not in response.keys()

Overall, moto does a great job of implementing the S3 API. It’s easy to install, feels just like the real S3, and doesn’t require any code changes.

Option 2: Botocore stubs

Botocore stubs allow you to mock out S3 requests with fake responses.

Below is a pytest fixture that creates an S3 stub. Since other S3 clients won’t use this stub, we also need to patch get_s3 and replace its return value with the stub - thereby forcing all S3 clients in the Recipe class to use our stub.

import datetime
import json
from dateutil.tz import tzutc
from io import BytesIO
from unittest.mock import patch

import boto3
from botocore.stub import Stubber, ANY
from botocore.response import StreamingBody
import pytest

from recipe import Recipe, S3_BUCKET


@pytest.fixture
def s3_stub():
    """Pytest fixture that mocks the get_s3 function with a S3 client stub
    
    Yields a Stubber for the S3 client
    """
    s3 = boto3.client("s3")
    stubber = Stubber(s3)

    with patch("recipe.get_s3", return_value=s3):
        yield stubber

Then, we can stub out responses for the put_object and get_object S3 APIs. With those stubs in place, we can run the test that creates and subsequently fetches a Recipe.

def test_create_and_get(s3_stub):
    # Stub out the put_object response
    # Note: These stubs are incomplete - I omitted things such as
    # HTTP headers for brevity
    put_object_response = {
        "ResponseMetadata": {
            "RequestId": "5994D680BF127CE3",
            "HTTPStatusCode": 200,
            "RetryAttempts": 1,
        },
        "ETag": '"6299528715bad0e3510d1e4c4952ee7e"',
    }
    put_object_expected_params = {"Bucket": ANY, "Key": ANY, "Body": ANY}
    s3_stub.add_response("put_object", put_object_response, put_object_expected_params)

    # Create the StreamingBody that will be returned by get_object
    encoded_message = json.dumps(
        {"name": "nachos", "instructions": "Melt cheese on chips"}
    ).encode("utf-8")
    raw_stream = StreamingBody(BytesIO(encoded_message), len(encoded_message))

    # Stub out the get_object response
    get_object_response = {
        "ResponseMetadata": {
            "RequestId": "6BFC00970E62BC8F",
            "HTTPStatusCode": 200,
            "RetryAttempts": 1,
        },
        "LastModified": datetime.datetime(2020, 4, 6, 5, 39, 29, tzinfo=tzutc()),
        "ContentLength": 58,
        "ETag": '"6299528715bad0e3510d1e4c4952ee7e"',
        "ContentType": "binary/octet-stream",
        "Metadata": {},
        "Body": raw_stream,
    }
    get_object_expected_params = {"Bucket": ANY, "Key": ANY}
    s3_stub.add_response("get_object", get_object_response, get_object_expected_params)

    # Activate the stubber
    with s3_stub:
        recipe = Recipe(name="nachos", instructions="Melt cheese on chips")
        recipe.save()

        recipe = Recipe.get_by_name("nachos")
        assert recipe.name == "nachos"
        assert recipe.instructions == "Melt cheese on chips"

While botocore stubs are functional, I don’t like working with them for several reasons:

They require a lot more prep. Creating stubs is time-consuming. Even if you run the real code interactively and copy the response, some things need to be replaced - such as the StreamingBody above.
They’re fragile and fake. Responses are returned first in, first out - so if you call the S3 APIs in a different order than you added the responses, it will throw an error. If you have a bug in how you call the API, it might not be caught.
To make the stubs look somewhat realistic, you have to mock many fields that your code doesn’t care about and bloat your tests with fake responses.
They leak implementation details from the module being tested. For example, if a module switched from using s3.list_objects to s3.list_objects_v2, the test would fail because it depends on a specific API being called. This creates an unnecessary dependency on the private API of the module, instead of testing the public API.

Option 3: Localstack

A third option is localstack, which allows you to bring up an entire AWS cloud stack locally.

First, we need to bring up localstack. I choose to do this with docker-compose.

version: "3.7"
services:
  tests:
    image: s3_testing:latest
    networks:
      - app
    entrypoint:
      - /app/wait-for-it.sh
      - -t
      - "30"
      - localstack:4572
      - --
      - pytest
      - test/
    environment:
      - AWS_ACCESS_KEY_ID=fake
      - AWS_DEFAULT_REGION=fake
      - AWS_SECRET_ACCESS_KEY=fake
  localstack:
    image: localstack/localstack
    ports:
      - "4566-4599:4566-4599"
    networks:
      - app
    environment:
      - SERVICES=s3
networks:
  app:
    driver: bridge

Next, we mock get_s3 again and this time replace it with an S3 client that is connected to localstack.

from unittest.mock import patch

import boto3
import pytest

from recipe import Recipe, S3_BUCKET


@pytest.fixture
def s3_localstack():
    s3 = boto3.client("s3", endpoint_url="http://localstack:4572")
    s3.create_bucket(Bucket=S3_BUCKET)
    with patch("recipe.get_s3", return_value=s3):
        yield s3

With this mock in place, we can run the same tests that we ran with moto.

def test_create_and_get(s3_localstack):
    Recipe(name="nachos", instructions="Melt cheese on chips").save()

    recipe = Recipe.get_by_name("nachos")
    assert recipe.name == "nachos"
    assert recipe.instructions == "Melt cheese on chips"


def test_get_does_not_exist(s3_localstack):
    with pytest.raises(s3_localstack.exceptions.NoSuchKey):
        recipe = Recipe.get_by_name("sandwich")

Localstack is extremely easy to use, but it takes almost 30 seconds to spin up on my machine.

Recommendation

Both moto and localstack are very powerful and easy to work with. Both solutions do a good job of implementing the S3 API, and they also support other AWS services including EC2, RDS, Lambda, and more. They can both be used to test code in other languages in addition to Python.

Localstack is probably the closest thing to actually connecting to AWS, but for my simple use case presented above, I can’t justify the extra overhead and time required to spin up the stack. Therefore, I recommend moto as it’s the most lightweight solution that properly implements the S3 API.

For more complicated projects that are testing S3 performance, localstack could be a good choice. Botocore stubs don’t make the cut.

What did I miss?

Thanks for reading! If you have any feedback, I’d love to hear from you - follow me on Twitter or message me on LinkedIn.