MongoDB With Serverless

Written by Pete Corey on Jun 6, 2016.

Last week I wrote about how excited I was about AWS Lambda’s pricing model. Fueled by that excitement, I spent some time this week experimenting on how I could incorporate Lambda functions into my software development toolbox.

As a Meteor developer, I’m fairly intimately associated with MongoDB (for better or worse). It goes without saying that any Lambda functions I write will most likely need to interact with a Mongo database in some way.

Interestingly, using MongoDB in a Lambda function turned out to be more difficult that I expected.

Leveraging Serverless

Rather than writing, deploying and managing my Lambda functions on my own, I decided to leverage one of the existing frameworks that have been built around the Lambda platform. With nearly nine thousand stars on its GitHub repository, Serverless seems to be the most popular platform for building Lambda functions.

Serverless offers several abstractions and seamless integrations with other AWS tools like CloudFormation, CloudWatch and API Gateway that help make the micro-service creation process very simple (once you wrap your head around the massive configuration files).

Using the tools Serverless provides, I was able to quickly whip up a Lambda function that was triggered by a web form submission to an endpoint. The script would take the contents of that form submission and store them in a MongoDB collection called "events":

"use strict";

import _ from "lodash";
import qs from "qs";
import { MongoClient } from "mongodb";

export default (event, context) => {

    let parsed = _.extend(qs.parse(event), {
        createdAt: new Date()
    });

    MongoClient.connect(process.env.MONGODB, (err, db) => {
        if (err) { throw err; }
        db.collection("events").insert(parsed);
        db.close();
        context.done();
    });

};

Unfortunately, while the process of creating my ES6-based MongoDB-using Lambda function with Serverless was painless, the deployment process turned out to be more complicated.

MongoDB Module Problems

Locally, I was using Mocha with a Babel compiler to convert my ES6 to ES5 and verify that my script was working as expected. However, once I deployed my script, I ran into problems.

After deploying, submitting a web form to the endpoint I defined in my project resulted in the following error:

{
  "errorMessage": "Cannot find module './binary_parser'",
  "errorType": "Error",
  "stackTrace": [
    "Function.Module._load (module.js:276:25)",
    "Module.require (module.js:353:17)",
    "require (internal/module.js:12:17)",
    "o (/var/task/_serverless_handler.js:1:497)",
    "/var/task/_serverless_handler.js:1:688",
    "/var/task/_serverless_handler.js:1:17260",
    "Array.forEach (native)",
    "Object.a.12../bson (/var/task/_serverless_handler.js:1:17234)",
    "o (/var/task/_serverless_handler.js:1:637)"
  ]
}

At some point during the deployment process, it looked like the "binary_parser" module (an eventual dependency of the "mongodb" module) was either being left behind or transformed beyond recognition, resulting in a broken Lambda function.

Over Optimized

After hours of tinkering and frantic Googling, I finally made the realization that the problem was with the serverless-optimizer-plugin. Disabling the optimizer and switching to using ES5-style JavaScript resulted in a fully-functional Lambda.

While I could have stopped here, I’ve grown very accustomed to writing ES6. Transitioning back to writing ES5-style code seemed like an unacceptable compromise.

While weighing the decision of forking and hacking on the serverless-optimizer-plugin to try and fix my problem, I discovered the serverless-runtime-babel plugin. This new plugin seemed like a promising alternative to the optimizer. Unfortunately, after removing the optimizer form my project and adding the babel plugin, I deployed my Lambda only to receive the same errors.

Webpack Saves the Day

Finally, I discovered the serverless-webpack-plugin. After installing the Webpack plugin, and spending some time tweaking my configuration file, I attempted to deploy my Lambda function…

Success! My ES6-style Lambda function deployed successfully (albeit somewhat slowly), and successfully inserted a document into my MongoDB database!

PRIMARY> db.events.findOne({})
{
        "_id" : ObjectId("5751e06e1aba0e0100313db7"),
        "name" : "asdf",
        "createdAt" : ISODate("2016-06-03T19:54:22.139Z")
}

MongoDB With Lambda

While I still don’t fully understand how the optimizer or babel plugins were corrupting my MongoDB dependencies, I was able to get my ES6-style Lambda function communicating beautifully with a MongoDB database. This opens many doors for exciting future projects incorporating Lambda functions with Meteor applications.

Check out the full serverless-mongodb project on GitHub for a functional example.


While working on this project, some interesting ideas for future work came up. In my current Lambda function, I’m re-connecting to my MongoDB database on every execution. Connecting to a Mongo database can be a slow operation. By pulling this connection request out of the Lambda handler, the connection could be re-used if several executions happen in quick succession. In theory, this could result in significantly faster Lambda functions, cutting costs significantly.

Finding explicit details on this kind of container sharing is difficult. The information that I’ve been able to find about it is incomplete at best, but it’s definitely an interesting area to look into.

Anatomy of an Assessment

Written by Pete Corey on May 30, 2016.

I’ve been quitely offering Meteor security assessments as a service for nearly a year now. In that time, I’ve worked with some amazing teams to analyze and assess the state of security in their Meteor applications.

An assessment is an in-depth, hands-on dive into your Meteor application. The goal of each assessment is to tease out any weak points that may leave your application susceptible to malicious users and outside attackers.

Why Security?

Security is fundamental to everything we do as software creators. It is an underlying assumption that makes everything we do possible. We spend countless hours building an effective team, developing amazing software and nurturing trust with our users, but all of that falls to the floor without security.

Imagine your company is doing well. Your application is a pleasure to use, and your user base is rapidly growing. You’ve attracted investors and you’ve built yourself an amazing team.

But suddenly, everything changes. A malicious user has managed to find and exploit a severe vulnerability within your application. Their attack has negatively impacted hundreds users.

The hard earned trust between those affected users and your company vanishes instantly. Other users, when they learn of the attack, quickly begin to lose trust as well. Now, one of the first results when people google your product is a scathing TechCrunch article outlining the gory details of the attack. Soon, investors lose interest. With their lack of support and a rapidly dwindling user base, you realize that you won’t be able to make payroll this month.

The question of “why security?” is answered simply: Because everything we do depends on it.

How Do Assessments Work?

Before an assessment begins, I like to have a high-level discussion about your business, your application, and your users. This conversation lends insight into what you need and expect from an assessment. I also like to end this discussion with a quick architectural overview and a walkthrough of your application. This sets me up to get moving quickly once the assessment begins.

During the assessment, I sweep from the back-end of your application toward the front. Each assessment starts with a thorough review of any collection schemas being used, keeping a careful eye out for any type or structural inconsistencies or weaknesses that might lead to issues.

The bulk of each assessment is spent reviewing data paths between the client and the server. Meteor methods, publications, collection validators, and server-side routes are the primary targets of inspection. Each of these areas are reviewed with the following in mind:

  • Trusted fields are always being used, where applicable (e.g., this.userId).
  • All user actions are correctly authenticated and properly authorized.
  • All user provided data is thoroughly validated and sanitized.
  • User provided data is only trusted when appropriate.
  • Data is not being inadvertently or unexpectedly leaked to the client.
  • The risk of “Denial of Service” attacks are mitigated through proper error handling, rate limiting, and unblocking, when appropriate.

Next, attention shifts to the front-end of the application. I review the application’s Content Security Policy, investigate potential avenues for front-end attacks, and look for leaking secrets and configuration values.

Lastly, I run the project’s Meteor and Node.js dependencies through several automated scanners (Package Scan, NSP, and Snyk) looking for known vulnerabilities. As the results of these scans are sometimes only applicable in specific circumstances, I review the results and determine if they pose a threat to your application.

What Can I Expect From an Assessment?

While the most apparent goal of an assessment is to find vulnerabilities within your Meteor application, my real motivation is to help you build confidence around the security process.

It’s my hope that you leave the assessment with a more thorough understanding of Meteor security and how it applies to your application. In all of my communications, I try to include as much, if not more, “why” than “what”, in an attempt to equip you with the knowledge required to keep your application secure once I leave.

The final deliverables of an assessment include an in-depth report discussing the scope of the assessment, the methodologies used, an overview of each finding, and my final thoughts and suggestions in regards to your immediate course of action. The overview of each finding includes the finding’s severity, a brief description outlining why the issue exists and examples of how it can be exploited, a summary of the finding’s impact, and steps for remediation.

I like to present this final report in either a screen-sharing session, or an in-person meeting so that we can discuss the results in-detail. It’s my goal that you leave this final meeting with a complete understanding of everything in the report, along with a clear path forward.

Take a look at an example assessment report to get a clearer picture of what to expect.

Interested in an Assessment?

If you value the security of your Meteor application, I’d love to hear from you. Enter your email address below and I’ll send you a short questionnaire about your application. From there, I’ll reach out and start a discussion.

I’m looking forward to hearing from you!

AWS Lambda First Impressions

Written by Pete Corey on May 24, 2016.

Lately, I’ve been paying quite a bit of attention to AWS Lambda.

Lambda is an Amazon Web Service designed to run small pieces of code in response to external stimuli (an endpoint is hit, a document is inserted into a database, etc…). The beautiful thing about Lambda is that your code is designed to run once, and you’re only charged for the amount of time your code is running.

A Node.js Script

To make things a little more concrete, let’s talk about my first baby-steps into working with Lambda.

I have a script-based tool that automates Bitcoin lending on the Poloniex exchange. Pre-Lambda, I implemented this tool as a Node.js script that spun up a local server, and executed a job every 15 minutes to “do stuff” (💸 💸 💸).

I wanted to move this script off of my local machine (mostly so I could close my laptop at night), so I began investigating my hosting and pricing options. On the low end of things, I could spin up a small DigitalOcean droplet for five dollars per month. Not bad, but I knew I’d be unnecessarily paying for quite a bit of idle server time.

I even considered buying a Raspberry PI for around forty dollars. I figured the upfront-costs of buying the device would be payed for within a year. After that initial investment, the power requirements would be negligible.

Meets AWS Lambda

Finally, I found Lambda. I quickly and painlessly modified my Node script to run once, manually deployed it to Lambda, and added a schedule trigger to run my script once every fifteen minutes.

Fast forward past a couple hours of fiddling and my script was working!

After monitoring my script for several days, I noticed that it took between one to two seconds to execute, on average. I added an execution hard-stop duration of three seconds to my Lambda function. With that, I knew that I would be charged for, at most, three seconds of up-time every fifteen minutes.

Using that data and Lambda’s pricing sheet, I calculated that at three seconds per execution with an execution every fifteen minutes, the yearly cost for running my script was, at most, at just under twenty two cents zero dollars.

I was shocked. $0.22/year! Thanks to Lambda’s free tier, hosting my script was free! Comparing that to DigitalOcean’s $60/year, or a Raspberry PI’s upfront cost of $40+ dollars, I had a clear winner.

Looking Forward

My first introduction to AWS Lambda left me impressed. Further research has left me even more excited. The possibilities of an scalable on-demand, event-driven infrastructure seem very attractive.

While I’m not totally re-assessing my software development stack, I’m definitely making a little room for Lambda. I’m already thinking about how I could have used it in the past to build more elegantly engineered, and cheaper solutions.