AWS: DynamoDB + API Gateway – The Correct Way

I went through the hard way so you don’t. This is not a tutorial but more of a post to skip all step to fetch and clean data with DynamoDB + API Gateway.
Before I say anything, I want to thank aws for creating this and second scrutinize them to not give proper documentation.

I was working on a project that would load data inside a DyanmoDB database periodically (like once a month of same/updated) data. Everything went great until I connected the dynamodb to an API gateway to act as a proxy.

1] Formatting DynamoDB data to normal JSON in AWS API Gateway

So if you manage to create an API endpoint to hit for dynamo db and see some weird JSON data, “YOU ARE NOT ALONE”. I hope someone sends this message to us from proxima centauri. Either way, there are a lot of posts and documentation on how to clean the data up but none of them has a direct way, some have JS libraries, some have datatype conversions etc etc. You don’t need a lamda to clean up JSON, you can clean it up directly in the API.

Reference: Formatting DynamoDB data to normal JSON in AWS Lambda

So we solve this issue directly inside aws API gateway using mapping templates.

Open the api gateway > Select the resource > Method Execution (Integration Response) > Mapping Templates

#set($inputRoot = $input.path('$'))
{
    "project_id": "$inputRoot.Items[0].project_id.N",
    "project_proto": {
#foreach($element in $inputRoot.Items[0].project_proto.M.keySet())
  "$element": "$inputRoot.Items[0].project_proto.M.get($element).S"
    #if($foreach.hasNext),#end
    
#end
    },
    "project_dependencies": {
#foreach($element in $inputRoot.Items[0].project_dependencies.M.keySet())
  "$element": "$inputRoot.Items[0].project_dependencies.M.get($element).S"
    #if($foreach.hasNext),#end
#end      
    }
}

The above example retrieves one row of data from the table. The primary key is project_id and retrieves two more columns of data which are maps. So a JSON like this…

{
  "Count": 1,
  "Items": [
    {
      "project_proto": {
        "M": {
          "someservice": {
            "S": "0.1.1"
          },
          "google": {
            "S": "1.3.1"
          }
        }
      },
      "project_dependencies": {
        "M": {
          "@something/some-service-utils": {
            "S": "^11.0.7"
          },
          "lodash": {
            "S": "^4.17.4"
          }
        }
      }
    }
  ],
  "ScannedCount": 1
}

Will become this…

{
  "project_id": "61",
  "project_proto": {
    "someservice": "0.1.1",
    "google": "1.3.1"
  },
  "project_dependencies": {
    "@something/some-service-utils": "^11.0.7",
    "lodash": "^4.17.4"
  }
}

2] How to get entire DynamoDB data in json.

The table was small. Around 1000 rows but we wanted the whole data inside the JSON and not just one endpoint. Moreover, did not want to write any external lamda with some scan or query functions OR making 1000 API calls to that endpoint.

To solve this, you need to:

API Name > Method Execution > Integration Request > Mapping Templates > Create one if there isn’t any “application/json”

{
    "TableName": "ddb.devops.somedbname",
    "ProjectionExpression": "project_id, project_name, project_owner, project_language, project_url"
}

This will act as a “getAll” function but please note, this is only if the db is small. Otherwise, you should probably stick to the scan and query functions provided by AWS.

Working with Scans in DynamoDB

If there is something funky you are doing and are stuck, feel free to reach out. I’m always interested in learning and doing something new!