Tidbit: API Gateway as a S3 Proxy – CloudFormation script with Serverless Framework

Quick helpful article to get you to setup an API gateway which acts as a S3 proxy using Cloudformation script.

This is a Tidbit, basically whenever you see Tidbit: in the title,
it means I am going to simply throw some helpful code without
explaining too much around it.

What you can use this for?

This is a cheap solution for your application to upload files via a REST API leveraging API gateways authorization options while optionally making the uploaded object available to public, you can even pull private objects via this API by passing the correct Authorization header.

Here is a simple depiction of a possible flow:

The code

I have uploaded the entire code on GitHub here.

Make sure you go through the serverless.yml file and replace stuff.

In case you don’t know what serverless framework is, I have written an article to explicitly remedy that here.

Here is the excerpt of the Cloudformation script that is available in the repository:

Resources:
    s3filesbucket:
      Type: 'AWS::S3::Bucket'
      Properties:
        AccessControl: PublicReadWrite
        BucketName: ${env:bucketNamePrefix}<your bucket name>
        VersioningConfiguration:
          Status: Suspended
        Tags:
          - Key: Environment
            Value: ${opt:stage}
        WebsiteConfiguration:
          IndexDocument: index.html
    route53BucketDNS: #this adds a DNS recordset which can be used for public viewing of objects
      Type: "AWS::Route53::RecordSet"
      Properties: 
        HostedZoneId: <hosted zone id>
        Name: ${env:bucketNamePrefix}<your bucket name>
        ResourceRecords:
          - ${env:bucketNamePrefix}<your bucket name>.s3-website-us-east-1.amazonaws.com
        TTL: 300
        Type: CNAME
    s3Proxy:
      Type: "AWS::ApiGateway::RestApi"
      Properties:
        BinaryMediaTypes: #add or substract MIME types here
          - image/png
          - image/jpg
          - image/gif
          - image/x-icon
          - application/octet-stream
        Description: Storage service for ${opt:stage} environment
        FailOnWarnings: false
        Name: ${env:apiGatewayPrefix}fileStorage #api gateway name
    s3ProxyAuthorizer: #custom authorizer for the API gateway which adds stuff to the bucket
      Type: "AWS::ApiGateway::Authorizer"
      Properties:
        AuthorizerResultTtlInSeconds: 300
        AuthorizerUri: arn:aws:apigateway:us-east-1:lambda:path//2015-03-31/functions/arn:aws:lambda:us-east-1:<your acc id>:function:${opt:stage}<auth fn name>/invocations
        IdentitySource: method.request.header.Authorization
        IdentityValidationExpression: ^Bearer.+
        Name: CommonAuthorizer
        RestApiId:
          Ref: "s3Proxy"
        Type: TOKEN
    s3ProxyAnyMethod:
      Type: "AWS::ApiGateway::Method"
      Properties:
        ApiKeyRequired: false
        AuthorizationType: CUSTOM
        AuthorizerId:
          Ref: "s3ProxyAuthorizer"
        HttpMethod: ANY
        Integration:
          Credentials: arn:aws:iam::<your acc id>:role/s3ProxyRole
          IntegrationHttpMethod: ANY
          IntegrationResponses:
            - StatusCode: 200
          PassthroughBehavior: WHEN_NO_MATCH
          RequestParameters:
            integration.request.header.Content-Disposition: method.request.header.Content-Disposition
            integration.request.header.Content-Type: method.request.header.Content-Type
            integration.request.header.x-amz-acl: method.request.header.x-amz-acl
            integration.request.path.key: method.request.querystring.key
          Type: AWS
          Uri: arn:aws:apigateway:us-east-1:s3:path/${env:bucketNamePrefix}<your bucket name>/{key}
        MethodResponses:
          - StatusCode: 200
        RequestParameters:
          method.request.header.Content-Disposition: false
          method.request.header.Content-Type: false
          method.request.header.x-amz-acl: false
          method.request.querystring.key: false
        ResourceId:
          Fn::GetAtt: 
            - "s3Proxy"
            - "RootResourceId"
        RestApiId:
          Ref: "s3Proxy"
    ApiGatewayDeploymentthisiswhatiwillreplace:
      Type: 'AWS::ApiGateway::Deployment'
      Properties:
        RestApiId:
          Ref: "s3Proxy"
        StageName: ${opt:stage}
      DependsOn:
        - s3ProxyAnyMethod

Points to note:

1. The last part of the YAML declares an API Gateway deployment which will make the deployed changes available to public, the predicament is that unless you change something in that section (and there is nothing to change there ever), Cloudformation will not detect that as a change and will never deploy your changeset even though it will update every change, this is solved by replacing the aptly named thisiswhatiwillreplace part of the name with epoch time during the build process.
2. The build is via AWS Codebuild (Get started with it).
3. Go through the entire script, edit as required, I have replaced some account specific stuff with placeholders like or etc.

The script will deploy the following:

1. The API gateway acting as the proxy and deploy it.
2. An S3 bucket corresponding to the ‘stage’ that you deploy in.
3. Create a DNS recordset in Route 53 to point to your bucket to have a custom domain name to access your public objects.

Calling the service

Put an object

PUT https://<your api id here>execute-api.us-east-1.amazonaws.com/prod?key=blah.png HTTP/1.1
Content-Type: image/png
User-Agent: Fiddler
Host: <your api id here>.execute-api.us-east-1.amazonaws.com
Authorization: Bearer <some auth token>
Content-Disposition: inline
x-amz-acl: public-read
Content-Length: 205523

<binary data here>

The header x-amz-acl: public-read will make the object available to public, to keep it private, simply omit the header.

Get a private object

GET https://<your api id here>.execute-api.us-east-1.amazonaws.com/prod?key=blah.png HTTP/1.1
Content-Type: image/png
User-Agent: Fiddler
Host: <your api id here>.execute-api.us-east-1.amazonaws.com
Authorization: Bearer <some auth token>
Content-Disposition: inline

Delete an object

DELETE https://<your api id here>.execute-api.us-east-1.amazonaws.com/prod?key=blah.png HTTP/1.1
Content-Type: image/png
User-Agent: Fiddler
Host: <your api id here>.execute-api.us-east-1.amazonaws.com
Authorization: Bearer <some auth token>
Content-Disposition: inline

If you liked this article, you can choose to follow this blog/subscribe to email alerts (floating follow button {bottom-right} or below comments in mobile) so that you know when any future posts come about.