A gotcha in running apps in AWS Fargate

Recently I’ve been migrating some ECS services to AWS Fargate. I have relatively smooth experience in dealing with apps in one VPC. However, I bumped into some undocumented problem when trying to deploy one app in a different VPC.

In short, I was trying to deploy my docker app into Fargate with tasks running in private subnets balanced by a ALB in public subnets. The resource in AWS is similar to the cloudformation at https://github.com/nathanpeck/aws-cloudformation-fargate/blob/master/service-stacks/private-subnet-private-loadbalancer.yml

CannotPullContainerError: error pulling image configuration: Get https://docker-images-prod.s3.amazonaws.com/registry…

I double checked all security groups, NATs, rout tables etc and have made 100% sure that Internet is accessible from within the private subnets.  However, it looked like ECS tasks are not able to pull docker images.

Wait…, it was not the image itself not being able to pull, it was just the configuration, and look at that URI, it is S3.  This clue led me to the solution.

It turned out that my VPC has got a VPC endpoint set up. But the route tables are not updated for the private subnets (that were added to the VPC later).

Once the route tables were updated with the VPC endpoint, the tasks passed the Load Balancer health check.

 

 

 

 

 

 

Leave a comment