01 Sep 2021

Yohan Beschi
Developer, Cloud Architect and DevOps Advocate

Accessing Private AWS Resources

To ensure maximum security most AWS resources must be created in private subnets, hence, without public endpoints. But during development, for operations and/or for administration purposes, we sometimes need to access them.

In this article we will explore the main solutions to access these resources without compromising the security and, taking into consideration the maintenance and running costs (costs may slightly vary depending on the region).

Table of Contents

GUI Bastion

Let’s start with the easiest option, a Windows Server deployed in a public subnet. The user can login to the Bastion via RDP to access Windows user interface, and therefore, to any tool that can be installed on the machine (e.g. web browsers, SQL Clients, IDEs, etc.) as if they were working on their local machine, with the advantage to be able to resolve private DNS (DNS records defined in a private hosted zone associated with the VPC into which the EC2 instance has been deployed).

To secure the bastion, we can configure the security group to only allow access from a specific IP on the RDP port (3389).

A Windows Bastion requires at least a t3.small instance type which will have a monthly running cost of $30 if the instance is up and running all the time.

Unfortunately, this solution has a lot of limitations:

RDP is known to have some security vulnerabilities.
When accessing the bastion from a company network the outgoing IP is the same for all the employees.
Provisioning a Windows Server AMI with all the required tools is not an easy task. It will take time and the same goes for any security patching or update.
By default, only two concurrent users are allowed without extra charges. To have more users we must purchase Client Access Licenses (CALs) and consequently manage users.

To have a Linux Bastion with a user interface we have few options like VNC, X2Go or Apache Guacamole and except the CALs, they share the same limitations as a Windows Bastion, adding more complexity to the provisioning, more security risks and sometimes lag makes the user experience not so great.

Amazon WorkSpaces

Amazon WorkSpaces is like a GUI Bastion but as a managed service which can be deployed in a private subnet. Each user has its own machine (Windows or Ubuntu) with a user interface, and users are managed via Simple AD or Microsoft Active Directory. We can even create custom bundles.

Unfortunately:

Amazon WorkSpaces comes at a price which is either a $30 flat-fee per user per month or $7.25 per user per month + $0.26 per hour on-demand for the bare minimum (2 vCPU, 4 GB Memory, root volume of 80 GB and user volume of 10 GB).
Creating a bundle cannot be fully automated.
Managing users and WorkSpaces is time consuming.
WorkSpaces are sometimes quick slow even with a good internet connection.

Self-Managed VPN

A VPN can be used in two ways:

to secure public resources, all traffic is redirected to the VPN and the VPN outgoing IP is used in the resources’ security group;
to have the users’ local machines part of our private network, only DNS resolutions (public and private) and access to specific routes (usually private subnets CIDRs) will go through the VPN.

A VPN does not need more than a t3.micro instance type (around $10/month) for a dozen users.

We can even have an architecture to allow users to request temporary credentials. Unfortunately, this kind of setup add more complexity, more resources to manage and therefore, more elements to secure. Considering that a VPN, by itself, is already extremely complex to configure correctly.

Moreover, with a VPN, if users’ local machines become part of our private network it increases the surface of attack, adding components we have almost zero control over. A compromised machine (with a Trojan for example) could try to access the whole private network without the user knowledge.

OpenVPN Cloud

An alternative to self-managed VPNs is to use OpenVPN Cloud, which offers 3 concurrent connections for free. Unfortunately, we can only use OpenVPN Cloud as a secured proxy in front of our public resources (we won’t be able to directly access our private network). Furthermore, with OpenVPN Cloud we do not have static IPs out of the box. To do so, we need to configure a connector which is basically an EC2 with an EIP (Launch Connector on AWS). But this means having an additional resource and EIPs dynamic attachment to manage to ensure high availability.

To have more than 3 concurrent users it will cost around $75/month (OpenVPN Cloud pricing).

AWS VPN Client

The last, but not least solution, as it is the most secure one if we want a VPN, is to use AWS VPN Client, which creates dedicated Elastic Network Interfaces in each subnet we need access to.

Unfortunately, AWS VPN Client is not cheap and not everyone can afford it. Each AWS Client VPN endpoint association (one in each subnet) cost $0.10 per hour (around $73/month) and like NAT Gateways these associations cannot be stopped/paused when not used. Which means that a regular infrastructure with 2 private subnets in 3 Availability Zones for 2 environments costs almost $900/month. To which we need to add the clients’ connections (around $8/month per user for 8 hours a day for 20 days).

But even with a managed service, even if we delegate the VPN configuration to the vendor, it still does not change the fact that our local machine becomes part of our private network and users machines security is part of our responsibility.

SSH Bastion

An SSH Bastion is the less expensive solution (with a running cost between $4 and $8 per month - t3.nano or t3.micro instance types) and the easier to setup (launching an Linux EC2 instance in a public subnet and having a mechanism to create users, retrieving public keys in S3 for example). On the other hand, it is the less versatile and reserved to technical users.

Furthermore, an SSH Bastion does not offer private DNS resolution, accessing a private web app from a browser requires port forwarding, etc.

AWS Systems Manager Session Manager

AWS Systems Manager Session Manager is like an SSH Bastion but much more secured. However, it is limited to EC2 instances. The EC2 instance can be in a private subnet and no rule in the instance security group is required.

Furthermore, with AWS Systems Manager Session Manager we can easily enable logging and auditing session activity, to track every command executed during a session.

Finally, AWS Systems Manager Session Manager allows to start actual SSH Sessions and do Port forwarding.

To connect to an EC2 instance in SSH-like mode, through the AWS Console or locally with the AWS CLI (and the Session Manager plugin for the AWS CLI) we need:

The SSM Agent installed on the EC2 instance we want to connect to
The EC2 instance must have an instance profile with minimal Session Manager permissions
The user starting a session must have the right IAM permissions to do so

Using AWS Systems Manager Session Manager is completely free, but like an SSH Bastion is primarily dedicated to technical users and have the same limitations (no private DNS resolution, etc.).

It is worth mentioning as-well that to start a session the instance ID is required and even with properly tagged resources, EC2s launched by an Auto Scaling Group require an extra step to retrieve the actual ID of the EC2 we want to access.

AWS Cloud9

AWS Cloud9 is based on a different concept and should only be used for development purposes in a development AWS account without any access to any other environment.

AWS Cloud9 is a web-based IDE (visually similar to Visual Studio Code) with an integrated terminal. If the AWS Cloud9 environment is configured without ingress access it can be deployed in a private subnet and accessed via Systems Manager (indirectly as we access the environment via an URL).

In theory, AWS Cloud9 is a great concept. From any (low spec) device with a keyboard and a mouse, we can have access to a very powerful computer. However, it has some drawbacks:

Cloud9 IDE does not offer as much capability as VSCode, JetBrain products, etc.
we always need to have an internet connection and even if we are working in the cloud, it does not mean we always need to be connected
by default, AWS Cloud9 works with temporary credentials with very limited permissions. The only way to lift this limitation is to modify the EC2 instance profile which is the same for all the Cloud9 environments started in an account (named AWSCloud9SSMAccessRole) or simply change the instance profile.

With AWS Cloud9 we only pay the underlying EC2 instance and most of the time the t2.micro instance type is more than enough. As AWS Cloud9 has an auto-paused feature the monthly cost per user is usually less than $4 per user.

Amazon AppStream 2.0

The last AWS managed service we are going to explore is Amazon AppStream 2.0. AppStream allows us to run almost any application in a web browser.

An AppStream fleet can be deployed in private subnets with no particular ingress rule on the security group. Unlike, Amazon WorkSpaces or AWS Cloud9, users do not have a dedicated instance. AppStream manages a fleet of EC2 instances in an Auto Scaling Group for which we have control over the scale-in and scale-out policies.

Furthermore, Amazon AppStream 2.0 comes with:

auditing capability
control over the clipboard (from and to the AppStream instance)
control over files upload and download
user home folder synchronization into AWS S3

Users can be easily managed with AppStream 2.0 user pool or Single Sign-on.

Before being able to use AppStream, we first need to build an image, by launching an Image Builder with a Windows Server Base image, installing the required softwares, creating an application catalogue to tell AppStream which application to stream (specifying a name, an executable path and an icon). Finally, we can create the image and even share it with other AWS Accounts.

AppStream image creation can be automated but like Amazon WorkSpaces images it is not an easy task.

For Amazon AppStream to run smoothly, even with extremely limited usage, we need at least stream.standard.medium instances (2 vCPU and 4 GB Memory) which costs $0.10/hour when used. Unused instances are instances to which no user is connected. They are considered stopped but are not free, they cost $0.025/hour.

In definitive, Amazon AppStream 2.0 is a fantastic service to ensure maximum security and limit access to graphical interfaces through a web-browser (a web-browser inside a web-browser), an SQL client, etc.

DevOps Infrastructure

This chart is a simple example of a dedicated DevOps AWS account, with one DevOps VPC per environment (AWS Account). The only objective of this setup is to avoid breaking environments isolation and therefore, security. For the DevOps VPCs to be able to access private networks we have multiple VPC Peerings which are almost free, except for cross Availability Zones data transfer.

Both Lab and Production VPCs use:

AWS CodeBuild (potentially called by a single AWS CodePipeline pipeline) to deploy/update the Infrastructure as Code, execute Database migrations and manage the EKS Cluster
Amazon AppStream 2.0 with a web-browser to access Kubernetes and Prometheus Dashboards and an SQL client to query the Database.

The Lab VPC has one additional resource, an SSM bastion (or more precisely one bastion per user) with an IAM Role attached to the EC2 instance profile which allow to assume an Administrator role in the Lab environment (the EC2 itself should not have Administrator permissions over the DevOps account). We could even deploy a single SSM bastion (one for all the users) for other environments (other DevOps VPCs) with reduced permissions to be able to execute kubectl commands.

In this example, we have a Kubernetes cluster only reachable through a private endpoint, therefore, to be able to test our infrastructure as code, managing users, namespaces, deployments, etc. everything need to be executed from a resource having access to it. But developing in a Terminal is not ideal. We need one last piece to complete our puzzle. Most IDEs allow remote SSH. All our source code will be stored on the bastion and all commands executed from it, but everything will be done from our local IDE.

As already mentioned with AWS Systems Manager Session Manager we can start SSH Sessions, but this requires a little configuration.

First, the EC2 must be launched with a tag that can be used to easily retrieve its instance ID (i.e. User: beschiyo).

USER=beschiyo
PROFILE=spikeseed-labs
REGION=us-east-1

INSTANCE_ID=$(aws ec2 describe-instances \
  --filters Name=tag:User,Values=$USER \
            Name=instance-state-name,Values=running \
  --profile $PROFILE --region $REGION --no-cli-pager \
  --query "Reservations[].Instances[].InstanceId" | jq -r '.[0]')

Next, we need to generate an SSH Key Pair (locally):

IDENTITY_FILE=bastion_demo
ssh-keygen -b 2048 -t rsa -f $IDENTITY_FILE -q -N ""

We then upload the public key into the Bastion with an SSM Command:

PUB_KEY=$(cat ~/.ssh/$IDENTITY_FILE.pub)
COMMAND="echo $PUB_KEY > /home/ec2-user/.ssh/authorized_keys"

aws ssm send-command --instance-ids $INSTANCE_ID \
  --document-name AWS-RunShellScript \
  --parameters commands="$COMMAND" \
  --profile $PROFILE --region $REGION --no-cli-pager

And finally, we add the SSM configuration for our bastion instance in our local ~/.ssh/config:

Host $INSTANCE_ID
  IdentityFile ~/.ssh/$IDENTITY_FILE\n
  User ec2-user
  ProxyCommand aws ssm start-session \
    --target %h --document-name AWS-StartSSHSession \
    --parameters 'portNumber=%p' \
    --profile $PROFILE --region $REGION

Now, within our IDE (an extension may be required) we can connect to our bastion. To make things easier, in the bastion user data we can install all the necessary development tools (e.g. git, AWS CLI v2, Terraform, etc.) and clone the repository.

All these steps can be added to a bash script and when a new bastion is launched, we only need to execute the script to have everything setup.

Conclusion

Despite all the solutions at our disposal only a few of them should be used to avoid compromising the security. The last example may be seen as overly complex for a simple project, but similarly to one AWS account per environment and one central AWS Account to share resources, having one DevOps account is another best practice that should be enforced for security and resources segregation. Adding an AWS Account slightly increase the complexity during the setup phase, but considerably reduce the operational overhead by reducing the number of resources in the environment accounts, centralizing the CI/CD pipeline(s) and even offering an easy kill switch if a DevOps resource is compromised.