Leaving Bastion Hosts Behind Part 1: GCP

Introduction

Any enterprise running virtual machines in the cloud needs to securely manage them, which is commonly done with Remote Desktop Protocol (RDP) or Secure Shell (SSH). One problem that arises is how to manage this access without exposing the management protocols to the internet, leaving them open to various types of attacks. Historically, it has been a best practice to implement bastion hosts to limit the exposure of the management protocols. However, there are some disadvantages to that approach. Recently, the big three cloud providers, Amazon Web Services (AWS), Google Cloud Platform (GCP), and Azure, have all released services that provide an alternative solution. We’ll be publishing a series of blog posts on these solutions, detailing the alternatives from each provider in its own blog post. The last blog post of the series will cover Netskope Private Access (NPA), which provides a Zero Trust Network Access (ZTNA) solution that is easy to deploy and can secure management access across all three providers.

In this post, we’ll first review what bastion hosts are, what the difficulties are with them, and then present the general model that all of the alternative solutions follow. Finally, we’ll examine the GCP services, OS Login, and Identity-Aware Proxy (IAP) in more detail to show how they can be used as an alternative to bastion hosts.

Bastion Hosts

Before looking at the alternatives, we will review what bastion hosts are and the general issues enterprises have with them.

What are bastion hosts?

Bastion hosts are computers that are deliberately exposed on a public network to enable access to a private network. Once a user has connected to the bastion host, they are able to access additional virtual machines that are not accessible from the internet. Because they are prone to attacks, bastion hosts should be appropriately hardened. These hosts should also be logging SSH sessions and sending the logs to a centralized repository.

Bastion hosts are usually hosted in separate subnets from the rest of your internal infrastructure, so the networks can be segmented. The bastion hosts are located within a publicly available subnet, while the more sensitive virtual machines are hosted in private subnets.

Why run bastion hosts?

Bastions provide security against brute force attacks on your sensitive virtual machines, by removing the need to open the SSH port to the internet. They allow an enterprise to consolidate access and reduce the attack surface of their infrastructure. Monitoring can also be consolidated and enhanced for sensitive workloads, so alerts attributed to bastions may be addressed faster than other alerts in the organization’s security monitoring tools.

Problems with Running Bastions

The problem with running bastion hosts is that it’s additional infrastructure you have to maintain yourself. Patching, monitoring, and keeping them running creates more load on your administrators. If you need to support a large user base, then you may need to run a lot of bastion hosts, and those all just add to your cloud provider bill.

In addition, there is a feature of SSH called SSH multiplexing, which may be used to attack your organization. It could allow a malicious actor to pivot from a user’s compromised laptop to your servers in the cloud. If you are able to use an alternative to bastion hosts, which we’ll discuss in more detail later in this post, then you can effectively eliminate the possibility of SSH multiplexing as a threat.

Another drawback of bastion hosts is that you must still manage the SSH keys. You may have a separate solution for this, but the use of bastions does not directly help with this problem.

General Overview of Alternatives

All three of the major cloud providers have created services that will give you an alternative to managing your own bastions, and, in some cases, they provide more than one alternative. This blog post series will not present all alternatives. It is focused on the alternatives that generally take the following approach:

• Provide a cloud service that users will access with the cloud providers’ identity and access management (IAM) credentials.
• Once authenticated, the cloud service typically creates an encrypted tunnel with port forwarding, which runs SSH or RDP for the user.

The benefits of this general approach include:

• Public IP addresses are not required in order to access the virtual machines.
• It eliminates the possibility of compromising an entire organization with SSH multiplexing attacks.
• In some cases, disabling a user’s IAM credentials also removes SSH or RDP access.
• Cloud audit logs will capture metadata for RDP or SSH sessions, and in some cases, full session logs are easy to collect through the provider’s service.

Presentation of the Alternatives

Since the implementation in each cloud provider is so different, we will break down information about each solution into the following categories:

1. Networking Considerations
2. Virtual Machine Configuration
3. Identity Management
4. Logging

GCP: OS Login and IAP

In GCP, there are two different services that provide functionality around access to your virtual machines (VMs). Using both together makes it very easy to decommission (or avoid creating) your own bastion hosts. The first service, OS Login, links SSH credentials to each Google identity. This allows GCP admins to easily manage access to VMs at either an instance or project level with IAM permissions. The second service, Identity-Aware Proxy (IAP), authenticates Google identities and can be configured to provide TCP forwarding for SSH or RDP access.

Networking Considerations

• You must first enable IAP in the GCP project that contains the VMs you want the users to access. As with other services offered by GCP, the APIs are not enabled by default.
• You must configure firewall rules to allow SSH or RDP traffic from the IAP address block, and apply that rule to the target VMs.

There is no need to assign Public IP addresses to the VMs, and the firewall rules don’t need to allow any traffic from the internet.

Virtual Machine Configuration

Users can apply permissions to allow connections from the IAP at the individual VM level (or it could be applied at a higher level). However, there are no other changes required to the VMs themselves to enable IAP.

In many cases, there are no VM changes required to use OS Login. However, there are two particular situations where additional changes may be required at the VM level:

1. If custom images (that are not built upon a standard image from GCP) are being used for the VMs. In this case, you must install the Guest Environment.
2. If the administrator of the GCP environment has not enabled OS Login at the project level. In this case, you must enable OS Login on each instance.

Identity Management

For OS Login, you must grant one of the following roles (or another role containing the same permissions) to users who should be able to access your VMs:

• roles/compute.osLogin (for regular access)
• roles/compute.osAdminLogin (for administrator level access)

Each user has the option of adding SSH keys to their identity for use with OS Login. However, if there is no key associated, GCP will generate ephemeral keys for the user. This is a great benefit, as you don’t have to implement a separate solution to manage SSH keys.

Once OS Login has been enabled, it’s very easy to enable Two-Factor Authentication (2FA) for SSH. If you already use GSuite, it can leverage the 2FA that’s already there, you just need to enable it in GCP. This can provide some great value for free if you are a GSuite customer.

For IAP, you must grant the following role (or another role that contains the same permissions) at the project or instance level: roles/iap.tunnelResourceAccessor. You may even specify the ports (such as port 22 for SSH) that the user is allowed to access on the VMs.

Logging

Without any additional configuration, the IAP service logs metadata about the SSH connection. The IAP authorization events for SSH tunneling are sent to the cloud audit logs in Stackdriver, so admins are able to browse these events right from the Stackdriver console, or wherever those logs are centralized. For users who are new to GCP and the logs, it may be a little difficult to find these events. There are generally two types of events generated by users in “cloudaudit” logs within GCP:

• Admin Activity – Examples include where someone has created a resource or modified permissions.
• Data Access – Examples include reading metadata or accessing data in object storage that requires authentication.

When attempting to tunnel into a VM with SSH, the IAP is requested to authorize the user. Information about this request, called “cloud.security.gatekeeper.AuthorizeUserRequest”, is actually sent to the Data Access logs. The log entries for these events provide a lot of useful information, including:

• The primary email of the Google identity that made the request
• The destination IP address and port (if no public IP is associated, you will see the RFC 1918 address)
• The instance ID of the target VM
• The IP address of the originating request
• Timestamp of the authorization
• If the authorization was granted or denied

Since these are only authorization events, you will not see any more details about the SSH session, such as what users did while they were connected, how many bytes were transferred, or when the session ended. If you require more of this type of information, you can install the Stackdriver agent on your VMs to send this information to Stackdriver. 

Conclusion

Using a combination of GCP’s OS Login and IAP will give you a very effective way to provide the protection of bastion hosts without having to manage much on your own. In particular, it provides the following advantages:

• The target VMs do not require external IP addresses and are not required to allow any ingress traffic from the internet.
• Since the SSH keys can be tied to a Google identity, you can avoid managing SSH keys manually.
• It’s very easy to enable 2FA for SSH.
• You are automatically provided simple information about who’s successfully accessing the VMs in the cloud audit logs.
• The time to set this up in GCP is also very minimal, making it a great solution for organizations that may not have a lot of experience with GCP.

Some disadvantages include:

• At the time of writing, OS Login is not supported by:
  • Google Kubernetes Engine (GKE) clusters
  • Fedora CoreOS images
  • Windows Server images
  • SQL Server images
• It will take some additional effort to obtain granular details (such as a record of console commands) from the SSH sessions once a user has been authenticated.

If you are using VMs in GCP, we believe it would be worth your time to evaluate if OS Login and IAP will solve your access use cases before deploying your own bastion hosts.