Bastion host security: what to check before replacing yours

Someone on your team says your bastion host is outdated and you should replace it with a zero-trust service.

Meanwhile, you've already got it behind a VPN, access is restricted, and it works with the database tools your team uses. You're going to want a better reason to replace it than "it's old".

There are good reasons to switch. Managing individual SSH keys, patching another server, and keeping track of who still has access all take work. Depending on your setup, a managed service can handle some of that for you.

Before you switch though, you need to check that your team can still do their jobs - particularly when production is broken.

Table of Contents

• What is a bastion host
• Check what needs fixing
• Work out how people will access the database
• Check what gets logged
• Try removing someone's access
• Have a way in when the usual method fails
• Test it before removing the old host

What is a bastion host

For administrative access, a bastion host is a hardened server you connect through to reach private systems. You might SSH into it to reach another server, or use it to tunnel a connection from your laptop to a database.

It doesn't have to be publicly accessible. You can put a bastion behind a VPN, or use another access service to connect to it.

Likewise, having a bastion doesn't automatically mean everyone shares an SSH key. You still choose how people authenticate and what they're allowed to reach.

When reviewing bastion host security, those are the details you need to look at.

Check what needs fixing

Start with how your current setup works. Can you tell who logged in? Do former employees still have working keys? Who patches the machine?

If people use it as a shared workspace, check what they leave behind. Database exports, private keys, and credentials in shell history are all things you'll want to avoid accumulating on a machine used to access production.

Also check what the host can reach. Access to one database shouldn't give someone an unnecessary route to every other system in the network.

You might decide to fix the existing setup, or replace it to reduce the ongoing work. For example, AWS Systems Manager Session Manager lets you open native shell sessions on managed servers without inbound SSH ports or SSH keys. You control who can connect to which servers through IAM policies.

That's useful if you're trying to stop maintaining SSH keys. You still need to configure the permissions and keep the agent working.

Work out how people will access the database

Connecting to a server's shell and connecting your database client to a private database are different tasks. Make sure the proposed replacement handles the one your team needs.

For shell access, Session Manager can use the SSM agent running on the target server.

For a private database, remote-host port forwarding uses a managed node that can reach the database. That node needs to resolve the database's hostname and connect to its port. The database itself doesn't need to be managed by Systems Manager.

In other words, the connection goes:

Your database client
  -> local forwarded port
  -> Session Manager connection to a managed node
  -> private database

You still need that middle machine, or another supported way of forwarding the connection. "We'll use SSM" doesn't explain where it comes from if all your applications run on Fargate.

AWS also supports port forwarding through an ECS task, with ECS Exec enabled and the required task permissions. You could create a temporary task for database access and remove it afterwards. You'll need to handle setup and cleanup, but you can avoid keeping a VM running for occasional connections.

Before switching, have someone try it with their usual database client. Check authentication and TLS certificate verification as well as whether the connection opens. You don't want the team figuring this out during their next incident.

Check what gets logged

Be specific about what you need from audit logs.

Knowing that someone opened a connection is different from knowing which commands or SQL statements they ran.

Session Manager supports CloudTrail records of API calls, and you can configure session output logging to S3 or CloudWatch Logs. However, AWS doesn't provide session-content logging for SSH or port-forwarding sessions through Session Manager.

So if you're forwarding a database connection, don't assume you'll get a record of every query. If your organisation requires that, you'll need database auditing or another way to capture those actions and associate them with the person who performed them.

Try a connection, perform a harmless action, and go looking for the records. It's better to discover a gap now than when someone asks who changed production last week.

Try removing someone's access

Once you've checked that someone can connect, check that you can stop them connecting too.

Use a test account with the permissions a team member would normally have. Remove those permissions and try again. Also check what happens to a session that's already open - blocking new connections doesn't necessarily end existing ones.

If you're granting temporary access, verify that it expires when you expect.

Remember the old credentials during the migration. Removing someone from the new service won't help if they can still use an old SSH key or a shared database password.

Have a way in when the usual method fails

Imagine a server starts failing and you need to log in to investigate. Unfortunately, the SSM agent has also stopped working.

If that's your only way in, you're now debugging how to access the machine before you can debug the original problem.

Replacing the instance might get the service running again. But if you need to understand what happened, you'll need either enough logs and other saved diagnostics, or another way to inspect it.

Work out how you'll handle this before removing your existing access method. Your fallback might be a separately controlled connection, a supported console, or a documented recovery procedure that replaces the machine and preserves the evidence you need.

Check what that fallback depends on. An SSH connection tunnelled through Session Manager still needs the SSM agent. It won't help when that agent is the thing that broke.

Write the steps in a runbook, including who can use emergency access, how to get the necessary credentials, and what to clean up afterwards. Have someone else try it in a test environment with the normal access method unavailable.

Test it before removing the old host

Pick someone who didn't set up the replacement and have them work through the tasks your team normally performs. They should be able to connect using their own account, run the tools they need, and find the relevant logs afterwards.

Then have them follow the emergency procedure. Fix any missing instructions or permissions before retiring the old host and its credentials.

Keep checking access after changes to permissions, networking, or agent configuration. Your application can be running perfectly while nobody can log in to administer it. Even a successful TCP check on an SSH port won't tell you whether a team member can authenticate.

Make testing access part of preparing people for on-call. It's a much better time to find out their account doesn't work than when they've just been paged.