Building Your Secure Foundation

#

In the world of infrastructure automation, your control box serves as the nerve center of your operations. Whether you’re orchestrating GitOps workflows, managing Terraform deployments, or coordinating Ansible playbooks, this single machine becomes the gateway to your entire infrastructure ecosystem. That’s precisely why I never skip the hardening process – a compromised control box isn’t just a security incident, it’s a potential catastrophe that could cascade across your entire infrastructure.

In the previous article, we explored the strategic considerations for where to locate your control box. Now that you’ve chosen the perfect location for your infrastructure command center, it’s time to build it properly.

Today, I’ll walk you through my systematic approach to hardening a fresh Debian 12 installation, transforming it from a default system into a secure foundation ready for serious infrastructure work. This isn’t just about checking security boxes; it’s about building confidence in the platform that will manage your critical systems.

Setting Up Your Hardened Control Box

#

So, you’re looking to set up a hardened control box, but a crucial question arises: if your daily work machine is already compromised, and an attacker potentially has elevated rights, how can you possibly set up a secure and hardened system? I’m glad you asked, because this is the classic “chicken and egg” dilemma of the situation.

The best advice is to never use your daily computer for these tasks. If you have an old laptop—perhaps one that can no longer run Windows 11—consider using it. Install Linux on it and dedicate it as your administrative tasks machine.

But why would you even need a control box in the first place? You’ll want your control box to run 24/7, automating tasks that reside on it. You’ll then use your secure laptop only for direct administrative work.

For maximum security, especially in a home office environment, consider an even more isolated setup. If your control box is a Mini PC, you could connect a dedicated monitor and keyboard directly to it. This approach completely eliminates the need for remote access protocols like SSH, which, despite being secure, still represent a potential attack surface. By interacting with the control box directly, you reduce the risk of network-based attacks targeting your administrative sessions. This method might be less convenient, but it provides the highest level of isolation for your most sensitive tasks.

If your control box is directly connected to a monitor and keyboard, and you don’t require remote access, you can skip the SSH related parts of this article.

Roadmap

#

We start by establishing a secure foundation. We will do this by performing the following tasks:

• Update system and configure automatic security updates
• Secure SSH access
• Set up a firewall
• Remove services that you don’t need
• Set up audit and security tools
• Review kernel parameters
• Install management tools (kubectl, ansible, packer, talosctl, SOPS + Age, git)

Update System and Configure Automatic Security Updates

#

Before we begin hardening our control box, we need to ensure our system is up-to-date and will stay current with security patches. A control box that manages critical infrastructure must never fall behind on security updates.

Initial System Update

#

Start by updating the package lists and upgrading all installed packages:

# Update package lists
sudo apt update

# Upgrade all packages
sudo apt upgrade -y

# Remove unnecessary packages
sudo apt autoremove -y

# Clean package cache
sudo apt autoclean

Configure Automatic Security Updates

#

For a control box, automatic security updates are crucial. We’ll use the unattended-upgrades package to handle this:

# Install unattended-upgrades if not already installed
sudo apt install unattended-upgrades apt-listchanges -y

# Configure automatic updates
sudo dpkg-reconfigure -plow unattended-upgrades

When prompted, select “Yes” to enable automatic updates.

Now let’s configure the automatic updates settings:

# Edit the unattended-upgrades configuration
sudo nano /etc/apt/apt.conf.d/50unattended-upgrades

Ensure these lines are uncommented and configured:

// Automatically upgrade packages from these origins
Unattended-Upgrade::Origins-Pattern {
    "origin=Debian,codename=${distro_codename},label=Debian-Security";
    "origin=Debian,codename=${distro_codename}-security,label=Debian-Security";
    "origin=Debian,codename=${distro_codename}-updates,label=Debian";
};

// Remove unused automatically installed kernel-related packages
Unattended-Upgrade::Remove-Unused-Kernel-Packages "true";

// Remove unused dependencies
Unattended-Upgrade::Remove-Unused-Dependencies "true";

// Automatically reboot if required
Unattended-Upgrade::Automatic-Reboot "true";

// Reboot time (24-hour format)
Unattended-Upgrade::Automatic-Reboot-Time "03:00";

// Send email notifications (optional)
//Unattended-Upgrade::Mail "your-email@example.com";

Configure the update schedule:

# Edit the periodic configuration
sudo nano /etc/apt/apt.conf.d/20auto-upgrades

Set these values:

APT::Periodic::Update-Package-Lists "1";
APT::Periodic::Unattended-Upgrade "1";
APT::Periodic::Download-Upgradeable-Packages "1";
APT::Periodic::AutocleanInterval "7";

Enable and start the unattended-upgrades service:

# Enable the service
sudo systemctl enable unattended-upgrades

# Start the service
sudo systemctl start unattended-upgrades

# Check the status
sudo systemctl status unattended-upgrades

Test Automatic Updates

#

You can test the automatic update process:

# Run a dry-run to see what would be upgraded
sudo unattended-upgrade --dry-run

# Check the logs
sudo tail -f /var/log/unattended-upgrades/unattended-upgrades.log

Security Update Verification

#

Set up a simple script to verify that security updates are being applied:

# Create a script to check for security updates
sudo nano /usr/local/bin/check-security-updates.sh

Add this content:

#!/bin/bash
# Check for available security updates

echo "Checking for security updates..."
apt list --upgradable 2>/dev/null | grep -i security

# Check last update time
echo "Last unattended-upgrades run:"
ls -la /var/log/unattended-upgrades/

Make it executable:

sudo chmod +x /usr/local/bin/check-security-updates.sh

Now your system is updated and configured for automatic security updates. This ensures your control box stays secure with the latest patches while you’re hardening other aspects of the system.

Secure SSH Access

#

To secure your remote access to your control box, you will have to:

• Configure secure and modern protocols
• Disable root login
• Set up SSH key authentication with private key
• Disable password authentication
• Choose a different port for the SSH server
• Install and configure Google Authenticator for 2FA
• Bonus: set up endlessh trap

Let’s get started!

Configure Secure And Modern Protocols

#

First, let’s back up the original SSH configuration and create a hardened version:

# Create a backup of the original configuration
sudo cp /etc/ssh/sshd_config /etc/ssh/sshd_config.backup

# Create our hardened configuration
sudo nano /etc/ssh/sshd_config

Replace the contents with this hardened configuration:

# Basic SSH Configuration
Port 22  # We'll change this later
Protocol 2
AddressFamily inet

# Authentication
PermitRootLogin no
PasswordAuthentication no
PermitEmptyPasswords no
ChallengeResponseAuthentication no
UsePAM yes
PubkeyAuthentication yes
AuthorizedKeysFile .ssh/authorized_keys

# Limit login attempts and timing
MaxAuthTries 3
LoginGraceTime 30
MaxStartups 2:30:10
MaxSessions 2

# Disable dangerous features
X11Forwarding no
AllowTcpForwarding no
GatewayPorts no
PermitTunnel no
AllowAgentForwarding no

# Use strong ciphers and algorithms
Ciphers chacha20-poly1305@openssh.com,aes256-gcm@openssh.com,aes128-gcm@openssh.com,aes256-ctr,aes192-ctr,aes128-ctr
MACs hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha2-256,hmac-sha2-512
KexAlgorithms curve25519-sha256@libssh.org,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256

# Logging
SyslogFacility AUTH
LogLevel VERBOSE

# Timeout settings
ClientAliveInterval 300
ClientAliveCountMax 2
TCPKeepAlive no

# Banner (optional)
Banner /etc/issue.net

Setup SSH Key Authentication with a Private Key

#

Generate a strong SSH key pair on your secure client machine (not the control box):

# On your client machine, generate an Ed25519 key pair
ssh-keygen -t ed25519 -b 4096 -C "controlbox-access-$(date +%Y%m%d)"

Follow the prompts to save the key with a strong passphrase. Now copy the public key to your control box:

# Copy the public key to the control box
ssh-copy-id -i ~/.ssh/id_ed25519.pub username@controlbox-ip

# Or manually copy it
cat ~/.ssh/id_ed25519.pub | ssh username@controlbox-ip "mkdir -p ~/.ssh && cat >> ~/.ssh/authorized_keys"

On the control box, secure the SSH directory:

# Set proper permissions
chmod 700 ~/.ssh
chmod 600 ~/.ssh/authorized_keys
chown -R $USER:$USER ~/.ssh

Disable Password Authentication

#

This is already configured in our SSH config above, but let’s verify:

# Check that password authentication is disabled
sudo grep -i passwordauthentication /etc/ssh/sshd_config

Choose Port for SSH Server

#

Edit the SSH configuration to use a non-standard port:

sudo nano /etc/ssh/sshd_config

Change the port line to something between 1024-65535:

Port 2222  # Or any port you prefer

Hint: It’s crucial to select a port other than 2222, as this port is frequently recognized as an alternative SSH port. Port scanners often prioritize well-known ports and may not conduct comprehensive range scans. Although you cannot entirely conceal your port from a full scan, opting for a less common port can offer a degree of stealth against partial scans targeting standard ports. I suggest leveraging AI to identify such a port. For illustrative purposes, I’ve used port 2222 as an example of a port to steer clear of.

2FA with Google Authenticator

#

Install the Google Authenticator PAM module:

# Install the necessary packages
sudo apt update
sudo apt install libpam-google-authenticator

# Configure Google Authenticator for your user
google-authenticator

Answer the prompts as follows:

• Do you want authentication tokens to be time-based? Yes
• Do you want me to update your “~/.google_authenticator” file? Yes
• Do you want to disallow multiple uses of the same authentication token? Yes
• Do you want to increase the time window? No
• Do you want to enable rate-limiting? Yes

Save the emergency scratch codes in a secure location!

Now configure PAM to use Google Authenticator:

sudo nano /etc/pam.d/sshd

Add this line at the top of the file:

auth required pam_google_authenticator.so

Update the SSH configuration to require both key and 2FA:

sudo nano /etc/ssh/sshd_config

Add or modify these lines:

ChallengeResponseAuthentication yes
AuthenticationMethods publickey,keyboard-interactive

Endlessh Trap

#

Install and configure endlessh to trap SSH brute force attempts:

# Install endlessh
sudo apt install endlessh

# Configure endlessh
sudo nano /etc/endlessh/config

Add this configuration:

# Port to bind to
Port 22

# Delay between bytes sent
Delay 10000

# Maximum number of clients
MaxClients 4096

# Log level
LogLevel 1

# Bind to IPv4 only
BindFamily 0

Since we’re using port 22 for the trap, make sure your SSH is on the different port we configured earlier.

We are using a privileged port so we need to change some more settings.

sudo nano /lib/systemd/system/endlessh.service

See the comments in this file:

## If you want Endlessh to bind on ports < 1024
## 1) run:
##     setcap 'cap_net_bind_service=+ep' /usr/local/bin/endlessh
## 2) uncomment following line
#AmbientCapabilities=CAP_NET_BIND_SERVICE
## 3) comment following line
PrivateUsers=true

Change the file accordingly to

## If you want Endlessh to bind on ports < 1024
## 1) run:
##     setcap 'cap_net_bind_service=+ep' /usr/local/bin/endlessh
## 2) uncomment following line
AmbientCapabilities=CAP_NET_BIND_SERVICE
## 3) comment following line
#PrivateUsers=true

And here is one more thing: apt install endlessh will install the binary in /usr/bin/endlessh instead of /usr/local/bin/endlessh as written in the comment, so you have to execute:

sudo setcap 'cap_net_bind_service=+ep' /usr/bin/endlessh

Also find this line

InaccessiblePaths=/run /var

and change it to

#InaccessiblePaths=/run /var

Now enable and start endlessh:

sudo systemctl enable endlessh
sudo systemctl start endlessh
sudo systemctl status endlessh

Now restart SSH to apply all changes:

# Test the configuration first
sudo sshd -t

# If no errors, restart SSH
sudo systemctl restart sshd

Now every time you connect to your control box you will be asked for

• your ssh private key passphrase (use pagent on windows or keyring on linux to provide it automatically)
• your password
• your current token

Firewall Setup

#

Install UFW

#

UFW (Uncomplicated Firewall) comes pre-installed on Debian 12, but let’s ensure it’s available:

# Install UFW if not present
sudo apt install ufw

# Check status
sudo ufw status

Configure SSH Port

#

Configure UFW to allow your custom SSH port:

# Allow your custom SSH port (replace 2222 with your chosen port)
sudo ufw allow 2222/tcp

# If you want to restrict to specific IPs
sudo ufw allow from YOUR_IP_ADDRESS to any port 2222

Permit Access To IP Addresses

#

You can restrict access to specific IP addresses or ranges:

# Allow specific IP address
sudo ufw allow from 192.168.1.100

# Allow IP range
sudo ufw allow from 192.168.1.0/24

# Allow specific IP to specific port
sudo ufw allow from 192.168.1.100 to any port 2222

# Deny all other SSH attempts (optional, as default is deny)
sudo ufw deny 22/tcp
# or if you install endlessh (see below)
sudo ufw allow 22/tcp

Enable the firewall:

# Enable UFW
sudo ufw enable

# Check the status
sudo ufw status verbose

Testing Your Configuration

#

Before disconnecting, test your new configuration:

# Test SSH connection with new port and 2FA
ssh -p 2222 username@controlbox-ip

# Check that root login is blocked
ssh -p 2222 root@controlbox-ip  # This should fail

# Verify firewall rules
sudo ufw status numbered

Summary

#

So far, we have setup unattended security update, secured remote access, and we configured a firewall. Your control box now has:

• Automatic daily security updates inlcuding reboot at night if required
• Hardened SSH configuration with modern cryptography
• Disabled root login and password authentication
• SSH key-based authentication with 2FA
• Custom SSH port to avoid automated attacks
• Endlessh trap on the default SSH port
• Properly configured firewall rules

In the next articles, we will address the remaining security measures including removing unnecessary services, setting up audit tools, reviewing kernel parameters, and installing essential management tools. Stay tuned!

Important: Always keep a backup method to access your control box (like console access) in case you lock yourself out during configuration. Test each step thoroughly before moving to the next one.