rodesia.org
Blog

Blog ·

How I secure my servers, after a cryptominer moved in

How I secure my servers, after a cryptominer moved in

A cryptominer lived on one of my servers for 11 days before I noticed. It got in through a self-hosted tool I hadn't updated, and nothing told me it was there.

Anything open on the internet gets attacked, all day. Bots try to log in to every server they find, nonstop. So this is my own setup and what I've learned along the way, in plain language, for anyone who runs a server. There's a checklist at the end, and a free skill that lets an AI agent run the whole audit for you.

1. Stop strangers from even knocking

The fix I use is Tailscale. It's an app you install on your laptop, phone and servers. It links them into your own private network, as if they sat in the same room, even when they're in different countries. Strangers on the internet can't see that network, let alone get in.

How I set it up:

  1. Close SSH (the remote login to your server) to the internet. Allow it only over Tailscale.
  2. Admin panels and dashboards are Tailscale only, never public.
  3. Default-deny rules. Each server can reach only what it needs, nothing more.
  4. Tag your servers, so one stolen laptop can't reach everything.
  5. Turn on 2FA for the account you log into Tailscale with. It's your new master key.
  6. Leave Funnel (Tailscale's public sharing) off unless you really mean it.
  7. Keep one way back in, like your host's web console, in case Tailscale is down.

It's free for personal use, and you can set it up in minutes.

2. Fewer ways in, fixed sooner

The miner got in through a self-hosted tool I hadn't updated, with a page that should never have been public. An attacker used a known bug in it to run their own code. All they wanted was free CPU to mine crypto. Two lessons came out of it.

Make the target smaller

Every program you run and every page you expose is a door. Fewer doors, fewer break-ins.

  • Uninstall what you don't use.
  • Never expose dev or test pages (previews, debug tools) to the internet.
  • Close every port you don't need, and put the rest behind Tailscale.
  • Don't run apps as root (the all-powerful admin user). If one is broken into, the damage stays small.

Keep everything up to date

Most break-ins use bugs that already have a fix. The attacker is just faster than your next update.

  • Turn on automatic security updates for the system (on Ubuntu, that's unattended-upgrades).
  • Rebuild and redeploy your containers regularly, so they pick up fixes too.
  • Let GitHub's Dependabot tell you when a library you use has a known hole, and fix it the same day.
  • Restart after kernel updates. An update you haven't restarted for isn't protecting you yet.

3. Watch everything

That miner ran at 98% CPU for 11 days and nothing told me. That was the real failure, more than the bug that let it in. Monitoring just means your servers tell you when something looks wrong, before your users do.

What I watch, and why:

  1. Is it up? A check from outside every minute. If the site is down, you hear it first.
  2. CPU stuck at 100%. Often a bug, sometimes a cryptominer that moved in.
  3. Every login to the server. With Tailscale these should be rare, so any surprise one is worth a look.
  4. Sudden floods of errors. Hundreds of failed requests in a minute usually means a bot is probing for weak spots.
  5. Disk space and backups. A full disk takes a site down, and a backup you never tested is just a hope.

Two rules make it work:

  • Few alerts, all of them real. If your phone buzzes for nothing, you'll stop looking, and miss the one that matters.
  • Keep logs somewhere other than the server. Whoever breaks in will try to delete them.

Good tools to start with, free or self-hosted:

  • Is it up: Uptime Kuma, or UptimeRobot
  • CPU, memory, disk: Netdata, or Prometheus + Grafana
  • Logs off the server: Grafana Loki, or Better Stack
  • Backups and scheduled jobs: Healthchecks.io, which alerts you when a job doesn't check in

4. Secrets

A secret is anything that lets someone act as you. Passwords, API keys, tokens, database addresses. Bots scan public code for them all day, and a leaked key gets used fast.

Where to keep them

  1. Never in your code, not even in a private repo. Git remembers everything, so a key you deleted later is still in the history.
  2. In a vault. That's an encrypted place for secrets that only opens with a master key. I use Ansible Vault. My secrets sit in the repo encrypted and only unlock when I deploy, and a test fails if a vault file is ever committed unencrypted. 1Password, Bitwarden Secrets Manager, Doppler and SOPS work well too.
  3. On the server, in a file only the app can read. Never baked into a Docker image, where anyone with the image can pull them out.
  4. Encrypted when you store them. One app I'm building keeps every connected account's tokens encrypted, with keys I can rotate. A copied database alone is useless.

Keeping them out of your logs

  • Never log full requests or headers. That's where tokens hide.
  • Show only the last 4 characters of a key, in logs and admin pages.
  • Redact before anything leaves your system, like error reports, support tools and AI helpers.
  • Turn off debug mode in production. Debug error pages love to print your settings.

If one leaks, change it at once. Use one key per app and per purpose, so you can cancel one without breaking everything.

And turn on GitHub's secret scanning with push protection. It stops a push that contains a key.

5. When something still gets through

The attacker who got into my server did something clever. They closed the hole behind them, so no one else could use it. Then they mined crypto on my CPUs until I noticed.

What I did, in this order:

  1. Kept the evidence. I copied what they'd planted and the CPU history off the server.
  2. Checked the whole server. Logins, services, scheduled jobs, outgoing connections. It had never left the tool's container.
  3. Updated the tool and removed their files.
  4. Changed every secret they could have seen. Passwords, keys, tokens. Then I gave the other apps new API keys.
  5. Locked the app down with no admin rights, read-only files and CPU limits. Next time a miner can't take the whole machine.

Backups are what made it calm instead of scary.

  • Every night, encrypted, stored off the server. If the server dies or is wiped, the backups don't go with it.
  • Databases are copied safely, never mid-write, and checked after each copy.
  • I keep 7 daily, 4 weekly and 6 monthly copies, so I can go back to before a break-in even if I notice late.
  • I can restore one site at a time, and every restore first saves the current state, so a restore can't make things worse.

Remember 3-2-1. Three copies, on two kinds of storage, one of them off-site. Even better, give your backups credentials that can add copies but never delete them.

And if you're ever broken into, rebuild from a clean copy. Don't try to "clean" the broken one.

6. Give everything only the access it needs

That intruder got full control of one app and still never reached the rest of my server. That's least privilege. When something is broken into, what else can it touch? Keep that answer as short as you can.

How I do it:

  1. Apps don't run as admin. My sites run as a plain user made just for them, with no special system powers. The app that was broken into now also has read-only files and a CPU limit.
  2. Each app sees only its own files. A site's container gets its own folders and nothing else on the disk.
  3. Servers can't talk to each other freely. My Tailscale rules start from "nothing allowed", and each server may open only the few connections it needs.
  4. One account and one key per job. I log in with my own account, and the software that runs the sites has its own. Every service has its own API key, so a leak opens one door and I can change that key without breaking the rest.
  5. Tools get the smallest permission that works. A tool that only reads gets read access. A deploy key covers one repo, never your whole GitHub account.

For people:

  • SSH keys, never passwords.
  • Two-factor login on every account that offers it.
  • Remove access the day someone no longer needs it.

Ask this of everything you run. If it were broken into tomorrow, what else could it reach?

7. Your web app's front door

In October a bot fired 62 SQL-injection attempts at one of my sites in 2 minutes. Every one bounced, and I had no idea that bot was coming. The front door was simply built to say no.

What makes it say no:

  1. Check the type of every input. The page number must be a number, so ?page=2 UNION SELECT… was rejected before it got anywhere near the database. Use your framework's validation, and never paste what a visitor typed into a database query.
  2. Rate limits. More than 40 requests in 10 seconds from one visitor gets "slow down". Logins have their own limit, so nobody can guess passwords at full speed.
  3. Security headers. A few lines of config tell browsers to use HTTPS only, not to guess file types, and not to let other sites show my pages inside theirs (a common trick to steal clicks).
  4. Forms carry a hidden token, so another site can't submit a form as you while you're logged in.
  5. A shield in front. Cloudflare sits before my servers, hides their address and soaks up floods before they arrive.

That bot taught me something too. At first my monitoring saw 62 attackers. It was one, faking a new address on every request. Cloudflare adds the real address last, and I had been reading the whole list. Now I only trust the last one.

Be careful what you trust. Anything a visitor sends you can be faked, even their address.

8. The accounts that own everything

You can lock down every server and still lose it all on one login page. Whoever gets into your email can reset every other password. Whoever gets your domain can point your site, and your email, anywhere they like.

Above your servers sit your email, your domain registrar, DNS, GitHub, your hosting, and anything that takes payments.

How to guard them:

  1. Two-factor login on all of them, and never by text message. Use an authenticator app or a hardware key. SMS codes can be stolen by moving your phone number to someone else's SIM card.
  2. Start with your email. It's the reset button for everything else.
  3. Keep the recovery codes offline, on paper or in your password manager. If you lose your phone, they're how you get back in.
  4. Lock your domain. Turn on the transfer lock at your registrar so nobody can move it without you.
  5. Don't hang everything on one inbox. A separate address for the registrar and money accounts means one hacked inbox doesn't open them all.
  6. Check who has access. Apps connected to your GitHub and Google accounts, old API tokens, people who have left. Remove anything you don't use.
  7. Change your keys now and then. I recently replaced mine for GitHub, Cloudflare, my email service and my AI provider. Because each key has one job, I knew exactly what each change would touch.

The whole thing as one checklist

Save it, and go through it once a month. Each line is one thing to check.

1. Lock the door

  • SSH and admin pages closed to the internet, reachable only over Tailscale
  • Tailscale rules start from "nothing allowed"
  • 2FA on your Tailscale login

2. Fewer ways in, fixed sooner

  • Unused apps and ports removed, no dev or debug pages public
  • Automatic security updates on, restart after kernel updates
  • Containers rebuilt regularly, Dependabot alerts fixed the same day

3. Watch everything

  • An outside check that your site is up, every minute
  • Alerts for CPU stuck high, new logins, error floods, a full disk
  • Logs kept off the server, and few alerts, all real

4. Secrets

  • No secrets in your code or git history
  • Secrets in a vault, server env files readable only by the app
  • Keys masked in logs, debug mode off, GitHub secret scanning on

5. Backups and recovery

  • Nightly, encrypted, off-site, following 3-2-1 (3 copies, 2 kinds of storage, 1 off-site)
  • A restore tested this month
  • A plan for a break-in. Keep evidence, change secrets, rebuild from clean

6. Least privilege

  • No app runs as admin, each sees only its own files
  • One account and one key per job
  • Read-only access when reading is enough

7. The front door

  • Every input type-checked, never pasted into a query
  • Rate limits everywhere, stricter on logins
  • Security headers, form tokens, only trust the address your proxy adds

8. The accounts above

  • 2FA by app or hardware key on email, registrar, DNS, GitHub, hosting, payments
  • Recovery codes offline, domain transfer lock on
  • Connected apps and old tokens reviewed

Let an AI run the audit

I turned this checklist into a free skill for Claude Code, security-audit on GitHub. It only looks. It never installs, changes or deletes anything, and it never prints a secret. You get the 5 most urgent fixes first, then pass, fail or unknown for every check, with evidence.

git clone https://github.com/mmarcos/security-audit
mkdir -p ~/.claude/skills
cp -r security-audit/security-audit ~/.claude/skills/
claude
> /security-audit

Using Codex or Cursor? The repo also has a shorter prompt you can paste in.

An AI agent with shell access can do a lot, so run it as a user without sudo if you can, and read what it proposes before approving commands. Treat the report as a starting point, not a certificate.