<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom"><channel><title>Raed Ouiriemmi — Field notes</title><link>https://ouiriemmiraed.me/blog/</link><atom:link href="https://ouiriemmiraed.me/feed.xml" rel="self" type="application/rss+xml"/><description>Incident write-ups and engineering notes on servers, security, RAG and full-stack apps.</description><language>en</language><item><title>A VS Code task turned my GitHub repos into malware droppers. Here is how the PolinRider campaign works and how I cleaned up.</title><link>https://ouiriemmiraed.me/blog/vscode-folderopen-task-github-malware-polinrider/</link><guid isPermaLink="true">https://ouiriemmiraed.me/blog/vscode-folderopen-task-github-malware-polinrider/</guid><pubDate>Thu, 17 Sep 2026 08:00:00 +0000</pubDate><description>How the PolinRider campaign plants a hidden VS Code folderOpen task and a fake .woff2 font in GitHub repos by rewriting commits, how to detect it and clean up.</description><content:encoded><![CDATA[<div class="note">
<p><strong>TL;DR</strong> — If you open cloned repositories in VS Code or Cursor: run <code>Tasks: Manage Automatic Tasks</code> → <em>Disallow</em>, grep your repos for <code>/0x/cls</code> and for <code>.woff2</code> files that do not start with the bytes <code>wOF2</code>, and check every <code>.vscode/tasks.json</code> for <code>"runOn": "folderOpen"</code>. Scan <strong>every branch</strong>, not just <code>main</code>.</p>
</div>
<h2 id="how-i-found-it">How I found it</h2>
<p>I was auditing my portfolio site's git history for an unrelated scroll bug when <code>git show --stat</code> showed something odd: a commit that added sixteen Font Awesome files, fifteen listed as <code>Bin</code> and one, <code>public/fonts/fa-solid-400.woff2</code>, listed as <code>1 +</code>. A font file that git thinks is a <strong>one-line text file</strong>.</p>
<p>It was one line of obfuscated Node.js, preceded by several hundred spaces so that it looks empty in an editor. And the name is subtly wrong: Font Awesome 5's solid face is <code>fa-solid-900</code>. The fifteen real fonts around it are camouflage.</p>
<p>Two more files completed the chain:</p>
<pre><code class="language-json">// .vscode/tasks.json  (abridged)
{
  &quot;label&quot;: &quot;eslint-check&quot;,
  &quot;type&quot;: &quot;shell&quot;,
  &quot;command&quot;: &quot;node ./public/fonts/fa-solid-400.woff2&quot;,
  &quot;runOptions&quot;: { &quot;runOn&quot;: &quot;folderOpen&quot; },
  &quot;presentation&quot;: { &quot;reveal&quot;: &quot;never&quot; },
  &quot;hide&quot;: true
}
</code></pre>
<pre><code class="language-json">// .vscode/settings.json
{
  &quot;task.allowAutomaticTasks&quot;: true,
  &quot;terminal.integrated.hideOnStartup&quot;: &quot;always&quot;
}
</code></pre>
<p><code>runOn: folderOpen</code> makes VS Code run the task when the folder is opened. <code>task.allowAutomaticTasks</code> removes the confirmation prompt. <code>reveal: never</code> and <code>hide: true</code> keep it out of the terminal and the task list. <strong>No click, no build, no <code>npm install</code> — opening the folder is the exploit.</strong></p>
<h2 id="what-the-payload-does">What the payload does</h2>
<p>Without executing anything, reading the de-obfuscated code was enough: it resolves its command-and-control address from a <strong>blockchain transaction</strong> (the "EtherHiding" technique — public RPC endpoints, so the traffic looks like ordinary web requests and the operators can move servers without touching the code), downloads a second stage, <code>eval</code>s it, and re-spawns itself detached so it survives the editor closing. Public analyses of the second stage describe an infostealer that goes after browser passwords and cookies, password-manager extensions, crypto wallets, and — the part that mattered for me — <strong>git credentials, GitHub CLI and GitHub Desktop tokens, and the OS credential store</strong>.</p>
<p>The campaign is tracked as <strong>PolinRider</strong> (with links to <em>TasksJacker</em> and the DPRK "Contagious Interview" operation). OpenSourceMalware counted <a href="https://github.com/OpenSourceMalware/PolinRider" target="_blank" rel="noopener">1,951 compromised public repositories from 1,047 owners as of April 2026</a> — I was one of about a thousand, not a target.</p>
<h2 id="the-part-that-fooled-me-commits-that-were-never-added">The part that fooled me: commits that were never "added"</h2>
<p>The malware never arrives in a commit of its own. It arrives in a <strong>rewrite of one of your legitimate commits</strong>: same message, same author date, same commit date to the second. In my history, the only differences between the real commit and its evil twin were the committer <em>name</em> (<code>Raed</code> instead of <code>Raed Ouiriemmi</code>) and the committer <em>time zone</em> (−0700, +0300… I am in UTC+1). <code>git diff</code> between the two, restricted to the site's files, was <strong>empty</strong>. It is designed to pass a quick review.</p>
<p>The tool that does this is a published indicator of compromise called <code>temp_auto_push.bat</code>: it reads the last commit's metadata, <strong>changes the system clock</strong> to the original timestamp, amends the commit with the payload, restores the clock and force-pushes. It also added a <code>.gitignore</code> entry for <em>itself</em>, so its future edits never showed in <code>git status</code>.</p>
<p>Which raises the real question — <em>who pushed?</em> Personal-account security logs on GitHub do not record pushes, and they do not export IP addresses. What does work: the <strong>repository activity endpoint</strong>, <code>GET /repos/{owner}/{repo}/activity</code>, readable with any read access. It records the actor, the server-side timestamp, <code>before</code> → <code>after</code> SHAs and whether it was a force-push. That is how I established that the pushes came from a third machine using my stolen credentials, not from my own PC.</p>
<h2 id="three-camouflages-one-payload">Three camouflages, one payload</h2>
<p>Scanning by file name found three infected repositories. Scanning by <strong>content</strong> found eight. The same payload hides in three ways:</p>
<ol>
<li>the fake <code>.woff2</code> plus the <code>folderOpen</code> task — the only variant that runs on its own;</li>
<li>a ~9 KB line <strong>appended after the closing <code>];</code> of <code>eslint.config.js</code></strong>, after a long run of whitespace;</li>
<li>appended to <code>postcss.config.mjs</code> / <code>postcss.config.js</code> / <code>next.config.ts</code> — executed <strong>on every build</strong> by the framework.</li>
</ol>
<p>The signatures that held up:</p>
<pre><code class="language-bash"># obfuscated C2 path + RPC provider + escaped &quot;http&quot;
grep -rlaE '/0x/cls|/0x/ls|TronGrid|u0068.{0,2}u0074.{0,2}u0074.{0,2}u0070' .

# a &quot;font&quot; whose magic bytes are wrong (real ones start with wOF2 / wOFF / OTTO)
find . -name '*.woff2' -exec sh -c 'head -c4 &quot;$1&quot; | grep -q wOF2 || echo &quot;FAKE: $1&quot;' _ {} \;

# auto-running VS Code tasks
grep -rl '&quot;runOn&quot;: *&quot;folderOpen&quot;' --include=tasks.json .
</code></pre>
<p>Two things I got wrong first:</p>
<ul>
<li><strong><code>windowsHide</code> is a terrible marker.</strong> It is a legitimate Node option; used alone it flagged sixty files in VS Code Server, Playwright and Copilot. The discriminating string is the C2 path.</li>
<li><strong>Counting occurrences discriminates nothing.</strong> The payload is minified onto one line, so a real infection contains <code>/0x/cls</code> exactly once — the same as a note that merely mentions it. The right test is <em>does this file contain executable code around the string</em>, e.g. <code>unref()</code> and <code>windowsHide</code> within a few hundred bytes of the URL.</li>
</ul>
<h2 id="cleanup-and-the-four-mistakes">Cleanup, and the four mistakes</h2>
<p>The obvious part: rewrite history to drop the files, and force-push.</p>
<pre><code class="language-bash">git filter-branch --force --prune-empty \
  --index-filter 'git rm -r --cached --ignore-unmatch .vscode public' \
  -- --all
</code></pre>
<p>Check the result by content, not by trust: the final tree must be byte-identical to the last clean commit, and no blob in the repository may match the signature. Then the mistakes.</p>
<p><strong>1. Old commits stay downloadable.</strong> After a force-push, the removed commits are unreachable — but GitHub does not garbage-collect immediately, and <code>raw.githubusercontent.com/&lt;owner&gt;/&lt;repo&gt;/&lt;old-sha&gt;/public/fonts/...</code> still answered <code>200</code> for weeks. Anyone with the SHA could still fetch the payload from <em>my</em> namespace. For repositories that can be recreated, <strong>delete and recreate</strong> the repository; for the one serving GitHub Pages with a custom domain, I had to ask GitHub Support for a garbage collection.</p>
<p><strong>2. Your local clone keeps the payload too.</strong> <code>git gc --prune=now</code> did nothing, because the old commit was still <em>reachable</em> from three places I had not thought of: <code>refs/original/</code> (filter-branch's backup), the <strong>remote-tracking ref</strong> <code>refs/remotes/origin/main</code> (which may live in <code>packed-refs</code>, not as a loose file), and the reflogs.</p>
<pre><code class="language-bash">rm -rf .git/refs/original .git/ORIG_HEAD .git/FETCH_HEAD
git update-ref refs/remotes/origin/main &lt;clean-sha&gt;
git reflog expire --expire=now --expire-unreachable=now --all
git gc --prune=now
git cat-file -e &lt;payload-blob-sha&gt; &amp;&amp; echo STILL THERE   # must fail
</code></pre>
<p>Verify with <code>git cat-file -e</code>, not with the size of <code>.git</code>.</p>
<p><strong>3. <code>grep -r</code> cannot see inside <code>.git</code>.</strong> Objects are zlib-compressed. To scan a repository's object database you iterate <code>git cat-file --batch-all-objects</code>; to scan a <code>.bundle</code> you have to clone it first. My "clean" declaration was wrong twice for this reason — the third scan found the payload in nine throwaway clones I had made <em>during the investigation</em>.</p>
<p><strong>4. Branches.</strong> One repository had four branches. I cleaned <code>main</code> and declared victory; the payload sat on the other three for two more days. <code>filter-branch -- --all</code>, then re-scan every branch.</p>
<h2 id="what-actually-stops-it">What actually stops it</h2>
<ul>
<li><strong>VS Code:</strong> <code>Tasks: Manage Automatic Tasks</code> → <strong>Disallow</strong>, and keep <a href="https://code.visualstudio.com/docs/editing/workspaces/workspace-trust" target="_blank" rel="noopener">Workspace Trust</a> on. A repository's <code>settings.json</code> can <em>ask</em> for automatic tasks; your user setting decides.</li>
<li><strong>Git hosting:</strong> a <strong>ruleset that blocks force-pushes</strong> on every repository. The entire incident, on every repository involved, was a sequence of force-pushes that nothing prevented.</li>
<li><strong>Credentials:</strong> treat every secret the infected machine could read as stolen — rotate from a <em>different</em> device: passwords, PATs, SSH keys, OAuth grants, and check for <strong>deploy keys</strong>, which are per-repository and invisible in your account settings. 2FA was on and it changed nothing: the attacker used an already-valid token, and 2FA only guards logins.</li>
<li><strong>Monitoring:</strong> I now run a <a href="https://ouiriemmiraed.me/projects/telegram-monitoring-bot/">read-only watcher</a> that records every branch SHA of every repository every 30 minutes, and scans changed repositories by content. A force-push or a branch that moves backwards is a Telegram message within the half hour.</li>
<li><strong>Take-home tests and "starter" repositories</strong> are the delivery vehicle. Open them in a container or a VM, never in your daily editor with your daily credentials.</li>
</ul>
<p>This is also why my GitHub repositories are private for now: the affected histories are being rewritten before anything goes public again. Until then, the code of any project on this site is available on request.</p>
<h2 id="sources">Sources</h2>
<ul>
<li><a href="https://github.com/OpenSourceMalware/PolinRider" target="_blank" rel="noopener">OpenSourceMalware — PolinRider IoCs and campaign tracking</a></li>
<li><a href="https://opensourcemalware.com/blog/polinrider-rides-again" target="_blank" rel="noopener">OpenSourceMalware — "PolinRider rides again"</a></li>
<li><a href="https://research.jfrog.com/post/hijacked-npm-vscode-tasks-blockchain/" target="_blank" rel="noopener">JFrog Research — hijacked npm packages and VS Code tasks with blockchain C2</a></li>
<li><a href="https://code.visualstudio.com/docs/editing/workspaces/workspace-trust" target="_blank" rel="noopener">VS Code docs — Workspace Trust</a></li>
</ul>]]></content:encoded></item><item><title>UFW says &quot;deny incoming&quot;. Your Docker containers are on the internet anyway.</title><link>https://ouiriemmiraed.me/blog/docker-published-ports-bypass-ufw/</link><guid isPermaLink="true">https://ouiriemmiraed.me/blog/docker-published-ports-bypass-ufw/</guid><pubDate>Wed, 16 Sep 2026 08:00:00 +0000</pubDate><description>Why ports published by Docker bypass UFW on Linux, how to check whether your containers are exposed right now, and three fixes (loopback, DOCKER-USER, ufw-docke</description><content:encoded><![CDATA[<h2 id="the-symptom">The symptom</h2>
<pre><code class="language-bash">$ sudo ufw status
Status: active
To         Action   From
--         ------   ----
22/tcp     ALLOW    Anywhere
80/tcp     ALLOW    Anywhere
443/tcp    ALLOW    Anywhere
</code></pre>
<p>Everything else should be dropped. Now the containers:</p>
<pre><code class="language-bash">$ docker ps --format '{{.Names}}\t{{.Ports}}'
jobsearch-postgres   0.0.0.0:5544-&gt;5432/tcp
jobsearch-redis      0.0.0.0:6390-&gt;6379/tcp
web-admin            0.0.0.0:80-&gt;80/tcp
</code></pre>
<p>From another machine, <code>nc -vz &lt;server&gt; 5544</code> connects. UFW never saw it.</p>
<h2 id="why-ufw-loses">Why UFW loses</h2>
<p>UFW is a front end for iptables that works on the <strong><code>INPUT</code></strong> chain — traffic addressed to the host itself. A published container port is not traffic to the host: Docker adds a DNAT rule in the <code>nat</code> table's <code>PREROUTING</code> chain that rewrites the destination to the container's address, and the packet then flows through the <strong><code>FORWARD</code></strong> chain, where Docker manages its own rules and puts them <em>before</em> anything UFW installs. Docker's documentation is explicit about it: rules in <code>INPUT</code> do not apply to published ports, and the supported place for your own filtering is the <a href="https://docs.docker.com/engine/network/packet-filtering-firewalls/" target="_blank" rel="noopener"><code>DOCKER-USER</code> chain</a>.</p>
<p>So <code>ufw deny 5544</code> is a rule for a packet that never reaches the chain the rule lives in.</p>
<h2 id="check-your-own-server-10-seconds">Check your own server (10 seconds)</h2>
<pre><code class="language-bash"># 1. anything published on all interfaces?
docker ps --format '{{.Names}}\t{{.Ports}}' | grep '0.0.0.0'

# 2. what does the DOCKER-USER chain say? (empty = &quot;RETURN&quot;, i.e. allow everything)
sudo iptables -L DOCKER-USER -n -v --line-numbers

# 3. the only test that counts: from OUTSIDE the box
nc -vz your.server.ip 5544
</code></pre>
<p><code>ss -tlnp</code> on the host is <em>not</em> enough: a published port shows up as <code>docker-proxy</code> listening on <code>0.0.0.0</code>, which looks like any other service and tells you nothing about what the firewall does with it.</p>
<p>In my case the mechanism was confirmed by an nginx container publishing port 80: its access log held 135,000 requests from the internet over four months. Port 80 is meant to be public, but it proved the path — and the same path applied to the database and the cache.</p>
<h2 id="fix-1-dont-publish-on-0000-do-this-first">Fix 1 — don't publish on 0.0.0.0 (do this first)</h2>
<p>A database or a cache should almost never have a public port at all. Bind to loopback:</p>
<pre><code class="language-yaml"># docker-compose.yml
services:
  postgres:
    ports:
      - &quot;127.0.0.1:5544:5432&quot;
  redis:
    ports:
      - &quot;127.0.0.1:6390:6379&quot;
</code></pre>
<p>Or drop the <code>ports:</code> entry entirely and let the other containers reach it over the compose network by service name — that is what the network is for. Other tools that ship with an open port by default deserve the same treatment: Ollama listens on <code>0.0.0.0:11434</code> with no authentication unless you set <code>OLLAMA_HOST=127.0.0.1:11434</code>.</p>
<h2 id="fix-2-filter-in-docker-user">Fix 2 — filter in DOCKER-USER</h2>
<p>For ports that must be published but not to the whole world, add rules to the chain Docker leaves for you. <code>ext_if</code> is your public interface (<code>ip route | grep default</code>):</p>
<pre><code class="language-bash"># allow an office / VPN range, drop everyone else, for every published port
sudo iptables -I DOCKER-USER -i eth0 ! -s 203.0.113.0/24 -m conntrack --ctstate NEW -j DROP

# or per port
sudo iptables -I DOCKER-USER -i eth0 -p tcp --dport 6390 -m conntrack --ctstate NEW -j DROP
</code></pre>
<p>Rules in <code>DOCKER-USER</code> are evaluated before Docker's own, and Docker never flushes the chain. They are not persistent by themselves: put them in <code>/etc/ufw/after.rules</code> (UFW loads it) or use <code>iptables-persistent</code>.</p>
<h2 id="fix-3-ufw-docker">Fix 3 — ufw-docker</h2>
<p>If you want to keep thinking in UFW terms, <a href="https://github.com/chaifeng/ufw-docker" target="_blank" rel="noopener"><code>ufw-docker</code></a> installs a <code>DOCKER-USER</code> block in <code>/etc/ufw/after.rules</code> and gives you commands such as <code>ufw-docker allow web-admin 80/tcp</code>. It is the least surprising option on a server where several people run <code>ufw allow</code> and expect it to mean something.</p>
<h2 id="what-i-changed">What I changed</h2>
<ol>
<li>The two containers were already stopped when I found this. Before they come back, their compose file publishes on <code>127.0.0.1</code> — or not at all, since only other containers ever talk to them.</li>
<li>The only publication left on <code>0.0.0.0</code> is nginx, which is public by design.</li>
<li>Still on the list: Ollama and my development backend run on the host, not in Docker, so UFW does protect them today — but binding them to loopback removes the dependency on a firewall rule being right.</li>
<li>Every audit checklist I use now starts with <code>docker ps</code> and the <code>DOCKER-USER</code> chain, before <code>ufw status</code> — because <code>ufw status</code> is the one that lies.</li>
</ol>
<p>The uncomfortable footnote: a Redis without a password reachable from the internet is not "some data exposure". Redis can be told to write its dump file anywhere — <code>~/.ssh/authorized_keys</code>, a cron directory — which is remote code execution. Check the six-week window in your logs, not just the fix.</p>]]></content:encoded></item><item><title>GitHub Pages, custom domain: &quot;Enforce HTTPS&quot; greyed out, certificate stuck, and the www CNAME that causes it</title><link>https://ouiriemmiraed.me/blog/github-pages-enforce-https-greyed-out-custom-domain/</link><guid isPermaLink="true">https://ouiriemmiraed.me/blog/github-pages-enforce-https-greyed-out-custom-domain/</guid><pubDate>Tue, 15 Sep 2026 08:00:00 +0000</pubDate><description>Why Enforce HTTPS stays greyed out on a GitHub Pages custom domain, what the certificate needs (A records, www CNAME, CAA) and how to unstick it.</description><content:encoded><![CDATA[<h2 id="the-symptom">The symptom</h2>
<p>Settings → Pages shows your custom domain with a green check, but <strong>Enforce HTTPS</strong> is greyed out. Sometimes with "certificate is being provisioned", sometimes with nothing. Meanwhile:</p>
<pre><code class="language-bash">$ curl -sI http://example.me/ | head -1
HTTP/1.1 200 OK          # served in clear, no redirect
$ curl -sI https://example.me/ | head -1
                         # TLS error, or a certificate for the wrong name
</code></pre>
<h2 id="read-the-real-state-from-the-api">Read the real state from the API</h2>
<p>The settings page hides the interesting field. The Pages API does not:</p>
<pre><code class="language-bash">gh api repos/OWNER/REPO/pages --jq '{status, cname, https_enforced, cert: .https_certificate}'
</code></pre>
<p><code>https_certificate.state</code> is what you want. The values you will meet:</p>
<div class="table-wrap"><table>
<thead>
<tr>
<th>state</th>
<th>meaning</th>
</tr>
</thead>
<tbody>
<tr>
<td><code>new</code> / <code>authorization_created</code></td>
<td>GitHub has asked Let's Encrypt; DNS must already be right</td>
</tr>
<tr>
<td><code>authorization_pending</code></td>
<td>the ACME challenge is waiting — usually DNS not propagated</td>
</tr>
<tr>
<td><code>authorized</code> / <code>issued</code></td>
<td>certificate exists; the checkbox becomes clickable within an hour</td>
</tr>
<tr>
<td><code>dns_changed</code></td>
<td>GitHub saw your DNS change and <strong>restarted from scratch</strong></td>
</tr>
<tr>
<td><code>errored</code></td>
<td>check <code>description</code> — CAA record, or a record pointing elsewhere</td>
</tr>
</tbody>
</table></div>
<p><code>dns_changed</code> is the one that looped for me: every time a record changed, the process restarted, and one record kept "changing".</p>
<h2 id="the-dns-that-actually-works">The DNS that actually works</h2>
<p>GitHub's <a href="https://docs.github.com/en/pages/configuring-a-custom-domain-for-your-github-pages-site/managing-a-custom-domain-for-your-github-pages-site" target="_blank" rel="noopener">documented setup</a> is short, and every deviation costs a certificate cycle:</p>
<pre><code class="language-text">example.me.        A      185.199.108.153
example.me.        A      185.199.109.153
example.me.        A      185.199.110.153
example.me.        A      185.199.111.153
www.example.me.    CNAME  OWNER.github.io.      # NOT example.me, NOT an A record
</code></pre>
<p>My mistake: <code>www</code> was a CNAME to the <strong>apex</strong> (<code>example.me</code>) instead of <code>OWNER.github.io</code>. Both resolve to the same addresses, so the site <em>worked</em> on <code>www</code> — but GitHub's certificate request lists both names, its verifier treats a <code>www</code> that does not point at <code>OWNER.github.io</code> as "changed", and the request restarts. The certificate that finally got issued after the fix covers both names:</p>
<pre><code class="language-bash">$ echo | openssl s_client -connect example.me:443 -servername example.me 2&gt;/dev/null \
  | openssl x509 -noout -ext subjectAltName
X509v3 Subject Alternative Name:
    DNS:example.me, DNS:www.example.me
</code></pre>
<p>Two more records to check:</p>
<ul>
<li><strong>CAA.</strong> If your zone has <em>any</em> CAA record, it must include <code>0 issue "letsencrypt.org"</code>, or issuance fails with a polite <code>errored</code>. No CAA record at all is fine. (Adding one that allows only Let's Encrypt is a cheap hardening step once everything works.)</li>
<li><strong>No AAAA records</strong> pointing anywhere else, and no leftover A record from the previous host — the verifier checks all of them.</li>
</ul>
<p>Verify from outside your own resolver, because your cache will lie to you for the whole TTL:</p>
<pre><code class="language-bash">dig +short example.me A @8.8.8.8
dig +short www.example.me CNAME @8.8.8.8
dig +short example.me CAA @8.8.8.8
</code></pre>
<h2 id="unsticking-a-certificate-that-is-being-provisioned-forever">Unsticking a certificate that is "being provisioned" forever</h2>
<p>If DNS is right and the state has been <code>authorization_pending</code> for more than a day, GitHub's job is stuck, not yours. What worked for me: in Settings → Pages, <strong>remove the custom domain, save, add it back, save</strong>. That discards the stuck request and starts a fresh one; with correct DNS the state went to <code>issued</code> within an hour. (Doing the same through the API did not help — the UI path seems to reset more.)</p>
<p>Then wait for the checkbox. It becomes clickable shortly <em>after</em> <code>issued</code>, not at the same moment. Tick it, and within about five minutes:</p>
<pre><code class="language-bash">$ curl -sI http://example.me/ | grep -i '^location\|^HTTP'
HTTP/1.1 301 Moved Permanently
location: https://example.me/
</code></pre>
<p>Check the whole matrix, not just the apex: <code>http://</code> and <code>https://</code> for the apex, <code>www</code>, and <code>OWNER.github.io</code>, plus any sub-pages. The <code>OWNER.github.io</code> redirect in particular kept pointing at <code>http://</code> until the checkbox was on.</p>
<h2 id="what-the-checkbox-does-not-give-you">What the checkbox does not give you</h2>
<ul>
<li><strong>HSTS.</strong> Pages does not send <code>Strict-Transport-Security</code>, so the first request of a new visitor can still go out in clear. If you put Cloudflare in front, enable HSTS there; otherwise accept it.</li>
<li><strong>Security headers</strong> in general — no CSP, no <code>X-Frame-Options</code>. A <code>&lt;meta http-equiv="Content-Security-Policy"&gt;</code> in the HTML covers most of the CSP value; clickjacking protection cannot be done from a meta tag.</li>
<li><strong>Renewal visibility.</strong> Certificates auto-renew, but nothing tells you if renewal fails. A weekly cron on any machine you own is enough:</li>
</ul>
<pre><code class="language-bash">echo | openssl s_client -connect example.me:443 -servername example.me 2&gt;/dev/null \
  | openssl x509 -noout -enddate
</code></pre>
<h2 id="bonus-page-build-failed-with-no-reason">Bonus: "Page build failed" with no reason</h2>
<p>Unrelated to HTTPS but found the same week: the legacy Pages build failed five times in a row on a commit that added three PNG icons generated with Python. Valid PNGs, no Jekyll, <code>.nojekyll</code> present. Removing the three files fixed it; the same generator's <code>apple-touch-icon.png</code> built fine. I never found the rule — but I did find that a build can also fail <em>transiently</em> for no reason at all, so before bisecting your commit, re-trigger the build and see if it goes green:</p>
<pre><code class="language-bash">gh api -X POST repos/OWNER/REPO/pages/builds
gh api repos/OWNER/REPO/pages/builds/latest --jq '{status, error: .error.message}'
</code></pre>]]></content:encoded></item><item><title>The site went down because Let&#x27;s Encrypt renewal had been failing for three months. Standalone vs. webroot behind nginx in Docker.</title><link>https://ouiriemmiraed.me/blog/letsencrypt-renewal-failing-behind-nginx-docker/</link><guid isPermaLink="true">https://ouiriemmiraed.me/blog/letsencrypt-renewal-failing-behind-nginx-docker/</guid><pubDate>Mon, 14 Sep 2026 08:00:00 +0000</pubDate><description>Outage from an expired Let&#x27;s Encrypt cert: certbot ran twice a day and failed every time (standalone vs nginx on :80), plus a second cert store inside Docker.</description><content:encoded><![CDATA[<h2 id="first-reflex-is-it-really-the-provider">First reflex: is it really the provider?</h2>
<p>When a site is "down", test SSH before anything else. If SSH answers, it is not a suspended account or an unpaid bill. Here it answered immediately — uptime 46 days, every container <code>Up</code>. Then:</p>
<pre><code class="language-bash">$ echo | openssl s_client -connect example.com:443 -servername example.com 2&gt;/dev/null \
  | openssl x509 -noout -dates
notBefore=May  6 17:43:12 2026 GMT
notAfter=Aug  4 17:43:11 2026 GMT     # yesterday
</code></pre>
<p>Expired. Every browser refuses the connection; to a non-technical person that is indistinguishable from "the server is off".</p>
<h2 id="why-renewal-had-been-failing-since-may">Why renewal had been failing since May</h2>
<p>The timer was doing its job:</p>
<pre><code class="language-bash">$ systemctl list-timers certbot.timer
NEXT                         LEFT     LAST                         PASSED   UNIT
Wed 2026-08-05 21:14:00 UTC  3h left  Wed 2026-08-05 09:02:11 UTC  8h ago   certbot.timer
</code></pre>
<p>Twice a day, for three months. And every run failed:</p>
<pre><code class="language-bash">$ sudo certbot renew --dry-run
Failed to renew certificate example.com with error:
Could not bind TCP port 80 because it is already in use by another process.
</code></pre>
<p>The renewal configuration told the whole story:</p>
<pre><code class="language-ini"># /etc/letsencrypt/renewal/example.com.conf
[renewalparams]
authenticator = standalone
</code></pre>
<p>The certificate had been <em>issued</em> with the <strong>standalone</strong> authenticator — certbot starts its own web server on port 80 to answer the ACME challenge. That works exactly once: while nginx is stopped, on the day you first set things up. From then on, nginx owns port 80, certbot cannot bind it, and renewal fails silently twice a day. No alert, because nothing was watching the timer's <em>result</em>.</p>
<h2 id="the-second-trap-two-certificate-stores">The second trap: two certificate stores</h2>
<p>Renewing was not enough. nginx ran in a container, and its <code>/etc/letsencrypt</code> was a <strong>Docker volume</strong> — a copy taken at setup time — not the host's <code>/etc/letsencrypt</code> that certbot writes to.</p>
<pre><code class="language-bash">$ docker inspect nginx --format '{{range .Mounts}}{{.Name}} -&gt; {{.Destination}}{{&quot;\n&quot;}}{{end}}'
app_certbot_conf -&gt; /etc/letsencrypt
app_certbot_www  -&gt; /var/www/certbot
</code></pre>
<p>So a successful renewal on the host would have changed nothing in the browser. Two stores, one updated by certbot, the other served by nginx, and nothing between them.</p>
<h2 id="the-fix-with-zero-downtime">The fix, with zero downtime</h2>
<p><strong>1. Switch to webroot.</strong> nginx already had the standard challenge location, pointing at the <code>certbot_www</code> volume:</p>
<pre><code class="language-nginx">location /.well-known/acme-challenge/ { root /var/www/certbot; }
</code></pre>
<p>So the challenge can be answered <em>through</em> nginx, with nginx running:</p>
<pre><code class="language-bash">sudo certbot certonly -n --webroot \
  -w /var/lib/docker/volumes/app_certbot_www/_data \
  --cert-name example.com -d example.com -d www.example.com -d app.example.com
</code></pre>
<p>This both issues a fresh certificate and rewrites the renewal config to <code>authenticator = webroot</code>, which is the real fix: future renewals no longer need port 80.</p>
<p><strong>2. Sync the store nginx reads.</strong> Copy <code>archive/</code> and <code>live/</code> (both — <code>live/</code> is symlinks into <code>archive/</code>) into the volume, then reload nginx gracefully:</p>
<pre><code class="language-bash">sudo rsync -a /etc/letsencrypt/archive/example.com/ /var/lib/docker/volumes/app_certbot_conf/_data/archive/example.com/
sudo rsync -a /etc/letsencrypt/live/example.com/    /var/lib/docker/volumes/app_certbot_conf/_data/live/example.com/
sudo docker kill -s HUP nginx      # same as nginx -s reload, no dropped connections
</code></pre>
<p><strong>3. Make step 2 automatic.</strong> A deploy hook runs after every successful renewal:</p>
<pre><code class="language-bash"># /etc/letsencrypt/renewal-hooks/deploy/sync-nginx-certs.sh   (chmod +x)
#!/bin/sh
set -e
V=/var/lib/docker/volumes/app_certbot_conf/_data
rsync -a /etc/letsencrypt/archive/example.com/ &quot;$V/archive/example.com/&quot;
rsync -a /etc/letsencrypt/live/example.com/    &quot;$V/live/example.com/&quot;
docker kill -s HUP nginx
</code></pre>
<p><strong>4. Prove it end to end.</strong></p>
<pre><code class="language-bash">sudo certbot renew --dry-run      # must say &quot;Congratulations, all simulated renewals succeeded&quot;
echo | openssl s_client -connect example.com:443 -servername example.com 2&gt;/dev/null \
  | openssl x509 -noout -enddate  # the NEW date, served by nginx, from outside
</code></pre>
<p>The dry run exercises the webroot path <em>and</em> the hook. If it passes, the timer will renew about 30 days before expiry and nginx will pick it up.</p>
<h2 id="what-i-would-have-needed-to-avoid-the-outage">What I would have needed to avoid the outage</h2>
<ul>
<li><strong>Monitor the outcome, not the schedule.</strong> <code>certbot.timer</code> being active means nothing. Alert on <code>certbot renew</code> exit status, or simply check the served certificate's expiry from outside every day and alert under 20 days. A five-line cron does it, and it belongs next to the other checks in a <a href="https://ouiriemmiraed.me/projects/telegram-monitoring-bot/">monitoring bot</a>.</li>
<li><strong>One certificate store.</strong> Mount the host's <code>/etc/letsencrypt</code> read-only into the nginx container instead of copying it into a volume. Then there is nothing to sync.</li>
<li><strong>Never issue with <code>standalone</code> on a machine that runs a web server.</strong> If you have to (first setup, nginx not yet configured), switch the authenticator to webroot the same day.</li>
<li><strong>Write the diagnosis down for the client.</strong> "The provider disabled the server" was not the first time this symptom got that explanation. A one-line reflex — <em>does SSH work?</em> — separates infrastructure incidents from application incidents in ten seconds.</li>
</ul>]]></content:encoded></item></channel></rss>
