The machine I wasn't worried about

We were reviewing a bunch of open pull requests and deciding which changes to merge into main that day, knowing that anything reaching that branch would be picked up by our CI pipeline and deployed to the production servers.

Among the changes were what looked like font files, which seemed strange because we couldn't see why we needed fonts there or who would have uploaded them, so we asked the developer about them.

"I didn't upload any fonts?"

When we took a closer look, though, what was inside didn't look like a font at all, and it became clear that we were looking at malicious code disguised as something far less suspicious.

What had started as a discussion about the day's deployments now meant investigating several developers' machines, usually more than one per person, to understand how the code had ended up in the repositories and whether it had executed anywhere. If it had, we also needed to understand what else it could have reached from there.

Before any of this happened, we had already put some useful boundaries in place, with production and staging secrets centrally held in our deployment infrastructure rather than committed to repositories or routinely distributed in local configuration files. Alongside that, we had two-factor authentication and endpoint protection, and we kept development separate from production.

Those precautions appear to have helped limit the consequences, because somebody with repository access wouldn't find our production secrets sitting alongside the source code. That gave us a useful boundary to work from while we investigated, although it still left us needing to establish whether malicious code had run somewhere with other access available to it.

An extensive examination of one developer's current machine found no evidence that the malicious payload had executed there, while the investigation produced strong evidence pointing towards an older machine that was no longer being used for the project. Endpoint protection had detected malicious material on that machine, and although the work had moved elsewhere, it still held an old credential associated with repository access.

The findings don't give us a complete account of every step in the incident, but that old credential explains why a machine we had stopped thinking about in connection with the project still belonged in the investigation. We no longer needed it for the work, yet as long as it retained permission to act on that work, somebody else could potentially use it to do so.

That was the part that stayed with me: a machine doesn't stop being part of your security perimeter because you stopped thinking about it, and if it still possesses authority, it still matters regardless of whether you intend to use it again.

It also put the familiar "I'm okay. I have antivirus" reassurance in a slightly different light, given that endpoint protection had detected malicious material on the older machine without making its remaining access irrelevant. We also had 2FA, but GitHub's command-line authentication can use a token or SSH key without asking for a second factor on every operation, so that wasn't a reason to disregard an existing credential either.

As we considered what code running on a developer machine might be able to use, the concern widened beyond that old credential to the ordinary account's access to SSH keys, source repositories, authenticated cloud tools, browser sessions and database connections, with configuration files or shell history potentially revealing further useful information.

Depending on the setup, some of that may be locked away or require fresh approval, but whatever is already readable or usable by the ordinary user may also be available to code running as that user. It doesn't necessarily need administrator privileges if the operation it wants to perform is one the developer is already allowed to perform, because it may have all the permission it needs without them.

What did the control actually buy us?

After the incident, we strengthened repository protections, including making GitHub enforce at least one peer review before changes can be merged into main.

Peer review was already part of our operating procedure, and we trusted our developers to follow it, which had never been a problem. The gap became apparent when we considered somebody else using a developer's credential, because they would inherit its permissions without having any reason to follow our procedure.

Our trust in the developer hadn't changed, but we needed the system to account for a request authenticated as that developer coming from malicious code instead. Since our pipeline automatically deploys from main, enforcing review in GitHub puts another person's approval between repository access and a production release, rather than allowing a credential subject to that rule to push arbitrary code straight into production.

We also restricted force pushes that rewrite branch history, removing a particular way to damage a repository while leaving us needing to examine what else the credential permits. There is an odd wrinkle here too, because a large, unexpected rewrite of repository history is quite a conspicuous way to announce your presence, whereas an attacker who responds to the restriction by using a quieter permitted operation may be harder to notice. The restriction remains useful, although we may need to look for different activity rather than assume the attacker has lost access.

I would want those protections in place again, just as I would want the precautions we had taken before the incident, while being more careful about what we expected each of them to settle. Requiring somebody else to approve a change helps when their review can challenge what's being proposed, just as monitoring helps when the activity we care about produces a signal we can recognise and respond to.

We had done quite a lot right, and those decisions appear to have helped, but I wanted us to follow the reasoning beyond that reassurance. If endpoint protection reduces the chance of malicious code running, we still need to consider what that code could do if it gets through, rather than treating the presence of protection as the end of the question.

The question followed us elsewhere

That question came up again when we discussed hardening developer workstations, where active endpoint protection and normally working without administrator or root privileges are sensible starting points. I would apply the same restraint to AI coding agents, keeping a person involved in consequential decisions rather than giving them blanket permission to download, install and execute arbitrary code.

The difficulty is deciding how far to take that when a developer's job involves downloading, building, modifying and executing code, because if we lock the machine down sufficiently, eventually we have prevented the developer from developing.

Containers, virtual machines and disposable environments can help separate that work from the rest of the workstation, although the separation depends on what we bring into them. If we mount the host source tree into a development container, forward an SSH agent and provide cloud credentials and access to the Docker daemon, we have brought considerable authority into it for convenience, so calling it a container tells us very little until we examine what those connections allow. Docker's security guidance explains why shared directories and daemon access deserve particular attention here.

A remote development server can keep code execution away from a personal laptop, but moving the same permissions with it leaves us needing to ask what code on that server could reach, or we may simply have changed where we investigate the next incident.

I found it more useful to think about how we could make compromising the development environment less useful to an attacker, giving a developer room to work while questioning whether that needs to include standing access to production or unrelated projects. The older machine made that concern particularly concrete, because access we no longer needed was still available for somebody else to use.

Following that reasoning also brought me back to the production secrets we had kept out of the repositories, because regardless of how carefully we store them, they eventually have to become usable somewhere.

If malicious code can act through that process, it may be able to use the same access through an existing authenticated connection or the process's recognised identity, without needing to steal a password at all. Its reach still depends on the permissions we gave the application, which is why those permissions matter even when we have done the right thing with secret storage.

Keeping secrets out of source control protects them from source-control compromise, but it doesn't necessarily protect them from arbitrary code executing where those secrets are ultimately used.

Once we follow the access that far, the same question applies to a CI runner, a build container, a production server and the application workload itself, wherever code can run with permission to read, change or invoke something that might take it into the next system.

What stayed with me

We came out of the incident with stronger repository protections and a clearer reason for enforcing a procedure our developers had already been following, while the older machine reminded us that moving the work doesn't retire the access. Removing authority it no longer needed would have changed how useful that machine could be to an attacker, even if malicious code still ran on it.

I still want us to prevent that first compromise wherever we can, but while we are doing so, I also want us to decide what happens when a preventative control fails, before we find out through an incident. That includes looking at the permissions we have accumulated and considering whether they are still needed, alongside how well we have protected the machine or process holding them.

The precautions we'd taken had helped, but I would want to preserve those benefits without treating them as certainty about what could happen next.

The question I want to keep asking, alongside "How well have we protected this?", is: "Assume this is compromised. What authority have we just handed the attacker, and where can they go next?"

If you are working through security and architecture decisions like these in your own systems, tell me what's going on.