This repository has been archived by the owner on Jun 16, 2022. It is now read-only.
-
Notifications
You must be signed in to change notification settings - Fork 3
April 16th Security Incident Retro
Joshua Zenn edited this page Apr 16, 2022
·
16 revisions
On 4-16-2022, a notice was sent from Heroku indicating a possible data breach regarding a system that stores credentials to authenticate with GitHub. After reviewing audit access logs, and pre-emptively revoking all Heroku OAuth tokens, it has been determined that there was no compromise of VirtuDoc services, application source code integrity, or user data.
- Unknown Time - Some unconfirmed sources estimate that starting on this date, several private repositories on GitHub that deploy to Heroku through GHA were cloned by suspicious sources, indicating a possible start date for the attack.
- Unknown Time - An employee at Travis CI disabled their OAuth token on @ARMmaster17's account.
- 12:03 AM - A notice was sent from Heroku explaining that a security incident had occurred, and that users would be notified if they were affected.
- Early Morning - Further information was sent from Heroku on their status page stating that as a precautionary measure, they are revoking ALL OAuth tokens for all customers.
- Early Morning - Various unconfirmed sources are stating that suspicious activity has also been seen with Travis CI OAuth tokens, indicating a possible larger supply-chain attack through an unknown vector.
- 10:45 AM - Email is seen by @ARMmaster17 from Heroku.
- 11:00 AM - OAuth tokens are disabled on @ARMmaster17's account for both Heroku. OAuth tokens for Travis CI were noted to have been already revoked by Travis CI staff.
- 12:10 PM - Audit complete of VirtuDoc repository. Results have been documented below.
Assuming no further updates from Heroku, no further action is being taken at this time.
- Because we conform to the 12-Factor Methodology, we have no hard-coded credentials in our database (which would have been leaked if we did not practice this because our repository was private).
- Our repository is already open-source, meaning that we are already encourage to follow safe DevOps practices when pushing new code.
- All hard-coded user accounts only work locally and on GHA, and not production. This means that no user accounts were compromised with theoretical access to real user data.
- By following the practice of
least-permission, the Heroku OAuth token did not have access to perform any malicious actions on the VirtuDoc repository. - If the above item was not true, force pushes to
mainare forbidden, and all changes must go through pull requests which requires a non-author sign-off, so a compromised write-token could not be used add malicious code to our application and push to Heroku. - All security scan API tokens are stored as GHA secrets, meaning that forged or bogus security reports cannot be released about our application.
- Because we push Docker containers, and not code slugs, if all of the above were not true and somebody force pushed malicious code to our
mainbranch, the deploy would fail because Heroku is expecting a packaged container, not code. Container deploys also require a separate secret key, which as currently understood is not part of the breach. - Database is protected by per-user access and Geo-IP firewalling, making database access by a bad actor very complex.
- Because we are using a sandbox account, a compromise of our Mailgun account due to a code push by a bad actor would not allow for us to be a vector for distributing scams or malware payloads through email.
- Our deploy pipeline should not have been using an OAuth token shared with @ARMmaster17's account. Although not related to this incident, it was noted that a breach in one of @ARMmaster17's repositories would affect Virtudoc, and vice-versa.
- For the duration of the incident response by Heroku, we are unable to automatically push new code to production. Instead containers must be built locally and then pushed to Heroku using the CLI tool.
The following is a worst-case scenario, assuming that all of the protections mentioned above are not true or a bad actor found a way to circumvent them. The purpose of this exercise is to show the financial and legal impact of this breach, had we not followed the practices we currently do.
- Actor obtains OAuth token from Heroku breach.
- Actor gains access to our private repository and clones it locally.
- Actor uses plaintext database credentials to access customer data (theoretical impact:
4 user accounts x $50,000 HIPAA breach fine = $200K fine plus 12 months of prison if found guiltyalso if any users live in the EU:up to $10,806,000 fine for GDPR violation). - Actor pushes code and deploys a crypto miner on Heroku using our account (theoretical impact: auto-banned from Heroku account, time lost looking for a new PaaS provider and migrating tooling to new platform).
- Actor logs into website using hard-coded credentials in repository. Pretends to be a doctor or admin and solicits further information from patients (theoretical impact:
$50,000 per Class 4 HIPAA violation (e.g. per patient)). - Actor pushes code that leverages our Mailgun emailing service (theoretical impact:
$16K per email sent under the CAN-SPAM act of 2003).
- GitHub account audits of both @ARMmaster17's account and @CS4850Group5A revealed no suspicious activity. All OAuth tokens have been revoked, so no future impact can occur from this incident.
- Deploys through GHA are no longer working, and will not work until OAuth is set up again.
- Once the incident is marked as resolved by the Heroku security team, @ARMmaster17 will set up a new OAuth token under the @CS4850Group5A account that will be used for Heroku -> GitHub communication.
- @ARMmaster17 will test that Docker containers can be deployed manually using the Heroku and Docker CLI. If the incident is not resolved before our Sprint 3 presentation, this method will be used to push new code.
- @ARMmaster17 will continue to monitor status updates from Travis CI to confirm that there is no impact from a second breach, and that the revocation of the OAuth token from TCI was an unrelated action.
- Use service accounts, not human accounts for CI/CD.
- Every repository should have its own credentials to a service.
- Even if your repository is private, always assume that anyone can read your code.
- Deploys should always require human approval at some stage in the process.
- Zero/low-trust architectures can severely limit a blast radius when done right.
- Repositories should have a "break-glass" procedure to deploy when their CI/CD is down or compromised.
- Sending signed containers is far more secure than directly hooking in your application's source code.