Continuous integration and continuous delivery (CI/CD) misconfigurations discovered within the widely-used TensorFlow machine learning framework raise concerns about potential supply chain attacks.
TensorFlow Vulnerabilities and the Risk of Supply Chain Attacks
Praetorian researchers Adnan Khan and John Stawinski highlighted vulnerabilities that could have allowed attackers to compromise TensorFlow releases on GitHub and PyPi by manipulating TensorFlow’s build agents through a malicious pull request.
Exploiting these misconfigurations could have enabled external attackers to upload malicious releases to the GitHub repository, achieve remote code execution on the self-hosted GitHub runner, and even obtain a GitHub Personal Access Token (PAT) for the tensorflow-jenkins user.
TensorFlow utilizes GitHub Actions to automate its software build, test, and deployment pipeline, with runners executing jobs in the workflow. GitHub’s documentation emphasizes the use of self-hosted runners with private repositories due to potential security risks associated with public forks.
The identified issue allowed any contributor to execute arbitrary code on the self-hosted runner by submitting a malicious pull request. Praetorian identified TensorFlow workflows executed on self-hosted runners, revealing that fork pull requests from previous contributors automatically triggered CI/CD workflows without requiring approval.
Further investigation exposed non-ephemeral self-hosted runners with extensive GITHUB_TOKEN permissions, allowing an attacker to upload releases, push code directly to the TensorFlow repository, and compromise the JENKINS_TOKEN repository secret.
The disclosure prompted TensorFlow’s project maintainers to implement crucial security measures by requiring approval for all workflows from fork pull requests and restricting GITHUB_TOKEN permissions to read-only for workflows running on self-hosted runners. These changes, implemented by December 20, 2023, aimed to enhance security and prevent unauthorized access.
Conclusive Thoughts
This incident showcases the rising threat of CI/CD attacks, particularly impacting AI/ML companies relying on significant compute power. Since more organizations automate their CI/CD processes, strict security measures become imperative to protect against potential vulnerabilities.
In addition, the researchers also disclosed vulnerabilities in other public GitHub repositories, including those associated with Chia Networks, Microsoft DeepSpeed, and PyTorch, reinforcing the need for ongoing security assessments in the evolving landscape of software development and deployment.
Milena Dimitrova
An inspired writer and content manager who has been with SensorsTechForum since the project started. A professional with 10+ years of experience in creating engaging content. Focused on user privacy and malware development, she strongly believes in a world where cybersecurity plays a central role. If common sense makes no sense, she will be there to take notes. Those notes may later turn into articles! Follow Milena @Milenyim
Follow Me: