The Oligo research team recently uncovered an ongoing attack campaign, dubbed ShadowRay, targeting a vulnerability in Ray, a widely utilized open-source AI framework. This vulnerability, currently unresolved and lacking a patch, poses a critical threat to thousands of companies and servers running AI infrastructure.
Exploiting this vulnerability allows attackers to seize control of companies’ computing power and potentially expose sensitive data. The flaw has been actively exploited for the past 7 months, affecting various sectors such as education, cryptocurrency, and biopharma. To mitigate risks, all organizations employing Ray are urged to assess their environments for exposure and examine any suspicious activity.
CVE-2023-48022 Has a Critical CVSS Score of 9.8
The security flaw, identified as CVE-2023-48022 with a critical CVSS score of 9.8, allows remote attackers to execute arbitrary code via the job submission API. This lapse in authentication control within Ray’s Dashboard and Client components grants unauthorized actors the ability to submit, delete, and retrieve jobs, as well as execute remote commands.
Despite being under active attack since September 2023, Anyscale, the developers and maintainers of Ray, have not addressed the issue as of now, citing long-standing design decisions. They intend to integrate authentication in a future release as part of a broader security strategy.
Observations by cybersecurity researchers reveal that hackers have successfully breached numerous Ray GPU clusters, potentially compromising a wealth of sensitive data. This includes production database passwords, SSH keys, access tokens, and the ability to manipulate models.
Furthermore, compromised servers have been observed to harbor cryptocurrency miners and tools for persistent remote access. The utilization of an open-source tool named Interactsh further complicates detection efforts, allowing attackers to operate clandestinely.
Infiltrating a Ray production cluster presents hackers with a lucrative opportunity. With valuable data and remote code execution capabilities, attackers can monetize their activities while remaining virtually undetectable.
Mitigation Strategies
Oligo proposes the following mitigation measures to minimize the risk of CVE-2023-48022-based exploits:
- Adhere to the best practices for securing Ray deployments.
- Initiate Ray within a secured, trusted environment.
- Implement firewall rules or security groups to thwart unauthorized access.
- Apply authorization atop the Ray Dashboard port (default: 8265):
- If Ray’s dashboard needs accessibility, deploy a proxy incorporating an authorization layer to the Ray API when exposing it over the network.
- Regularly monitor production environments and AI clusters for anomalies, including within Ray.
- Recognize that Ray relies on arbitrary code execution for functionality. Traditional Code Scanning and Misconfiguration tools may fail to detect such attacks, as Ray’s open-source maintainers (Anyscale) have marked them as disputed rather than bugs, constituting a feature at the time of writing.
- Avoid binding on 0.0.0.0 for simplicity. It’s advisable to utilize an IP address from an explicit network interface, such as the IP within your local network subnet or a trusted private VPC/VPN.
- Exercise caution with defaults. Verify settings thoroughly; sometimes tools presume familiarity with their documentation.
- Select appropriate tools. While open-source maintainers shoulder some responsibility, the technical onus of securing open source falls on users. Utilize tools designed to protect production workloads from the risks inherent in runtime use of open source.
Milena Dimitrova
An inspired writer and content manager who has been with SensorsTechForum since the project started. A professional with 10+ years of experience in creating engaging content. Focused on user privacy and malware development, she strongly believes in a world where cybersecurity plays a central role. If common sense makes no sense, she will be there to take notes. Those notes may later turn into articles! Follow Milena @Milenyim
Follow Me: