The healthcare industry is ground zero for AI companies and the rollout of their products: Microsoft tells us that AI is better than doctors at diagnosing complex medical conditions. Nvidia claims that its chatbot, a partnership with the startup Hippocratic AI, can outperform nurses on detecting over the counter drug toxicities.¹ AI firms suggest that their tools will improve workflow efficiency, increase diagnosis accuracy, and unlock “personalized treatment” or “precision” medicine. For some elite facilities, AI tools promise to enhance patient care through innovation. The Mount Sinai hospital system, for instance, has invested hundreds of millions of dollars in AI tools, turning their facilities into experimental labs. Amidst mounting financial pressures on the healthcare industry–from federal Medicaid/Medicare cuts, increased rates of uninsured patients, reduced access to primary care, and an increasingly corporate ownership structure—AI companies present their tools as a cost-effective and common-sense solution for under-resourced facilities.

But the reality of AI in the healthcare industry is far different:

For patient safety and confidentiality, the dangers loom large. Chatbots can fail to note whether a patient has drug allergies, a potentially life-threatening oversight. Ambient scribes can invent conditions and describe conversations that did not occur.
For healthcare workers, the risks are profound. While AI technologies are often adopted under the guise of “transforming” and “democratizing” healthcare, they are used to justify job displacement and undermine healthcare professionals’ judgment. The adoption of AI tools may lead to reduced staffing supports, diminished care for patients, stressed facility budgets, and untold reliance on corporate actors. The rollout of new so-called budget-saving AI tools like Abridge, Nabla, and Open Evidence may ironically increase operating costs because of labor-intensive implementation expenses and ongoing subscription requirements, among other factors.
Institutions are not immune to AI-related harm. AI tools may be drawing healthcare companies into greater debt, adding more risk-intensive working conditions, sabotaging patient care, and consolidating corporate power. Even though healthcare is the largest employment sector in the US and one of the only growing ones, new human management technologies may be turning the workplace into an auction where decision-making is obscured and trust is relegated to computational software.
AI firms may be accelerating the healthcare industry’s move away from democratic institutions, good governance practices, and public oversight. AI tools are centralizing and obscuring decision making over health and care. AI firms are building an opaque health infrastructure in which the data used to supposedly democratize information is held by a few players while access to the decision-making processes is largely impenetrable for the majority of us.

Healthcare is one of the most high-stakes sectors for AI deployment. And it has become a central site for legitimizing AI’s broader social value, with industry leaders frequently positioning healthcare AI as an unambiguous public good. Yet there remains limited independent scrutiny and evaluation of how AI tools are actually integrated into healthcare settings, how risks are distributed, and who bears the costs, despite significant public investment and rapid commercialization. This gap is the starting point for our current AI and Healthcare work.

* * *

The AI Now Institute has tracked the healthcare industry’s rapid adoption of large-scale AI tools over the past decade. Now, we are building an independent, participatory, and action-oriented research portfolio to interrogate the deployment of AI across healthcare systems. This work is focused on the consequences of this deployment for patient safety, worker dignity, public spending, and democratic accountability. Specifically, we are assessing (a) the experiences of nurses on-the-ground; (b) the shifting regulatory landscape; and (c) the political economy of the healthcare industry, including the business practices of companies selling and deploying AI technologies.

Healthcare workers are on the frontlines of Silicon Valley’s new AI tool deployment in medicine, be it in long-term care facilities where AI-sensor machines “watch” to see if a patient has fallen out of bed or in major hospital systems where nurses use automated text-to-speech tools like AI Scribe to take notes on patient conditions. Healthcare workers are also some of the first to see how AI firms experiment differently in different settings, be it acute (hospitals) or post-acute (long-term care and rehab facilities), private equity-owned or nonprofit, rural or urban, and unionized or non-unionized. Just as MRI machines and Da Vinci surgical robots are not accessible evenly across the US, neither are AI tools evenly deployed across healthcare infrastructure. Through collaborative research with healthcare workers, we are examining the promises, mechanics, limits, and geographies of new automated-decision-making technologies.

The regulatory landscape for healthcare and AI remains insufficient, and the appropriate standards for validation and testing of AI systems are underspecified. Healthcare AI bills tend to not focus on the critically important labor issues, and labor bills tackling AI tend to not focus on the particular nuances of the healthcare industry. For a time, the nonprofit Coalition for Health AI (CHAI) tried to fill some of these oversight gaps through a self-regulatory “AI Assurance” approach that would provide evaluation frameworks co-developed by tech firms and hospital systems. However, CHAI failed to develop independent benchmarks and, after three years, pivoted in its mission. The coalition’s priorities are now focused on “disclosure of conflicts of interest,” “protecting intellectual property,” and matching other priorities set by corporate stakeholders. Patient care, worker safety, and data privacy remain a wild west in the age of AI-meets-healthcare.

Any assessment of AI’s incursion into the healthcare industry must be paired with legal analysis of how AI firms and their boosters engage with state laws, government contracts, and regulatory institutions. Some AI firms seek to rewrite existing rules while others exploit lucrative gaps in healthcare regulation. Many AI tools, for instance, do not yet trigger FDA scrutiny or clinical trial requirements because they are not recognized as physical, medical devices. Other AI products like ambient scribes, algorithms designed for cost-saving on sepsis, and acuity scoring systems are not currently subject to governmental review even though they may significantly reshape healthcare workflows and impact medical outcomes for patients. Non-compliance with HIPAA is also common with AI tools that handle private conversations and sensitive clinical notes. To make these regulatory gaps permanent, the Healthcare Leadership Council—which represents Johnson & Johnson, Labcorp, Amazon, Mayo Clinic, McKesson, Merck, Mount Sinai Health System, AstraZeneca, UnitedHealth Group, Blackstone, BlueCross BlueShield, and more—is lobbying for even less regulatory oversight; this group is advancing a temporary ban on any state-level legislation about AI uses in healthcare.

AI’s entry into the healthcare industry—and the threats it poses—must be understood alongside the industry’s new political economy. Over the last fifty years, financialization, capital-intensive procedures, and the rise of business administrators in medicine have built an unprecedented, for-profit, health-care empire. Nearly twenty-five percent of community hospitals are investor-owned while more than 700 rural hospitals are at risk of closure. The dangers of this new model of medicine have worsened with the entry of private equity firms, which today control “more than four hundred hospices, nearly 10 percent of gastroenterologists, 8 percent of dermatologists, and more than 10 percent of oncologists.” Such facilities get

AI Now Institute

Message History

Delivery option(s)