Health systems sometimes develop a syndrome in which they test lots of new ideas and technologies that fail to scale up and end up in the dreaded “pilot graveyard.” Chad Jones, senior vice president of information systems at Texas-based heath system Baylor Scott & White Health, and Amy Goad, managing director at Dallas-based Sendero Consulting, recently sat down with Healthcare Innovation to discuss why this can happen in healthcare systems and some ways to avoid it.
Healthcare Innovation: I understand that both of you have given some thought to how health systems can take steps to make sure a higher percentage of digital health and IT pilot projects successfully scale up. Let me start by asking Chad whether you have had the experience of churning through pilots that didn’t scale up to the next level?
Jones: The short answer is yes. For me, there are a lot of reasons why those things happen. I think number one is insufficient planning and road-mapping and governance around where we we are going and what we are doing with our IT skills and resources and having really clear, transparent roadmaps on strategies about where we’re going. When you don’t have that, it actually allows this vacuum or opening for shiny objects or super enthusiastic people to be able to insert pilots that don’t really go anywhere.
Goad: I think healthcare is uniquely susceptible to this for a couple of reasons. Since the pandemic, the number of healthcare tech companies has grown tremendously, and for good reason, right? The pandemic opened the doors of innovation on payment models and other things. So healthcare leaders were inundated with all of this new technology with the pitch that this will make your patients better and make your health system better and more competitive, and help your workforce avoid burnout.
Someone might say, ‘Hey, I’ve got this new digital app that’s going to allow us to monitor our patients so much better.’ That’s kind of hard to say no to, right? And if it is hard to say yes to at scale, it’s easier to say, ‘Sure, let’s pilot it. Let’s just try it and see if it gets the results that we think it will.’ So it’s all well intended, and the promise of what they could be is exciting and important.
Jones: Amy raises a good point. I would also point to the private equity-backed spot solutions that have flooded our industry in the last four years. Billions of dollars of private equity money has been put to work on extremely niche micro solutions. They have flooded the zone.
Goad: Yes, and innovative system like Baylor Scott & White are targets. Sometimes the vendor will say ‘we’ll give it to you for free. If you do this for six months, we won’t charge you anything. We can have you co-develop it.’ So there is an enticing business case for that, right? It would be irresponsible for a system like Baylor to not at least entertain some of these.
HCI: Amy, when you’re called in to consult with a health system, are there certain common things you see as far as organizational structures or processes that are leading to this problem not being addressed? Is it setting up governance structures to prioritize projects and having a strong project management office?
Goad: Having a strong project management office is key. I think there are still going to be challenges, because many health systems have hospitals that are different entities; there are joint ventures; there is physician ownership. So even with a great governance structure, there are still going to be negotiations and issues to work through to appease everyone who is a rightful stakeholder. Understanding those dependencies can help level the playing field. We work with our clients to help answer some of the questions that, off the bat, could tell you: Is this going to work or is it not? Do you have the right baseline infrastructure to even do some of this? Some of these pilots require access to lots of data. Do you even have the right data infrastructure to allow this tool to be successful in your group? Sometimes if the answer is no, you can take it off the table pretty easily, in a very unemotional way. So governance is important. Even with governance, you still have people who have different priorities, and that’s what make healthcare complicated, right? You’ve got people looking at the same things from different angles. So governance can only do so much.
HCI: Sometimes pilots involve working with startup vendors on a project. Does that complicate things?
Jones: We’re an organization that is somewhat risk-averse by nature. Doing pilots is interesting, but we always ask how we could scale something before we even do the pilot. Let’s imagine it’s successful. What does it take to scale this? What does this look like if it’s deployed enterprise-wide at Baylor Scott & White Health?
We look at at the vendor viability. Oftentimes what we find is is a lot of these guys are too small and we won’t work with them. Because we know that even if this pilot is successful, we’re not going to go into a long-term relationship with a garage band, because there’s too much security risk. There’s too much vendor risk. They can’t sign a contract with liability and insurance that would satisfy our needs. For instance, there are a million garage bands doing AI right now. That’s where you need the discipline to do this upfront analysis before you waste your time and effort to do this cool pilot. Oftentimes we can look and see if there is an existing vendor that’s doing something similar.
HCI: AI is a great example of an area where people are talking a lot about governance, but a health system can have projects going on in revenue cycle management, in radiology, with these AI scribes for the EHR. How do you set up a structure to evaluate all of those types of things? Is it centralized or is it department by department?
Jones: We’re figuring that out right now. We’re trying to stand up an AI review committee, but it’s really hard, exactly for the reasons that you just said. Evaluating an AI tool in the revenue cycle space vs. a niche radiology tumor assessment tool — they’re very, very different. And thinking that this one committee will be able to look at an administrative bot and a clinical bot — they are such vastly different use cases with vastly different impacts, and candidly we’re struggling with on the governance of that. I think what will happen is that AI will rapidly become status quo, and then we can fall back on our existing processes and methods, so that we don’t actually need to introduce something totally new and unique to assess a particular AI attribute. I think all the vendors will end up incorporating some sort of AI into their solutions, and we’re going to learn to just accept that and bring that in.
HCI: Amy, do you want to add anything on the AI front?
Goad: AI or not, we work with clients to identify what the project is trying to accomplish. What’s their business objective? A lot of people say it’s AI, but really it’s just some automation. I think that the term AI is loosely applied, and we see a lot of people diverting it into the separate bucket, which is causing a lot of confusion, exactly as Chad just described. Who then is the decision maker? But if you treat it just like any other project, with some acknowledgement that there are some additional complications and other people that need to be at the table, it shouldn’t be a separate entity. It should follow the same path of vetting.
Going back to the original question about the pilot graveyards, sometimes people are very afraid to call it quits. There’s not always a clear ‘ding, ding, ding, this isn’t working.’ Someone will say, ‘Well, it’s kind of good and, maybe it just needs a little bit longer’ because they can’t pinpoint why it’s not working. Do we need to rework processes? Is it the tools? This is only a beta, and in version two or version three we’re actually going to get the value out of. But sometimes you need the discipline to say that if it not doing XYZ, then we’re going to turn it off and not pay for this anymore.
