Reimagining CRO Change Order Management with AI

‍

Joe Roualdes: All right, let's go ahead and kick things off. First of all, thanks everybody for joining today. My name is Joe Roualdes. I'm the VP of Marketing here at Condor Software. With me today is Jeff Jahnke, who's our Director of Customer Operations. We've got an exciting webinar today on Reimagining CRO Change Order Management with AI.

‍

This is the second webinar that we're doing in a series we're calling Practical Applications of AI in Clinical Finance and Ops. Bit of a mouthful, but you guys get the point. If you like today's webinar, there are a lot more sessions that we're gonna be doing, both virtual webinars and in-person workshops.

‍

We're also gonna be publishing a lot of content within this series, so keep your eyes tuned. We'll be sharing those with everybody. Some housekeeping before we get started. If you've got questions do me a favor, add it either into the chat or actually add it into the chat so everybody can see.

‍

And what we'll be doing throughout this session is if there is a question that we're getting a lot, I'll stop Jeff, he'll go ahead and answer that question. If not, what we'll do is we'll answer questions at the end of the session. With that, let's get started. Jeff?

‍

Jeff Jahnke: Thanks, Joe. Thanks everybody for joining.

‍

A couple months ago we were at the ABFO National Conference, and our CEO, Jen Kyle, delivered an amazing panel on AI. And this slide pretty much encapsulates the main takeaway, which everybody knows that they need to be using AI that AI can help them in their workflows, but they don't know how to start.

‍

And so a lot of the questions that I got just mingling with people, circulating around the room after the panel, was asking us at Condor to help share some practical hands-on examples of how to use AI in clinical finance. And so that's why we decided to do this series. So really excited to teach you all some of those practical applications.

‍

So the series that we're hosting and kicking off today is gonna cover a variety of topics over the course of several sessions. But today we're starting with potentially one of the biggest pain points that you all feel in both finance and clinical operations, which is the CRO change order review process.

‍

And we've done some work to help reimagine how folks can review and understand kind of what's going on under the hood of those CRO change order budgets. So today we're planning on reviewing a CRO change order end to end, surfacing risks and concerns before they become problems. We'll also be validating some of those findings and drafting follow-on language back to the CRO, requesting justification for risky line items So before we dive into the actual practical workflows here first I just wanna talk a little bit about who we are at Condor.

‍

Condor's a financial intelligence platform that's been purpose-built for life sciences R&D. We've partnered with all the big four and we're trusted by leading pharma organizations to help streamline their finance and clinical operations workflows. And some of the benefits that our customers are seeing, 90% forecast accuracy, 70% faster month to close, and up to 30% budget savings.

‍

And hopefully you walk away today with some of the new tools in your tool belt to help you, especially with that last one, some budget savings. 'Cause I know that CRO change orders can be one of those big ticket surprise type exercises where all of a sudden your study budgets balloon So why AI and why now?

‍

I'm just gonna do a quick little orientation here about where we're at with AI as a collective and where we're going. So I think most of us are probably in stage one on this view, which is essentially what you're familiar with in terms of a, a language model like a Copilot or a ChatGPT or a Claude.

‍

Basically, it's a chat window where you can have a conversation with the AI, and it can give you some general purpose results. Really good at summarizing, extracting, drafting emails, drafting messages. A lot of fast answers, but the reason those answers come very quickly is 'cause those models don't necessarily have a ton of context on your specific problem.

‍

It's very generalized. The second stage that we're moving towards it, for the most part, are workflow embedded AIs. That might be where you're using a particular app. Like for example, if NetSuite rolled out a, an AI agent in their app and were able to provide you with some general context about your particular problem.

‍

The process is consistent, but the, the reasoning is still very generic. And then the third stage is purpose-built AI. This is where these models are trained to be domain experts in your area of profession profession. And, in the case of some of the models that we are developing here at Condor on our ontology, on our knowledge graph, these AI models think and understand about your particular protocol, your particular clinical trial.

‍

They understand all the ins and outs, all the nuances, and therefore can give very reasoned precise answers about the problems that you're all facing. So keep all of these three stages in mind today, 'cause we're gonna visit all three of them over the course of the hour that we're spending together

‍

So let's also talk a little bit about CRO change orders just to underscore kind of how big and prevalent of a concern or a problem this is throughout the industry. They're uniquely painful. I'm sure all of us have felt that pain. I've had the opportunity to see it on both sides because I used to work for a CRO and was the architect of a lot of those change order budgets.

‍

I'm in a unique position to share a little bit about this. First, let's talk about protocol amendments because those are the main catalysts for the majority of change orders, I would say. If we look at clinical trials, protocols with at least one substantial amendment have ballooned over the last few years.

‍

They went from basically common to universal. So for all phases of studies, phase one, phase two, phase three, from two thousand thirteen to two thousand fifteen, about fifty-seven percent of clinical trials had a protocol amendment, and that has now ballooned up to seventy-six percent.

‍

So three out of every four have at least one major protocol amendment. And many have many more. Looking at phase three, you can see that gap is even wider. Sixty-six percent typically have a protocol amendment, and now up to eighty-two percent. So nearly every single phase three clinical trial, regardless of indication, is having a major protocol amendment, which is then resulting in a CRO change order on the back end.

‍

And you can see that's just one, on average, three point three major protocol amendments on a phase three trial Also, the later the phase, the larger the bill on average from a CRO change order. You can see that phase one on average... Or I'm sorry, phase two on average, CRO change orders are about $141,000.

‍

Phase three on average, they end up being $535,000, which is about 3.8x. I think, though, that all of us that have seen these change orders know that depending on the indication, depending on the length of the trial, depending on the complexity, that $535,000 is a truly a mean. It can get substantially larger than that.

‍

And, there's also a real operational impact from these change orders. They take about three months to negotiate on average, and that's three months of time where, yeah, the trial's progressing, but there's a waiting period. There's kind of a a delay to your timeline, which may result in not, you know, on the back end, a, a drug being released to market later than we'd all hope.

‍

You also see that CROs tend to lowball their bids. So when a study's first getting started, they count on that change order process to make up the margin. They wanna entice you to sign with them with a low cost, knowing that they're going to change order you down the road to get that budget back up to where it should have been from the initial bid and then beyond.

‍

Because obviously signing with a CRO is a significant commitment, and untangling that commitment midstream is very difficult to do. They know this, and they take advantage of that.

‍

And the burden of reviewing that lands both on clin ops and on finance. So let's talk a little bit about the change order budgets themselves. Oftentimes they contain hidden costs. Unit price increases may not be super clear in terms of how they impact or how they relate to the legitimate scope changes contained within those change orders.

‍

The explanations on every single line item can be a sentence or two long and not give a ton of detail. Negotiation asymmetry. These change orders are in development at the CRO, sometimes for months before they reach your desk, and then the CRO will say, you've got a week or two to review this."

‍

And it's up to you to untangle all of the complexity and understand what's really hiding in that document. And that's a tall order. They count on that asymmetry to get you to agree to things that are hidden in those budgets. Another big topic of concern is retrospective recognition. It's not always clear what work that is scoped within that change order has already occurred.

‍

Change order... cROs are notoriously bad at recognizing out of scope, even the best ones. Because the change orders are constantly evolving, that latest version of the change order might not have made it into your unit reporting. Or if you're on a milestone contract, you might not even get visibility into the work that's actually being performed at that given time, and that results in a large surprise true-up bill at the end of the day.

‍

And then the last part, no system of record really. And I alluded to that with the trackers, right? The out-of-scope reporting's not great. Inconsistent unit pricing, even across trials or within trials. Version control chaos. You're sometimes going back and forth on four or five different versions of a change order.

‍

I've seen changes revert to where, the CRO agrees with a sponsor's pushback on something, but then in a subsequent version, they revert back to the original version, and it doesn't get caught 'cause that was already addressed, that sort of thing. And it turns into weeks of delay. There's not a lot of reporting on how long the average change order takes to negotiate because it's so variable.

‍

I've seen them take up to, and even sometimes over a year, a calendar year to negotiate a change order budget. I think most usually try to get wrapped up within a quarter, but, but let's talk about this over the course of your entire portfolio too, right? Let's just take a, a biotech that might have eight active studies at a given time.

‍

That eight active studies times 3.3 on average protocol amendments results in 40 to 50 change orders over the course of those eight studies including ancillary vendors. That's roughly $10 million in unplanned, unbudgeted cost in motion, and it's up to you and your team to absorb the review and make sure that your company is not getting taken advantage of during those negotiations

‍

And to make matters even worse, if we try to put an index on this, you can see that over the last decade or so, the volume of change orders has increased about two times, while the average head count of a finance team at a biotech or life sciences company has remained relatively flat. So that difference, the arrow there, that's the gap that your team is having to absorb on top of their regular day job of forecasting cycles, accounting closes, et cetera, et cetera.

‍

And that gap, that gets closed either with tooling or on long weekends

‍

So now that we've identified the core problem and how, in general how AI can help, let's take a look at a live demo here. What I have here is a change order document with all of the different project parameters that are changing, the budget summary, and the budget detail. And you've got the CRO comments and all that good stuff here.

‍

Now, what I'm gonna do is I have Microsoft Copilot open. Now, the, the wonderful thing about Microsoft Copilot is it comes bundled with a Microsoft 365 license. So this is something that after this webinar you all could do on your own. It's something maybe that you all could do right now if you happen to have it open and have a change order in process or change order handy.

‍

I encourage you all to type it in. Now what I'm gonna do is I'm gonna drop a prompt in here And this is what the prompt says. It says, "I'm a file- finance director at a biotech company. I've received the attached change order budget from our CRO for a phase three study. Please analyze it and give me a summary of the total cost change and the top five largest cost increases, a breakdown of direct fees versus pass-through costs versus investigator costs, any line items that appear to be retroactive or cover work already performed, and a list of five questions I should ask the CRO before approving this change order.

‍

Format the output as a brief memo I can share with my CFO." Now, what this prompt is doing is giving the model a lot of context on what I need. So I'm telling it the role that it should assume, I'm telling it specifically the activity I need it to perform, and I'm giving it a list of deliverables to return to me.

‍

That's just general AI advice. The more context that you give the model before submitting a prompt, the better the output from that prompt is gonna be, especially this, "I'm a finance director at a biotech company." Anytime I'm working with an AI model, regardless of the task, I'm trying to give it the persona that it should embody so that it knows how it should be thinking about the problem.

‍

And so what I'm gonna do is drop in that change order Excel file And then I'm going to hit send.

‍

And now what Copilot is doing is drafting that memo that I asked for with those different components. So here's the executive summary. They're requesting an increase of two point two five million. The budget's increasing from ten point six to twelve point nine three. It's going from thirteen sites to twenty sites.

‍

We're keeping the enrollment targets the same. The study duration has not changed. Here are the largest cost increases by category. Clinical monitoring is growing by about one point three. Investigator fees are going up by almost seven hundred thousand dollars. The thing that you can see here, though, is this looks pretty solid in terms of, the explaining what the changes are.

‍

And that is a big problem when you're reviewing CRO budgets, is summarizing what those changes are into something that's digestible. Because those CRO budget grids, as can be hundreds of line items long. You're having to go through sometimes with a ruler to look at each line individually and make an assessment.

‍

So this does a really good job of teasing out what is changing in the change order

‍

So yeah, genuinely useful. It was quick. We did that in 30 seconds. Once you have a prompt that you can develop, it's super fast. Solid starter questions for the CRO, the CFO ready memo, and probably the most important part of this is no procurement needed. 99% of you already have this on your desktop as part of your Microsoft accounts.

‍

But where it stops is, you saw it, it gave you a lot of what's changing, but nothing about why it's changing or how it's changing or if it should be changing. We didn't tie any units back to the original contract. It has no knowledge of your actual clinical trial, so it can't tell you if any of those changes are reasonable or not.

‍

It didn't spot any unlabeled retroactive work. It just said, "This is what the change is." There's a million dollars of additional monitoring, which is helpful, but not defensible. There's no benchmark or understanding of what normal looks like. There's no relating to other clinical trials within your portfolio that might have had similar scope changes.

‍

There's no relation to clinical trials from other sponsors that might be similar in terms of indication or study design. And there is a different answer every run. The numbers are pretty deterministic, but if you plug and do that, that same prompt five, six times, you're gonna get the same flavor of answer, but five or six different answers.

‍

Joe Roualdes: A- and really quickly, Jeff, for folks who are new to using LLMs for this use case, if you were frantically trying to screen grab Jeff's prompt, we're gonna have a blog post that we're gonna publish on Monday next week that will include all of the recommendations that Jeff has, plus the prompt, so you have that and also know which files that you wanna include within that prompt.

‍

So we're gonna make it really easy in a step-by-step guide. It'll come out Monday. You'll get it via email. So if you screen grabbed, wait for the email. It'll include more detail

‍

We really appreciate everybody showing up and everybody hanging around. It's great to see the numbers stay consistent throughout. As Jeff and I mentioned, this is one of webinars that we're gonna be doing within our practical applications of AI in clinical finance and ops.

‍

The next ones will be coming up shortly. We're gonna use the feedback you all gave us today to figure out which we prioritize. We are also gonna be bringing these on the road live. So if you're going to up- upcoming ABFO events and FORMA events, et cetera, we're gonna be doing some live workshops in person where you can bring your laptop and do things live with us.

‍

We'll actually help you do things in real time. So really excited about this, really appreciate the feedback. Big thanks to everybody for showing up and big thanks to Jeff for taking the time to walk everybody through everything. We'll see you all soon. Thanks.

‍

Jeff Jahnke: Thanks, everybody.