Real-World Data and Evidence Generation with Dr. Hilary Marston

Q&A with FDA Podcast | Transcript

Dr. Roach: Welcome to “Q&A with FDA,” from the FDA’s Division of Drug Information. In this podcast series, we answer some of the most frequently asked questions that we’ve received from the public. My name is Dr. Sara Roach, and today we are joined by Dr. Hilary Marston, FDA’s Chief Medical Officer and primary clinical advisor to the FDA Commissioner Dr. Robert Califf. Dr. Marston is with us today to discuss real world evidence, real world data, and how FDA continues to advance clinical trials and promote reliable evidence to support the safety and effectiveness of medical products, including drug products.

Hi Dr. Marston, thanks for being here today!

Dr. Marston: Thanks so much for inviting me.

Dr. Roach: For a long time, the FDA gold standard of drug evaluation has been well-designed clinical trials. For some time now though, the Agency has been conveying expectations concerning both real-word evidence and clinical studies using real-world data that are submitted to the FDA to support regulatory decisions. Can you describe how the agency is combining these approaches to support regulatory decisions?

Dr. Marston: Sure, and thanks for the question. There continues to be considerable interest in using real-world data to generate real-world evidence to support regulatory decisions about effectiveness. But, before diving in, I want to make sure that we’re clear on the terms here, because they’re often used interchangeably, which leads to some confusion. So, real-world data are data relating to patient health status or the delivery of health care, that are routinely collected from a variety of sources. For example, data derived from electronic health records, medical claims data, data from product or disease registries, and data gathered from sources such as digital health technologies – all of these can inform information about a patient’s health status.

Real-world evidence, on the other hand, is derived from the analysis of real-world data, and it’s the clinical evidence about the usage and potential benefits and risks of a medical product. So, FDA has used real-world data primarily in its evaluation of safety for some time now, and in limited circumstances, to inform decisions about effectiveness.

Dr. Roach: What we now call real-world data have actually been used for years. Can you explain more about FDA’s history of using real-world data?

Dr. Marston: You’re absolutely right – so real-world data are not new in clinical research. The sources of data, however, have expanded greatly, right – we are swimming in a sea of data at this point, both in our daily lives, but also in our interactions with the health care system. So a great example of this is digital health technologies or DHTs. They have the potential to expand our understanding of patients’ interactions with drugs, with treatments, with potential devices, well beyond the clinic. So an example of that – we know that there is absolutely a phenomenon called white coat hypertension. Digital health technologies have the potential to take that and really minimize the effect that that can have on findings in a clinical trial. So we get beyond what’s measured in the clinic and get into the setting of the real world, of the patient.

Dr. Roach: I’m sure our audience would love to know what’s new in this area.

Dr. Marston: Yes. Well first, we’ve expanded our use of real-world data in the post-market space. So we’ve been using real-world data in the post-market space for some time to monitor and evaluate the safety of approved drugs. But, that’s expanded with systems like our Sentinel System. So the Sentinel System uses electronic health data, such as medical claims and pharmacy dispensing data, to execute some of our pharmacoepidemiologic queries and studies.

Not only do we perform those sorts of studies in collaboration with Federal partners like CMS (the Centers for Medicare & Medicaid Services) or the Veterans Health Administration, but we also use this with some of our international partners. So for example, the Clinical Practice Research Datalink captures data from the United Kingdom, and they’re very rich data sources in the national, in their NHS system.

Dr. Roach: Can you talk about some of the challenges in collecting real-world data to assess the safety and efficacy of drug products?

Dr. Marston: Yes. The current premarket system for generating and evaluating evidence, such as clinical trials – I think we can all agree that that works reasonably well. Not to say that there aren’t studies that end up being poorly designed, but in large part it is working quite well. The post-market phase, however, often fails to address questions that can only be addressed in the setting of clinical practice. And often they’re left unanswered because of a separation from clinical practice and research clinics. So the fragmentation of the health care and the clinical research system is really a false fragmentation, and there’s a lack of organization around common, transparent, high-quality information that would help us bridge that gap.

So what we need here is an evidence generation ecosystem that could help to overcome the challenges that are inhibiting collaboration. So, for example, advances in the availability and analysis of real-world data – they have the potential to streamline and improve the efficiency of clinical studies. Together with new trial designs - those offer the potential to meet patients where they are.

FDA is doing its part to support this sort of innovation, whether it’s through recent guidances that we’ve published and just finalized, the Decentralized Trials Guidance, or dedicated programs like the newly launched CDER Center for Clinical Trial Innovation, or C3TI.

Dr. Roach: Earlier you defined the difference between real-world data and real-world evidence. Can you expand upon real-world-evidence?

Dr. Marston: Yes, and as I mentioned, real-world evidence has been used historically to gather information about effectiveness, but on a very limited basis. More recently, FDA has taken steps to strengthen systems for generating and gathering data, and for analyzing those data. And there’s increasing interest both in the FDA, but also certainly in the product development community and patient communities in using real-world evidence for regulatory decision-making in the pre-market setting, including to support a determination regarding a product’s effectiveness and to the inform benefit-risk analysis that’s the bread and butter of our pre-market evaluation.

Real-world data have also been used to inform study design and endpoint selection. Examples of this – you can use real world data in order to identify clinical trial sites, in order to identify clinical trial volunteers themselves. This can help you make your study more efficient, and ultimately lead to answers more quickly.

Study design is also an essential part of producing real-world evidence. So, these trial designs include non-interventional studies as well as interventional studies such as randomized trials. And that actually bears emphasizing again – randomized trials are certainly part of real-world evidence. And that’s an area of misunderstanding that we’ve seen sometimes.

The strengthening of real-world data systems, in turn, provides reliable evidence that’s needed to demonstrate the safety and effectiveness of medical products, and ultimately that means we get better quality information that will help the use of these products in clinicians’ hands, and ultimately benefit patients and families.

Dr. Roach: There must be many challenges that reviewers have to consider when evaluating real-world data and evidence. Can you describe some of these challenges?

Dr. Marston: Glad to do it. And first I should emphasize that I am not a reviewer, so this is somewhat secondhand for me. But what I understand is that realizing the potential of real-world evidence in regulatory decision-making presents some challenges. I think we’re all excited about the potential, but we need to be careful that it really, that we’re using the data in a responsible way to really understand the performance of these products.

So, examples here: real-world data are not collected in controlled clinical research settings. So that means that we’ll have electronic health records and medical claims data. They may not capture all of the data elements needed to answer a question of interest. There might be missing data, data discrepancies. And one should also note that the electronic health record is really set up as a billing system. So the financial incentives that drive collection of that information are really set up to optimize reimbursement as opposed to research needed for our purposes. In multi-site trials, this can be a particular challenge, because data formatting from one center to another can be different. So examples of this – there might be differences in the way that lab values are formatted. So hemoglobin A1c might be entered differently in, in different electronic health records.

This is something that the Oncology Center of Excellence is trying to address in partnership with HHS’ Office of the National Coordinator for Health Information Technology, or ONC. And there are also partnerships between CDRH and ONC to look at that lab data question.

Another issue is the incorporation of scheduled assessments that may be required for a research protocol but aren’t part of routine clinical care. So individuals just aren’t necessarily brought in at the exact time point that you may want them to be brought in for a clinical research protocol.

In addition, we find that data from the routine-care settings are focused on patients at the bedside, or in the clinic. That’s completely appropriate, but for us, we need to consider the data’s reliability and its relevance to the regulatory question at hand. So, an example of this - information on clinical measures of disease severity can be lacking in health insurance claims, so we need to find better ways of examining the unstructured data in electronic health records in order to fill in some of those gaps.

Another issue here is safeguarding patient privacy and informed consent. You can imagine that that would offer different challenges when you’re looking at the entire electronic health record versus a snapshot in a clinical research setting.

Dr. Roach: You mentioned informed consent. How do these new systems and technologies mesh with human subject protection?

Dr. Marston: Well, first I’ll say that FDA is committed to protecting clinical trial participants and helping to ensure that the clinical research enterprise welcomes a breadth of participants who receive relevant and accessible information about participating. All of the technologies that we’ve been talking about, all of the methodologies, have the potential to expand the clinical research tent to reach populations that have been beyond the reach of the clinical research enterprise in the past. But we need to do it in a responsible way that makes sure that participants are protected.

So, recently, we issued a final guidance, "Informed Consent: Guidance for IRBs, Clinical Investigators, and Sponsors," and the intent of the guidance is to assist institutional review boards (or IRBs), clinical investigators and sponsors involved in clinical investigations of FDA-regulated products in carrying out their responsibilities related to informed consent. So the guidance provides FDA’s recommendations regarding informed consent and describes the regulatory requirements to help ensure the protection of rights and welfare of people participating in clinical trials.

More recently, we published a draft guidance “Key Information and Facilitating Understanding in Informed Consent”, and this is trying to solve a problem that I think we all see in the clinical research enterprise, which is that informed consents have gotten very complicated, and in fact are getting to the point where they’re not informative, which is a obviously a key problem. So what we’re trying to do in this guidance is really get to the distilled information of what’s critically important for a potential participant to understand before they sign on to a trial.

Together with a proposed rule to enhance the protection of human subjects, we hope this will invite that sort of broader participation in clinical research that I was mentioning.

Technology can also facilitate and potentially enhance the informed consent process. Regardless of how the informed consent is presented, whether its paper or electronic, use of videos, visuals, the process must provide sufficient opportunity for the potential volunteer to consider whether to participate.

Dr. Roach: How is FDA supporting expansion of the application of real-world evidence?

Dr. Marston: To integrate real-world evidence with traditional clinical trial evidence, we’re going to need collaboration across the board—so that’s among regulators, health care providers, academia, the clinical research community, and of course, the patients themselves. To help this, FDA actually has a research portion of its real-world evidence efforts. This supports a number of different efforts to understand the performance of real-world data and real-world evidence in providing the sort of information that we need for our regulatory purposes.

So for example, RCT DUPLICATE assess the ability of electronic health record-extracted data to replicate results from landmark traditional clinical trials. Importantly, it also developed a way to assess the likelihood of successful recapitulation. So researchers considering using real-world evidence techniques could use that sort of tool to understand their likelihood of an effective approach. Those sorts of research projects, we hope are going to improve the quality of real-world data and real-world evidence submissions to the agency.

Dr. Roach: We receive questions from health care professionals, clinical investigators and industry professionals who are interested in submitting real world evidence to the FDA (for example, to evaluate the drug-outcome association). What resources for understanding real-world data and evidence do you typically recommend first?

Dr. Marston: Thanks for that question. And first I would refer you to an excellent webpage that the FDA has, with all of its various resources, related to real world-data and real-world evidence. This is a program that's been led out of the Office of Medical Policy in particular by Khair ElZarrad and John Concato, and it really is an excellent compendium of various approaches to generating real-world data, the expectations that the agency has, and ways to interact with the agency.

So a couple of guidances that I'd want to recommend first. So one of these guidances, “Real-World Data: Assessing Registries to Support Regulatory Decision-Making for Drug and Biological Products,” provides considerations for sponsors proposing to use a registry, or to design a registry to support regulatory decision-making about a drug’s effectiveness or safety.

There’s another guidance, “Data Standards for Drug and Biological Product Submissions Containing Real-World Data,” addressing considerations for the use of data standards currently supported by the FDA in applicable drug submissions containing study data derived from real-world data sources.

And in addition, our CDRH colleagues have also developed guidance. So they have one, “Use of Real-World Evidence to Support Regulatory Decision-Making for Medical Devices,” and that updates and clarifies how FDA evaluates real-world data to determine if it is sufficient to be used in regulatory decision making about medical devices, and provides expanded recommendations for sponsors collecting real-world data.

Dr. Roach: That is incredibly helpful. The website that you referenced will be linked with the transcript of this recording on this episode's webpage. Is the Agency currently accepting submissions using real-world data?

Dr. Marston: Yes, I’m happy to say we are capable of reviewing submissions that include or even rely entirely on real-world data. FDA has already accepted real-world evidence to support drug and medical device product approvals. And many of these are in the setting of oncology and rare diseases.

Looking forward, we’re focused on not only promoting inclusion of real-world data used to generate real-world evidence in submissions but internally we’re ensuring consistency in our review of these submissions, and working with a variety of federal and external partners to help put in place an “evidence-ready” system.

Dr. Roach: We recently released a “Q&A with FDA” podcast episode on the Role of Artificial Intelligence, or AI, in Clinical Trial Design which you can find on our podcast playlist. There’s also a growing amount of interest in using AI in the realm of real-world data collection and evidence generation. Can you share with us FDA’s perspective on the use of AI in regulatory submissions?

Dr. Marston: Well that’s a great episode, and I highly encourage our audience to listen to it if they haven’t already. I also want to say I think we all know that the area of artificial intelligence is an evolving one, by the nature of it, and the FDA is evolving its thinking and developing its thinking along with that.

There are some new initiatives—including CDER and CBER’s PDUFA VII Advancing Real-World Evidence Program. This program seeks to improve the quality and acceptability of real-world evidence approaches in support of new intended labeling claims, including approval of new indications of approved medical products or to satisfy post-approval study requirements. Sponsors who are selected for that program are given the opportunity to meet with Agency staff—before protocol development or study initiation—to discuss the use of real-world evidence in product development. The Program is an optional pathway for sponsors submitting real-world evidence proposals. Sponsors are still able to engage with the agency through traditional means. There’s a semi-annual submission deadline for that, those meeting requests, and they are March 31st and September 30th, so one of them is coming up. And we’ve linked to the website in the podcast episode webpage.

We also recently issued a paper, “Artificial Intelligence & Medical Products: How CBER, CDER, CDRH, and OCP are Working Together.” OCP is within my office, the Office of Combination Products. That’s a lot of acronyms to say together, but I think the most important thing here is to recognize that multiple areas of the FDA are working together as we build our understanding and our plans in this area.

The FDA is also exploring ways to use AI technologies in our own internal operations, whether its regular business processes or actually our regulatory processes. This could benefit both the FDA and the public by streamlining workflows, and ultimately helping the quality of both the review that we’re doing, but also the quality of medical products that are eventually accessible to patients.

AI tools could help bring faster productivity and efficiency to our work – But also will allow us to automate some of the repetitive administrative functions, just like any workplace, and enable our expert staff to focus their time on more meaningful activities. That’s really the goal here.

Dr. Roach: You’ve shared so many examples, agency activities and resources today. Any last thoughts?

Dr. Marston: We have several ongoing agency initiatives related to real-world data and real-world evidence, and I really hope listeners will stay connected with FDA to learn about that progress. This is a very dynamic space so please stay tuned!

Dr. Roach: Thank you again, Dr. Marston.

Thanks for listening to “Q&A with FDA.” The full podcast and transcript of this recording is available at fda.gov/qawithfda. Many of our episodes offer continuing education credits for health care professionals, so be sure to visit this webpage for more details. If you are looking for additional learning or continuing education credit opportunities, including live and home study webinars, check out fda.gov/CDERLearn and fda.gov/DDIWebinars. And if you have questions about this episode, or anything drug-related, email us at druginfo@fda.hhs.gov.

U.S. Food and Drug Administration

Real-World Data and Evidence Generation with Dr. Hilary Marston

Resources: