Comment from Fayez Rahman
AnonymousSupportIndividual
Summary: The commenter suggests including specific language regarding Large Language Models (LLMs) in the draft guidance to highlight their role in generating reports and protocols. They advocate for distinguishing between public LLMs, which they view as risky, and internal, site-specific LLMs that can support junior engineers and improve operational efficiency.
Comment: I recommend the addition/ introduction of LLM in line 29. " A subset of AI that can be used in the documentation and analysis stage is large language model, LLM. LLM refers to models capable of generating human-like text that can be further trained to generate validation documents, e.g. ChatGPT." I can see the AI making a decision but the LLM component preparing a user-friendly report so I encourage an additional discussion on LLMs.
Question: line 47/48: Does operational efficiency include the use of LLMs to generate protocols or asking LLMs questions regarding a process CPP or CQA? In my opinion, the use of LLMs to generate such protocols or provision of incorrect information can impact patient safety as an error, such as ommission of a test that challenges CPPs and CQAs can have an impact on product quality and patient safety. I am hoping the guideline specifies that the use of generic and public LLMs should not be permissible due to bias and erros. However, an internal LLM trained on a site process can be used to generate protocols that require site review and provide guidance for junior engineers.
Question: Regarding model risk, is it fair to say that the risk decreases over time since the ML component is being trained with more data so there should be less bias moving forward therefore periodic review can decrease in frequency, and dependency on release can be increased? Can we include in the risk assessment a measure of how much data has the AI been trained on or a consideration to its accuracy?