We welcome Laura Patuya from C Rake presenting the structural evaluation framework for assessing LLM
quality performance in LC contexts.
So Laura, are you already in the Zoom meeting link?
OK, so good.
Good afternoon and everyone.
I'm Laura.
I'm also an actor and and today along with my colleague Jinja, you know, I'm going to talk about how
to LLMS at scale very frankly to that before.
So by this morning we know that probably the NCA.
They Teil is nicht um bit of them gigolent and concept and so me and the two evaluates and it and
i'll can internet see a contact.
The first one is human, so a human take the LNM output and it's very time consuming, expensive
actually thinking to read about it like we have the review by colleague and Arthur Donaldson and
managed to.
Sorry, another to make that comparison with the ground truth.
It's a classical machine running strategy, but instead are receivable because they're really just
one sitting right entry.
I see this subjective and in addition to that we don't have any ground truth database available yet
so.
2. Am approach will we i on this.
That is evaluation based on criteria with the perspective to use LM attitude.
So the objective of the three developer structure and reproduce all evaluation framework tailored to
LCA and this can be applied to different applicable like to completely from LLM.
Uns for intensiv LL Dings und Probleme Unit Spezies to dete man und es geht das.
And well, to track performance and problem during the training of women.
So let's get into this, into this variation framework, I said.
Gostap and i do gostap and it's the.
Question definition.
So here find the goal and the configuration we want to assess by choosing the specific FCA tag,
retext the LLM settings, for example, the model we evaluate the content to evaluate.
And then we we define a set of, We define, sorry, a set of questions to test the performance of this
very operation.
And those questions are sent.
We'll in one two if @me the insert of all den test and nova and.
Act as a bread and together with other Internet and just describe later on, then provide the event
and important to me.
That i can't have that we in to that so you know.
Plantation set aiming at recalibrating evaluation from all the configuration to ensure that the
judge works with and can be.
So let's look at the evaluation step.
So for one configuration.
Will be answers and we then provide the judge LLM with some additional criteria those evaluation
yeah are generate some are and they are quicker for any FCA task and some are tailored to pass in
we also.
It's a think for the scoring scale or the.
1234 for each question.
And then we can right go to the standards. They're a people of.
For answers that reflect LC subjectivity and this would help the judge to make the evaluation, but
they are different from the question we can use in the evaluation and that will assign to its
criteria question or justification.
So then we ended up with a bunch of comments of each criteria and then we come just like statistic
to come to the mean the sound of additional maybe we you're interested in and weighted score.
Why not?
But how to both results are very good and open question it question and it causes the question of
the acceptable level of quality performance, if we might.
So it we plug the number and and there's bice cool. We can't can.
Operation of two you'll see how many, so if 5.
C'est écrit plein de.
To achieve and whether there be 2/3.
Free.
Je ne vais pas encore.
OK, I will now start the full graph from some tested we have done on specific LCA task.
So the first one is the.
Golends coom Definition, so we've that Institutional trumping leaks the ios lars and also that you'm
an validays and is very chanaging because you need to rea i'm going to godance o'ditinition in a
rose, so we've need them i can simplication that's i'm just about I'm meting to you, that's i'll in
dentary thata base.
And we said that you can have many hallucinations if you don't provide umm.
The list of all the process and flows to the adelan or the white the will.
You will just make up some flow, process, geography combination and the last task we test is the.
Inventory generation and or concusion is that is super chanaging to have a real goal standard for
Prozess. It's on was in.
Possible and even more for technology.
We also learned that human validation cannot be done by anyone, it has to be done by an expert on
the process and the also that.
And we think a full inventory for a Prozess me be is a bit ambitious @the stage for a ldm and it me
be that or suit that for a structuring and ähnlich all TS that's it up so we have to.
This death manual.
But we're gerity wording on that @the mees and bipline to II was to that.
Many more context and many more configuration, so the idea is to provide some information directly
to.
Will be written, read by the code and then send to an.
And so all the prom, the output generate that and the egaduy central be them and the time atometic
lee so a worker terencey walking on that to ier number of as alls and feedbacks prom the crimward.
So as a discussion, we can highlight maybe the strength of the framework we propose.
So we systematically reveal the performance trends across LCA context.
The LLM configuration, it can be applied to any LCA task as long as you provide the appropriate
criteria for that.
It supports both human and LLM judges, but we recommend the LLM as a judge to scale the evaluation.
It also reduces the risk of LLM overfit to evaluation question and answer because if the ground
truth is made available then LLM could learn from them.
So here the gold standard are supposed to guide the judge, but are not directly used to evaluate the
outputs.
And the evaluation questions should be updated regularly to avoid this overfit.
And this framework is is will be ready to use very soon, but of course we will need some
improvement.
So as a future work we will need to test it on a larger set of configuration to strengthen also the
data.
The sorry criteria definition.
We also need to investigate on what's it an acceptable minimum quality threshold and to develop
indicator to support the interpretation of the different scores.
It would be so interesting to compare it to the LLM as a judge approach to a human review to kind of
validate it.
And creating a collaborating collaborative ground truth database would be crucial to reflect the
field subjectivity and to help make LLMS smaller.
Thank you.