Hi Barry, All,

IMHO you're raising quite a significant issue here, which is most
probably much more generic than just about interpreting confidence
values for a single model or a set of different models, derived
through different algorithms. In general, all the dimensionless values
that are present in the results need to be clearly defined. Take the
caco-2 model as an example. Do you really know what's the meaning of
caco2 permeability -4.55 for benzene, as it is calculated by
ToxPredict? I personally don't know, but I guess that reading the
article ToxPredict is referring to
(http://www.ncbi.nlm.nih.gov/pubmed/16959190) could help understanding
this. Moreover, some users certainly do know quite well the definition
of this value and probably would find such explanations redundant.

It would have been quite nice if we had definitions for all such types
of values we're currently dealing with in OpenToxipaedia and then
referring to those (human-readable) definitions from the web based
apps would have been straightforward...

Best regards,
Vedrin

On 2 June 2011 09:51, Barry Hardy <barry.hardy at douglasconnect.com> wrote:
> (I am expanding this comms exchange on lazar model confidences with
> Christoph as it would probably benefit from further discussion on what our
> general framework and approach to confidences and communications of
> confidences should be)
>
> I read material Andreas Maunz sent over just before the AXLR8 meeting, and
> felt after reading it I at least had a reasonable (although superficially
> shallow) understanding of the maths of your generalising to
> significance-weighted Tanimotos, smoothing nearest neighbour similarities,
> and adding the gaussian smoothing exponential.  However making the next
> steps needs further work and interaction:
> a) How to understand the above maths (and others) more clearly and deeply?
> An interaction discussion along the path "developer - communicator - user"
> is probably needed. Otherwise I worry that converting the maths used into a
> simplified explanation in English may result in incorrect statements. They
> will at least need a review.
> b) Even then it is hard to understand the values in practice, so we need
> several examples with several models to get a better feel for the meaning of
> the numbers
> c) The ToxCreate help says that "For most models confidence > 0.025 is a
> sensible (hard) cutoff to distiguish between reliable and unreliable
> predictions." and you can tell people that, but the first reaction to a
> prediction that has a confidence of 0.026 as being a reliable prediction is
> confusion, with the first reaction often being the opposite comprehension.
> So redefining the index (even 1-x?) would be helpful for first meaning
> comprehensions siutations. Could we even have a classification to the index?
> - strongly confident .... very unconfident ... that users could understand
> more easily?
> d) Then also we have to prepare help explaining the maths and concepts if a
> way that is easy to understand (probably leaving out the maths)
>
> Another issue, is that different models using different methods to
> communciate confidences in predictions will also be difficult for users to
> grasp. Could a classification approach on diverse confidences somehow
> "normalise meanings" for users?
>
> Barry
>
>>> - Also, the developers will have to communicate with the "tutorial
>>> developer", and perhaps even react. For example, try to explain to
>>> someone what a confidence of 0.08 for a lazar model prediction means. I
>>> am not even sure yet from the material how I would redefine it to be
>>> somewhat intuitive.
>>
>> The last point is a good example why developers should not write
>> tutorials. I have explained lazar confidence ~1000 times (and an
>> explanation pops up if you click at the word in ToxCreate) so I simply
>> do not realize if a proper definition is missing in a tutorial. For this
>> reason I need someone who less involved to spot such problems. I can of
>> course try to give an explanation (e.g. as in ToxCreate), but I cannot
>> judge if it is understandable for someone with a different background.
>>
>> Mapping lazar confidences to something more intuitive (ie real
>> probabilities) is possible, it is on my list, but we just did not have
>> the time to implement it.
>>
>> Best regards,
>> Christoph
>>
>>
>
> _______________________________________________
> Partners mailing list
> Partners at opentox.org
> http://www.opentox.org/mailman/listinfo/partners
>