At times, Sarah Nexus and Derek Nexus will disagree in overall calls. The frequency of these ‘disagreements’ will vary with test datasets – as do all measures of performance. That they can disagree is a clear demonstration of independence of the two products.
Several publications have illustrated how an expert can improve performance by evaluating in silico predictions (see below). This is really important and why we have worked hard to provide sufficient information to support an expert analysis. This includes a measure of confidence (expected accuracy) in a prediction for Sarah Nexus (we already do this for Derek Nexus with the likelihood levels – see Judson et. al.). In addition, Sarah Nexus also presents an explanation of the prediction in the form of relevant hypotheses and the strength of the support these provide. Finally, all the underlying data is just a click away. This is significantly different from ‘black box’ statistical models where the user can have an understanding of the model’s accuracy against a test set, but can’t assess an individual prediction. Sarah Nexus, like Derek Nexus, presents all the information an expert requires in order to come to an informed decision when models disagree. We believe this allows more informed decision than the conservative ‘if one model predicts positive, consider it positive’ approach which while increasing sensitivity, does so at the expense of the number of false positive predictions.
Judson PN., Stalford SA., and Vessey J. (2012). Assessing confidence in predictions made by knowledge-based systems. http://pubs.rsc.org/en/Content/ArticleLanding/2013/TX/C2TX20037F
Dobo, K. L., Greene, N., Fred, C., Glowienke, S., Harvey, J. S., Hasselgren, C., Jolly, R., et al. (2012). In Silico Methods Combined with Expert Knowledge Rule out Mutagenic Potential of Pharmaceutical Impurities: An Industry Survey. Regulatory Toxicology and Pharmacology, 62(3), 449–55. doi:10.1016/j.yrtph.2012.01.007
Sutter, A., Amberg, A., Boyer, S., Brigo, A., Contrera, J. F., Custer, L. L., Dobo, K. L., et al. (2013). Use of in silico systems and expert knowledge for structure-based assessment of potentially mutagenic impurities. Regulatory toxicology and pharmacology : RTP, (May). doi:10.1016/j.yrtph.2013.05.001