There is no simple answer here.
While extensive analyses of predictive performance of different models with different data sets is possible, it will not ultimately answer the key question – ‘which model is appropriate for this compound?’.
Two approaches are common in these situations. One is a conservative one where in the event of conflicting predictions, it is considered positive. This can improve sensitivity, but at the cost of an increased number of false-positive predictions (and all the down-stream activities that then result).
The second approach is to allow an expert to review and decide upon the overall prediction. The latter approach has been described in 2 recent publications. We support this by providing measures of expected accuracy for every prediction, and through the provision of transparent predictions with sufficient information for an expert to review.
Dobo, K. L., Greene, N., Fred, C., Glowienke, S., Harvey, J. S., Hasselgren, C., Jolly, R., et al. (2012). In Silico Methods Combined with Expert Knowledge Rule out Mutagenic Potential of Pharmaceutical Impurities: An Industry Survey. Regulatory Toxicology and Pharmacology, 62(3), 449–55. doi:10.1016/j.yrtph.2012.01.007
Sutter, A., Amberg, A., Boyer, S., Brigo, A., Contrera, J. F., Custer, L. L., Dobo, K. L., et al. (2013). Use of in silico systems and expert knowledge for structure-based assessment of potentially mutagenic impurities. Regulatory toxicology and pharmacology : RTP, (May). doi:10.1016/j.yrtph.2013.05.001