AutoQSAR Best Practices for Proteins

The analysis of biologics data using machine learning methods opens up new opportunities and at the same time presents potential hazards due to the complexity and diversity of datasets involved. To successfully navigate the pitfalls, it is important to take into consideration emerging guidelines and best practices in the field of machine learning applied to proteins and other types of biologics.

Sample Size

  • The number of datapoints should be no less than the number of features.

  • A common general guideline is that there should be 10 times more data points than features.

  • In practice the number of data points required is dependent on the complexity of the problem, type of features, quality of data and the ML method one intends to employ

  • Reserving separate sets of data to be used for model validation during and predictive performance testing is strongly recommended.

Assay Data

  • Identify and exclude any data that are not not suitable for machine learning e.g.

    • unreliable data with too much variability in measurement

    • obvious outliers

  • Assess bias in the dataset (e.g. over representation of active or inactive examples)

  • Avoid datasets with too narrow a range of values. The data’s dynamic range needs to be at least double the experimental error in order to have a chance at achieving meaningful correlations between experimental and predicted values.

  • Heterogeneous data should be avoided e.g. data from different species, cell lines or experimental protocols.

  • Be aware of experimental assay limits and the way they may influence training of regression models

In silico data

  • Ensure the amino acid sequence used to build models matches constructs used to generate experimental data

  • Adopt a consistent protocol in protein preparation across training and prediction molecules, e.g. if you prepare protein at ph 7, in training, you ought to do the same for prediction too

  • Structure-derived descriptors are not suitable for proteins that lack a well ordered three-dimensional structure. For such proteins, sequence-based descriptors may be preferable.

Data preparation

  • Find and remove duplicates

  • Normalize the data if features have different ranges e.g. protein expression data from different batches or measurements from different models of an instrument

  • Data sets should be balanced in terms of bin populations for categorical model fitting. You should, for instance, avoid creating models for data sets where 99.9% of predictions are ‘inactive’ and 0.1% of data is ‘active’. Such models tend to have very high accuracy simply by predicting all samples to be ‘inactive’. Instead, randomly select data such that the distribution of molecules in bins is within an order of magnitude.

  • Create and employ an external hold out set for model validation purposes

Feature selection

  • Select a maximally informative subset of features. To achieve this, AutoQSAR applies a selection procedure to eliminate features whose correlation exceeds a certain threshold. The threshold can be adjusted via the “Advanced Options” menu in the GUI

Model Building

  • Decide, based on the type of data available, whether a classification or regression model is appropriate

  • Employ the optional capability to create and employ an external hold out set for model validation purposes

  • Data is expected to be somewhat normally distributed for continuous model fitting. If you have a bifurcated distribution, creating two continuous models with a categorical model to decide what continuous model to employ may be more appropriate.

  • When building a categorical model based on cutoffs, it is recommended to discard compounds that fall in some gray area (e.g., stable if ΔΔG < -1.0 kcal/mol, unstable if ΔΔG > 1.0 kcal/mol, discarded if -1.0 ≤ ΔΔG ≤1.0 kcal/mol).