Comparative Evaluation of Tandem MS Search Algorithms Using a Target-Decoy Search Strategy

Abstract

Peptide identification of tandem mass spectra by a variety of available search algorithms forms the foundation for much of modern day mass spectrometry-based proteomics. Despite the critical importance of proper evaluation and interpretation of the results generated by these algorithms there is still little consistency in their application or understanding of their similarities and differences. A survey was conducted of four tandem mass spectrometry peptide identification search algorithms, including Mascot, Open Mass Spectrometry Search Algorithm, Sequest, and X! Tandem. The same input data, search parameters, and sequence library were used for the searches. Comparisons were based on commonly used scoring methodologies for each algorithm and on the results of a target-decoy approach to sequence library searching. The results indicated that there is little difference in the output of the algorithms so long as consistent scoring procedures are applied. The results showed that some commonly used scoring procedures may lead to excessive false discovery rates. Finally an alternative method for the determination of an optimal cutoff threshold is proposed. Peptide identification of tandem mass spectra by a variety of available search algorithms forms the foundation for much of modern day mass spectrometry-based proteomics. Despite the critical importance of proper evaluation and interpretation of the results generated by these algorithms there is still little consistency in their application or understanding of their similarities and differences. A survey was conducted of four tandem mass spectrometry peptide identification search algorithms, including Mascot, Open Mass Spectrometry Search Algorithm, Sequest, and X! Tandem. The same input data, search parameters, and sequence library were used for the searches. Comparisons were based on commonly used scoring methodologies for each algorithm and on the results of a target-decoy approach to sequence library searching. The results indicated that there is little difference in the output of the algorithms so long as consistent scoring procedures are applied. The results showed that some commonly used scoring procedures may lead to excessive false discovery rates. Finally an alternative method for the determination of an optimal cutoff threshold is proposed. Systems biology has come to occupy a central role in modern biological research. MS-based proteomics is an integral subset of the wide ranging technologies contributing to the systems biology toolbox. MS-based proteomics, in turn, encompasses several commonly used techniques, the most powerful of which is, arguably, high throughput fragmentation of complex protein mixtures that have been enzymatically digested and separated prior to their introduction into a tandem mass spectrometer. These fragmentation events (also referred to as tandem MS or MS2 events) are submitted to any of a number of available peptide identification search algorithms. Results are returned in the form of peptide sequences with one or more scores with which to evaluate the likelihood that the resulting sequence is correct. The first use of such an algorithm was reported by Eng et al. in 1994 (1Eng J.K. McCormack A.L. Yates III, J.R. An approach to correlate tandem mass spectral data of peptides with amino acid sequences in a protein database.J. Am. Soc. Mass Spectrom. 1994; 5: 976-989Crossref PubMed Scopus (5315) Google Scholar). The algorithm, to be named Sequest, enabled the high throughput and automated interpretation of tandem mass spectra against a protein sequence library. The Mascot search algorithm, an outgrowth of the MOWSE (molecular weight search) project (2Pappin, D. J. C., Rahman, D., Hansen, H. F., Bartlet-Jones, M., Jeffery, W., and Bleasby, A. (1996) Chemistry, mass spectrometry and peptide-mass databases: evolution of methods for the rapid identification and mapping of cellular proteins. Mass Spectrom. Biol. Sci. 135–150Google Scholar), was released in 1998 (3Perkins D.N. Pappin D.J. Creasy D.M. Cottrell J.S. Probability-based protein identification by searching sequence databases using mass spectrometry data.Electrophoresis. 1999; 20: 3551-3567Crossref PubMed Scopus (6661) Google Scholar). More recently two open source search algorithms have become available including the Open Mass Spectrometry Search Algorithm (OMSSA) 1The abbreviations used are: OMSSA, Open Mass Spectrometry Search Algorithm; CIEF, capillary IEF; ΔCn, Sequest delta correlation number; E, expectation; FDR, false discovery rate; HUPO, Human Proteome Organization; nano-RPLC, nanoflow reversed-phase- LC; RSp, Sequest preliminary score ranking; Sp, Sequest preliminary score; Xcorr, Sequest cross-correlation score; TPP, TransProteomic Pipeline; XML, extensible markup language. (4Geer L.Y. Markey S.P. Kowalak J.A. Wagner L. Xu M. Maynard D.M. Yang X. Shi W. Bryant S.H. Open mass spectrometry search algorithm.J. Proteome Res. 2004; 3: 958-964Crossref PubMed Scopus (1148) Google Scholar) and X! Tandem (5Craig R. Beavis R.C. TANDEM: matching proteins with tandem mass spectra.Bioinformatics. 2004; 20: 1466-1467Crossref PubMed Scopus (1943) Google Scholar) distributed by the National Center for Biotechnology Information (NCBI) and the Global Proteome Machine Organization, respectively. Although each implementation is different, these search algorithms operate under the same general principles. Namely they first find peptides whose theoretical mass approximates the experimental mass determined by the mass spectrometer for the precursor ion within an in silico digest of the sequence library being used. The search space is typically limited by parameters including mass tolerance, enzyme specificity, numbers of missed cleavages, and amino acid modifications. The algorithms next consider the fragment ions generated within the tandem mass spectrometer. The fragment ion masses are similarly compared with an in silico fragmentation of the peptides that passed the first criteria. The fragment ions must approximate the masses of the theoretical fragment ion masses within a defined mass tolerance. Typically only a subset of possible fragmentation ions are considered depending on the type of mass spectrometer and the fragmentation mechanism used. For example, only b and y ions are generally considered in the case of ion trap mass spectrometers using collision-induced dissociation. The details of this implementation differ among the algorithms and are not documented in all cases. In addition, the methods used to assign scores are very different as described in a recent review (6Sadygov R.G. Cociorva D. Yates III, J.R. Large-scale database searching using tandem mass spectra: looking up the answer in the back of the book.Nat. Methods. 2004; 1: 195-202Crossref PubMed Scopus (327) Google Scholar). As noted, each algorithm outputs one or more scores in response to a query for each tandem mass spectrum. In some cases these scores are dependent on the search parameter inputs and/or sequence library used in addition to the quality of the tandem mass spectrum. This capability is intended to allow the search engine to output a probability consistent with these parameters. However, not all search algorithms consider all parameters in their score assignment. A combination of a lack of documentation together with a general lack of understanding of the underlying algorithms contributes to inconsistent application of any given search algorithm within and across research groups. Ideally a search algorithm would perform an initial search of a relatively small portion of the submitted dataset to determine the appropriate search parameters such as precursor and fragment ion mass accuracy, number of missed cleavages, monoisotopic or average masses, etc. to remove this aspect of user bias. This is now as search results their into such as Eng J.K. A. R. L. H. D. J.A. D.J. W. Yang L. H. R. with the of peptide sequences by mass Biol. PubMed Google Eng J. J. R. The Res. PubMed Scopus Google Scholar) and the Global Proteome Machine database R. Beavis R.C. Open source for and protein identification Proteome Res. 2004; 3: PubMed Scopus Google Scholar). source of in the search output is in the of the of tandem mass spectra to be typically referred to as a are generated by with the data of the of the mass spectrometer on which the data were Open source for data into such as and have recently become available as has an open source to to are as not more parameters available to the of a as there are parameters that the search a lack of documentation and understanding of the algorithms to inconsistent For example, spectral H. R.G. Yates III, J.R. A for and of protein in 2004; PubMed Scopus Google Scholar) is as a and parameters that have a on the of such a are such parameter available to of the within of the same precursor within a be together to a a to the number of the to a more or This parameter may the of a search as together may any of that For example, mass spectrometers used for proteomics use a relatively wide For any given there may be of mass that is for fragmentation together with the ion of resulting in a fragmentation of this of interpretation by the search The to a dataset is the interpretation of the scores output by the search algorithms. A wide variety of scoring procedures have been by with methods for false rates. This is of the Sequest algorithm as the scores not the probability of the as is the case with Mascot, OMSSA, and X! Tandem. scoring are used for Sequest, two of which were used by the Human Proteome to results their Proteome D.J. M. R. H. R. J.S. A. R. Eng J. H. J.S. J. J. X. R. X. R. A. A. A. M. H. of the Proteome results the with and a dataset of proteins and a 5: PubMed Scopus Google Scholar) and are as a for that The scoring procedures are referred to as and criteria. Sequest has recently a score used within the Mascot has two for search an score and a The score is to be used as a threshold for and the score is to be used as a threshold for Mascot has recently an in addition to the and X! Tandem as the score for The scores is However, a user must still is an appropriate cutoff A user that an or probability of or each intended to to a or likelihood of a would be However, have that these scores with search parameters, sequence and as has been reported for false databases tandem mass Proteome Res. 2004; 3: PubMed Scopus Google Probability-based evaluation of peptide and protein tandem mass spectrometry and the Proteome Res. PubMed Scopus Google Scholar). of the search algorithm have to data evaluation methods an algorithm for database search Am. Soc. Mass Spectrom. PubMed Scopus Google A. R. to the of peptide by and database PubMed Scopus Google Yates III, J.R. Probability-based of protein using a PubMed Scopus Google based of peptides and the of in protein Proteome Res. 2004; 3: PubMed Scopus Google of results using a mass threshold data Proteome Res. PubMed Scopus Google Scholar), the use of these methods has not been and not approach that is to with by techniques, data and search parameters is the target-decoy sequence library search J. S.P. of with tandem mass spectrometry for protein the Proteome Res. PubMed Scopus Google W. S.P. evaluation of mass spectrometry used in proteomics Methods. PubMed Scopus Google S.P. search for in protein by mass Methods. 3: Scopus Google Scholar). In this method the protein sequence library to be is and and the is to the of the sequence library. This a sequence library within the search with the same number of sequence and amino acid as the or sequence library. any that would a search be in the search of the library. The target-decoy search an initial of search results and application of based on the false discovery determined the is to that the are only in that a to a sequence is only a for a false and is not a false to as This initial be by of spectra proteins to which biological is proteins be by et al. J.A. D. Eng J.K. An and of several available search and of 5: PubMed Scopus Google Scholar) have recently and compared a number of tandem MS search algorithms. However, was using parameters for input into each of the search algorithms, and search parameters were such as precursor and fragment ion mass a dataset was for that may have been the search algorithms with which was Finally were based on a limited of tandem mass spectra In this four search Mascot, OMSSA, Sequest, and X! Tandem. data data search algorithm and sequence library were by using a search parameters, and the same target-decoy sequence library in all searches. Results using commonly scoring procedures were compared with using a consistent cutoff as determined by the target-decoy determined The and of each algorithm was as as the algorithms of protein Finally an method for appropriate scoring is proposed. and have been described in J. Proteome of using a capillary with PubMed Scopus Google W. J. of using capillary PubMed Scopus Google Scholar). the of was and were with were using by with on a trap and to of the resulting digest was separated into by capillary by nanoflow with using a ion trap ions were a mass of was enabled with the most ions of the precursor ion for tandem MS with enabled and to MS and tandem MS were to a of were for MS and for MS2 search data were using the of distributed with The used to A of the the may be in the methods The by were used as the input to the were and to using an This was used as the input to the Mascot, OMSSA, and X! Tandem search The sequence library the was used to a target-decoy sequence library for searching. An was used to the sequence the to each protein in the protein sequence in the and the now to the search were Mascot and X! Tandem were on a with a and of used the search precursor ion mass tolerance, fragment ion mass tolerance, enzyme one missed monoisotopic precursor mass OMSSA, and monoisotopic fragment ion mass and a of and a of An parameter to was used that that one of a number of the most fragment ions of the theoretical In this case the parameter was to be that X! Tandem the precursor mass the algorithms. The is such that the search is defined by a to the precursor ion the precursor in the case of the algorithms. were to query peptide protein experimental and theoretical masses, and were in were to using available Systems the of Mascot were to using the available with the TransProteomic The is available the Proteome were to using available Systems the of Sequest results were to using the within X! Tandem were to using the The data were into a database for and discovery were to the method of et al. W. S.P. evaluation of mass spectrometry used in proteomics Methods. PubMed Scopus Google were by and the was by the of the and A all that the MS2 threshold of the algorithms is in the form of an as and peptide were to all and to which they MS2 a search query that results in a peptide some peptides peptides with proteins A protein was a peptide to or not that peptide to Tandem mass spectra were the of a proteomics of of by with A generated was used as input into four tandem mass spectrometry search Mascot, OMSSA, Sequest, and X! Tandem. search parameters were used for each search including the sequence in this case a target-decoy of the subset of The score for each of the algorithms is in to and peptides are The results of this search are in Results are only a MS2 for each algorithm with the of Mascot and Mascot, OMSSA, and X! Tandem scores are reported as the same scoring method used for sequence Mascot scores on a to scores to is X! Tandem scores to a of and Sequest probability score to Results across the scoring may be in An is defined as the probability that the this case a peptide sequence matching an experimental tandem mass would by the be For example, an of that a is as as a false an of the by one in given to probability an is based on the probability of an a of that is A probability the likelihood that an given a of that For example, given a peptide with a probability of with possible in the sequence library within the defined search parameters, an of would of search results and Sequest scoring and Tandem high high not not Open in a is that to and spectra not to the results by the or probability an of would be that the number of spectra would be of the number of spectra the in 1: This is in with an of an of a and the method used to false by the target-decoy false This for X! Tandem For Mascot this an and for this an This to the in the likelihood of the fragmentation across the possible peptides D.N. the of false identification using ion in mass with a mass spectral Am. Soc. Mass Spectrom. PubMed Scopus Google proteomics by mass Biol. PubMed Scopus Google Scholar). not in the search algorithm are the in a These any peptides whose are not defined by the search cleavages, including protein as as such as and be that the sequence typically used are as proteins are and as algorithms they the sequences by the variety of These and to the in the for an For this a target-decoy search approach has in that these as to a sequence as a sequence S.P. search for in protein by mass Methods. 3: Scopus Google Scholar). Mascot the user with two an ion score and an The ion score is compared with two that are for each the score and the score is the reported threshold used in most The score is typically the score in the case of peptides and in the case of the results of a Mascot search using the score as the threshold using the and scores as and and using the a threshold to a MS2 The scoring using the and scores is this is an to remove which have a to be false This results in a dataset with an in the number of proteins Sequest a case determined threshold scores are used in research the in are some commonly used by HUPO, including a and high of Sequest results are using determined to a MS2 for each Sequest two The first score is a of several The cross-correlation score is a cross-correlation of the theoretical and experimental Typically a different threshold is used for each to the difference in probability of a to A score the difference the two spectra is given in the form of the delta correlation the scores are such that the score is to and the difference is used is the preliminary score and the preliminary score that are scores first as a to the number of spectra to to cross-correlation The Sequest score is a probability score in the most recent of the the results by using a MS2 are very consistent for each of the algorithms. Mascot, X! Sequest and Sequest peptides and peptides and proteins. The protein for each algorithm is consistent as and a MS2 The across a of MS2 is in across a very wide there is little difference search algorithms in the of MS2 to protein false discovery rates. The the importance of the MS2 results as small in this have a on protein For example, a MS2 results in protein of for all algorithms. A MS2 the protein to A MS2 a protein FDR, and a MS2 results in protein in of that the proteins reported are false This has been in the of different Sequest W. A. R.G. and the to of the 5: PubMed Scopus Google Scholar). The same was Sequest high and The high returned an MS2 of and a protein of the a MS2 and a protein these the number of peptide and the using the a MS2 However, protein to proteins in the case of the high and proteins for the of and respectively. the number of protein and respectively. In this case the are and the quality of search results and the of the The high are an a protein of data interpretation and to across In the used Mascot score is a relatively an MS2 of and a protein of with the score as an threshold Mascot returned more results with a MS2 and protein However, as be the number of proteins to proteins the Mascot and score more proteins the Sequest scoring In all a MS2 with the for two peptides protein the protein to an of Mascot E, OMSSA, and X! Tandem each only one protein with two Sequest This method of with very high However, the method the number of false For example, that as were in the to be false that proteins with a two peptides protein would not an of data quality are the average number of MS2 and peptide protein in of these is as an in the of a protein identification Yates III, J.R. Probability-based of protein using a PubMed Scopus Google Scholar). In peptide are as they lead to in and sequence of the this and X! Tandem Sequest and Mascot with to MS2 MS2 events protein an X! Tandem MS2 Mascot MS2 and Sequest MS2 The number of MS2 very spectral the of each protein are determined by the number of MS2 The algorithms are much more with to the number of peptides protein with X! Tandem and Sequest peptides protein on Mascot and and Sequest resulting in The returned by the scoring are a of the high protein false resulting use of these scoring As false protein the sequence false not any given protein or subset of resulting in a number of proteins with a peptide This to of MS2 protein and peptides As in the number of peptide the number of protein of MS2 and peptide protein by algorithm peptide and MS2 MS2 MS2 MS2 Open in a the in of MS2 of MS2 a MS2 returned more MS2 X! the next very false discovery with Mascot, Sequest, or X! Tandem. returned MS2 and proteins with an average of MS2 and peptides respectively. proteins by the number of peptides mapping to similarly returned the with proteins or more two and a peptide of peptides protein by of proteins with of proteins with MS2 MS2 MS2 MS2 Open in a the number of proteins by each algorithm and in combination with each of the algorithms. This is by in A proteins by one peptide and proteins by two peptides that all are in was very the algorithms on a of proteins with all algorithms or proteins with one or two respectively. or proteins were by two of the four algorithms with one or two respectively. proteins by one or more peptides or proteins by two or more In the numbers of proteins by Mascot, Sequest, and X! Tandem were all that of is to that the proteins by were by a of peptides This an average of peptides which is only the peptides protein of protein by of of proteins with one or more peptides by all X! X! Tandem and X! Tandem and X! Tandem and and and Sequest and X! OMSSA, and X! OMSSA, and X! Sequest, and Sequest, and proteins of proteins with two or more peptides by all X! X! Tandem and X! Tandem and X! Tandem and and and Sequest and X! OMSSA, and X! OMSSA, and X! Sequest, and Sequest, and proteins Open in a and may be the numbers of and and in a In this case the protein of the are may use the protein and the number of false protein to evaluate the and of each The results are in In this case the number of is to the number of proteins in the protein sequence library the number of proteins by any algorithm This number is the of the number of proteins by all algorithms the of false proteins by all algorithms The number of false is the number of proteins by any algorithm the number of proteins by that The number of is the number of protein the number of false the use of the target-decoy search approach the by all algorithms is The of Mascot and Sequest are X! Tandem and of algorithm and Open in a An was with to the of MS2 to proteins. For this the a to the MS2 for each of the algorithms The in is the of using a of to across the results of algorithms that output scores across The MS2 the of an algorithm as S.P. protein identification by a library of tandem mass 2004; PubMed Scopus Google Scholar), and the MS2 to protein may a method for not to perform the sequence library search and documented to evaluate search results and and This the for results a MS2 as However, this the of the protein are This on the intended use of the the results are being used to a protein library for a is to on the of and to use a such as a MS2 with a for two peptides In the cases the results are being used in with experimental may be to use results as a being the which lead to a false that four commonly used peptide identification search algorithms perform and more or the results are the use of a target-decoy protein sequence search library These results were by the complex protein a by using a ion trap mass spectrometer. A number of tandem mass spectra were and output to a that as a input to all search algorithms. parameters were used to perform each of the and all used an target-decoy protein sequence library. These to to that the search algorithms These results these data may be For example, search algorithms may perform using tandem mass spectra resulting different or different tandem mass is to that only the peptide scoring of the search algorithms is in this The algorithms that the output for each search are not results by any of the search may be different the underlying peptide output in this to the output their input data the algorithms The open source algorithm is in that that of each of the algorithms by the number of MS2 by may the implementation of spectral The proteomics would were scoring to be as a score Eng J.K. Beavis R.C. M. for and database scoring algorithms using the search PubMed Scopus Google Scholar) in X! Tandem or were to be as a search into the for Systems TransProteomic of the search algorithm a may to is that the some for data evaluation and A target-decoy search is one as relatively results different algorithms for the same input data the of MS2 protein is that would be relatively for each of the search algorithm to in their data and of these have the of being to false is, the data reported be by all of the up to the search the to the and to the search parameters This is false to and some algorithms not consider all of the search parameters in their probability or scoring In this of high throughput proteomics and is critical that data evaluation and procedures are used and that their use is as and as with