Sanity Checks for Lottery Tickets: Does Your Winning Ticket Really Win\n the Jackpot?

Abstract

There have been long-standing controversies and inconsistencies over the\nexperiment setup and criteria for identifying the "winning ticket" in\nliterature. To reconcile such, we revisit the definition of lottery ticket\nhypothesis, with comprehensive and more rigorous conditions. Under our new\ndefinition, we show concrete evidence to clarify whether the winning ticket\nexists across the major DNN architectures and/or applications. Through\nextensive experiments, we perform quantitative analysis on the correlations\nbetween winning tickets and various experimental factors, and empirically study\nthe patterns of our observations. We find that the key training\nhyperparameters, such as learning rate and training epochs, as well as the\narchitecture characteristics such as capacities and residual connections, are\nall highly correlated with whether and when the winning tickets can be\nidentified. Based on our analysis, we summarize a guideline for parameter\nsettings in regards of specific architecture characteristics, which we hope to\ncatalyze the research progress on the topic of lottery ticket hypothesis. Our\ncodes are publicly available at:\nhttps://github.com/boone891214/sanity-check-LTH.\n