2023 · 18 citations · 24 references
ComputingComputer ScienceKnowledge RepresentationKnowledge TransferMachine LearningDeep LearningGraph AlgorithmsGraph AnalyticsGraph MiningGraph-based LearningTransfer LearningFine-tuningGraph GenerationGraph Neural NetworksGraph Representation LearningGraph Transfer LearningPretraining
In recent years, graph pre-training has gained significant attention, focusing on acquiring transferable knowledge from unlabeled graph data to improve downstream performance. Despite these recent endeavors, the problem of negative transfer remains a major concern when utilizing graph pre-trained models to downstream tasks. Previous studies made great efforts on the issue of what to pre-train and how to pre-train by designing a variety of graph pre-training and fine-tuning strategies. However, there are cases where even the most advanced "pre-train and fine-tune" paradigms fail to yield distinct benefits. This paper introduces a generic framework W2PGNN to answer the crucial question of when to pre-train (.e., in what situations could we take advantage of graph pre-training) before performing effortful pre-training or fine-tuning. We start from a new perspective to explore the complex generative mechanisms from the pre-training data to downstream data. In particular, W2PGNN first fits the pre-training data into graphon bases, each element of graphon basis (i.e., a graphon) identifies a fundamental transferable pattern shared by a collection of pre-training graphs. All convex combinations of graphon bases give rise to a generator space, from which graphs generated form the solution space for those downstream data that can benefit from pre-training. In this manner, the feasibility of pre-training can be quantified as the generation probability of the downstream data from any generator in the generator space. W2PGNN offers three broad applications: providing the application scope of graph pre-trained models, quantifying the feasibility of pre-training, and assistance in selecting pre-training data to enhance downstream performance. We provide a theoretically sound solution for the first application and extensive empirical justifications for the latter two applications.
24
Aditya Grover, Jure Leskovec · 2016 · 10.9K citations
Network Motifs: Simple Building Blocks of Complex Networks
Ron Milo, Shai S. Shen-Orr, Shalev Itzkovitz et al. · Science · 2002 · 7.5K citations