Node Feature Extraction by Self-Supervised Multi-scale Neighborhood\n Prediction

Abstract

Learning on graphs has attracted significant attention in the learning\ncommunity due to numerous real-world applications. In particular, graph neural\nnetworks (GNNs), which take numerical node features and graph structure as\ninputs, have been shown to achieve state-of-the-art performance on various\ngraph-related learning tasks. Recent works exploring the correlation between\nnumerical node features and graph structure via self-supervised learning have\npaved the way for further performance improvements of GNNs. However, methods\nused for extracting numerical node features from raw data are still\ngraph-agnostic within standard GNN pipelines. This practice is sub-optimal as\nit prevents one from fully utilizing potential correlations between graph\ntopology and node attributes. To mitigate this issue, we propose a new\nself-supervised learning framework, Graph Information Aided Node feature\nexTraction (GIANT). GIANT makes use of the eXtreme Multi-label Classification\n(XMC) formalism, which is crucial for fine-tuning the language model based on\ngraph information, and scales to large datasets. We also provide a theoretical\nanalysis that justifies the use of XMC over link prediction and motivates\nintegrating XR-Transformers, a powerful method for solving XMC problems, into\nthe GIANT framework. We demonstrate the superior performance of GIANT over the\nstandard GNN pipeline on Open Graph Benchmark datasets: For example, we improve\nthe accuracy of the top-ranked method GAMLP from $68.25\\%$ to $69.67\\%$, SGC\nfrom $63.29\\%$ to $66.10\\%$ and MLP from $47.24\\%$ to $61.10\\%$ on the\nogbn-papers100M dataset by leveraging GIANT.\n