From Canonical Correlation Analysis to Self-supervised Graph Neural\n Networks

Abstract

We introduce a conceptually simple yet effective model for self-supervised\nrepresentation learning with graph data. It follows the previous methods that\ngenerate two views of an input graph through data augmentation. However, unlike\ncontrastive methods that focus on instance-level discrimination, we optimize an\ninnovative feature-level objective inspired by classical Canonical Correlation\nAnalysis. Compared with other works, our approach requires none of the\nparameterized mutual information estimator, additional projector, asymmetric\nstructures, and most importantly, negative samples which can be costly. We show\nthat the new objective essentially 1) aims at discarding augmentation-variant\ninformation by learning invariant representations, and 2) can prevent\ndegenerated solutions by decorrelating features in different dimensions. Our\ntheoretical analysis further provides an understanding for the new objective\nwhich can be equivalently seen as an instantiation of the Information\nBottleneck Principle under the self-supervised setting. Despite its simplicity,\nour method performs competitively on seven public graph datasets. The code is\navailable at: https://github.com/hengruizhang98/CCA-SSG.\n