Tokenized Heterogeneous Graph Transformer with Enhanced Local and Global Representation Learning

Abstract

Graph Transformers have demonstrated superiority in handling complex heterogeneous graphs in recent years. However, existing models still face several inherent challenges: (1) reliance on manually designed meta-paths to encode explicit local graph heterogeneity; (2) inability to capture fine-grained global information from distant yet relevant nodes. To address these limitations, we introduce THFormer, a novel node tokenized heterogeneous graph Transformer that learns expressive node representations by incorporating local and global perspectives. From the local perspective, we employ multiple subsequences for different heterogeneous types to explicitly encode local semantic relations, eliminating the need for manually designed meta-paths. From the global perspective, we design a local masking and global sampling mechanism to construct global structural (semantic) sequences, effectively capturing fine-grained global structural (semantic) information. Subsequently, THFormer separately feeds the resulting global and local sequences into standard Transformer layers as model inputs. Since these sequences represent two distinct views of the same target node, their corresponding outputs are naturally aligned to generate self-supervisory signals for model training, further enhancing the expressiveness and reliability of the target node representation. Extensive experiments are conducted to validate the efficacy of THFormer, and the quantitative performance gains are 0.24%, 0.31%, 0.51%, and 0.81% on DBLP, ACM, IMDB, and Freebase, respectively. The experimental results demonstrate the superiority of THFormer over representative heterogeneous graph neural networks and graph Transformer models.