A well-functioning pretrain-finetune paradigm on knowledge graph with great generalizability and interpretability. It formulates complex logical queries as masked predictions on graph patterns and introduces a two-stage masked pre-training strategy. It also proposes a KG triple transformation method to enable transformer to handle KG elegantly and a mechanism to unify different tasks of knowledge graph problems.
