Legend has it that when Tang Dynasty Princess Wencheng first came to Lhasa, she calculated that Lhasa had superior geographical conditions, with a white dragon in the east, a green tiger in the south, a rose finch in the west and a holy turtle in the north. After hearing her calculations, Tubo King Songtsan Gambo conducted a survey of the northern suburbs of Lhasa himself, and actually found a giant rock resembling a reclining turtle in the Nyangri Gully. As a result, a castle five-stories high was erected on the giant rock and named Giant Rock Castle, or Pabangka in Tibetan. Legend has it that the castle rose nine stories high, had stone walls cemented together with smelted iron, and was bound tightly to the rock with iron chains.

The Six Syllable Prayer Words, written by Tubo Minister Tome Sangbozha, were carved on the giant rock. Legend has it that when the Tubo minister returned from his studies in South Asia, King Songtsan Gambo arranged for him to create the Tibetan script, and then studied the script from the minister, who was appointed "the standardization tutor."

The rock castle was damaged when Lang Darma suppressed Buddhism in Tubo in the mid-9th century. In the 11th century, however, Pangdowa and Zhadowa, two monks from the Gardain Sect of Tibetan Buddhism, built a two-story monastery called Pabangka on the ruins to facilitate their practicing of Buddhism. Because of its long history and significance in Buddhist study, the small monastery is now so famous that Dalai Lamas of various generations came to the monastery for celebrations after taking vows or winning Geshi, the highest degree in Buddhism.

The monastery has been restored to its old grandeur.