The Kyoto National Museum stands on a site formerly occupied by part of the Hōjūji-dono residence of Retired Emperor Go-Shirakawa (1127-1192). Rengeō-in Temple and its main hall Sanjūsangen-dō were also a part of this sprawling residence. The temple's Great South Gate is currently situated directly south of the museum's South Gate and the new Heisei Chishinkan Wing's entrance.

In the sixteenth century, the site was part of Hōkō-ji Temple, home to a Great Buddha Hall built by warlord Toyotomi Hideyoshi (1537-1598) during the Momoyama period (1573-1615). The historical remains of Hōkō-ji's walled outer corridor and South Gate, which were built by Hideyoshi's son Toyotomi Hideyori (1593-1615), are also located on the museum grounds and are designated as an Historic Site. The locations of their foundation stones have been incorporated as design elements in the reflecting pools in front of the new wing and inside the lobby. The Heisei Chishinkan Wing was designed with great consideration toward this area's rich history.

The fountain located between the Main Gate (West Gate) and the Meiji Kotokan (Main Hall) was part of the original design of the museum grounds, though over the years its size and location have changed slightly. The initial plan to draw water from the Lake Biwa Canal was not realized. In 1968, Matsushita Electric Company donated the current circular fountain, which has undergone upgrades over time and continues to be enjoyed by museum visitors. Originally, this pool was surrounded by flower beds and a grassy lawn, and the grounds were open as public leisure space. In 1950, after Rodin's The Thinker was installed in the museum garden, the need to protect this open-air artwork resulted for the first time in an admission fee to enter the museum grounds. Today, there are numerous cultural properties exhibited throughout the gardens.

When the museum first opened, works from the collection were exhibited according to artistic category, but beginning in autumn 1915 with the special exhibition in celebration of Emperor Taishō's enthronement, the museum began showing objects chronologically. Then in 1924, after the museum came under the administration of Kyoto City, it again began exhibiting according to artistic genre, a practice that continues today.

As a means of highlighting the permanent collection, the museum began in 1898 to organize special New Year's exhibitions. From 1901 to 1910, the New Year's exhibition featured works related to the annual New Year's poetry reading at the imperial court and to the Chinese zodiac animal of the coming year. Following the opening of the Heisei Chishinkan Wing in 2014, this tradition of an annual New Year's zodiac animal exhibition has been revived.

The Kyoto National Museum is known for housing masterworks from temples and shrines in Kyoto, which are entrusted on long-term loan. In addition, the museum acquires works through purchase and donation. Some of the important collections that have been donated to the museum are as follows.

• 1931  Objects and archives related to Sakamoto Ryōma (1836-1867), one of Japan's most beloved historical figures, were donated by Sakamoto Yatarō and have become one of the Kyoto National Museum's signature collections.
• 1953  The Moriya Collection, a world-renowned assemblage of early Buddhist manuscripts amassed by Moriya Kōzō (1876-1954), was donated to the museum.
• 1960  The Ueno Collection, comprising Chinese painting and calligraphy collected by Ueno Riichi (1848-1919), one of the founders of the Asahi Shimbun newspaper company, was given to the museum.
• 1956  Rodin's The Thinker was entrusted to the museum on loan in 1950. A generous gift from Ōmiya Kurakichi (1881-1972), president of the Takara Shuzō Company, helped support the subsequent purchase of this masterpiece by the museum.
• 1999-2000  The Suma Collection of modern Chinese painting and calligraphy, amassed by the Japanese diplomat Suma Yakichirō (1892-1970), was donated to the museum. Suma acquired many of his works through personal friendships with celebrated contemporary Chinese artists of the day

Incidentally, one of the museum's early acquisitions was the National Treasure Reading in a Bamboo Grove, attributed to fifteenth-century ink painter Shūbun (n.d.). It was purchased in 1905 for 750 yen from Myōchi-in Temple; however, when the museum was transferred to Kyoto City and renamed the Kyoto Gift Museum of Kyoto, this masterwork was moved into the collection of the Tokyo National Museum, where it remains today.