The earthworks at Dunharrow were originally made by the Men of the Mountains, and were known to predate the Third Age. In drafts of The Return of the King from volume VIII of The History of Middle-earth, we're told that '[h]ere they had dwelt under the shadow of the Dark Years...', so we can be sure that Dunharrow was in use during the Second Age, but it's not impossible that it dated back even further than this.

The reference to a 'temple' (harrow in English place names, specifically indicating a heathen place of worship) refers to Dunharrow's presumed original purpose. This purpose had been long forgotten by the later Third Age, and to the Rohirrim it was simply a highly defensible place of refuge among the mountains.