During the War of the Elves and Sauron, and as Sauron was moving against Eriador, Elrond led an army of Elves of Lindon to the region. Forced northward toward the headwaters of Bruinen by the armies of Sauron, Elrond established a stronghold.

This became a refuge for those seeking shelter from Sauron's onslaught in Eriador, and Elrond's and Celeborn's Elves were soon joined by many fleeing the Sack of Eregion. By S.A. 1700, Sauron had overrun all of Eriador and Imladris was besieged. The siege ended when the forces of Gil-galad (the High King of the Noldor, who lived in the Grey Havens) routed Sauron's armies and drove him out of Eriador.

A Council held at that time made Elrond Gil-galad's vice-regent in Eriador, and Imladris his seat. Gil-galad bestowed Vilya upon Elrond in secret. Aided by the power of the Elven ring, Elrond made Imladris one of the chief seats of Elvish strength in the west in the latter part of the Second Age. Imladris remained an Elvish stronghold in Eriador to the end of the Second Age and into the Third.

At the end of the Second Age, Elendil and Gil-galad formed the Last Alliance of Elves and Men to challenge Sauron, and their host halted for a while at Imladris before crossing the Misty Mountains. Isildur's wife and his youngest son, Valandil, were in Imladris at that time, and Isildur was journeying back to them after Sauron's defeat when he was ambushed at the Gladden Fields. After receiving the shards of Narsil, Valandil took up his role as King of Arnor and left Imladris for Annúminas.

After the death of Gil-galad, Elrond remained in Imladris. During the Third Age it was a refuge and sanctuary; many Elves gathered there, and it was the chief dwelling of the High Elves in Eriador. The heirs of Isildur were also harboured there due to their kinship with Elrond, as descendants of his brother Elros. Rivendell maintained this relationship with the North Kingdom throughout the Third Age, and Elrond and his people remained steadfast allies to the heirs of Isildur.

Danger came to Eriador around T.A. 1300 when the witch-kingdom of Angmar was established in the north-east beyond the Ettenmoors. It was late in the reign of Arveleg I that Rivendell was besieged by Angmar. After an incursion by Angmar into Eriador in T.A. 1409, the Elves of Rivendell joined those of Lindon in subduing the power of the Witch-king for many years.

After the end of the North-kingdom in T.A. 1975 Aranarth, the first Chieftain of the Dúnedain, had his son fostered in Rivendell, as were all subsequent sons of the chieftains. The heirlooms and treasures of the house of Isildur, including the shards of Narsil, were also kept there after Arvedui's death. Among these were also some of the few remaining documents containing the history of Númenor.

In the Quest of Erebor, Bilbo Baggins stopped off at Rivendell with the Dwarves on the way to the Lonely Mountain and also on the way back to the Shire with Gandalf. Bilbo retired there after his 111th birthday, writing his memoir 'There and Back Again'.

Years later, Frodo Baggins and his companions journeyed to Rivendell, where they met Bilbo. Several other Elves, Dwarves and Men had also arrived at Rivendell on separate errands; at the Council of Elrond they learned that all of their errands were related to the fate of the One Ring. At that Council, the Fellowship of the Ring was formed.