Founded by the Elves of Lindon at the beginning of the Second Age (S.A. 1), the Grey Havens were afterwards used by any of the Elves (whether Noldor, Sindar, or Silvan Elf) to leave Middle-earth for Valinor. Círdan the Shipwright was Lord of the Havens. He was one of the wisest and most foresighted of the Elves, and became the oldest known elf in Middle-earth. However, he also acknowledged the authority of Gil-galad, King of Lindon and last High King of the Noldor.

In S.A. 600, Entulessë, a Númenórean ship headed by Captain Vëantur, arrived in the Havens. It was the first time the Númenóreans, ancestors of Aragorn and the Men of the West, landed in Middle-earth.

When Sauron invaded Eriador in S.A. 1695 in the War of the Elves and Sauron, Gil-galad called Númenor for aid. Then King Tar-Minastir sent out a great navy; but it was delayed, and did not reach the coasts until S.A. 1700. By that time all of Eriador fell under Sauron's control, save only besieged Imladris (Rivendell), and he reached the line of the river Lhûn. Nevertheless, though he killed many elves, he dared not attempt to take the Havens or Lindon, feeling they were too strong and because they had the help of the Númenóreans. Indeed, the Havens became Sauron's bane in that chapter of history, for the Númenóreans were able to land in Middle-earth at the Havens and push him back beyond the Misty Mountains, ending the Dark Years.

In S.A. 3431 the Elves of Lindon led by Gil-galad and Círdan marched eastward from the Havens, and were awaited by the High King of Arnor and Gondor, Elendil, at Amon Sûl (later known as Weathertop). The combined host marched towards Imladris, where they joined with Lord Elrond. In the battles and long siege of Barad-dûr that followed, many of the Elves of Lindon were killed, and Gil-galad was burned to death by the hand of Sauron in the final battle. Greatly weakened by his fight against Gil-galad and Elendil, Sauron was killed by Elendil's son Isildur, and the One Ring taken from him. Thus, in S.A. 3441, the Second Age ended, and the Third Age began. Isildur refused Círdan and Elrond's entreaties that he destroy the One Ring by casting it into the Crack of Doom, claiming it as a weregild for his father's death.

About T.A. 1000 the Istari or Wizards arrived in Middle-earth at the Grey Havens. Círdan, upon meeting Gandalf, gave him Narya, the Elven Ring of Fire.

In T.A. 1975, Prince Eärnur of Gondor brought a Gondorian fleet to the Grey Havens to aid Arthedain against Angmar. The fleet was so massive that the ships of Gondor filled the Grey Havens, as well as Forlond and Harlond.

In T.A. 3018, Círdan sent Galdor of the Havens "on an errand" to Rivendell and he became Círdan's representative at the Council of Elrond.

Following the defeat of Sauron and the crowning of King Aragorn Elessar in the War of the Ring, Gandalf, Elrond, and Galadriel left Middle-earth for Valinor from the Grey Havens. The Last Riding of the Keepers was accompanied by Bilbo Baggins and Frodo Baggins, who were given the honour of taking the Straight Road because they had both been a Ring-bearer. Years later Fairbairn family tradition held that Samwise Gamgee in Fo.A. 61 also sailed West from the Grey Havens because he too had been a Ring-bearer, even if just briefly. It is also said that after the death of King Elessar, Legolas sailed to Valinor and took Gimli with him.

It is unclear how long the Grey Havens lasted into the Fourth Age, as the number of Elves dwindled with the coming of the "Age of Men". Supposedly Círdan remained at the Havens until the last ship left from Middle-earth for the Blessed Realm.