He returned to France after 9 Thermidor (27 July 1794) and immediately took up the royalist cause as one of the agents of Louis XVIII. He became a member of the Royalist Committee in Paris, and for his activism he was once exiled to Menton.
Under the First Restoration he was appointed Minister of the Interior ( 13 May 1814 - 19 March 1815 ). In his brief term he appointed Pierre-Paul Royer-Collard and director of the library François Guizot secretary general.
The Second Restoration, he had the title of Minister of State. Elected deputy by the department of Gers, he opted for the Chamber of Peers with the title of Count (as of 31 August 1817) and Duke (as of 30 April 1821). He resigned his peerage 9 January 1832, shortly before his death.
He was appointed member of the Académie française by royal decree of 21 March 1816 . He was also elected member of the Academy of Inscriptions and Belles-Lettres on 12 August 1816.
He left manuscripts on the history of Louis XV and Louis XVI and a travel journal for the U.S. and Canada but did nothing to have them published.