He was born in Edinburgh, and educated at Edinburgh University and Gottingen University. After studying law in London, he was called to the bar in 1879, and was a rather successful lawyer. A few years before, he had been elected a Liberal member of Parliament for the Scottish seat at East Lothian. In 1895, he helped found the London School of Economics. In 1905, he was appointed Secretary of State for War in Henry Campbell Bannerman's administration. Haldane, a prominent Liberal Imperialist and close associate of Herbert Henry Asquith, was a strong advocate of British commitments on the continent, and took great steps in preparing the army for participation in a possible European war by establishing the British Expeditionary Force. His tenure also saw the creation of the Imperial General Staff the Territorial Army the Officer Training Corps and the Special Reserve. He was raised to a peerage in 1911, becoming the Viscount Haldane of Cloan. Upon Lord Loreburn's retirement in 1912, Haldane succeeded him as Lord Chancellor, but was forced to resign in 1915, after being falsely accused of pro-German sympathies.

As the war progressed, Haldane moved more and more to the left, and when Ramsay MacDonald and the Labour Party came to power in 1924 with very little support in the House of Lords, Haldane was recruited to serve once again as Lord Chancellor.

Haldane also served as first chancellor of the University of Bristol and wrote several philosophical works, the best known of which is The Reign of Relativity (1921), which dealt with the philosophical implications of the theory of relativity.

Lord Haldane's brother was respiratory physiologist John Scott Haldane, and his sister was author Elizabeth Haldane.