When Davis announced his resignation Grossman ran to succeed him as leader of the Ontario Progressive Conservative Party, however, the party's grassroots had tired of the progressivism of the Davis and John Robarts years and chose the more conservative Frank Miller as leader in early 1985. Miller ran a disastrous election campaign, however, and the Tories ended up losing government for the first time in 42 years. Miller resigned as Grossman was chosen as the new leader of the Tory party on November 25, 1985 and became Leader of the Opposition to the minority government of Liberal Premier David Peterson.

The Peterson government became very popular and Grossman's Tories, in opposition for the first time since the 1940s, had a very difficult time adjusting to their new role. When the Liberals called an early election for the fall of 1987 the Grossman Tories tried to campaign from the right but Grossman's history as a Red Tory made his new found conservatism less than credible and, in any case, the public was not in a mood to return to the Tories. Peterson won a majority government and the Conservatives were reduced from 52 seats to 16 falling to third place behind the Ontario New Democratic Party. Grossman lost his own seat and promptly resigned being replaced by Andy Brandt as interim leader.

In 1997 Larry Grossman died at the age of 53 from brain cancer