After playing from 1945 to 1951 with Dizzy Gillespie, Jazz at the Philharmonic, Milt Jackson, and Ella Fitzgerald (who was then his wife), Brown played with the Oscar Peterson trio for fifteen years. After leaving the Trio he became a manager and promoter as well as a performer. Among the artists he managed were the Modern Jazz Quartet and Quincy Jones. In his later years he recorded and toured extensively with pianist Gene Harris.