Grandy was born in Sioux City, Iowa, the youngest of three sons of William Grandy, who worked in his father's insurance business, and his wife Bonnie. When Grandy was eleven, his father died of a heart attack. His mother died of an aneurysm a year later.

The young Grandy was raised by Margaret Avery, his mother's best friend (a widow who later married his father's doctor). He went to public schools until ninth grade, after which he attended high school at Phillips Exeter Academy in Exeter, New Hampshire, as his father and brothers had done. At Exeter, Grandy was the roommate of David Eisenhower, the grandson of President Dwight D. Eisenhower, and was later best man at Eisenhower's wedding to Julie Nixon, the daughter of then–president-elect Richard Nixon, in December 1968. Grandy graduated magna cum laude with a bachelor's degree in English studies from Harvard University in 1970. In 1971, he joined an improvisational group at Harvard called "The Proposition" and then rejoined the show when it moved off-Broadway in 1971. Among his co-stars was future Saturday Night Live star Jane Curtin. Although of service age, Grandy had a high lottery number and was not drafted for Vietnam.