Written By: Ben Cosgrove

Of the three Kennedy brothers John, Robert and Edward who ascended to the national political stage in the 1950s and ’60s, it was arguably the middle brother, Bobby, who best embodied the enormous contradictions at play within that famed American family.

There was, for example, RFK’s fraught relationship with liberals and with American liberalism in general. As the author and historian Sean Wilentz once wrote while reviewing a largely unflattering biography of Kennedy in the New York Times:

Robert F. Kennedy always irked liberals; and they always irked him. . . . Kennedy’s association with the reckless Sen. Joseph McCarthy in the 1950s forever tainted his reputation in some reform circles. As his brother’s presidential campaign manager in 1960, and thereafter as attorney general, he struck many liberals as ruthless in the pursuit of power and reluctant in the pursuit of principle, especially regarding civil rights. Kennedy, for his part, regarded his liberal critics as hopeless, sanctimonious losers who put purity above political realism, and who seemed to think that sure-fire defeat was inherently noble.

That Bobby Kennedy was, like his brothers and many of his other relatives, past and present, a titanically driven individual is hardly news. There’s a reason, after all, that he’s still despised today, five decades after his death, by some liberals and most conservatives: he did not fit into a neat, ideological box and then as now neither side knew what to do with a man who refused to act and speak according to their expectations and their rules.

Then there was his relationship with Lyndon Johnson a man who, according to virtually everyone who knew both men, hated Bobby Kennedy with an intensity matched only by RFK’s loathing for his brother’s successor as president.

But Kennedy also had an intellectual and in public, at least an emotional poise that makes most present-day American politicians seem glib and trifling by comparison. (Is there a sitting U.S. senator or representative whom one can picture quoting Herodotus or Sophocles, from memory, as Kennedy so often did?)

Of course, like his brothers especially John Robert Kennedy was also able to immediately and powerfully connect with crowds in a way that most politicians can only envy, and there were certainly people who saw greatness in him and in his future.

“He is one of the half-dozen men in the country today qualified for top political leadership,” one of Lyndon Johnson’s advisers told LIFE writer Robert Ajemian. “He really cares about right and wrong. He cares about people.”

Here, LIFE.com shares photos most of which never ran in LIFE magazine of Kennedy and his extended and immediate family in 1964. The pictures, by LIFE’s George Silk, capture a man who, as Robert Ajemian wrote in the magazine’s July 3, 1964, issue, “had shouldered massive burdens” in the six months since his brother John was gunned down in Dallas the previous November.

A major preoccupation of Bob Kennedy’s in the past six months [Ajemian wrote] has been his family and now it includes his brother’s children, Caroline, who is 6, and John, who is 3. Jackie Kennedy brings them out almost every day to their uncle’s home, Hickory Hill, five miles outside Washington. Bob and [his wife] Ethel spend as much time with them as with their own brood of eight. “They think of it as their own home,” says Jackie Kennedy. “Anything that comes up involving a father, like father’s day at school, I always mention Bobby’s name. Caroline shows him her report cards.”

But even surrounded by so many loved ones, and so busy with speeches and appearances around the country, the rawness of the loss of his older brother was, it seems, never far away. After a speech in Pittsburgh, a reporter asked Kennedy, “What do you miss most about your brother?”

“Kennedy looked startled,” Ajemian reported, “and stared at the reporter as he sought the exact answer. His face softened and he said, ‘Just that he’s not here.'”

Four months after the LIFE cover story, Robert F. Kennedy was elected as the Democratic U.S. Senator from New York. He served until June 6, 1968, when he was assassinated by a gunman named Sirhan Sirhan, while campaigning in Los Angeles for his party’s presidential nomination. Robert Kennedy was 42, four years younger than John Kennedy was when he was killed.

Robert Kennedy with his three dogs, Hickory Hill, Va., 1964.

Robert Kennedy at home in 1964

George Silk The LIFE Picture Collection/Shutterstock

Robert F. Kennedy, 1964.

Robert Kennedy at home in 1964

George Silk The LIFE Picture Collection/Shutterstock

Under a tree hung with swings at Hickory Hill, Bob and Ted have a last talk about Bob's possible plans for entering the New York Senate race.

Robert Kennedy at home in 1964

George Silk The LIFE Picture Collection/Shutterstock

On his way to the office, Kennedy stands in hall at Hickory Hill with wife, Ethel, scanning the headlines while his daughter Kerry, 4, tugs for a goodbye kiss.

Robert Kennedy at home in 1964

George Silk The LIFE Picture Collection/Shutterstock

John Kennedy Jr., who plays at Hickory Hill often, hands Braumas, a Newfoundland, rubber bone as he says hello. In distance, sister and cousin ride ponies.

Robert Kennedy at home in 1964

George Silk The LIFE Picture Collection/Shutterstock

Ethel finishes her meal as Michael, 6, holds Christopher, one, in lap and 7-year-old Courtney tempts baby by testing his food. Joseph, 11, looks out window.

ROBERT KENNEDYRobert Kennedy at home in 1964

George Silk The LIFE Picture Collection/Shutterstock

Caroline Kennedy, legs tangled in a cousin's, tumbles off a submerged Pierre Salinger. They lost a watery jousting match with Bob and two of his daughters.

Robert Kennedy at home in 1964

George Silk The LIFE Picture Collection/Shutterstock

John Kennedy's daughter has eagerly joined Bob's family. She loves to push Christopher in his stroller, stops to cling for a moment to her Uncle Bob. 'She's my pal,' he says fondly. But there is a special feeling for Caroline who, at 6, understands the tragedy of her father's death as her brother John does not.

Robert Kennedy at home in 1964

George Silk The LIFE Picture Collection/Shutterstock

Robert Kennedy with his niece, Caroline; his son, Christopher (in pram); and his daughter, Courtney, 1964.

Robert Kennedy at home in 1964

George Silk The LIFE Picture Collection/Shutterstock

Bob swings his nephew on terrace. 'Jack made John the mischievous, independent boy he is,' says Jackie Kennedy. 'Bobby is keeping that alive.'

Robert Kennedy at home in 1964

George Silk The LIFE Picture Collection/Shutterstock

Robert F. Kennedy eats breakfast with his family, 1964.

Robert Kennedy at home in 1964

George Silk The LIFE Picture Collection/Shutterstock

Robert Kennedy at home with some of his kids, 1964.

Robert Kennedy at home in 1964

George Silk The LIFE Picture Collection/Shutterstock

Robert Kennedy and Pierre Salinger joust with kids in Kennedy's pool, 1964.

Robert Kennedy at home in 1964

George Silk The LIFE Picture Collection/Shutterstock

Robert Kennedy, Pierre Salinger and kids at RFK's Virginia home, 1964.

Robert Kennedy at home in 1964

George Silk The LIFE Picture Collection/Shutterstock

John Kennedy Jr. (left) and his sister Caroline and their cousins David, Kerry and Courtney Kennedy, 1964.

Robert Kennedy at home in 1964

George Silk The LIFE Picture Collection/Shutterstock

Robert Kennedy with John Kennedy Jr. and his daughter Courtney, 1964.

Robert Kennedy at home in 1964

George Silk The LIFE Picture Collection/Shutterstock

Robert Kennedy with his daughter, Kerry, 1964.

Robert Kennedy at home in 1964

George Silk The LIFE Picture Collection/Shutterstock

Robert F. Kennedy and his wife Ethel putting their one-year-old son, Christopher, down for a nap, 1964.

Robert Kennedy at home in 1964

George Silk The LIFE Picture Collection/Shutterstock

Robert Kennedy and John Kennedy Jr., Virginia, 1964.

Robert Kennedy at home in 1964

George Silk The LIFE Picture Collection/Shutterstock

More Like This

people

A Tribute to Couplehood

people

Why “Voluptua” Was Too Hot For TV

people

Catherine Deneuve: The Eyes Have It

people

Laughing With the Stars

people

Paul Newman: Now There Was a Star

Frank Sinatra backstage with Sammy Davis Jr. and Natalie Wood during Davis' run on Broadway in the play, Golden Boy, New York, 1965. people

Insider’s View: LIFE Goes Backstage with the Stars