The music critic Chris Weingarten, a longtime critic of Rap Genius, pointed out on Twitter that this is likely a reference to Tougher than Leather, the Run D.M.C. album and movie directed by Rick Rubin. Hi, baby.
https://twitter.com/1000TimesYes/status/314119093361987585

See this as being a hoarder of references.

On first impression I’m seeing mid-Eighties anti-apartheid activist wordplay here, but whatever (reminded of a music critic comparing “Horchata” line “Here comes a feeling you’ve thought you’d forgotten” to “It’s been a long time/shouldn’t have left you without a dope beat to step to” from “I Know You Got Soul” by Eric B and Rakim, very much an era-specific thing)

Ezra Koenig worked as an intern at The Walkmen’s recording studio in Harlem. And it’s worth noting Outkast' Tomb of The Boom as a referent here, since everyone has listened to Speakerboxxx/Love Below

Another interpretation is Ezra’s love for music. While his friend’s girl is off somewhere else, his girl is song, kept safe by his side and gathering up as much of it as he can.

Interspersed with some riff on Pachebel’s Canon, a direct reference to Souls of Mischief’s Step To My Girl:

Every time I see you in the world
You always step my girl
But ask yourself homeboy — Why is that?

http://www.youtube.com/watch?v=HvBmr_b-Rpg

(the “back back way back” continues the reference)

An alternate explanation is that stepping to the girl means stepping to the beat of the music. It could also mean that other fans are starting to listen to the music the music he’s listening to.

Batmanglij stated:

That song is about people who can be possessive over music they love, and it’s also about other songs.

The last king of Lydia, Croesus was so famously rich that his name became a byword for wealth in the expression rich as Croesus.

Croesus was so rich that he had every guest take as much gold as he could carry, upon leaving.

The story of Croesus' life can be found in the Histories by Herodotus, the first history text ever written, which is a required part of Columbia University’s Core Curriculum.

Doubt it’s related, but I’m reminded of one of the early Talking Heads singles “I Feel It In My Heart” in which it is totally impossible to believe that David Byrne is singing from the heart. The line that follows about the better girlfriend reinforces it, thinking of “Girlfriend is Better.”

This might also be a reference to the Tegan and Sara/Tiësto joint, “I feel it in my Bones.”

Yes, like mousy and meek, and yes, like Modest Mouse the band — and aging out of the era in which that band was popular.

Modest Mouse got its name from the Virginia Woolf story “The Mark on the Wall,” and Koenig refers to British Modernism in many other songs:

“I wish I could hit upon a pleasant track of thought, a track indirectly reflecting credit upon myself, for those are the pleasantest thoughts, and very frequent even in the minds of modest, mouse-coloured people, who believe genuinely that they dislike to hear their own praises.”

Ready for the House is a record by Jandek, maybe used here to refer to something analogous to Ezra Koenig’s post-hippie domesticity vibe on his college blog Internet Vibes.

See a lot of nostalgia for the early Vampy Weeks days here. At times the post-hippie domesticity post is derivative of an essay on Wes Anderson and Brooklyn and premature aging called “Captain Neato” by the cultural critic Christian Lorentzen for the New York magazine n+1. (Lorentzen deliberately draws on a bunch of stuff Norman Mailer wrote for Partisan Review in the Fifties and Sixties on New York City hipsters and urban change, and Koenig cites Lorentzen’s essay in the comments section of the post linked here.) Koenig’s de facto mentor Dave Longstreth of Dirty Projectors was involved with K Records, and Jandek is a huge influence on that crowd, as well as on other groups that sound like Modest Mouse.

Weirdly, and probably coincidentally, there are very similar promo shots of Modest Mouse and Vampire Weekend posing with some gramophone-type thing

Vampire Weekend

Modest Mouse

Bruce Springsteen, Dylan Thomas, have at it. Could, indeed, be a reference to James Dean’s death in light of his perceived lifestyle.

In the context of the first two lines, you could also think of the infamous Chappaquiddick incident involving Ted Kennedy. In 1969, Kennedy took his apparent mistress Mary Jo Kopechne away from a party on Cape Cod and, apparently drunk driving, drove off a bridge, killing his passenger. He swam away from the scene of the accident.

The “Kennedy curse” is the series of family deaths that followed Joe Kennedy’s decision to lobotomize his eldest daughter Rosemary in the mid-1940s. The job was botched, and she was permanently incapacitated. After Kennedy’s oldest son Joe Jr. was killed in a World War II Allied version of a kamikaze mission (actually before the lobotomy), other daughter Kathleen (“Kick”) died in a plane crash with her British lord non-Catholic husband, then Jackie pre-O miscarried, then of course JFK later on, RFK, JFK Jr., and several of RFK’s children to drug overdose and skiing accidents. Some of this album was written on Cape Cod, adding to the resonance.

Racing culture, especially horseracing, is a pretty prominent Irish American trope — gambling, alcohol, vagaries of fate and all that, like David Milch said on Luck (despite not being Irish). Vampire Weekend lyricist Ezra Koenig has a longtime fixation on Irish Catholics as fetishists of spiritual loss, as chronicled on his college era blog Internet Vibes in a post called “The St. Patrick’s Day Massacre”

Here is my plug for that Joe Kennedy biography The Patriarch, which is like a Vampire Weekend song in book form, which is to say Vampire Weekend is a book on the American century in song form, which the writer Elif Batuman (The Possessed) said first, not me.

Read a joke by a music critic on the excellent Rap Genius-hating message board I Love Music that Koenig fails here to take his own advice in “Walcott” to escape the ghetto of Hyannis Port. The Kennedy family keeps its compound there as everyone dies off.

I thought of Diane Court from Say Anything here, though for what it’s worth Diane Young is the name of a salon on the Upper East Side that markets itself as an anti-aging establishment

A pun on “live fast, die young.” A homophone for “dying young”. Per their Pitchfork interview:

“I had this feeling that the world doesn’t want a song called ‘Dying Young’,“ says Koenig, "it just sounded so heavy and self-serious, whereas ‘Diane Young’ sounded like a nice person’s name.”