The best song wasn't the single, but you weren't either
Frank compares a girl/guy to the best song on an album. While the best songs on any given album usually aren’t the ones heard on the radio (the single is the song used to promote the album), this person isn’t single as in they have a significant other. Often the most promising mates are already taken.
Taken in the context of the song’s topic (Frank is likely creating a fictional world here), this line also works as a statement that a character in the song never wanted to dig deeper into what the world has to offer (album tracks), but only swallows what’s given to him/her (singles).
To help improve the meaning of these lyrics, visit “Sweet Life” by Frank Ocean Lyrics and leave a comment on the lyrics box