We go, no baby, me go
Sean is talking about the two going away together, but only he can’t come with her. Only she can leave as it’s only a one person trip.
The song also about his girl leaving, “no baby, me go” is a way the of her telling him she’s leaving him.
To help improve the meaning of these lyrics, visit "Don't Look Down" by Kanye West (Ft. Big Sean, Lupe Fiasco & Yasiin Bey) Lyrics and leave a comment on the lyrics box