Picked her up at four and I went (holding hands)
Murs is picking her up at 4 o'clock and taking her out.
However, he might be referring to baseball. He was hoping to get to first base. Technically, fourth base is a home run, which can be defined as sex. So he could have picked her up to have sex and then went (as in busted a load on her muck)
To help improve the meaning of these lyrics, visit “Baby Girl (Holding Hands)” by Murs (Ft. 9th Wonder) Lyrics and leave a comment on the lyrics box