I wanted what they had back in the fourth grade
For a good portion of his life, his family has been poor (then turned middle-class). He has also spoken about how he had many dreams and wishes in the fourth grade (My Girls). He frequently speaks how he wishes he had what a lot of kids had back then when he was a kid like the chance to go on vacations.
That’s why in the next line he describes how he goes on vacations with his family now because he can. His fame has granted him the privilege to engage in endeavors he wasn’t able to as a kid.
To help improve the meaning of these lyrics, visit "We Ain't Them" by Childish Gambino Lyrics and leave a comment on the lyrics box