Maybe its because I had you girl
Sean looks back at the time when he was too broke to do anything, and realizes that spending too much money on Ashley might’ve been why. Alternativley that he spent too much time with this girl, instead of making his keep.
In a different way, he could be tying this line into those were the times I felt the richest in a sweet way. Even though all they did was spend time by the pool and he was so broke he couldn’t take her to the zoo he felt the richest because he had Ashley, because she was his girl.
To help improve the meaning of these lyrics, visit "Ashley" by Big Sean (Ft. Miguel) Lyrics and leave a comment on the lyrics box