It’s a cardigan,
A knit sweater that fastens in the front, a typical garment in the men’s fashion industry. Heems is contrasting it against the just-mentioned “pullover,” a term that is often used interchangeably with “cardigan” but in fact refers to a similar garment that isn’t fastened, but instead must be “pulled over” the head.
The “pullover” and “it’s a cardigan” exchange is a reference to a scene in the movie Dumb and Dumber
To help improve the meaning of these lyrics, visit "Nutmeg" by Das Racist Lyrics and leave a comment on the lyrics box