Don't ask him to pull up his pants up, it irks him
Joey is actually referring to Dyme here and “irks” as in Steve Urkel. In the show Family Matters where Steve starred in you could catch him always pullin' his pants up.
Irks is a synonym for “bothers”.
To help improve the meaning of these lyrics, visit "Swank Sinatra" by DyMe-A-DuZiN (Ft. Capital STEEZ, CJ Fly & Joey BADA$$) Lyrics and leave a comment on the lyrics box