Unicorns and wizard sleeves
This is a sexual innuendo. wizard sleeves is slang to describe a woman’s vagina and unicorn is an obvious phallic symbol.
Both these are also a part of mythical culture which comes up later.
To help improve the meaning of these lyrics, visit "Castle" by Macklemore and Ryan Lewis (Ft. Macklemore) Lyrics and leave a comment on the lyrics box