I left my crutch over the rainbow with my riches
He is saying he was born without, nor does he want to rely on, a crutch (Family wealth, assets, businesses etc.). He also is saying that he has no riches and he doesn’t expect to “strike gold”. “Over the rainbow” being a non existent, unrealistic place.
To help improve the meaning of these lyrics, visit "Prosperity" by Aesop Rock Lyrics and leave a comment on the lyrics box