Then I’m off to London watchin' this bridge that we built destroy
It looks like he has lost a battle or link with something/someone (i.e. the burning of a bridge). To emphasize the point, he references one of the grandest and most iconic bridges in the world, the London Bridge. Said bridge has been destroyed and rebuilt many a time, which ties well with this line.
To help improve the meaning of these lyrics, visit "Smallville" by XV Lyrics and leave a comment on the lyrics box