The “fifth guest” here is the phone going off while Tom’s mistress tries to reach him. Everyone in the room knows what is really going on now, and it leads to a very tense situation.
Daisy usually puts up with Tom’s white supremacy rants and boasts about his polo ponies. But even he knows better than to tick her off while the woman he’s cheating on her with calls him during dinner.
And Nick and Jordan are just the uncomfortable bystanders to Tom and Daisy’s disagreement. Some people might have enjoyed watching all this tension and drama going on, but Nick just wants to get out of there at this point.