I'm not only at the party, but I am the party,” the Oscar-nominated actor tells PEOPLE of his style philosophy ...