There is no problem if a silk handkerchief or similar item is in a man’s pocket, and it does not invalidate prayers.