Speaker comes on during phone call

Love my 6p. But every time I use the phone and put it to my ear it switches to speaker and I have to toggle the speaker on and off. The speaker icon does not turn on when it first happens. Am I missing a setting or do I have a problem with the phone? Thanks for any help.

Sent from my Nexus 6P using Tapatalk