Have you tried different setting sprays? It sounds like the setting spray you are using is leaving a stick feel to your skin and that attracts little hairs etc.
Yes I agree. Some setting sprays have a 'plastic-like sticky' finish to them. This could be the issue.
One solution is a two-step process:
1) Lightly set your setting spray with a blot powder, or no-colour powder, this will make the setting spray less sticky. (Do NOT use a setting or loose powder, as it's too powdery).
2) Gently mist your face with canned facial spray water, like Evian, Avene, Olehenriksen, or Herbivore, Just to name a few. The best way to do this is to mist your face at a full arm's length away, or spray the air and walk through the mist.
This will get rid of the overly matte powdery look while still maintaining the hold of the setting spray. Don't spray too much or too close as this will defeat the purpose and make the setting spray "sticky" again.
Also try using a velour powder puff to apply the blot / no-colour powder. As powder brushes sometimes shed tiny bristles that will stick to the setting spray.
Good luck.
HtH