The thing with minerals, is that they are rocks and ores. And when you grind them into a fine powder, those individual particles have sharp edges to them and those sharp edges cut into the skin at the micoscopic level, hence your skin is reacting to it.
It's exactly the same theory behind Boric Acid. Yes it's a fine powder, but that fine powder cuts into the exoskeletal structure of insects (cockroaches, bed bugs, ants, earwigs, etc) and thus they die of dehydration (and stomach poisoning when they ingest it, as these types of social insects are excessive groomers).
Yes, companies add fillers and binders to them so that the number of microscopic edges to the minerals will be less so that it feels smooth, but the fillers and binders don't smooth out the edges of the minerals.
Example: fine beach sand (minerals in Mineral make-up). You add flour to it (filler). The result is that the beach sand may feel smoother, but the flour didn't make the edges of the sand particles smoother.
However, some people can react to the fillers or binders in a product as well, but not that many, as fillers are in all make-ups, not just mineral based make-ups.