I would keep up with the shampoo 2x a week at least, but whenever you run out of baby shampoo, I would recommend not using that. Get a shampoo for color treated hair instead, it keeps the bristles more moist and allows less dryness and breakage to happen.
After you wash, get a paper towel, put a little brush cleaner on it, and work the brush through it. Then lay it flat and let it dry.
For now, since you are still using the baby shampoo on your brushes, put a little conditioner on after you shampoo them, let it sit for a few minutes, then rinse really well. Let me know if that makes a difference. The little bristles might be coming from bacteria, but they also might be coming from the baby shampoo.