Where did you get the 10k from exactly?
Tried finding the original source, but it was somewhere last year so no clue. However the FBI says the KKK has about 2,000 members nationally. The FBI says the Aryan Brotherhood has 15,000ish active members in prisons and out. So maybe 20k not 10k as I remembered. Those are you two biggest groups.
There are cross over between the different groups though. Also worth noting, much like in our jails, these groups are joined as a form of protection so outside of jail they are not always active members.
The main reason I say that their numbers are small is that the highest estimate of the Charlottesville Unite The Right Rally was 250 and that was from various different organisations. The "anniversary" the next year drew only 30 people. I Believe they wan't people to think there are much more of them than there are to give themselves credibility.
Obviously the size does not really matter as a few people can carry out a devastating attack. Luckily they don't do suicide missions.