That is correct.
The single biggest reason the Khoi and the San, two different tribes, were virtually wiped out was because of smallpox as they had no natural immunity to it. Two or more outbreaks of smallpox, starting in 1734 through to 1755 or thereabouts, by all accounts reduced the population to 10% or less of its original number according to one historic record.
With reduced numbers the remaining Khoi and San fled the Cape northwards to get away from the disease, others may have mingled into what is known as our coloured population today, the rest were probably shot by the white settlers as they became marginalised and stole to survive.
Basically in less than a hundred years of the settlers arriving in the Cape, the Khoi and the San who had been living in the area for thousands if not millions of years were reduced to a small band of people in the Northern Cape, Namibia (Nama) and Botswana. The Germans further reduced the Nama by half during the period 1904 and 1907.
I feel really sorry for these Khoi or San people, they have the historic claim to most of South Africa as we know it today, but are few and humble. Let's see what the Government does for them and how important minority groups are to them.