There are a few reasons for the disparity in the number of people in your ChMS and Gloo.
Gloo only pulls people over with a phone number. More specifically, Gloo only pulls people over with a "Mobile" phone number.
Check to make sure all your ChMS phone numbers are in the "Mobile" spot of a person's profile instead of marked as "Home."
Gloo only pulls people over that are older than 13.
Check to see if they have a birthday noted in your ChMS. If they are under 13, they will not be brought over.
Gloo removes any duplicate phone numbers.
Check to see if you have multiple people with the same phone number in your ChMS. If you do, only one will be brought over.
Gloo only pulls over people that are "active" in a ChMS.
Check to see that the people you want to integrate are still marked as "active."