There are a few reasons for the disparity in the number of people in your ChMS and Gloo.
When a contact has a number, Gloo only pulls over people with a "Mobile" phone number.
Check to make sure all your ChMS phone numbers are in the "Mobile" spot of a person's profile instead of marked as "Home."
Gloo only pulls people over that are older than 13.
Check to see if they have a birthday noted in your ChMS. If they are under 13, they will not be brought over.
Gloo removes any duplicate phone numbers.
Check to see if you have multiple people with the same phone number in your ChMS. If you do, only one will be brought over.
Gloo only pulls over people that are "active" in a ChMS.
Check to see that the people you want to integrate are still marked as "active."
Note: Gloo now has the ability to pull over contacts with only an email address (no phone number required).