If you need to remove a member from your community, you can do it from people management. After you have removed the user, they will no longer have access to your community. The user will keep their CrowdWorks profile and other access rights they have.
If they have not yet registered their profile, you can also withdraw their invitation. Read this on how to withdraw a user invitation
How to remove a person
Navigate to manage > people
Find the person you wish to remove and tick the box next to their name
Click remove access from the top menu
Confirm your choice
You can always send them a new invitation if you want to add them back to your community.