If you need to remove a member from your community, you can do it from people management. After you have removed the user, they will no longer have access to your community. The user will keep their CrowdWorks profile and other access rights they have.
If they have not yet registered their profile, you can also withdraw their invitation. Read this on how to withdraw a user invitation
How to remove a person
Navigate to manage > people
Click on the 3 dots on the right-hand side
Click the "Remove access" option
Confirm your selection
You can always send them a new invitation if you want to add them back to your community.