If you need to remove a user from your community, you can do it from user management. After you have removed the user, they will no longer have access to your community. The user will keep their CrowdWorks profile and other access rights they have.
Navigate to settings > users
Click on the 3 dots on the right-hand side
Click the "Remove access" option
Confirm your selection
You can always send them a new invitation if you want to add them back to your community.
Confirm your selection to remove access: