To remove a label from a task or project, click on the edit icon and click on the label in the edit mini dialog to remove it.