sudo apt remove virtualbox virtualbox-*

Note that this will not remove the virtual machines and the files associated with the operating systems you installed using VirtualBox.