sudo apt remove virtualbox virtualbox-*
Note that this will not remove the virtual machines and the files associated with the operating systems you installed using VirtualBox.