Outlook certainly can use a Personal Address Book if it is configured to use one.
Rather than exporting to a CSV file, if you do have Contacts, export them to a PST file instead. Much easier to copy back in again. In fact you don't even need to import them if you do it that way. Just open up the PST file with the contacts, select them all and copy to your new PST file's contact list.
---
But regarding your larger problem, if you press the Shift key while you delete files, they won't go to the Deleted Items folder and you can delete them. Hopefully this will help you reduce the size of your current PST file.
There are several ways to help this process. One is to look at the folder sizes to see which ones are taking up the most space and start with these first. Another is to show a Size entry in the column, sort by size, and shift-delete the larger files.
After you reduce the number of e-mails in it, you may need to compact the file aftewards.