To confirm, the Outlook addin does not attempt to keep copies of your address books. It requests the latest copy of the address book from your system every time it is displayed.
First of all, please check that you are not viewing the wrong address book.
Please make sure the address book you have selected is the one you want to use.
If your Outlook Contact Group/Folder is not visible then you will need to do the following:
1) Right click on the Outlook Contact Group/Folder
2) Select Properties
3) Select the 'Outlook Address Book' tab
4) Select the 'Show this folder as an e-mail address book' checkbox
5) Click 'OK'