Every so often a traveling peddler could pass through, selling extremely rare and unique items.  But you can only purchase these items with gold or gems.  Certain items he sells for gold, others for gems.  The items available would be random and change every time.  You can exchange silver for gold and gems with the banker/money changer.  But the bank won't be open every day.  And not necessarily on the day the traveling peddler/merchant appears.  Also, the banker will only allow you to exchange a certain amount of silver each day.  But you can increase your favor/credit with the banker by investing gems in your account, and  thereby be allowed to exchange a larger amount of silver per day.   

The kinds of items the peddler might sell could be things like elixirs that permanently increase certain attributes/stats, weapons with bonuses against certain factions, dragon meat that ranks up and levels you at the same time, etc.