If you installed rubygem with some sore of package management system, it probably provides for an uninstall capability. If not, here is one way to remove it.

1. Find out where ruby installs its libraries. If you don’t know where your libraries are located, you can execute the following command:
   ruby -e ‘puts $:’
   On my machine, this outputs:
   /usr/lib/ruby/site_ruby/1.8
   /usr/lib/ruby/site_ruby/1.8/i386-freebsd5
   /usr/lib/ruby/site_ruby
   /usr/lib/ruby/1.8
   /usr/lib/ruby/1.8/i386-freebsd5
2. Change into the /usr/lib/ruby/site_ruby/1.8 directory. In there you should find the following files that belong to rubygems:
   gemconfigure.rb
   rubygems.rb
   ubygems.rb
   You should also see the directory "rubygems"
   Remove these files and the directory with the following commands:
   cd /usr/lib/ruby/site_ruby/1.8
   rm -r gemconfigure.rb rubygems.rb ubygems.rb rubygems
3. In order to remove all the gems on your computer, remove the gems directory under /usr/lib/ruby with the following command:
   rm -r /usr/lib/ruby/gems
   If your ruby libraries are installed somewhere other than /usr/lib/ruby, make the appropriate changes to the above commands.

Source: http://lists.rubyonrails.org/pipermail/rails/2006-May/038251.html