Utility Belt exposes a small bug in JRuby. The JRuby team knows about it and are working on it. It involves their Readline implementation and goes beyond the extent of my fu, but I'll be putting a simple fix in Utility Belt svn soonish.
Utility Belt won't have its next release until at least late Feb, due to general scheduling insanity in my life, but for the time being, avoid using vi or emacs from within JRuby's IRB with Utility Belt. (TextMate integration still works fine.)
Update: The simple fix is in svn; that's the good news. The bad news is it's possible the backticks thing is a limitation of Java's ability to communicate with the world outside its virtual machine. One JRuby developer commented that the ticket might be a "can't fix."