Update: Jose Valim corrects me on this: your application being created with bundler is just a default, as with prototype, activerecord, erb and test/unit.
Rumors abound regarding Bundler annoying people in various ways. Before I earn myself a stern talking-to, let me stress that the rumors which abound are indeed rumors, and they may only abound because that's what rumors do. I mean how many times have you heard of a pickle abounding? Rumors abound. The Dude abides. That's how it is.
I haven't used Bundler myself, and Jeremy McAnally says Bundler is awesome. Always take rumors with a grain of salt. All I know is people are talking. I've even seen Bundler cause the odd interwebz skirmish here and there. I hear it changed too much, too fast. I hear it breaks Heroku. I hear it's the best thing since sliced bread. I hear a ton of talk and none of it interests me. None of it means anything.
Bundler is not the only way to manage gem dependencies in the Ruby ecosystem. In fact gems themselves are not the only packaging system in the Ruby ecosystem. The moment I get some spare time, I plan to get in the habit of using Rip, and if everything goes according to my plans, I'm never using gems again in my life after about mid-2010. I just think gems are the spawn of the devil. So I don't give a damn about Bundler, because I don't want to use it either way. I don't care how good it is; in my opinion, its foundation is too rotten to redeem. I hear great things about the OS X Subversion wrapper app Versions, but I'm never going to use it, because I favor Git over Subversion. RubyGems are a Subversion that have not yet met their Git.
(I mean that very precisely. I knew the CollabNet people when they first started building Subversion, back in 98 or 99. I was in their offices, helping them set up furniture. They did the world a gigantic favor moving everybody off of CVS. But there were more moves to make, and Git changed my life. Likewise, RubyGems were a fantastic move for 2003, but their sun is setting. It's only a matter of time.)