(James Bennet's blog software won't let me post a comment this long, so I'm putting it here instead. It's a reply to this post.)
Andrew Ingram: as Philip Brocoum said, the magic incantation you must painstakingly memorize, if you dare to meddle with supernatural forces and summon up the request object from the distant, mystic realms within the innards of Rails, is you type the word "request."
Seriously, how is that magical? How is that hard to remember? I can't imagine anybody sits down and goes "oh my God, in order to get at a request, I have to remember to type 'request'. How am I going to remember that? I'd better make some flash cards." Is that you? I don't think it's you. I don't think it's anybody.
Anyway, I hate to say this, but I know nothing about Django and have no interest in Django, aside from a mild curiousity and having heard a few good things. The only interesting thing in that "I hate magic: why Django rocks" article *to me* was the term "magic." I said this on Reddit, debating Django's finer points based on my rant is like seeing a bull charge and asking if the matador's cape should have been a different shade of red. I wish I had something useful to say about Django but I don't, and I don't pretend to.
Reg - you can lead a horse to water but you can't make him drink. Thanks all the same tho.
All the people who want to debate about me, i.e., all you crazy ad hominem types and the guy who thinks I brought a shitstorm on my head or whatever - well, sorry, I realize this is ironic, but I just don't think the topic is that interesting.
Likewise the topic of mysterious and/or surprising behavior in Rails, I don't think it's a very interesting topic to write about. It's a very interesting topic to write *code* about, and I think it's very interesting what's going on in Rails with the heavily Merb-influenced refactoring and differentiating of public vs. private APIs, but if you have opinions about that, it's a lot more useful to either express them in code or complain about them in more detail, so that people who can solve the problems know what the problems are. I admire that effort but it's just not my personality type, there are a bunch of other things I'll be working on before I spend much time on that myself.
All the comments etc. out of the way, James, I think your argument is reasonable but timid. You acknowledge that calling something magic is lame, but you don't call it something else. You downgrade it from magic to "magic." Those quotation marks are not strong enough to undo the damage that this awful, inappropriate variable naming does.
I mean basically if I can paraphrase you're saying that you have to find a balance between clever tricks and drudge code. To some extent, you're agreeing with something I said. At the end of my post, my conclusion is that since a lot of people are new to this whole phenomenon of code which writes code, or code which rewrites itself or modifies itself, there are a lot of people doing it wrong and making various mistakes as part of the ongoing effort to discover that balance. Implicit (but apparently not explicit) is the assumption that you need to go through those mistakes in order to get to that balance. You're going to get through those mistakes quicker if you look at your techniques as techniques then you are if you wave your hands and complain about nonspecific "magic."
For instance, you tell this story about Django moving from code generation to in-memory code generation. If I recall correctly, Rails went through a transition where a lot of model attribute/association methods were handled by method_missing and are now generated. In other words, Rails probably encountered the opposite problem because it went in the opposite direction. Now we're not talking about magic. Now we're talking about how one framework discovered they didn't like how they were using code generation, and preferred an in-memory solution, while the other framework made almost exactly the opposite decision. That's an interesting contrast. If we were to look at the histories of the two projects and the specifics of these design changes, and their contexts, we would probably learn something of value to both communities about the appropriateness of these strategies at different times and in different contexts. The big difference between this and a dialogue about "magic" is that *this* is USEFUL. People babbling about "magic" are not doing anyone any favors. They're just wasting everybody's time, and I think that's very greedy.
However, there is also in my opinion a massive flaw with your argument, which is that so-called "magical" features are not necessarily clever tricks at all. In some cases, they're very egregious, blatant self-indulgence. In other cases, they achieve great terseness. A very terse code base is a wonderful thing. However there is a downside. The Lisp community is a little autistic, and I think part of the reason is that their code gets too terse, because you can modify the language so much that you almost have a different dialect of Lisp for every single program you write. That's a judgement call, just like the judgement call of clever tricks vs. drudge work, but it's a different type of judgement call.
The "clever tricks vs. drudge work" spectrum is a valid spectrum, but it's not the only spectrum. It might not even be the most important spectrum. Zed Shaw just tweeted me that he thinks the practical problem with "magical" code is that it's difficult to obtain a clear, simple stack trace when you're invoking methods that don't map directly to any actual method definition anywhere in your code base. And the thing is, as far as I understand it, people were complaining about this with Lisp before I was even born. But the question becomes, is it even about clever tricks vs. drudge work? Maybe it's just that you can't debug complex, self-modifying code with a debugging tool designed for simple, literal code. Maybe it's just that when you graduate to more sophisticated forms of programming, that little pocket calculator we call the stack trace becomes totally fucking useless and we have to throw it away.
Maybe at that stage we need a debugger. I'm no fan of debuggers, but a lot of Smalltalk people tell me that the Smalltalk debugger is a wonderful thing. They're used to working with power, and they refer to this as their power tool. Maybe they know something. Since they've been working with these kinds of language features for a very long time, and they swear by this tool, maybe tools of this nature are more valuable than I personally ever realized. And certainly asking this question, about when you have chosen to use unusual programming features, is it not reasonable to expect other things in your approach to programming to change as well, this question is just so much more interesting than all these ridiculous people waving their arms and peeing their undies over "magic."