"Magic that sometimes works and sometimes doesn't is fun for small projects..."

SQL has a magical compiler. It's actually a lot more magical in some ways than GHC, because good implementations use statistics on the actual input data and a sophisticated cost-based optimizer. And SQL is certainly suitable for large projects.

I mostly program in C, so you don't have to convince me that predictability of code fragments has its benefits.

But it has a lot of costs, too. Sometimes, good performance fundamentally requires a lot of complexity, so if your compiler is not helping you (being "magical") then the complexity is forced on the programmer. Once the programmer takes on that complexity, they also take on the burden of additional maintenance cost and additional cost to add new capabilities forever.

So, you have to consider whether you really have a net increase in predictability by using more predictable (but less helpful) tools. People complain about garbage collection and virtual memory and NUMA for the same reasons, but those are ubiquitous now -- dealing with the annoyances simply takes less effort than managing the memory yourself.

You could argue that laziness, in particular, is not worth the trade-off (though it has worked out pretty well for SQL).

SQL has EXPLAIN PLAN. SQL without EXPLAIN PLAN is useless, for exactly this reason: you have no idea how it will perform.

You can output various internal representations from GHC. I believe that, or something similar, can show you a lot of information about the kinds of transformations done.

I'm not really a GHC expert, but that doesn't seem like a fundamental problem that can't be solved, and I assume it is (to some extent) solved already.

On a large project you optimize the hotspots using an excellent profiler...

You also want reasonable and understandable default behavior so you can avoid problems before they even occur.

Different languages make different tradeoffs.

It's not that easy to see efficiency from the source code even in a low level language like C. It shouldn't really be the job of the programmer to see from the source code if stuff gets efficiently executed. Profilers are for that, and if you really wan't to know what gets executed, you can always read the generated assembly. Code should express the intention, not the implementation of the execution. That is what high level languages are for.

In SQL, you don't even specify the algorithm, the cost-based optimizer chooses the algorithm for you (that is, if it's a good SQL implementation).

Except that it isn't magic. It is simply a different method of evaluation. Calling this "magic" is the same as calling strict evaluation in C++ magic.

I am talking about things like strictness analysis, automatic unboxing, etc. not lazy evaluation. And even an 'optimization' as simple as common subexpression elimination on a lazy language can introduce space leaks, e.g. http://hackage.haskell.org/trac/ghc/ticket/917.

I don't understand. Strictness analysis and automatic un-boxing are "free" optimizations and are directly related to laziness. So you can't talk about one without the other.

To clarify, I meant that even if you exclude the claimed unpredictable performance of lazy evaluation itself, you still have the unpredictability of optimizations that are performed on top of lazy evaluation like strictness analysis and unboxing.

My point was simply that the unpredictability of these optimizations is irrelevant in my opinion. They simply make your programs run faster. I rarely if ever have seen a library that relies on these optimizations triggering. Usually strictness, when absolutely necessary, is specifically annotated by the developer with bang patterns. Same with unboxing with the UNPACK pragma.

If everything is annotated then obviously you don't have predictability problems. I was talking about relying on automatic optimizations. In practice, do Haskell users rely mostly on the explicit annotations rather than the 'magic' optimizations?

I think if you polled a bunch of devs, you'd come up with a bunch of things you "shouldn't have to do" in order to write efficient code. It varies by framework, and GHC/Haskell is generally unapologetic about what you have to do to write fast Haskell programs. It is a language nerd's language, after all.

The really important advice boils down to the simple things. Like not using the default String type for heavy text processing since it is a [Char]. In any language a string represented as a linked list is going to be slow. It is still useful, just slow. But use Data.Text instead and you'll likely see a massive performance gain due to the use of proper data structures and algorithms (when processing a lot of text).

Having a solid grasp of the fundamentals gets you quite far.