Node is maddening for one simple reason: asynchronous code shouldn't need to look like asynchronous code, and in fact it's much safer and less verbose if it doesn't.
The whole performance argument is completely beside the point. Yes, we get it, in-process asynchronous parallelism is where its at. But it simply doesn't follow that we need to live in inversion-of-control hell.
Even the simplest node program ends up drowning in error-handling code, because there's no way to automatically propagate errors back to the right place. Every function needs to handle error conditions immediately, because you can't throw an error up the stack, because your stack is completely meaningless.
This is a huge step backward. It feels like coding directly against system calls in C.
It doesn't need to be this way. There's no reason your language runtime can't be smart enough to switch contexts automatically, so that your code gets to be "blocking" even though the actual OS-level process is never blocked. Erlang does this, Stackless Python does this, any Lisp with good old call-with-current-continuation does this.
It's absolutely a step back and the Node community is way too harsh on those that point this out. But they are probably making the right call by not forking JavaScript. If you want a synchronous style its easy enough with node-fibers. The problem is mostly that there isn't a cultural consensus around a node-fibers based ecosystem of libraries.
This is absolutely correct. The lack of consensus is the heart of the problem. There are several nice solutions (node-fibers being one), but none of them is dominant, so none of them work out of the box with everything else.
I'm convinced the only way you get this kind of consensus is by building a feature into your language's standard library. This is one big reason that languages with robust "batteries included" standard libraries have been successful. It makes it so much easier to interoperate when everyone can just assume the same abstractions are always available.
For those interested in the fibers way of doing things, I recommend checking out https://github.com/scriby/asyncblock. It provides an easy-to-use abstraction to get the best of both worlds (straight-line code without blocking).
I don't think there really needs to be a fibers based ecosystem. Isn't it best if modules don't rely on fibers such that they can be reused in either context?
> It feels like coding directly against system calls in C.
I disagree. Coding directly against system calls is much harder in C, because you don't have the advantage of closures and flexible typing. But, that aside, node is intended to be a very low-level library that facilitates higher-level extensions and abstractions in userland. It is more like C than it is like Python, and that is by design.
Node is JavaScript on the server. It is not JavaScript-like, or Compiles-to-JavaScript, or Erlang-with-semicolons-and-braces on the server. We don't mess around with the language runtime, we take it mostly as-is, and there is a huge benefit to doing that.
When and if V8 implements generators (as they are likely to do somewhat soon), then I expect we'll see a lot of experimentation in this area in userland modules. They'll have to be run with a --harmony_generators flag, most likely, but they won't need to be compiled or do scary bad-touch things with threads and stacks.
When the area has been explored a bit in those userland modules, and one or a few of them are popular and good and intuitive to use, and V8 moves generators out from behind a flag, and they're fast enough to be used in node without introducing performance regressions, then we'll investigate adding something like this to node-core.
Part of the reason why you meet such backlash from people in the Node community when you complain about "callback hell" is that the model is very simple, and it really is not as bad as it looks at first. JavaScript's bulky "function" keyword does make CPS quite a bit uglier than it is in Scheme, but it's a very reasonable approach to the problem which is extremely extensible.
The crappiest part in my opinion is doing `if (er) return cb(er)` all the damn time. Domains make that a little bit easier, but you're just trading one bit of boilerplate for another, so I don't know. Generators are probably the ideal approach to that problem, but I'm personally not sure they're worth the complexity cost they introduce. I am often wrong, and try to be quick to admit it. We'll see how they change the shape of things once they're a real thing and not just an idea.
In the meantime, use named functions. Use the "async" utility. Use Stream interfaces and .pip() them to one another. And most of all, Don't write big apps! Write small modules that each do one thing, and assemble them into other modules that do a bigger thing. You can't get into callback hell if you don't go there.
If you're writing really sensitive things then callbacks are great, but I strongly disagree that this should be considered "good enough" for app-level code, but with the current state of node as a community it seems like would be very very awkward to even introduce something else right now. node-fibers is definitely annoying since you have to compile but even if you didn't you'd still be wrapping all of your favorite existing libraries to have a nice interface. As for back-pressure etc there's no reason why that doesn't work well if not better and implicitly with sync-style, many languages do this already and do it really well. I love node but the core community needs to stop being ignorant towards other concepts thinking that callbacks are simply "the way to go" when they're simply codesmell for many if not most applications.
Stuff like "hey @nodejs people lets make a website called http://fibersarestupid.com in which we provide education on how to use callbacks and streams" certainly doesn't help, it just makes node as a community look childish, maybe the site should be called iDontUnderstandFibersThereforeIDismissThem.com... come on.
> The crappiest part in my opinion is doing `if (er) return cb(er)` all the damn time.
That is heart of my complaint. And it's why I made an analogy to system calls, because there you end up doing the same thing -- manually propagating error codes.
> It is more like C than it is like Python, and that is by design.
I agree, which is why the original article simply makes no sense when it presents Node as a competitor to Ruby. They aren't really comparable.
Like I said in my nodeconf talk. "Callbacks are Hard... in C!" JS callbacks are a very elegant tool for a very hard problem. In node you have the power and responsibility to manually decide when your thread of execution stops and when is resumes. That's what is hard. Coroutines are another tool for the same problem, but they come with their own set of problems and complexities. Callbacks at least are very simple to understand and reason about.
It seems like so many arguments against Node, or Javascript in general, is based on ignorance of Javascript and programming techniques that you would never do in other languages anyways (like deep nesting).
I'm disappointed that the discussion has (yet again) turned into debating the merits of Node rather than the much more important topic of where to place the MVC logic (client vs server), regardless of the language/framework chosen. Node was just a personal preference for the OP, based on his performance requirements.
The reason these types of arguments are generally perceived as being ignored is because they generally argue a point that everyone who has experience with asynchronous programming already realizes. You need to change your level of abstraction. That can be through language choice, or through flow control programs, or streams, or switching to a monadic coding style or replacing nesting with a more functional approach.
If you prefer that the language does that for you, then that's great. There's enough viable options out there to fit anyone's preferred asynchronous style. Node exposes a lot of the low level elements of asynchronous programming (much like the low level programming you could do in C). You can stick with that, or use a flow control library, or employ any other number of viable strategies to avoid deeply nested callback soup. You get to pick the level of abstraction you work on, and not everyone wants to work on the highest levels of abstraction (where the language or framework handles everything and you just write synchronous style code).
The node community could probably do a better job of directing people to good documentation on those different levels of abstraction (it's a bit silly to assume someone will know to use streams, or monads to clean up the style of their code). When node first started to gain traction, it felt like almost everyone wrote a flow control library. Now the problem is solved in almost every way imaginable. The problem is that new users don't know where to look to find those 3rd party solutions, or how to evaluate the tradeoffs.
You're right to find it maddening. A lot of other people did as well. It doesn't need to be maddening though, and while you have a valid point, it's not especially relevant given the number of possible solutions (you just have to know how to find the style you're after).
The problem with the "solve it with a library" answer is interoperability. You and your dependencies usually need to agree on which paradigm you're going to use.
The Node ecosystem clearly has no consensus on this point. So you spend a lot of time gluing pieces together.
And I do know how to find the style I'm after. At the moment I'm doing a lot with Q, and I have also used node-fibers. But in either case, you end up doing a lot of plumbing, because you end up depending on other people who chose different (or non-existent) high-level flow control abstractions.
Another point: you're implying that not blocking the OS level process is in itself good thing. But that then requires that your runtime implements its own scheduler, in effect duplicating work the OS is already doing.
This is true, and in an ideal world we would definitely leave it up to the operating system. I would love to see the Linux kernel get so good as massive parallelism that we can simply do it that way.
But as far as I know, it's still not competitive. Which makes sense, since the kernel offers far stronger separation guarantees.
Your argument is valid. But Node's lightweight-ness comes exactly from the async non-blocking model. The more concurrent connections you have, the bigger is it's advantage. Node can scale without the need to increase the memory as much.
If you like to have a sync model on node, fine, add a layer like threads-a-gogo and you are ready. By doing so, you give up some of the lightweight-ness and will have increased memory consumption. It is your choice and it depends on your use case. If you expect just a few hundred connections or you have lots of memory, you don't need to think about node.
This completely misses the point. You don't get rid of non-blocking single OS thread. You get rid of the horrible API to use it. Node says "here's a low level API, now write an ad-hoc version of green threads on top of it". But other languages have been saying "we already wrote and tested a solid green threads implementation on the low level async API for you, you can just use that" for years now.
Even green threads increase memory consumption and add implementation overhead.
Anyway, the call for a sync model is nearly as old as node is and I agree that a solid thread implementation should be part of node for those who like to code this way.
Show me the benchmarks. I have no reason to believe that the buggy, ad-hoc versions being written over and over again in every node app are faster than writing one good version and using it.
And it is not a call for a sync model. It is a call for a sane API.
I got you with your call for a well designed API and I am with you.
About a benchmark, I have none, but it is the nature of threading. Threads need to store and switch contexts. Even if it is just a few 100KBs with 100000 connections this easily multiplies to a gig of additional memory use. In the real world this is often a few MBs per thread making that scale even impossible for a mid class server.
No that's not how green threads work, which was the whole point. You can have an API that presents you with "threads", but not have any actual threads underlying it. It is just a state machine running async, event driven code under the hood. The overhead of such a system is no higher than it is using the naive and error prone event loop approach.
Programming against system calls doesn't suffer from this problem. In fact, if you use synchronous system calls, the OS does exactly what you're asking for: switching contexts when appropriate.
Sounds like what you want is a good Promises library. They will guarantee that errors float back up to the original caller (if not handled directly). Promises might be baked into ES6. For now these are the best 2 implementations:
But until the Node community can agree on a standard for this, library writers can't make their APIs dependent on these techniques, so we're stuck at the lowest-common-denominator.
And even if everybody was using Q, it's still an awful lot of boilerplate compared to saying the same thing in a language with coroutines.
And there are painful design choices that make it unlikely everyone will ever agree on a promise library. For example: Q guarantees that promises will resolve on a different stack than where they were created. This is nice, it helps you reason about the code. But there are places where you absolutely need to resolve a promise on the same stack (Node's IO handlers, or parts of the window.openDatabase API), and dealing with the edge cases is really gross.
Library authors won't, and I would argue shouldn't, make their code dependent on a particular promises implementation. When/if either ES standards body or Node authors choose a method that will become the default. Until then individuals will use the library they like, or none at all, and we've gotten by without promises in the last 10 years that JS has exploded in popularity.
By the way, Q can easily wrap node-style callbacks in a single line.
The whole performance argument is completely beside the point. Yes, we get it, in-process asynchronous parallelism is where its at. But it simply doesn't follow that we need to live in inversion-of-control hell.
Even the simplest node program ends up drowning in error-handling code, because there's no way to automatically propagate errors back to the right place. Every function needs to handle error conditions immediately, because you can't throw an error up the stack, because your stack is completely meaningless.
This is a huge step backward. It feels like coding directly against system calls in C.
It doesn't need to be this way. There's no reason your language runtime can't be smart enough to switch contexts automatically, so that your code gets to be "blocking" even though the actual OS-level process is never blocked. Erlang does this, Stackless Python does this, any Lisp with good old call-with-current-continuation does this.