Take the fastest webserver written for Java and for Erlang. Now run static requests around 2048 bytes, 10k connections, 3 req/s on each connection for 5 minutes. Measure the 99.9th percentile latency.
Erlang wins by a large margin because the JVM has to garbage collect and block everyone while doing so. This is not desirable in a soft-realtime system.
I'd definitely not order one as the strict superset of the other.
The TechEmpower Benchmarks measures the maximal number of requests per second you can churn through a server. They don't care for the number of connections, but simply pick the best result from trying different connection counts.
Also, they don't really measure latency correct I think, because they make the mistake of "coordinated omission" (Look it up, Gil Tene, of Zing/Azul fame is its main discoverer).
I was very careful in my wording. You need 10.000 connections, each running 3 req/s for 30.000 req/s. So we are not trying to maximize the system. We are trying to saturate it over a long period with a constant load. And you need to use wrk2 (also from Gil Tene) because otherwise the load generator and the System-under-test ends up coordinating among themselves, hiding the fact that some of the 10.000 connections are starved to get more req/s through on a subset.
The Java frameworks are very very good at handling 512 connections and throwing as many requests through those as possible. What they fail at, is 10.000 connections. What usually happens in bad load generators is that they allow the Java code to starve, say 9500 of those connections while getting all the good numbers on the last 500. wrk2 doesn't allow that to happen, and starvation is counted against the SUT.
As for the source, you would have to wait. I have the raw data, but I have not yet written the blog post. I think I've given you all the information you need to carry out the test if you are so inclined here.
I deleted my comment because I realize the benchmarks don't cover a 5 minute test. It's unproven whether the Java implementations need to in fact GC over that time. I'm skeptical though -- they probably reuse all their objects, and I think it would be interesting to prove that.
Also, the benchmarks do in fact test all the way up to 16,384 simultaneous connections.[1]
I bet if you published a test case that proves Java GC impacting the max latency over 5 minutes, that would get the attention of TechEmpower and make a case for extending the test past 15 seconds.
> It's unproven whether the Java implementations need to in fact GC over that time. I'm skeptical though -- they probably reuse all their objects
Are you saying that it's common for java web-frameworks to automatically reuse objects and that they do not cause any GC?
> Also, the benchmarks do in fact test all the way up to 16,384 simultaneous connections.
Read jlouis comment once more. It's not that they don't try many connections at once, it's that they take the best result that they get. Big difference.
EDIT:
Looking at the code for the Java Netty/JSON test shows that it does not reuse objects. The biggest takeaway from these tests is that it's hard to do performance testing...
> Erlang wins by a large margin because the JVM has to garbage collect and block everyone while doing so.
Not even close (certainly not when using Quasar). Whether or not you have large GC pauses depends on how you use the heap. If you only allocate objects that live for the duration of the request (which can be enforced by your choice of JVM language) you get the same GC behavior as Erlang (only more general), and a much, much, much, better computation performance, due to HotSpot having one of the world's most advanced optimizing compilers.
The TechEmpower Web Framework Benchmarks score Java 5X faster than Erlang at a simple JSON test. (There's an even simpler plaintext test but no one was implemented it for Erlang.)
However it's worth nothing the test only runs for 15 seconds. It might be an interesting addition to run it for 10 minutes and measure 99.9% latency, as jlouis proposes, and prove one of you correct.
It also shows that Erlang is 15X faster than Java (depending on what framework you compare) and that the Java Netty/JSON test has 8X more code than the Erlang Cowboy/JSON counterpart ;)
>It also shows that Erlang is 15X faster than Java
That's nonsense. jlouis challenged us to "Take the fastest webserver written for Java and for Erlang", not the slowest ones. And that's exactly what the TechEmpower benchmarks do and they give Java a ~500% advantage.
As for codesize, if you don't like netty then the very next ranked result is servlet3-cass which delivers nearly the same performance -- again, approx 5X faster than the fastest Erlang impl -- in just a few lines of code. Surely you were aware of this when you wrote that comment.
There was a smiley in the end you know... My point was that these "tests" really shows nothing. It's a very specific scenario and if you read the comment from jlouis you would know that his scenario was very different and would yield a different result.
You're trying to dismiss the benchmark because you don't like the result. But actually the tests do show max latency with high concurrency which is the question at hand, and AFAICS the only factor not covered is a minutes-long test that might surface any GC impacts. So they are an excellent base for comparing performance.
Again if you or anyone can prove a significant GC impact serving up text or Json definitely hit up the techempower folks and make a case for extending the test length. It might help improve Erlang's standing which right now utterly sucks in comparison to Java performance (probably because of the maturity of the JVM).
HotSpot doesn't matter that much in this test. You end up spending 80% of the time in the kernel and when I tested, the Erlang code only spent a fraction of it's time in the emulator loop. In other words, the speed comes from everything but the optimizing compiler. It is from the runtimes.
Sure I'll just fork over $8,000 and get started... The beauty of BEAM is that I don't have to know or worry about projects like Zing, BEAM "just works." (Another mark against Zing just because you brought it up -- where's the source code?)
It's not that you don't have to know or worry about groundbreaking projects like Zing, but that they don't exist. The resources put into advancing the Erlang ecosystem are a tiny fraction of those put into the JVM ecosystem, and it shows.
You say BEAM "just works", but it "just works" if you can put up with its non-stellar performance or happen to play directly to its strengths. Any technology just works if you happen to need exactly what it provides. Java and the JVM, however, like the Linux kernel, are a huge project with vast resources, wide reach and very wide applicability from small embedded devices to mainframes. So, naturally, there are a lot of things to choose from, but you are guaranteed to find one that suits your need -- no matter what it is.
Too many Erlang projects start out thinking Erlang is good enough, and then find that they need to write more and more of their code in C. This just cannot happen with the JVM. Not that it's appropriate for everything (it's bad for command-line utilities and Java SE is inappropriate for constrained environments), but it's much harder to miss. You don't need to know or worry about Zing unless you need what it provides (worst-case latency of under 50us); similarly for other tens of thousands of JVM libraries and tools. All you need to know is that you can find a high-quality implementation of whatever you might need down the line.
As for Zing's source code: Zing is proprietary, but as the JVM ecosystem is so big, there's a similar open-source attempt by Red Hat and intended to go into OpenJDK called Shenandoah[1]. Its code is here: http://icedtea.classpath.org/hg/jdk9-shenandoah/hotspot/
I'm aware of Zing and I'm pretty sure it would fare way better in this kind of problem. However, I'm inclined to guess that every time you sink 1 hour of work into BEAM, there is somewhere between 5-15 hours sunk into OpenJDK and other VMs in Java space. It wins many competitions by brute force alone.
Furthermore, Erlang has some things against it in the speed department:
It is forced functional.
It is dynamically typed (A tracing JIT can somewhat alleviate this problem).
But I think the BEAM architecture is pretty sound. It has a lot of things in common with a micro kernel, architecturally. And while micro kernels are not popular currently, they see a lot of use in the embedded space, on sattelite's and so on.
Erlang wins by a large margin because the JVM has to garbage collect and block everyone while doing so. This is not desirable in a soft-realtime system.
I'd definitely not order one as the strict superset of the other.