> I'd really like to see numbers on how dtrace handles many millions of probes at once (which is what xray is handling).
It's not uncommon to have 10 or 100 million functions in some of these programs. I have strong doubts that dtrace has the same overhead given 200 million probes.
I see now. That's a fair question, and I'm not aware of data either way. I just tried a pretty simple experiment inspired by Brendan's that suggests that on my machine, the overhead is about 1450ns per probe for as many as 140,000 probes:
> AFAICT, there is no more generic function entry probe than what that blog post describes. But i'd love to be wrong, and understand how dtrace is going to determine what instructions are a function entry in several ns :P
Well, DTrace as architected is always going to pay the cost of a context switch into the kernel for each probe, and I think it's fair to take Brendan's result of 600ns as a lower bound of the per-probe overhead, at least on his machine. However, once in the kernel, for a typical native program (i.e., not JIT), I expect DTrace would only record the current userland thread instruction pointer. Names are typically resolved asynchronously by the consumer. So I would be surprised if it really was much slower, especially given the result above, but I too would like to see data.
I'm not saying that DTrace solves all problems or even that the OP should have used it instead. It's certainly true that for the special case of userland function boundary tracing, one might expect to do better by skipping the context switch (at the expense of much functionality, including any ability to correlate with broader system activity). But since DTrace was brought up, I wanted to help clarify the uncertainty about what it can do and what its overhead is.
I see now. That's a fair question, and I'm not aware of data either way. I just tried a pretty simple experiment inspired by Brendan's that suggests that on my machine, the overhead is about 1450ns per probe for as many as 140,000 probes:
https://gist.github.com/davepacheco/a12a0d45d55f0d7a28c312c2...
> AFAICT, there is no more generic function entry probe than what that blog post describes. But i'd love to be wrong, and understand how dtrace is going to determine what instructions are a function entry in several ns :P
Well, DTrace as architected is always going to pay the cost of a context switch into the kernel for each probe, and I think it's fair to take Brendan's result of 600ns as a lower bound of the per-probe overhead, at least on his machine. However, once in the kernel, for a typical native program (i.e., not JIT), I expect DTrace would only record the current userland thread instruction pointer. Names are typically resolved asynchronously by the consumer. So I would be surprised if it really was much slower, especially given the result above, but I too would like to see data.
I'm not saying that DTrace solves all problems or even that the OP should have used it instead. It's certainly true that for the special case of userland function boundary tracing, one might expect to do better by skipping the context switch (at the expense of much functionality, including any ability to correlate with broader system activity). But since DTrace was brought up, I wanted to help clarify the uncertainty about what it can do and what its overhead is.