Phantom still gets blocked, as it reveals itself in the header. I've had success...

rectangletangle · on July 14, 2017

You can set the user-agent with Phantom

    var webPage = require('webpage');
    var page = webPage.create();
    page.settings.userAgent = 'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/37.0.2062.120 Safari/537.36';

http://phantomjs.org/api/webpage/property/settings.html

dhruvkar · on July 14, 2017

While that's true, user-agent isn't the only thing in the header that reveals PhantomJS[0].

You could take the time to build in spoofs for these issues. But for testing (and scraping), you're going to be better off if your headless browser is the same as your GUI browser.

0: https://blog.shapesecurity.com/2015/01/22/detecting-phantomj...

eriknstr · on July 14, 2017

I think I read recently that Chrome / Chromium is now able to run without having to use xvfb, so now truly headless.