Custom Headers

If you want to pass your own custom headers to scrape a website then you can do that by using custom_headers=true paramter. This parameter will let you pass your own headers to the request constructor. This can be helpful when you are trying to scrape a website that requires login cookies or something similar. There is absolutely no extra cost to using this parameter.

Our API already manages headers and it is designed to increase the success rate of scraping any website. Only use this feature when either you want to scrape something which is behind an authentication wall or you are expecting something in response headers.


curl --header "X-customheader: bar" \

Last updated