pax_global_header00006660000000000000000000000064132612753150014517gustar00rootroot0000000000000052 comment=d56c981bfebdb3a53f6437be8c823888c15155d5 cloudflare-scrape-1.9.5/000077500000000000000000000000001326127531500151265ustar00rootroot00000000000000cloudflare-scrape-1.9.5/.gitignore000066400000000000000000000003601326127531500171150ustar00rootroot00000000000000# Byte-compiled / optimized / DLL files __pycache__/ *.py[cod] # C extensions *.so # Distribution / packaging .Python env/ build/ develop-eggs/ dist/ downloads/ eggs/ .eggs/ lib/ lib64/ parts/ sdist/ var/ *.egg-info/ .installed.cfg *.egg cloudflare-scrape-1.9.5/LICENSE000066400000000000000000000020611326127531500161320ustar00rootroot00000000000000The MIT License (MIT) Copyright (c) 2015 Anorov Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. cloudflare-scrape-1.9.5/MANIFEST.in000066400000000000000000000000321326127531500166570ustar00rootroot00000000000000include LICENSE README.md cloudflare-scrape-1.9.5/README.md000066400000000000000000000242541326127531500164140ustar00rootroot00000000000000cloudflare-scrape ================= A simple Python module to bypass Cloudflare's anti-bot page (also known as "I'm Under Attack Mode", or IUAM), implemented with [Requests](https://github.com/kennethreitz/requests). Cloudflare changes their techniques periodically, so I will update this repo frequently. This can be useful if you wish to scrape or crawl a website protected with Cloudflare. Cloudflare's anti-bot page currently just checks if the client supports Javascript, though they may add additional techniques in the future. Due to Cloudflare continually changing and hardening their protection page, cloudflare-scrape requires Node.js to solve Javascript challenges. This allows the script to easily impersonate a regular web browser without explicitly deobfuscating and parsing Cloudflare's Javascript. Note: This only works when regular Cloudflare anti-bots is enabled (the "Checking your browser before accessing..." loading page). If there is a reCAPTCHA challenge, you're out of luck. Thankfully, the Javascript check page is much more common. For reference, this is the default message Cloudflare uses for these sorts of pages: Checking your browser before accessing website.com. This process is automatic. Your browser will redirect to your requested content shortly. Please allow up to 5 seconds... Any script using cloudflare-scrape will sleep for 5 seconds for the first visit to any site with Cloudflare anti-bots enabled, though no delay will occur after the first request. Installation ============ Simply run `pip install cfscrape`. The PyPI package is at https://pypi.python.org/pypi/cfscrape/ Alternatively, clone this repository and run `python setup.py install`. Dependencies ============ * Python 2.6 - 3.x * **[Requests](https://github.com/kennethreitz/requests)** >= 2.0 * **[Node.js](https://nodejs.org/)** * Your computer or server may already have it (check with `node -v`). If not, you can install it with `apt-get install nodejs` on Ubuntu and Debian. Otherwise, please read [Node's installation instructions](https://nodejs.org/en/download/package-manager/). `python setup.py install` will install the Python dependencies automatically. Node is the only application you need to install yourself. Updates ======= Cloudflare modifies their anti-bot protection page occasionally. So far it has changed maybe once per year on average. If you notice that the anti-bot page has changed, or if this module suddenly stops working, please create a GitHub issue so that I can update the code accordingly. * Many issues are a result of users not updating to the latest release of this project. Before filing an issue, please run the following command: ``` pip show cfscrape ``` If the value of the version field is not the latest release, please run the following to update your package: ``` pip install cfscrape -U ``` If you are still encountering a problem, open an issue and please include: * The full exception and stack trace. * The URL of the Cloudflare-protected page which the script does not work on. * A Pastebin or Gist containing the HTML source of the protected page. * The version number from `pip show cfscrape`. [This issue comment is a good example.](https://github.com/Anorov/cloudflare-scrape/issues/3#issuecomment-45827514) Usage ===== The simplest way to use cloudflare-scrape is by calling `create_scraper()`. ```python import cfscrape scraper = cfscrape.create_scraper() # returns a CloudflareScraper instance # Or: scraper = cfscrape.CloudflareScraper() # CloudflareScraper inherits from requests.Session print scraper.get("http://somesite.com").content # => "..." ``` That's it. Any requests made from this session object to websites protected by Cloudflare anti-bot will be handled automatically. Websites not using Cloudflare will be treated normally. You don't need to configure or call anything further, and you can effectively treat all websites as if they're not protected with anything. You use cloudflare-scrape exactly the same way you use Requests. (`CloudflareScraper` works identically to a Requests `Session` object.) Just instead of calling `requests.get()` or `requests.post()`, you call `scraper.get()` or `scraper.post()`. Consult [Requests' documentation](http://docs.python-requests.org/en/latest/user/quickstart/) for more information. ## Options ### Existing session If you already have an existing Requests session, you can pass it to `create_scraper()` to continue using that session. ```python session = requests.session() session.headers = ... scraper = cfscrape.create_scraper(sess=session) ``` Unfortunately, not all of Requests' session attributes are easily transferable, so if you run into problems with this, you should replace your initial `sess = requests.session()` call with `sess = cfscrape.create_scraper()`. ### Delays Normally, when a browser is faced with a Cloudflare IUAM challenge page, Cloudflare requires the browser to wait 5 seconds before submitting the challenge answer. If a website is under heavy load, sometimes this may fail. One solution is to increase the delay (perhaps to 10 or 15 seconds, depending on the website). If you would like to override this delay, pass the `delay` keyword argument to `create_scraper()` or `CloudflareScraper()`. There is no need to override this delay unless cloudflare-scrape generates an error recommending you increase the delay. ```python scraper = cfscrape.create_scraper(delay=10) ``` ## Integration It's easy to integrate cloudflare-scrape with other applications and tools. Cloudflare uses two cookies as tokens: one to verify you made it past their challenge page and one to track your session. To bypass the challenge page, simply include both of these cookies (with the appropriate user-agent) in all HTTP requests you make. To retrieve just the cookies (as a dictionary), use `cfscrape.get_tokens()`. To retrieve them as a full `Cookie` HTTP header, use `cfscrape.get_cookie_string()`. `get_tokens` and `get_cookie_string` both accept Requests' usual keyword arguments (like `get_tokens(url, proxies={"http": "socks5://localhost:9050"})`). Please read [Requests' documentation on request arguments](http://docs.python-requests.org/en/master/api/#requests.Session.request) for more information. *User-Agent Handling* The two integration functions return a tuple of `(cookie, user_agent_string)`. **You must use the same user-agent string for obtaining tokens and for making requests with those tokens, otherwise Cloudflare will flag you as a bot.** That means you have to pass the returned `user_agent_string` to whatever script, tool, or service you are passing the tokens to (e.g. curl, or a specialized scraping tool), and it must use that passed user-agent when it makes HTTP requests. If your tool already has a particular user-agent configured, you can make cloudflare-scrape use it with `cfscrape.get_tokens("http://somesite.com/", user_agent="User-Agent Here")` (also works for `get_cookie_string`). Otherwise, a randomly selected user-agent will be used. -------------------------------------------------------------------------------- ### Integration examples Remember, you must always use the same user-agent when retrieving or using these cookies. These functions all return a tuple of `(cookie_dict, user_agent_string)`. **Retrieving a cookie dict through a proxy** `get_tokens` is a convenience function for returning a Python dict containing Cloudflare's session cookies. For demonstration, we will configure this request to use a proxy. (Please note that if you request Cloudflare clearance tokens through a proxy, you must always use the same proxy when those tokens are passed to the server. Cloudflare requires that the challenge-solving IP and the visitor IP stay the same.) If you do not wish to use a proxy, just don't pass the `proxies` keyword argument. These convenience functions support all of Requests' normal keyword arguments, like `params`, `data`, and `headers`. ```python import cfscrape proxies = {"http": "http://localhost:8080", "https": "http://localhost:8080"} tokens, user_agent = cfscrape.get_tokens("http://somesite.com", proxies=proxies) print tokens # => {'cf_clearance': 'c8f913c707b818b47aa328d81cab57c349b1eee5-1426733163-3600', '__cfduid': 'dd8ec03dfdbcb8c2ea63e920f1335c1001426733158'} ``` **Retrieving a cookie string** `get_cookie_string` is a convenience function for returning the tokens as a string for use as a `Cookie` HTTP header value. This is useful when crafting an HTTP request manually, or working with an external application or library that passes on raw cookie headers. ```python import cfscrape request = "GET / HTTP/1.1\r\n" cookie_value, user_agent = cfscrape.get_cookie_string("http://somesite.com") request += "Cookie: %s\r\nUser-Agent: %s\r\n" % (cookie_value, user_agent) print request # GET / HTTP/1.1\r\n # Cookie: cf_clearance=c8f913c707b818b47aa328d81cab57c349b1eee5-1426733163-3600; __cfduid=dd8ec03dfdbcb8c2ea63e920f1335c1001426733158 # User-Agent: Some/User-Agent String ``` **curl example** Here is an example of integrating cloudflare-scrape with curl. As you can see, all you have to do is pass the cookies and user-agent to curl. ```python import subprocess import cfscrape # With get_tokens() cookie dict: # tokens, user_agent = cfscrape.get_tokens("http://somesite.com") # cookie_arg = "cf_clearance=%s; __cfduid=%s" % (tokens["cf_clearance"], tokens["__cfduid"]) # With get_cookie_string() cookie header; recommended for curl and similar external applications: cookie_arg, user_agent = cfscrape.get_cookie_string("http://somesite.com") # With a custom user-agent string you can optionally provide: # ua = "Scraping Bot" # cookie_arg, user_agent = cfscrape.get_cookie_string("http://somesite.com", user_agent=ua) result = subprocess.check_output(["curl", "--cookie", cookie_arg, "-A", user_agent, "http://somesite.com"]) ``` Trimmed down version. Prints page contents of any site protected with Cloudflare, via curl. (Warning: `shell=True` can be dangerous to use with `subprocess` in real code.) ```python url = "http://somesite.com" cookie_arg, user_agent = cfscrape.get_cookie_string(url) cmd = "curl --cookie {cookie_arg} -A {user_agent} {url}" print(subprocess.check_output(cmd.format(cookie_arg=cookie_arg, user_agent=user_agent, url=url), shell=True)) ``` cloudflare-scrape-1.9.5/cfscrape/000077500000000000000000000000001326127531500167145ustar00rootroot00000000000000cloudflare-scrape-1.9.5/cfscrape/__init__.py000066400000000000000000000215111326127531500210250ustar00rootroot00000000000000import logging import random import re import subprocess from copy import deepcopy from time import sleep from requests.sessions import Session try: from urlparse import urlparse except ImportError: from urllib.parse import urlparse __version__ = "1.9.5" DEFAULT_USER_AGENTS = [ "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Safari/537.36", "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Ubuntu Chromium/65.0.3325.181 Chrome/65.0.3325.181 Safari/537.36", "Mozilla/5.0 (Linux; Android 7.0; Moto G (5) Build/NPPS25.137-93-8) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.137 Mobile Safari/537.36", "Mozilla/5.0 (iPhone; CPU iPhone OS 7_0_4 like Mac OS X) AppleWebKit/537.51.1 (KHTML, like Gecko) Version/7.0 Mobile/11B554a Safari/9537.53", "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:60.0) Gecko/20100101 Firefox/60.0", "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.13; rv:59.0) Gecko/20100101 Firefox/59.0", "Mozilla/5.0 (Windows NT 6.3; Win64; x64; rv:57.0) Gecko/20100101 Firefox/57.0" ] DEFAULT_USER_AGENT = random.choice(DEFAULT_USER_AGENTS) BUG_REPORT = """\ Cloudflare may have changed their technique, or there may be a bug in the script. Please read https://github.com/Anorov/cloudflare-scrape#updates, then file a \ bug report at https://github.com/Anorov/cloudflare-scrape/issues."\ """ ANSWER_ACCEPT_ERROR = """\ The challenge answer was not properly accepted by Cloudflare. This can occur if \ the target website is under heavy load, or if Cloudflare is experiencing issues. You can potentially resolve this by increasing the challenge answer delay (default: 8 seconds). \ For example: cfscrape.create_scraper(delay=15) If increasing the delay does not help, please open a GitHub issue at \ https://github.com/Anorov/cloudflare-scrape/issues\ """ class CloudflareScraper(Session): def __init__(self, *args, **kwargs): self.delay = kwargs.pop("delay", 8) super(CloudflareScraper, self).__init__(*args, **kwargs) if "requests" in self.headers["User-Agent"]: # Set a random User-Agent if no custom User-Agent has been set self.headers["User-Agent"] = DEFAULT_USER_AGENT def is_cloudflare_challenge(self, resp): return ( resp.status_code == 503 and resp.headers.get("Server", "").startswith("cloudflare") and b"jschl_vc" in resp.content and b"jschl_answer" in resp.content ) def request(self, method, url, *args, **kwargs): resp = super(CloudflareScraper, self).request(method, url, *args, **kwargs) # Check if Cloudflare anti-bot is on if self.is_cloudflare_challenge(resp): resp = self.solve_cf_challenge(resp, **kwargs) return resp def solve_cf_challenge(self, resp, **original_kwargs): sleep(self.delay) # Cloudflare requires a delay before solving the challenge body = resp.text parsed_url = urlparse(resp.url) domain = parsed_url.netloc submit_url = "%s://%s/cdn-cgi/l/chk_jschl" % (parsed_url.scheme, domain) cloudflare_kwargs = deepcopy(original_kwargs) params = cloudflare_kwargs.setdefault("params", {}) headers = cloudflare_kwargs.setdefault("headers", {}) headers["Referer"] = resp.url try: params["jschl_vc"] = re.search(r'name="jschl_vc" value="(\w+)"', body).group(1) params["pass"] = re.search(r'name="pass" value="(.+?)"', body).group(1) except Exception as e: # Something is wrong with the page. # This may indicate Cloudflare has changed their anti-bot # technique. If you see this and are running the latest version, # please open a GitHub issue so I can update the code accordingly. raise ValueError("Unable to parse Cloudflare anti-bots page: %s %s" % (e.message, BUG_REPORT)) # Solve the Javascript challenge params["jschl_answer"] = self.solve_challenge(body, domain) # Requests transforms any request into a GET after a redirect, # so the redirect has to be handled manually here to allow for # performing other types of requests even as the first request. method = resp.request.method cloudflare_kwargs["allow_redirects"] = False redirect = self.request(method, submit_url, **cloudflare_kwargs) redirect_location = urlparse(redirect.headers["Location"]) if not redirect_location.netloc: redirect_url = "%s://%s%s" % (parsed_url.scheme, domain, redirect_location.path) return self.request(method, redirect_url, **original_kwargs) return self.request(method, redirect.headers["Location"], **original_kwargs) def solve_challenge(self, body, domain): try: js = re.search(r"setTimeout\(function\(\){\s+(var " "s,t,o,p,b,r,e,a,k,i,n,g,f.+?\r?\n[\s\S]+?a\.value =.+?)\r?\n", body).group(1) except Exception: raise ValueError("Unable to identify Cloudflare IUAM Javascript on website. %s" % BUG_REPORT) js = re.sub(r"a\.value = (.+ \+ t\.length).+", r"\1", js) js = re.sub(r"\s{3,}[a-z](?: = |\.).+", "", js).replace("t.length", str(len(domain))) # Strip characters that could be used to exit the string context # These characters are not currently used in Cloudflare's arithmetic snippet js = re.sub(r"[\n\\']", "", js) if "toFixed" not in js: raise ValueError("Error parsing Cloudflare IUAM Javascript challenge. %s" % BUG_REPORT) # Use vm.runInNewContext to safely evaluate code # The sandboxed code cannot use the Node.js standard library js = "console.log(require('vm').runInNewContext('%s', Object.create(null), {timeout: 5000}));" % js try: result = subprocess.check_output(["node", "-e", js]).strip() except OSError as e: if e.errno == 2: raise EnvironmentError("Missing Node.js runtime. Node is required and must be in the PATH (check with `node -v`). Your Node binary may be called `nodejs` rather than `node`, in which case you may need to run `apt-get install nodejs-legacy` on some Debian-based systems. (Please read the cfscrape" " README's Dependencies section: https://github.com/Anorov/cloudflare-scrape#dependencies.") raise except Exception: logging.error("Error executing Cloudflare IUAM Javascript. %s" % BUG_REPORT) raise try: float(result) except Exception: raise ValueError("Cloudflare IUAM challenge returned unexpected answer. %s" % BUG_REPORT) return result @classmethod def create_scraper(cls, sess=None, **kwargs): """ Convenience function for creating a ready-to-go CloudflareScraper object. """ scraper = cls(**kwargs) if sess: attrs = ["auth", "cert", "cookies", "headers", "hooks", "params", "proxies", "data"] for attr in attrs: val = getattr(sess, attr, None) if val: setattr(scraper, attr, val) return scraper ## Functions for integrating cloudflare-scrape with other applications and scripts @classmethod def get_tokens(cls, url, user_agent=None, **kwargs): scraper = cls.create_scraper() if user_agent: scraper.headers["User-Agent"] = user_agent try: resp = scraper.get(url, **kwargs) resp.raise_for_status() except Exception as e: logging.error("'%s' returned an error. Could not collect tokens." % url) raise domain = urlparse(resp.url).netloc cookie_domain = None for d in scraper.cookies.list_domains(): if d.startswith(".") and d in ("." + domain): cookie_domain = d break else: raise ValueError("Unable to find Cloudflare cookies. Does the site actually have Cloudflare IUAM (\"I'm Under Attack Mode\") enabled?") return ({ "__cfduid": scraper.cookies.get("__cfduid", "", domain=cookie_domain), "cf_clearance": scraper.cookies.get("cf_clearance", "", domain=cookie_domain) }, scraper.headers["User-Agent"] ) @classmethod def get_cookie_string(cls, url, user_agent=None, **kwargs): """ Convenience function for building a Cookie HTTP header value. """ tokens, user_agent = cls.get_tokens(url, user_agent=user_agent, **kwargs) return "; ".join("=".join(pair) for pair in tokens.items()), user_agent create_scraper = CloudflareScraper.create_scraper get_tokens = CloudflareScraper.get_tokens get_cookie_string = CloudflareScraper.get_cookie_string cloudflare-scrape-1.9.5/setup.py000066400000000000000000000013351326127531500166420ustar00rootroot00000000000000import os import re from setuptools import setup base_path = os.path.dirname(__file__) with open(os.path.join(base_path, 'cfscrape', '__init__.py')) as fp: VERSION = re.compile(r'.*__version__ = "(.*?)"', re.S).match(fp.read()).group(1) setup( name = 'cfscrape', packages = ['cfscrape'], version = VERSION, description = 'A simple Python module to bypass Cloudflare\'s anti-bot page. See https://github.com/Anorov/cloudflare-scrape for more information.', author = 'Anorov', author_email = 'anorov.vorona@gmail.com', url = 'https://github.com/Anorov/cloudflare-scrape', keywords = ['cloudflare', 'scraping'], include_package_data = True, install_requires = ['requests >= 2.0.0'] )