././@PaxHeader0000000000000000000000000000003300000000000010211 xustar0027 mtime=1635700103.248204 pywebcopy-7.0.1/0000777000000000000000000000000000000000000010433 5ustar00././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1635575442.0 pywebcopy-7.0.1/LICENSE0000666000000000000000000000104600000000000011441 0ustar00Copyright 2019 Raja Tomar Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.././@PaxHeader0000000000000000000000000000003300000000000010211 xustar0027 mtime=1635700103.248204 pywebcopy-7.0.1/PKG-INFO0000666000000000000000000001504000000000000011530 0ustar00Metadata-Version: 2.1 Name: pywebcopy Version: 7.0.1 Summary: Python library to clone/archive pages or sites from the Internet. Home-page: https://github.com/rajatomar788/pywebcopy/ Author: Raja Tomar Author-email: rajatomar788@gmail.com License: Apache Software License Platform: UNKNOWN Classifier: License :: OSI Approved :: Apache Software License Classifier: Programming Language :: Python Classifier: Programming Language :: Python :: 2.6 Classifier: Programming Language :: Python :: 2.7 Classifier: Programming Language :: Python :: 3.6 Classifier: Programming Language :: Python :: 3.7 Classifier: Programming Language :: Python :: 3.8 Classifier: Programming Language :: Python :: 3.9 Classifier: Programming Language :: Python :: 3.10 Classifier: Programming Language :: Python :: Implementation :: CPython Classifier: Programming Language :: Python :: Implementation :: PyPy Description-Content-Type: text/markdown License-File: LICENSE ``` ____ _ __ __ ______ / __ \__ _| | / /__ / /_ / ____/___ ____ __ __ / /_/ / / / / | /| / / _ \/ __ \/ / / __ \/ __ \/ / / / / ____/ /_/ /| |/ |/ / __/ /_/ / /___/ /_/ / /_/ / /_/ / /_/ \__, / |__/|__/\___/_.___/\____/\____/ .___/\__, / /____/ /_/ /____/ ``` `Created By : Raja Tomar` `License : Apache License 2.0` `Email: rajatomar788@gmail.com` PyWebCopy is a free tool for copying full or partial websites locally onto your hard-disk for offline viewing. PyWebCopy will scan the specified website and download its content onto your hard-disk. Links to resources such as style-sheets, images, and other pages in the website will automatically be remapped to match the local path. Using its extensive configuration you can define which parts of a website will be copied and how. ## What can PyWebCopy do? PyWebCopy will examine the HTML mark-up of a website and attempt to discover all linked resources such as other pages, images, videos, file downloads - anything and everything. It will download all of theses resources, and continue to search for more. In this manner, WebCopy can "crawl" an entire website and download everything it sees in an effort to create a reasonable facsimile of the source website. ## What can PyWebCopy not do? PyWebCopy does not include a virtual DOM or any form of JavaScript parsing. If a website makes heavy use of JavaScript to operate, it is unlikely PyWebCopy will be able to make a true copy if it is unable to discover all of the website due to JavaScript being used to dynamically generate links. PyWebCopy does not download the raw source code of a web site, it can only download what the HTTP server returns. While it will do its best to create an offline copy of a website, advanced data driven websites may not work as expected once they have been copied. ## Installation `pywebcopy` is available on PyPi and is easily installable using `pip` ```shell $ pip install pywebcopy ``` You are ready to go. Read the tutorials below to get started. ## First steps You should always check if the latest pywebcopy is installed successfully. ```pydocstring >>> import pywebcopy >>> pywebcopy.__version___ 7.x.x ``` Your version may be different, now you can continue the tutorial. ## Basic Usages To save any single page, just type in python console ```python from pywebcopy import save_webpage save_webpage( url="https://httpbin.org/", project_folder="E://savedpages//", project_name="my_site", bypass_robots=True, debug=True, open_in_browser=True, delay=None, threaded=False, ) ``` To save full website (This could overload the target server, So, be careful) ```Python from pywebcopy import save_website save_website( url="https://httpbin.org/", project_folder="E://savedpages//", project_name="my_site", bypass_robots=True, debug=True, open_in_browser=True, delay=None, threaded=False, ) ``` ### Running Tests Running tests is simple and doesn't require any external library. Just run this command from root directory of pywebcopy package. ```shell $ python -m pywebcopy --tests ``` ### Command Line Interface `pywebcopy` have a very easy to use command-line interface which can help you do task without having to worrying about the inner long way. - #### Getting list of commands ```shell $ python -m pywebcopy --help ``` - #### Using CLI ``` Usage: pywebcopy [-p|--page|-s|--site|-t|--tests] [--url=URL [,--location=LOCATION [,--name=NAME [,--pop [,--bypass_robots [,--quite [,--delay=DELAY]]]]]]] Python library to clone/archive pages or sites from the Internet. Options: --version show program's version number and exit -h, --help show this help message and exit --url=URL url of the entry point to be retrieved. --location=LOCATION Location where files are to be stored. -n NAME, --name=NAME Project name of this run. -d DELAY, --delay=DELAY Delay between consecutive requests to the server. --bypass_robots Bypass the robots.txt restrictions. --threaded Use threads for faster downloading. -q, --quite Suppress the logging from this library. --pop open the html page in default browser window after finishing the task. CLI Actions List: Primary actions available through cli. -p, --page Quickly saves a single page. -s, --site Saves the complete site. -t, --tests Runs tests for this library. ``` - #### Running tests ```shell $ python -m pywebcopy run_tests ``` ### Authentication and Cookies Most of the time authentication is needed to access a certain page. Its real easy to authenticate with `pywebcopy` because it uses an `requests.Session` object for base http activity which can be accessed through `WebPage.session` attribute. And as you know there are ton of tutorials on setting up authentication with `requests.Session`. Here is an example to fill forms ```python from pywebcopy.configs import get_config config = get_config('http://httpbin.org/') wp = config.create_page() wp.get(config['project_url']) form = wp.get_forms()[0] form.inputs['email'].value = 'bar' # etc form.inputs['password'].value = 'baz' # etc wp.submit_form(form) wp.get_links() ``` You can read more in the github repositories `docs` folder. ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1635575442.0 pywebcopy-7.0.1/README.md0000666000000000000000000001262600000000000011721 0ustar00``` ____ _ __ __ ______ / __ \__ _| | / /__ / /_ / ____/___ ____ __ __ / /_/ / / / / | /| / / _ \/ __ \/ / / __ \/ __ \/ / / / / ____/ /_/ /| |/ |/ / __/ /_/ / /___/ /_/ / /_/ / /_/ / /_/ \__, / |__/|__/\___/_.___/\____/\____/ .___/\__, / /____/ /_/ /____/ ``` `Created By : Raja Tomar` `License : Apache License 2.0` `Email: rajatomar788@gmail.com` PyWebCopy is a free tool for copying full or partial websites locally onto your hard-disk for offline viewing. PyWebCopy will scan the specified website and download its content onto your hard-disk. Links to resources such as style-sheets, images, and other pages in the website will automatically be remapped to match the local path. Using its extensive configuration you can define which parts of a website will be copied and how. ## What can PyWebCopy do? PyWebCopy will examine the HTML mark-up of a website and attempt to discover all linked resources such as other pages, images, videos, file downloads - anything and everything. It will download all of theses resources, and continue to search for more. In this manner, WebCopy can "crawl" an entire website and download everything it sees in an effort to create a reasonable facsimile of the source website. ## What can PyWebCopy not do? PyWebCopy does not include a virtual DOM or any form of JavaScript parsing. If a website makes heavy use of JavaScript to operate, it is unlikely PyWebCopy will be able to make a true copy if it is unable to discover all of the website due to JavaScript being used to dynamically generate links. PyWebCopy does not download the raw source code of a web site, it can only download what the HTTP server returns. While it will do its best to create an offline copy of a website, advanced data driven websites may not work as expected once they have been copied. ## Installation `pywebcopy` is available on PyPi and is easily installable using `pip` ```shell $ pip install pywebcopy ``` You are ready to go. Read the tutorials below to get started. ## First steps You should always check if the latest pywebcopy is installed successfully. ```pydocstring >>> import pywebcopy >>> pywebcopy.__version___ 7.x.x ``` Your version may be different, now you can continue the tutorial. ## Basic Usages To save any single page, just type in python console ```python from pywebcopy import save_webpage save_webpage( url="https://httpbin.org/", project_folder="E://savedpages//", project_name="my_site", bypass_robots=True, debug=True, open_in_browser=True, delay=None, threaded=False, ) ``` To save full website (This could overload the target server, So, be careful) ```Python from pywebcopy import save_website save_website( url="https://httpbin.org/", project_folder="E://savedpages//", project_name="my_site", bypass_robots=True, debug=True, open_in_browser=True, delay=None, threaded=False, ) ``` ### Running Tests Running tests is simple and doesn't require any external library. Just run this command from root directory of pywebcopy package. ```shell $ python -m pywebcopy --tests ``` ### Command Line Interface `pywebcopy` have a very easy to use command-line interface which can help you do task without having to worrying about the inner long way. - #### Getting list of commands ```shell $ python -m pywebcopy --help ``` - #### Using CLI ``` Usage: pywebcopy [-p|--page|-s|--site|-t|--tests] [--url=URL [,--location=LOCATION [,--name=NAME [,--pop [,--bypass_robots [,--quite [,--delay=DELAY]]]]]]] Python library to clone/archive pages or sites from the Internet. Options: --version show program's version number and exit -h, --help show this help message and exit --url=URL url of the entry point to be retrieved. --location=LOCATION Location where files are to be stored. -n NAME, --name=NAME Project name of this run. -d DELAY, --delay=DELAY Delay between consecutive requests to the server. --bypass_robots Bypass the robots.txt restrictions. --threaded Use threads for faster downloading. -q, --quite Suppress the logging from this library. --pop open the html page in default browser window after finishing the task. CLI Actions List: Primary actions available through cli. -p, --page Quickly saves a single page. -s, --site Saves the complete site. -t, --tests Runs tests for this library. ``` - #### Running tests ```shell $ python -m pywebcopy run_tests ``` ### Authentication and Cookies Most of the time authentication is needed to access a certain page. Its real easy to authenticate with `pywebcopy` because it uses an `requests.Session` object for base http activity which can be accessed through `WebPage.session` attribute. And as you know there are ton of tutorials on setting up authentication with `requests.Session`. Here is an example to fill forms ```python from pywebcopy.configs import get_config config = get_config('http://httpbin.org/') wp = config.create_page() wp.get(config['project_url']) form = wp.get_forms()[0] form.inputs['email'].value = 'bar' # etc form.inputs['password'].value = 'baz' # etc wp.submit_form(form) wp.get_links() ``` You can read more in the github repositories `docs` folder.././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1635700103.2013147 pywebcopy-7.0.1/pywebcopy/0000777000000000000000000000000000000000000012454 5ustar00././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1635575442.0 pywebcopy-7.0.1/pywebcopy/__init__.py0000666000000000000000000001464600000000000014600 0ustar00""" ____ _ __ __ ______ / __ \__ _| | / /__ / /_ / ____/___ ____ __ __ / /_/ / / / / | /| / / _ \/ __ \/ / / __ \/ __ \/ / / / / ____/ /_/ /| |/ |/ / __/ /_/ / /___/ /_/ / /_/ / /_/ / /_/ \__, / |__/|__/\___/_.___/\____/\____/ .___/\__, / /____/ /_/ /____/ PyWebCopy is a free tool for copying full or partial websites locally onto your hard-disk for offline viewing. PyWebCopy will scan the specified website and download its content onto your hard-disk. Links to resources such as style-sheets, images, and other pages in the website will automatically be remapped to match the local path. Using its extensive configuration you can define which parts of a website will be copied and how. What can PyWebCopy do? PyWebCopy will examine the HTML mark-up of a website and attempt to discover all linked resources such as other pages, images, videos, file downloads - anything and everything. It will download all of theses resources, and continue to search for more. In this manner, WebCopy can "crawl" an entire website and download everything it sees in an effort to create a reasonable facsimile of the source website. What can PyWebCopy not do? PyWebCopy does not include a virtual DOM or any form of JavaScript parsing. If a website makes heavy use of JavaScript to operate, it is unlikely PyWebCopy will be able to make a true copy if it is unable to discover all of the website due to JavaScript being used to dynamically generate links. PyWebCopy does not download the raw source code of a web site, it can only download what the HTTP server returns. While it will do its best to create an offline copy of a website, advanced data driven websites may not work as expected once they have been copied. # Copyright 2020; Raja Tomar # See license for more details # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # http://www.apache.org/licenses/LICENSE-2.0 # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. ..todo:: *1. Fix parser breaks on empty string. *2. Fix WebPage links and files method emptying the response stream. *3. Fix meta element in the head of the WebPage. *4. Add threading flag in the cmd and api. *5. Fix the MultiParser should be sub class of GenericElement. 6. Fix Asynchronous http requests for the sub elements. 7. Fix infinite nesting of anchor links requests on a single element. *8. Fix concurrent delay b/w requests to be domain specific. """ import logging import warnings logging.getLogger(__name__).addHandler(logging.NullHandler()) __all__ = ['save_website', 'save_webpage'] def save_page(url, project_folder=None, project_name=None, bypass_robots=None, debug=False, open_in_browser=True, delay=None, threaded=None,): """Easiest way to save any single webpage with images, css and js. example:: from pywebcopy import save_webpage save_webpage( url="https://httpbin.org/", project_folder="E://savedpages//", project_name="my_site", bypass_robots=True, debug=True, open_in_browser=True, delay=None, threaded=False, ) :param url: url of the web page to work with :type url: str :param project_folder: folder in which the files will be downloaded :type project_folder: str :param project_name: name of the project to distinguish it :type project_name: str | None :param bypass_robots: whether to follow the robots.txt rules or not :param debug: whether to print deep logs or not. :param open_in_browser: whether or not to open a new tab after saving the webpage. :type open_in_browser: bool :param delay: amount of delay between two concurrent requests to a same server. :param threaded: whether to use threading or not (it can break some site). """ from .configs import get_config config = get_config(url, project_folder, project_name, bypass_robots, debug, delay, threaded) page = config.create_page() page.get(url) if threaded: warnings.warn( "Opening in browser is not supported when threading is enabled!") open_in_browser = False page.save_complete(pop=open_in_browser) save_web_page = save_webpage = save_page def save_website(url, project_folder=None, project_name=None, bypass_robots=None, debug=False, open_in_browser=False, delay=None, threaded=None): """Crawls the entire website for html, images, css and js. example:: from pywebcopy import save_website save_website( url="https://httpbin.org/", project_folder="E://savedpages//", project_name="my_site", bypass_robots=True, debug=False, open_in_browser=True, delay=None, threaded=False, ) :param url: url of the web page to work with :type url: str :param project_folder: folder in which the files will be downloaded :type project_folder: str :param project_name: name of the project to distinguish it :type project_name: str | None :param bypass_robots: whether to follow the robots.txt rules or not :param debug: whether to print deep logs or not. :param open_in_browser: whether or not to open a new tab after saving the webpage. :type open_in_browser: bool :param delay: amount of delay between two concurrent requests to a same server. :param threaded: whether to use threading or not (it can break some site). """ from .configs import get_config config = get_config(url, project_folder, project_name, bypass_robots, debug, delay, threaded) crawler = config.create_crawler() crawler.get(url) if threaded: warnings.warn( "Opening in browser is not supported when threading is enabled!") open_in_browser = False crawler.save_complete(pop=open_in_browser) ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1635575442.0 pywebcopy-7.0.1/pywebcopy/__main__.py0000666000000000000000000000616500000000000014556 0ustar00# Copyright 2020; Raja Tomar # See license for more details import os import sys import optparse import six from pywebcopy.__version__ import __title__ from pywebcopy.__version__ import __version__ from pywebcopy.__version__ import __description__ from pywebcopy import save_webpage from pywebcopy import save_website parser = optparse.OptionParser( usage='%prog [-p|--page|-s|--site|-t|--tests] ' '[--url=URL [,--location=LOCATION [,--name=NAME ' '[,--pop [,--bypass_robots [,--quite [,--delay=DELAY]]]]]]] ', version=__version__, prog=__title__, description=__description__ ) #: Multiple saving modes. options = optparse.OptionGroup(parser, 'CLI Actions List', 'Primary actions available through cli.') options.add_option('-p', '--page', action='store_true', help='Quickly saves a single page.') options.add_option('-s', '--site', action='store_true', help='Saves the complete site.') options.add_option('-t', '--tests', action='store_true', help='Runs tests for this library.') parser.add_option_group(options) #: Required params parser.add_option('--url', type='string', help='url of the entry point to be retrieved.') parser.add_option('--location', type='string', help='Location where files are to be stored.') #: Optional params parser.add_option('-n', '--name', default=None, type='string', help='Project name of this run.') parser.add_option('-d', '--delay', type='float', help="Delay between consecutive requests to the server.") #: Optional flags parser.add_option('--bypass_robots', default=True, action='store_true', help='Bypass the robots.txt restrictions.') parser.add_option('--threaded', default=False, action='store_true', help='Use threads for faster downloading.') parser.add_option('-q', '--quite', default=False, action='store_true', help='Suppress the logging from this library.') parser.add_option('--pop', default=False, action='store_true', help='open the html page in default browser window after finishing the task.') args, remainder = parser.parse_args() # type checks if bool(args.page) or bool(args.site): if not args.url or not isinstance(args.url, six.string_types): parser.error("--url option requires 1 string type argument") if args.location and not isinstance(args.location, six.string_types): parser.error("--location option requires 1 string type argument") if args.name and not isinstance(args.name, six.string_types): parser.error("--name option requires 1 string type argument") if args.page: save_webpage( url=args.url, project_folder=args.location, bypass_robots=args.bypass_robots, open_in_browser=args.pop, debug=not args.quite, delay=args.delay, threaded=args.threaded, ) elif args.site: save_website( url=args.url, project_folder=args.location, bypass_robots=args.bypass_robots, open_in_browser=args.pop, debug=not args.quite, delay=args.delay, threaded=args.threaded, ) elif args.tests: os.system('%s -m unittest discover -s pywebcopy/tests' % sys.executable) else: parser.print_help() sys.exit(1) ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1635700087.0 pywebcopy-7.0.1/pywebcopy/__version__.py0000666000000000000000000000053600000000000015313 0ustar00# Copyright 2020; Raja Tomar # See license for more details __title__ = 'pywebcopy' __url__ = 'https://github.com/rajatomar788/pywebcopy/' __author__ = 'Raja Tomar' __email__ = 'rajatomar788@gmail.com' __license__ = 'Apache Software License' __version__ = '7.0.1' __description__ = 'Python library to clone/archive pages or sites from the Internet.' ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1635575442.0 pywebcopy-7.0.1/pywebcopy/configs.py0000666000000000000000000002324500000000000014464 0ustar00# Copyright 2020; Raja Tomar # See license for more details import logging import os import sys import tempfile from functools import partial from requests.structures import CaseInsensitiveDict from six import text_type from six import string_types from .__version__ import __title__ from .__version__ import __version__ from .urls import HIERARCHY from .urls import get_host from .urls import secure_filename from .session import default_headers __all__ = [ 'ConfigHandler', 'get_config', 'default_config', 'safe_file_types', 'safe_http_headers' ] logger = logging.getLogger(__name__) def add_stderr_logger(name=__title__, level=logging.DEBUG): """ Helper for quickly adding a StreamHandler to the logger. Useful for debugging. Returns the handler after adding it. """ # This method needs to be in this __init__.py to get the __name__ correct # even if this library is wrapped within another package. root = logging.getLogger(name) #: If there is already a stderr logger then we don't need to bother. for h in root.handlers: if isinstance(h, logging.StreamHandler): if h.stream == sys.stderr: h.disabled = False h.setLevel(level) return h handler = logging.StreamHandler(sys.stderr) handler.setFormatter( logging.Formatter( '%(levelname)-8s - %(name)s:%(lineno)d - %(message)s' ) ) root.addHandler(handler) root.setLevel(level) root.debug('Added a stderr logging handler to logger: %s', name) return handler # FIXME: Do something about these. safe_file_types = [ 'text/*', 'image/*', 'font/*', 'application/pdf', 'application/json', ] safe_http_headers = { "Accept-Language": "en-US,en;q=0.9", 'User-Agent': "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:70.0) " "Gecko/20100101 Firefox/70.0 PyWebCopyBot/%s" % __version__ } #: Base configuration with preconfigured values. default_config = { 'debug': False, 'project_url': None, 'project_name': None, 'project_folder': None, 'threaded': None, 'thread_join_timeout': None, 'tree_type': HIERARCHY, # TODO: Allow a `last-modified-time` overwrite mode 'overwrite': False, 'bypass_robots': False, 'http_cache': False, 'http_headers': default_headers(**safe_http_headers), 'delay': None, # TODO: Disabled for now until I figure it out. # 'allowed_file_types': safe_file_types, # TODO: domain blocking and whitelisting } class ConfigError(AttributeError, TypeError): """Bad config value or operation.""" class ConfigHandler(CaseInsensitiveDict): """Provides functionality to the config instance which stores and provides configuration values in every module. """ def __repr__(self): # pragma: no cover return '' % self.get('project_name', 'Not Set') def __getattribute__(self, item): """Dynamic method of name `get_(key)` and `set_(key)` generation for all of the keys available. for example to change the `project_url` key instead of using dictionary like operation you would do `.get_project_url()` instead of `['project_url']`. `.set_project_url(new)` instead of `['project_url'] = new`. """ if isinstance(item, string_types) and item.startswith('set_'): if item[4:] in self: return partial(self.__setitem__, item[4:]) elif isinstance(item, string_types) and item.startswith('get_'): if item[4:] in self: return partial(self.__getitem__, item[4:]) return super(ConfigHandler, self).__getattribute__(item) def resolve_url(self): """Resolves any redirects in the url and sets the final url as base url.""" raise NotImplementedError() def reset_config(self): """Resets all to configuration to default state.""" self.update(default_config) def reset_key(self, key): self.update({key: default_config.get(key)}) def is_set(self): """Checks whether the configuration has required attributes or not.""" try: assert self.get('project_folder') is not None assert self.get('project_url') is not None assert self.get('project_name') is not None except AssertionError: return False else: return True def setup_paths(self, project_folder, project_name): """Fills the project_name, project_name and its dependent keys after evaluation. .. version changed :: 6.0.0 Added string type checks and os based path normalising. .. version changed :: 6.1.0 FIX: fixed path issue when using relative path for project_folder .. version changed :: 6.3.0 FIX: Removed file based logging. FIX: Disabled dir change on setup :param project_name: new name of the project :param project_folder: folder where to store all the downloaded files """ if not isinstance(project_name, string_types): raise ConfigError("project_name value must be a string") if not isinstance(project_folder, string_types): raise ConfigError("project_folder value must be a string!") if os.altsep: project_folder = project_folder.replace(os.altsep, os.sep) if project_folder.find(os.sep) < 0: # pragma: no cover raise ConfigError("Project_folder path doesn't seem to be a valid path.") project_folder = os.path.abspath(project_folder) norm_p = os.path.join( os.path.normpath(project_folder), os.path.normpath(project_name) ) self.set_project_name(project_name) self.set_project_folder(norm_p) if not os.path.exists(norm_p): os.makedirs(norm_p) def setup_config(self, project_url=None, project_folder=None, project_name=None, overwrite=False, bypass_robots=False, debug=False, delay=None, threaded=None): """Sets up the complete config parts which requires a project_url to be present. Complete configuration is done here and subject to change according to application structure You are advised to use only the .setup_path() method if you get any unusual behaviour """ self.set_overwrite(overwrite) self.set_bypass_robots(bypass_robots) self.set_debug(debug) self.set_delay(delay) self.set_threaded(threaded) self.set_project_url(project_url) self.setup_paths(project_folder, project_name) #: Add a stderr logger to this library. if debug: add_stderr_logger(level=logging.DEBUG) #: Log this new configuration to the log file for debug purposes logger.debug(str(dict(self))) def create_context(self): if not self.is_set(): raise ConfigError("Config is missing required attributes!") from .urls import Context return Context.from_config(self) def create_session(self): if not self.is_set(): raise ConfigError("Config is missing required attributes!") from .session import Session return Session.from_config(self) def create_crawler(self): if not self.is_set(): raise ConfigError("Config is missing required attributes!") from .core import Crawler return Crawler.from_config(self) def create_page(self): if not self.is_set(): raise ConfigError("Config is missing required attributes!") from .core import WebPage return WebPage.from_config(self) def get_config(project_url, project_folder=None, project_name=None, bypass_robots=False, debug=False, delay=None, threaded=None): """Create a ConfigHandler instance and return it. If the project_folder is not supplied it will use the users Tempdir. :param project_url: project_url of the web page to work with :type project_url: str :param project_folder: folder in which the files will be downloaded :type project_folder: str :param project_name: name of the project to distinguish it :type project_name: str | None :param bypass_robots: whether to follow the robots.txt rules or not :param debug: whether to print deep logs or not. :param delay: amount of delay between two concurrent requests to a same server. :param threaded: whether to use threading or not (it can break some site). """ if not isinstance(project_url, string_types): raise ConfigError("Expected string type, got %r" % project_url) if project_folder and not isinstance(project_folder, string_types): raise ConfigError("Expected string type, got %r" % project_folder) if not project_folder: logger.debug('No project folder provided, %temp% dir will be used instead.') project_folder = tempfile.gettempdir() if not project_name: project_name = '_'.join( map(secure_filename, map(lambda x: text_type(x), filter(None, get_host(project_url))))) logger.debug('No project name provided, generated from url: %s' % project_name) ans = ConfigHandler(default_config) ans.setup_config( project_url=project_url, project_folder=project_folder, project_name=project_name, bypass_robots=bypass_robots, debug=debug, delay=delay, threaded=threaded, ) return ans ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1635575442.0 pywebcopy-7.0.1/pywebcopy/core.py0000666000000000000000000002236100000000000013762 0ustar00# Copyright 2020; Raja Tomar # See license for more details import logging import operator import os from lxml.html import HTMLParser from lxml.html import XHTML_NAMESPACE from lxml.html import parse from requests.models import Response from .elements import HTMLResource from .helpers import RewindableResponse from .schedulers import crawler_scheduler from .schedulers import default_scheduler from .schedulers import threading_crawler_scheduler from .schedulers import threading_default_scheduler __all__ = ['WebPage', 'Crawler'] logger = logging.getLogger(__name__) class WebPage(HTMLResource): """ WebPage built upon HTMLResource element. It provides various utilities like form-filling, external response processing, getting list of links, dumping html and opening the html in the browser. """ @classmethod def from_config(cls, config): """It creates a `WebPage` object from a set config object. Under the hood it checks whether the config is set or not, then it creates a `session` using the `config.create_session()` method. It then creates a `scheduler` based on whether the threading is enabled or not. It also defines a `context` object which stores the path metadata for this structure. """ if config and not config.is_set(): raise AttributeError("Configuration is not setup.") session = config.create_session() if config.get('threaded'): scheduler = threading_default_scheduler( timeout=config.get_thread_join_timeout()) else: scheduler = default_scheduler() context = config.create_context() ans = cls(session, config, scheduler, context) # XXX: Check connection to the url here? return ans def __repr__(self): """Short representation of this instance.""" return '<{}: {}>'.format(self.__class__.__name__, self.url) element_map = property( operator.attrgetter('scheduler.data'), doc="Registry of different handler for different tags." ) def set_response(self, response): """ Set an explicit `requests.Response` object directly. It accepts a `requests.Response` object and wraps it in a `RewindableResponse` object. It also enables decoding in the original `urllib3` response object. You can use it like this import requests resp = requests.get('https://www.httpbin.org/') wp = WebPage() wp.set_response(resp) wp.get_forms() """ if not isinstance(response, Response): raise ValueError("Expected %r, got %r" % (Response, response)) response.raw.decode_content = True response.raw = RewindableResponse(response.raw) return super(WebPage, self).set_response(response) def get_source(self, buffered=False): """Returns the `requests.Response` object in a rewindable io wrapper. Contents can be consumed then the `.rewind()` method should be called to restore the contents of the original response. wp = WebPage() wp.get('http://httpbin.org/forms/') src = WebPage.get_source(buffered=True) contents = src.read() src.rewind() :param buffered: whether to return an Readable file-like object or just a plain string. :rtype: RewindableResponse """ raw = getattr(self.response, 'raw', None) if raw is None: raise ValueError( "HTTP Response is not set at the `.response` attribute!" "Use the `.get()` method or `.set_response()` methods to set it.") # if raw.closed: # raise ValueError( # "I/O operations are closed for the raw source.") # Return the raw object which will decode the # buffer while reading otherwise errors will follow raw.decode_content = True # fp = getattr(raw, '_fp', None) # assert fp is not None, "Raw source wrapper is missing!" # assert isinstance(fp, CallbackFileWrapper), \ # "Raw source wrapper is missing!" raw.rewind() if buffered: return raw, self.encoding return raw.read(), self.encoding def refresh(self): """Re-fetches the resource from the internet using the session.""" self.set_response(self.session.get(self.url, stream=True)) def get_forms(self): """Returns a list of form elements available on the page.""" source, encoding = self.get_source(buffered=True) return parse( source, parser=HTMLParser(encoding=encoding, collect_ids=False) ).xpath( "descendant-or-self::form|descendant-or-self::x:form", namespaces={'x': XHTML_NAMESPACE} ) def submit_form(self, form, **extra_values): """ Helper function to submit a `lxml` form. Example: wp = WebPage() wp.get('http://httpbin.org/forms/') form = wp.get_forms()[0] form.inputs['email'].value = 'bar' # etc form.inputs['password'].value = 'baz' # etc wp.submit_form(form) wp.get_links() The action is one of 'GET' or 'POST', the URL is the target URL as a string, and the values are a sequence of ``(name, value)`` tuples with the form data. """ values = form.form_values() if extra_values: if hasattr(extra_values, 'items'): extra_values = extra_values.items() values.extend(extra_values) if form.action: url = form.action elif form.base_url: url = form.base_url else: url = self.url return self.request(form.method, url, data=values) def get_files(self): """ Returns a list of urls, css, js, images etc. """ return (e[2] for e in self.parse()) def get_links(self): """ Returns a list of urls in the anchor tags only. """ return (e[2] for e in self.parse() if e[0].tag == 'a') def scrap_html(self, url): """Returns the html of the given url. :param url: address of the target page. """ response = self.session.get(url) response.raise_for_status() return response.content def scrap_links(self, url): """Returns all the links from a given url. :param url: address of the target page. """ response = self.session.get(url) response.raise_for_status() return response.links() def dump_html(self, filename=None): """Saves the html of the page to a default or specified file. :param filename: path of the file to write the contents to """ filename = filename or self.filepath with open(filename, 'w+b') as fh: source, enc = self.get_source() fh.write(source) return filename def save_complete(self, pop=False): """Saves the complete html+assets on page to a file and also writes its linked files to the disk. Implements the combined logic of save_assets and save_html in compact form with checks and validation. """ if not self.viewing_html(): raise TypeError( "Not viewing a html page. Please check the link!") self.scheduler.handle_resource(self) if pop: self.open_in_browser() return self.filepath def open_in_browser(self): """Open the page in the default browser if it has been saved. You need to use the :meth:`~WebPage.save_complete` to make it work. """ if not os.path.exists(self.filepath): self.logger.info( "Can't find the file to open in browser: %s" % self.filepath) return False self.logger.info( "Opening default browser with file: %s" % self.filepath) import webbrowser return webbrowser.open('file:///' + self.filepath) # handy shortcuts run = crawl = save_assets = save_complete class Crawler(WebPage): @classmethod def from_config(cls, config): """ It creates a `Crawler` object from a set config object. Under the hood it checks whether the config is set or not, then it creates a `session` using the `config.create_session()` method. It then creates a `scheduler` based on whether the threading is enabled or not. The scheduler is different from the `WebPage` objects scheduler due to its ability to process the anchor tags links to different pages. It also defines a `context` object which stores the path metadata for this structure. """ if config and not config.is_set(): raise AttributeError("Configuration is not setup.") session = config.create_session() if config.get('threaded'): scheduler = threading_crawler_scheduler( timeout=config.get_thread_join_timeout()) else: scheduler = crawler_scheduler() context = config.create_context() ans = cls(session, config, scheduler, context) # XXX: Check connection to the url here? return ans ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1635699982.0 pywebcopy-7.0.1/pywebcopy/elements.py0000666000000000000000000005762300000000000014657 0ustar00# Copyright 2020; Raja Tomar # See license for more details import errno import logging import os import re import sys import warnings from base64 import b64encode from contextlib import closing from datetime import datetime from functools import partial from io import BytesIO from shutil import copyfileobj from textwrap import dedent from lxml.html import HtmlComment from lxml.html import tostring from six import binary_type from six import string_types from six.moves.urllib.request import pathname2url from .__version__ import __version__ from .helpers import cached_property from .parsers import iterparse from .parsers import unquote_match from .urls import get_content_type_from_headers from .urls import relate logger = logging.getLogger(__name__) #: Binary file permission mode fd_mode = 0o777 #: Binary file flags for kernel based io fd_flags = os.O_CREAT | os.O_WRONLY if hasattr(os, 'O_BINARY'): fd_flags |= os.O_BINARY if hasattr(os, 'O_NOFOLLOW'): fd_flags |= os.O_NOFOLLOW def make_fd(location, url=None, overwrite=False): """Creates a kernel based file descriptor which should be used to write binary data onto the files. :rtype: int """ location = os.path.normpath(location) # Sub-directories creation which suppresses exceptions base_dir = os.path.dirname(location) try: os.makedirs(base_dir) except (OSError, IOError) as e: if e.errno == errno.EEXIST or ((os.name == 'nt' and os.path.isdir( base_dir) and os.access(base_dir, os.W_OK))): logger.debug( "[FILE] Sub-directories exists for: <%r>" % location) # dead on arrival else: logger.error( "[File] Failed to create target location <%r> " "for the file <%r> on the disk." % (location, url)) return -1 else: logger.debug( "[File] Sub-directories created for: <%r>" % location) try: # sys.audit("%s.resource" % __title__, location) sys.audit("os.open", location) if overwrite: fd = os.open(location, fd_flags | os.O_TRUNC, fd_mode) else: # raises FileExistsError if file exists fd = os.open(location, fd_flags | os.O_EXCL, fd_mode) except (OSError, PermissionError) as e: if e.errno == errno.EEXIST: logger.debug( "[FILE] <%s> already exists at: <%s>" % (url, location)) elif e.errno == errno.ENAMETOOLONG: logger.debug( "[FILE] Path too long for <%s> at: <%s>" % (url, location)) else: logger.error( "[File] Cannot write <%s> to <%s>! %r" % (url, location, e)) return -1 else: return fd def retrieve_resource(content, location, url=None, overwrite=False): """Retrieves the readable resource to a local file. ..todo:: Add overwrite modes: Overwrite or True, Update, Ignore or False :param BytesIO content: file like object with read method :param location: file name where this content has to be saved. :param url: (optional) url of the resource used for logging purposes. :param overwrite: (optional) whether to overwrite an existing file. :return: rendered location or False if failed. :rtype: string_types """ if content is None: raise ValueError("Content can't be of NoneType.") if location is None: raise ValueError("Location can't be of NoneType or empty str.") if url is None: raise ValueError("Url can't be of NoneType.") logger.debug( "[File] Preparing to write file from <%r> to the disk at <%r>." % (url, location)) fd = make_fd(location, url, overwrite) if fd == -1: return location with closing(os.fdopen(fd, 'w+b')) as dst: copyfileobj(content, dst) logger.info( "[File] Written the file from <%s> to <%s>" % (url, location)) return location def urlretrieve(url, location, **params): """ A extra rewrite of a basic `urllib` function using the tweaks and perks of this library. :param url: url of the resource to be retrieved. :param location: destination for the resource. :param params: parameters for the :func:`requests.get`. :return: location of the file retrieved. :rtype: string_types """ if not isinstance(url, string_types): raise TypeError("Expected string type, got %r" % url) if not isinstance(location, string_types): raise TypeError("Expected string type, got %r" % location) import requests with closing(requests.get(url, **params)) as src: return retrieve_resource( src.raw, location, url, overwrite=True) class GenericResource(object): def __init__(self, session, config, scheduler, context, response=None): """ Generic internet resource which processes a server response based on responses content-type. Downloadable file if allowed in config would be downloaded. Css file would be parsed using suitable parser. Html will also be parsed using suitable html parser. :param session: http client used for networking. :param config: project configuration handler. :param response: http response from the server. :param scheduler: response processor scheduler. :param context: context of this response; should contain base-location, base-url etc. """ self.session = session self.config = config self.scheduler = scheduler self.context = context self.response = None if response: self.set_response(response) self.logger = logger.getChild(self.__class__.__name__) def __repr__(self): return '<%s(url=%s)>' % (self.__class__.__name__, self.context.url) def __del__(self): self.close() def close(self): """Releases the underlying urllib connection and then deletes the response""" if self.response is not None: if hasattr(self.response, 'raw'): if hasattr(self.response.raw, 'release_conn'): getattr(self.response, 'raw').release_conn() del self.response @cached_property def filepath(self): """Returns if available a valid filepath where this file should be written.""" if self.context is None: raise AttributeError("Context attribute is not set.") if self.response is not None: ctypes = get_content_type_from_headers(self.response.headers) self.context = self.context.with_values(content_type=ctypes) return self.context.resolve() @cached_property def filename(self): """Returns a valid filename of this resource if available.""" return os.path.basename(self.filepath or '') @cached_property def content_type(self): """Returns a mimetype descriptor of this resource if available.""" if self.response is not None and 'Content-Type' in self.response.headers: return get_content_type_from_headers(self.response.headers) return '' @cached_property def url(self): """Returns the actual url of this resource which is resolved if there were any redirects.""" if self.response is not None: self.context = self.context.with_values(url=self.response.url) return self.context.url @cached_property def encoding(self): """Returns an explicit encoding if defined in the config else the encoding reported by the server.""" if self.response is not None: #: Explicit encoding takes precedence return self.config.get( 'encoding', self.response.encoding or 'ascii') return self.config.get('encoding', 'ascii') html_content_types = tuple([ 'text/htm', 'text/html', 'text/xhtml' ]) html_content_types.__doc__ = "Set of valid html mimetypes." def viewing_html(self): """Checks whether the current resource is a html type or not.""" return self.content_type in self.html_content_types css_content_types = tuple([ 'text/css', ]) css_content_types.__doc__ = "Set of valid css mimetypes." def viewing_css(self): """Checks whether the current resource is a css type or not.""" return self.content_type in self.css_content_types js_content_types = tuple([ 'text/javascript', 'application/javascript' ]) js_content_types.__doc__ = "Set of valid javascript mimetypes." def viewing_js(self): """Checks whether the current resource is a javascript type or not.""" return self.content_type in self.js_content_types def set_response(self, response): """Update the response attribute of this object. It also updates the content_type and encoding as reported by the server implicitly for better detection of contents.""" self.response = response #: Clear the cached properties self.__dict__.pop('url', None) self.__dict__.pop('filepath', None) self.__dict__.pop('filename', None) if hasattr(response, 'ok') and response.ok: self.__dict__.pop('content_type', None) self.__dict__.pop('encoding', None) self.context = self.context.with_values( url=response.url, content_type=self.content_type) def request(self, method, url, **params): """Fetches the Html content from Internet using the requests. You can any requests params which will be passed to the library itself. The requests arguments you supply will also be applied to the global session meaning all the files will be downloaded using these settings. :param method: http verb for transport. :param url: url of the page to fetch :param params: keyword arguments which `requests` module may accept. """ if params.pop('stream', None): warnings.warn(UserWarning( "Stream attribute is True by default for reasons." )) self.set_response( self.session.request(method, url, stream=True, **params)) def get(self, url, **params): """Initiates an `get` request for the given url. It uses the `.set_response()` method underneath to process the returned response. It is used to manually fetch the starting web-page. Example: wp = WebPage() wp.get('http://www.example.com/') wp.retrieve() """ return self.request('GET', url, **params) def post(self, url, **params): """Initiates an `post` request for the given url. It uses the `.set_response()` method underneath to process the returned response. It is required to submit forms. Example: wp = WebPage() wp.post('http://www.example.com/', data={'key': 'value'}) wp.retrieve() """ return self.request('POST', url, **params) def get_source(self, buffered=False): """ Returns a tuple with the response contents in either file-like object i.e. `RewindableResponse` if `buffered=True` or string format if ` buffered=False` and the encoding from the `.encoding` attribute. Example: wp.get(url=...) wp.get_source(buffered=False) "..." wp.get_source(buffered=True) "" Note: An Error would be raised if the `.context` attribute is not set. or the `.response` attribute is not set. :rtype: string_types | RewindableResponse """ if self.context is None: raise ValueError("Context not set.") if self.context.base_path is None: raise ValueError("Context Base Path is not set!") if self.context.base_url is None: raise ValueError("Context Base url is not Set!") if self.response is None: raise ValueError("Response attribute is not Set!") if getattr(self.response.raw, 'closed', True): raise ValueError( "I/O operations closed on the response object.") if not hasattr(self.response.raw, 'read'): raise ValueError( "Response must have a raw file like object!") self.response.raw.decode_content = True if buffered: return self.response.raw, self.encoding return self.response.content, self.encoding def retrieve(self): """Retrieves the readable resource to the local disk.""" if self.response is None: raise AttributeError( "Response attribute is not set!" "You need to fetch the resource using get method!" ) # XXX: Validate resource here? return self._retrieve() def _retrieve(self): #: Not ok response received from the server if not 100 <= self.response.status_code <= 400: self.logger.error( 'Status Code [<%d>] received from the server [%s]' % (self.response.status_code, self.response.url) ) if isinstance(self.response.reason, binary_type): content = BytesIO(self.response.reason) else: content = BytesIO(self.response.reason.encode(self.encoding)) else: if not hasattr(self.response, 'raw'): self.logger.error( "Response object for url <%s> has no attribute 'raw'!" % self.url) content = BytesIO(self.response.content) else: content = self.response.raw retrieve_resource( content, self.filepath, self.context.url, self.config.get('overwrite')) del content return self.filepath def resolve(self, parent_path=None): """Returns a relative url at which this resource should be accessed by the parent file. """ filepath = self.filepath if not isinstance(filepath, string_types): raise ValueError("Invalid filepath [%r]" % filepath) if parent_path and isinstance(parent_path, string_types): return pathname2url(relate(filepath, parent_path)) return pathname2url(filepath) class HTMLResource(GenericResource): """Interpreter for resource written in or reported as html.""" def parse(self, **kwargs): """Returns an `pywebcopy.parsers.iterparse` instance with the file-object returned from the `.get_source(buffered=True)`. :params kwargs: options to be passed to the `iterparse`. """ source, encoding = self.get_source(buffered=True) return iterparse( source, encoding, include_meta_charset_tag=True, **kwargs) def extract_children(self, parsing_buffer): """ Iterates over the `pywebcopy.parsers.iterparse` object and extract the elements which are handed over to the `scheduler` for processing. Then the final path of the element is updated in the `pywebcopy.parsers.iterparse` object. :param parsing_buffer: `iterparse` object. """ location = self.filepath for elem, attr, url, pos in parsing_buffer: if not self.scheduler.validate_url(url): continue sub_context = self.context.create_new_from_url(url) ans = self.scheduler.get_handler( elem.tag, self.session, self.config, self.scheduler, sub_context) self.scheduler.handle_resource(ans) resolved = ans.resolve(location) elem.replace_url(url, resolved, attr, pos) return parsing_buffer def _retrieve(self): if not self.viewing_html(): self.logger.info( "Resource of type [%s] is not HTML." % self.content_type) return super(HTMLResource, self)._retrieve() if not self.response.ok: self.logger.debug( "Resource at [%s] is NOT ok and will be NOT processed." % self.url) return super(HTMLResource, self)._retrieve() context = self.extract_children(self.parse()) # WaterMarking :) context.root.insert(0, HtmlComment(self._get_watermark())) retrieve_resource( BytesIO(tostring(context.root, include_meta_content_type=True)), self.filepath, self.context.url, overwrite=True) self.logger.debug('Retrieved content from the url: [%s]' % self.url) del context return self.filepath def _get_watermark(self): # comment text should be in unicode return dedent(""" * PyWebCopy Engine [version %s] * Copyright 2020; Raja Tomar * File mirrored from [%s] * At UTC datetime: [%s] """) % (__version__, self.response.url, datetime.utcnow()) class CSSResource(GenericResource): def parse(self): """Returns the `.get_source(buffered=False)`.""" return self.get_source(buffered=False) def repl(self, match, encoding=None, fmt=None): """ Schedules the linked files for downloading then resolves their references. """ fmt = fmt or '%s' url, _ = unquote_match(match.group(1).decode(encoding), match.start(1)) self.logger.debug("Sub-Css resource found: [%s]" % url) if not self.scheduler.validate_url(url): return url.encode(encoding) sub_context = self.context.create_new_from_url(url) self.logger.debug('Creating context for url: %s as %s' % (url, sub_context)) ans = self.__class__( self.session, self.config, self.scheduler, sub_context ) # self.children.add(ans) self.logger.debug("Submitting resource: [%s] to the scheduler." % url) self.scheduler.handle_resource(ans) re_enc = (fmt % ans.resolve(self.filepath)).encode(encoding) self.logger.debug("Re-encoded the resource: [%s] as [%r]" % (url, re_enc)) return re_enc # noinspection PyTypeChecker def extract_children(self, parsing_buffer): """ Runs the regex over the source to find the urls that are linked within the css file or style tag using the `url()` construct. """ source, encoding = parsing_buffer source = re.sub( (r'url\((' + '["][^"]*["]|' + "['][^']*[']|" + r'[^)]*)\)').encode(encoding), partial(self.repl, encoding=encoding, fmt="url('%s')"), source, flags=re.IGNORECASE ) source = re.sub( r'@import "(.*?)"'.encode(encoding), partial(self.repl, encoding=encoding, fmt='"%s"'), source, flags=re.IGNORECASE ) return BytesIO(source) def _retrieve(self): """Writes the modified buffer to the disk.""" if not self.viewing_css(): self.logger.info( "Resource of type [%s] is not CSS." % self.content_type) return super(CSSResource, self)._retrieve() if not self.response.ok: self.logger.debug( "Resource at [%s] is NOT ok and will be NOT processed." % self.url) return super(CSSResource, self)._retrieve() self.logger.debug( "Resource at [%s] is ok and will be processed." % self.url) retrieve_resource( self.extract_children(self.parse()), self.filepath, self.url, self.config.get('overwrite') ) self.logger.debug("Finished processing resource [%s]" % self.url) return self.filepath class JSResource(GenericResource): def parse(self): """Returns the `.get_source(buffered=False)`.""" return self.get_source(buffered=False) def repl(self, match, encoding=None, fmt=None): """ Schedules the linked files for downloading then resolves their references. """ fmt = fmt or '%s' url, _ = unquote_match(match.group(1).decode(encoding), match.start(1)) self.logger.debug("Sub-JS resource found: [%s]" % url) if not self.scheduler.validate_url(url): return url.encode(encoding) sub_context = self.context.create_new_from_url(url) self.logger.debug('Creating context for url: %s as %s' % (url, sub_context)) ans = self.__class__( self.session, self.config, self.scheduler, sub_context ) # self.children.add(ans) self.logger.debug("Submitting resource: [%s] to the scheduler." % url) self.scheduler.handle_resource(ans) re_enc = (fmt % ans.resolve(self.filepath)).encode(encoding) self.logger.debug("Re-encoded the resource: [%s] as [%r]" % (url, re_enc)) return re_enc # noinspection PyTypeChecker def extract_children(self, parsing_buffer): """ Runs the regex over the source to find the urls that are linked within the js file or script tag using the `url()` construct. ..todo:: It only recognises one type of url inside of the js. i.e. `url('example.com')`. Make it universal. """ source, encoding = parsing_buffer # P.S. There is one interesting Regex on this github repo under MIT license # https://github.com/GerbenJavado/LinkFinder/ source = re.sub( (r'url\((' + '["][^"]*["]|' + "['][^']*[']|" + r'[^)]*)\)' ).encode(encoding), partial( self.repl, encoding=encoding, fmt='url("%s")' ), source, flags=re.IGNORECASE ) return BytesIO(source) def _retrieve(self): """Writes the modified buffer to the disk.""" if not self.viewing_js(): self.logger.info("Resource of type [%s] is not JS." % self.content_type) return super(JSResource, self)._retrieve() if not self.response.ok: self.logger.debug("Resource at [%s] is NOT ok and will be NOT processed." % self.url) return super(JSResource, self)._retrieve() self.logger.debug("Resource at [%s] is ok and will be processed." % self.url) retrieve_resource( self.extract_children(self.parse()), self.filepath, self.url, self.config.get('overwrite') ) self.logger.debug("Finished processing resource [%s]" % self.url) return self.filepath class GenericOnlyResource(GenericResource): """Only retrieves a resource if it is NOT HTML.""" def _retrieve(self): if self.viewing_html(): self.logger.debug("Resource [%s] is of HTML type and must not be processed!" % self.url) return False return super(GenericOnlyResource, self)._retrieve() def resolve(self, parent_path=None): if self.viewing_html(): return self.context.url return super(GenericOnlyResource, self).resolve(parent_path=parent_path) class VoidResource(GenericResource): def get(self, url, **params): return None def get_source(self, buffered=False): return None def retrieve(self): return None # :) NullResource = VoidResource class UrlRemover(VoidResource): def resolve(self, parent_path=None): return '#' class AbsoluteUrlResource(VoidResource): def resolve(self, parent_path=None): return self.context.url class Base64Resource(GenericResource): def resolve(self, parent_path=None): source, encoding = self.get_source() import sys if sys.version > '3': if type(source) is bytes: return 'data:%s;base64,%s' % (self.content_type, bytes.decode(b64encode(source))) else: return 'data:%s;base64,%s' % (self.content_type, bytes.decode(b64encode(str.encode(source, encoding)))) else: return 'data:%s;base64,%s' % (self.content_type, b64encode(source)) def retrieve(self): #: There are no sub-files to be fetched. return None ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1635575442.0 pywebcopy-7.0.1/pywebcopy/helpers.py0000666000000000000000000003167700000000000014506 0ustar00# Copyright 2020; Raja Tomar # See license for more details import time import functools import threading from requests.compat import OrderedDict from six import BytesIO from six.moves.collections_abc import MutableMapping class RecentOrderedDict(MutableMapping): """ A custom variant of the OrderedDict that ensures that the object most recently inserted or retrieved from the dictionary is at the top of the dictionary enumeration. """ def __init__(self, *args, **kwargs): self._data = OrderedDict(*args, **kwargs) def __setitem__(self, key, value): if key in self._data: del self._data[key] self._data[key] = value def __getitem__(self, key): value = self._data[key] del self._data[key] self._data[key] = value return value def __delitem__(self, key): del self._data[key] def __iter__(self): return iter(self._data) def __len__(self): return len(self._data) def __contains__(self, value): return self._data.__contains__(value) def items(self): return self._data.items() def keys(self): return self._data.keys() class ConcurrentDelay(object): """ Blocking waiter which calculates the delay irrespective of the waiting call rather than just waiting on the specified time. for example if time between two calls for delay are 10 seconds apart but the timeout is set for 1 second then the second call will not block but a successive third call will be blocked for 1 second before completing. """ def __init__(self, timeout=1 / 10): self.timeout = None self.set_timeout(timeout) self.start_time = time.time() self._cond = threading.Condition() # self.logger = logger.getChild(self.__class__.__name__) def set_timeout(self, t): if not isinstance(t, (int, float)): raise ValueError("Expected int or float, got %r" % t) if t <= -1: raise ValueError("Delay cannot be smaller than 0") self.timeout = t def delay(self): """Delays until the timeout is completed""" with self._cond: current_time = time.time() diff_time = current_time - self.start_time # prevent none type values from waiting infinitely if isinstance(self.timeout, (int, float)) \ and not diff_time >= self.timeout: # self.logger.debug( # "Waiting on request for [%r] seconds!" % self.timeout) self._cond.wait(self.timeout - diff_time) self.start_time = current_time def lru_cache(maxsize=255, timeout=None): """lru_cache(maxsize = 255, timeout = None) --> returns a decorator which returns an instance (a descriptor). Purpose - This decorator factory will wrap a function / instance method and will supply a caching mechanism to the function. For every given input params it will store the result in a queue of maxsize size, and will return a cached ret_val if the same parameters are passed. Params - maxsize - int, the cache size limit, anything added above that will delete the first values entered (FIFO). This size is per instance, thus 1000 instances with maxsize of 255, will contain at max 255K elements. - timeout - int / float / None, every n seconds the cache is deleted, regardless of usage. If None - cache will never be refreshed. Notes - If an instance method is wrapped, each instance will have it's own cache and it's own timeout. - The wrapped function will have a cache_clear variable inserted into it and may be called to clear it's specific cache. - The wrapped function will maintain the original function's docstring and name (wraps) - The type of the wrapped function will no longer be that of a function but either an instance of _LRU_Cache_class or a functools.partial type. On Error - No error handling is done, in case an exception is raised - it will permeate up. """ class _LRU_Cache_class(object): # noinspection PyShadowingNames def __init__(self, input_func, max_size, timeout): self._input_func = input_func self._max_size = max_size self._timeout = timeout # This will store the cache for this function, format - # {caller1 : [OrderedDict1, last_refresh_time1], caller2 : [OrderedDict2, last_refresh_time2]}. # In case of an instance method - the caller is the instance, in case called # from a regular function - the caller is None. self._caches_dict = {} def cache_clear(self, caller=None): # Remove the cache for the caller, only if exists: if caller in self._caches_dict: del self._caches_dict[caller] self._caches_dict[caller] = [OrderedDict(), time.time()] def __get__(self, obj, obj_type): """ Called for instance methods """ return_func = functools.partial(self._cache_wrapper, obj) return_func.cache_clear = functools.partial(self.cache_clear, obj) # Return the wrapped function and wraps it to maintain the docstring and # the name of the original function: return functools.wraps(self._input_func)(return_func) def __call__(self, *args, **kwargs): """ Called for regular functions """ return self._cache_wrapper(None, *args, **kwargs) # Set the cache_clear function in the __call__ operator: __call__.cache_clear = cache_clear def _cache_wrapper(self, caller, *args, **kwargs): # Create a unique key including the types (in order to differentiate between 1 and '1'): kwargs_key = "".join(map(lambda x: str(x) + str(type(kwargs[x])) + str(kwargs[x]), sorted(kwargs))) key = "".join(map(lambda x: str(type(x)) + str(x), args)) + kwargs_key # Check if caller exists, if not create one: if caller not in self._caches_dict: self._caches_dict[caller] = [OrderedDict(), time.time()] else: # Validate in case the refresh time has passed: if self._timeout is not None: if time.time() - self._caches_dict[caller][1] > self._timeout: self.cache_clear(caller) # Check if the key exists, if so - return it: cur_caller_cache_dict = self._caches_dict[caller][0] if key in cur_caller_cache_dict: return cur_caller_cache_dict[key] # Validate we didn't exceed the max_size: if len(cur_caller_cache_dict) >= self._max_size: # Delete the first item in the dict: cur_caller_cache_dict.popitem(False) # Call the function and store the data in the cache (call it with # the caller in case it's an instance function - Ternary condition): cur_caller_cache_dict[key] = self._input_func( caller, *args, **kwargs) if caller is not None else self._input_func( *args, **kwargs ) return cur_caller_cache_dict[key] # Return the decorator wrapping the class (also wraps the instance to # maintain the docstring and the name of the original function): return lambda input_func: functools.wraps(input_func)(_LRU_Cache_class(input_func, maxsize, timeout)) _missing = object() class cached_property(property): """A decorator that converts a function into a lazy property. The function wrapped is called the first time to retrieve the result and then that calculated result is used the next time you access the value:: class Foo(object): @cached_property def foo(self): # calculate something important here return 42 The class has to have a `__dict__` in order for this property to work. """ # implementation detail: A subclass of python's builtin property # decorator, we override __get__ to check for a cached value. If one # chooses to invoke __get__ by hand the property will still work as # expected because the lookup logic is replicated in __get__ for # manual invocation. # noinspection PyMissingConstructor def __init__(self, func, name=None, doc=None): self.__name__ = name or func.__name__ self.__module__ = func.__module__ self.__doc__ = doc or func.__doc__ self.func = func def __set__(self, obj, value): obj.__dict__[self.__name__] = value def __get__(self, obj, *args, **kwargs): if obj is None: return self value = obj.__dict__.get(self.__name__, _missing) if value is _missing: value = self.func(obj) obj.__dict__[self.__name__] = value return value class CallbackFileWrapper(object): """ Small wrapper around a fp object which will tee everything read into a buffer, and when that file is closed it will execute a callback with the contents of that buffer. All attributes are proxied to the underlying file object. This class uses members with a double underscore (__) leading prefix so as not to accidentally shadow an attribute. """ def __init__(self, fp, callback=None): self.__buf = BytesIO() self.__fp = fp self.__callback = callback self.__once_done = False def __getattr__(self, name): # The vagaries of garbage collection means that self.__fp is # not always set. By using __getattribute__ and the private # name[0] allows looking up the attribute value and raising an # AttributeError when it doesn't exist. This stop things from # infinitely recurring calls to getattr in the case where # self.__fp hasn't been set. # # [0] https://docs.python.org/2/reference/expressions.html#atom-identifiers fp = self.__getattribute__("_CallbackFileWrapper__fp") return getattr(fp, name) def __is_fp_closed(self): try: return self.__fp.fp is None except AttributeError: pass try: return self.__fp.closed except AttributeError: pass return False def close(self): if self.__callback: # self.__callback(self.__buf.getvalue()) self.__callback() # We assign this to None here, because otherwise we can get into # really tricky problems where the CPython interpreter dead locks # because the callback is holding a reference to something which # has a __del__ method. Setting this to None breaks the cycle # and allows the garbage collector to do it's thing normally. self.__callback = None self.__once_done = True return self.__getattribute__("_CallbackFileWrapper__fp").close() def rewind(self): if self.__once_done: self.__fp.close() self.__fp = self.__buf self.__buf = BytesIO() self.__fp.seek(0) self.__once_done = False def read(self, amt=None): data = self.__fp.read(amt) self.__buf.write(data) if self.__is_fp_closed(): self.close() return data def _safe_read(self, amt): data = self.__fp._safe_read(amt) if amt == 2 and data == b"\r\n": # urllib executes this read to toss the CRLF at the end # of the chunk. return data self.__buf.write(data) if self.__is_fp_closed(): self.close() return data class RewindableResponse(object): """ Used by :class:`WebPage` to store current resource content to minimize the number of requests made while working with a page. """ def __init__(self, fp): self.fp = fp self.buffer = BytesIO() self.once_done = threading.Event() def __getattr__(self, name): fp = self.__getattribute__("fp") return getattr(fp, name) @classmethod def from_response(cls, response): ans = cls(response.raw) return ans def read(self, n=None): if self.fp.closed: self.once_done.set() data = self.fp.read(n) self.buffer.write(data) if self.fp.closed: self.once_done.set() return data def rewind(self): if not self.once_done.is_set(): return False self.fp.close() self.buffer.seek(0) self.fp = self.buffer self.buffer = BytesIO() ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1635575442.0 pywebcopy-7.0.1/pywebcopy/parsers.py0000666000000000000000000006037300000000000014516 0ustar00# Copyright 2020; Raja Tomar # See license for more details import inspect import logging import re import requests from lxml import etree from lxml.html import _nons from lxml.html import fromstring from lxml.html import tostring from lxml.html import XHTML_NAMESPACE from lxml.html.clean import Cleaner from lxml.html.defs import link_attrs from six import integer_types from six import string_types from six.moves.urllib.parse import urljoin from six.moves.collections_abc import Iterator __all__ = ['iterparse', 'MultiParser', 'Element', 'unquote_match', 'links'] logger = logging.getLogger(__name__) # Attributes which contains multiple links. srcset_attrs = frozenset([ 'srcset', 'data-srcset', 'src-set', 'imageset', ]) _iter_srcset_urls = re.compile(r"([^\s,]{4,})", re.MULTILINE).finditer _iter_css_urls = re.compile(r'url\((' + '["][^"]*["]|' + "['][^']*[']|" + r'[^)]*)\)', re.I).finditer _iter_css_imports = re.compile(r'@import "(.*?)"').finditer _archive_re = re.compile(r'[^ ]+') _parse_meta_refresh_url = re.compile(r'[^;=]*;\s*(?:url\s*=\s*)?(?P.*)$', re.I).search def unquote_match(s, pos): if s[:1] == '"' and s[-1:] == '"' or s[:1] == "'" and s[-1:] == "'" or \ s[:1] == '"' and s[-1:] == "'" or s[:1] == "'" and s[-1:] == '"': return s[1:-1], pos + 1 else: return s, pos class ElementBase(etree.ElementBase): def remove_csrf_checks(self): # Remove integrity or cors check from the file self.attrib.pop('integrity', None) self.attrib.pop('crossorigin', None) def replace_url(self, old_url, new_url, attr, pos): # Change the url in the object depending on the case if not isinstance(new_url, string_types): TypeError("Expected %r, got %r" % (string_types, new_url)) if not isinstance(pos, integer_types): TypeError("Expected %r, got %r" % (integer_types, pos)) if new_url is None or new_url == old_url: return if attr is None: new = self.text[:pos] + new_url + self.text[pos + len(old_url):] self.text = new else: cur = self.get(attr) if not pos and len(cur) == len(old_url): new = new_url # most common case else: new = cur[:pos] + new_url + cur[pos + len(old_url):] self.set(attr, new) #: For the new url to work properly we need to remove #: any sort of url check attributes present in element. self.remove_csrf_checks() def iterparse(source, encoding=None, events=None, include_meta_charset_tag=False, **kwargs): """Incrementally parse HTML document into ElementTree. TODO: 1. Make iterparse function take in a factory argument which defines the output of the generator. 2. Modify the links function to be a subclass of Iterator and it should be passable to iterparse as factory arg. 3. Make a additional no-op iterator and a forms iterator. """ encoding = encoding or 'iso-8859-1' # rfc default web encoding parser = etree.HTMLPullParser(events=events, encoding=encoding, **kwargs) lookup = etree.ElementDefaultClassLookup(ElementBase) parser.set_element_class_lookup(lookup) def iterator(): # try: while True: # yield from chain.from_iterable(map(filter_, parser.read_events())) # for i in chain.from_iterable( # (links(element) for event, element in parser.read_events()) # ): # yield i for event, element in parser.read_events(): for child in links(element): if child is None: continue yield child data = source.read(0o3000) if not data: break parser.feed(data) if include_meta_charset_tag: parser.feed(('' % encoding).encode( encoding, 'xmlcharrefreplace')) # try: # head = root.xpath( # "descendant-or-self::head|descendant-or-self::x:head", # namespaces={'x': XHTML_NAMESPACE} # )[0] # except (AttributeError, IndexError): # head = parser.makeelement('head') # root.insert(0, head) # #: Write the inferred charset to the html dom so that browsers read this # #: document in our specified encoding. # head.insert(0, parser.makeelement('meta', charset=encoding)) try: root = parser.close() except etree.XMLSyntaxError: parser.feed( ''.encode(encoding, 'xmlcharrefreplace')) root = parser.close() # parser could generate end events for html and # body tags which the parser itself inserted. # for event, element in parser.read_events(): # for child in links(element): # if child is None: # continue # yield child it.root = root # noinspection PyUnusedLocal root = None # XXX No implicit source closing # if close_source: # source.close() class IterParseIterator(Iterator): next = __next__ = iterator().__next__ it = IterParseIterator() it.root = None del IterParseIterator # close_source = False if not hasattr(source, "read"): # source = open(source, "rb") # close_source = True raise TypeError("Expected a readable object, got %r" % source) return it def links(el): tag = _nons(el.tag) attribs = el.attrib if tag == 'object': # pragma: no cover codebase = None if 'codebase' in attribs: codebase = el.get('codebase') yield el, 'codebase', codebase, 0 for attrib in ('classid', 'data'): if attrib in attribs: value = el.get(attrib) if codebase is not None: value = urljoin(codebase, value) yield el, attrib, value, 0 if 'archive' in attribs: for match in _archive_re.finditer(el.get('archive')): value = match.group(0) if codebase is not None: value = urljoin(codebase, value) yield el, 'archive', value, match.start() else: for attrib in link_attrs: if attrib in attribs: yield el, attrib, attribs[attrib], 0 # XXX Patch for src-set url detection for attrib in srcset_attrs: if attrib in attribs: urls = list(_iter_srcset_urls(attribs[attrib])) if urls: # yield in reversed order to simplify in-place modifications for match in urls[::-1]: url, start = unquote_match(match.group(1).strip(), match.start(1)) yield el, attrib, url, start if tag == 'meta': http_equiv = attribs.get('http-equiv', '').lower() if http_equiv == 'refresh': content = attribs.get('content', '') match = _parse_meta_refresh_url(content) url = (match.group('url') if match else content).strip() # unexpected content means the redirect won't work, but we might # as well be permissive and yield the entire string. if url: url, pos = unquote_match( url, match.start('url') if match else content.find(url)) yield el, 'content', url, pos itemprop = attribs.get('itemprop', '').lower() if itemprop == 'image': url = attribs.get('content', '') if url: yield el, 'content', url, 0 elif tag == 'param': valuetype = el.get('valuetype') or '' if valuetype.lower() == 'ref': yield el, 'value', el.get('value'), 0 elif tag == 'script' and el.text: urls = [ # (start_pos, url) unquote_match(match.group(1), match.start(1))[::-1] for match in _iter_css_urls(el.text) ] if urls: # sort by start pos to bring both match sets back into order # and reverse the list to report correct positions despite # modifications urls.sort(reverse=True) for start, url in urls: yield el, None, url, start elif tag == 'style' and el.text: urls = [ # (start_pos, url) unquote_match(match.group(1), match.start(1))[::-1] for match in _iter_css_urls(el.text) ] + [ (match.start(1), match.group(1)) for match in _iter_css_imports(el.text) ] if urls: # sort by start pos to bring both match sets back into order # and reverse the list to report correct positions despite # modifications urls.sort(reverse=True) for start, url in urls: yield el, None, url, start if 'style' in attribs: urls = list(_iter_css_urls(attribs['style'])) if urls: # yield in reversed order to simplify in-place modifications for match in urls[::-1]: url, start = unquote_match(match.group(1), match.start(1)) yield el, 'style', url, start # HTML style and script tags cleaner cleaner = Cleaner() cleaner.javascript = True cleaner.style = True class MultiParser(object): # pragma: no cover """Provides apis specific to scraping or data searching purposes. This contains the apis from the requests-html module. Most of the source code is from the MIT Licensed library called `requests-html` courtesy of kenneth, some code has been heavily modified to fit the needs of this project but some apis are still untouched. :param html: html markup string. :param encoding: optional explicit declaration of encoding type of that web page :param element: Used internally: PyQuery object or raw html. """ def __init__(self, html=None, encoding=None, element=None): self._lxml = None self._pq = None self._soup = None self._html = html # represents your raw html self._encoding = encoding # represents your provided encoding self.element = element # internal lxml element self._decoded_html = False # internal switch to tell if html has been decoded self.default_encoding = 'iso-8859-1' # a standard encoding defined by w3c def request(self, method, url, **params): """Fetches the Html content from Internet using the requests. You can any requests params which will be passed to the library itself. The requests arguments you supply will also be applied to the global session meaning all the files will be downloaded using these settings. :param method: http verb for transport. :param url: url of the page to fetch :param params: keyword arguments which `requests` module may accept. """ resp = requests.request(method, url, stream=True, **params) resp.raise_for_status() self._html = resp.content def get(self, url, **params): return self.request('GET', url, **params) def post(self, url, **params): return self.request('POST', url, **params) def get_forms(self): """Returns a list of form elements available on the page.""" return fromstring( self.html ).xpath( "descendant-or-self::form|descendant-or-self::x:form", namespaces={'x': XHTML_NAMESPACE} ) def submit_form(self, form, url=None, **extra_values): """ Helper function to submit a form. .. todo:: check documentation. You can use this like:: wp = WebPage() wp.get('http://httpbin.org/forms/') form = wp.get_forms()[0] form.inputs['email'].value = 'bar' # etc form.inputs['password'].value = 'baz' # etc wp.submit_form(form) wp.get_links() The action is one of 'GET' or 'POST', the URL is the target URL as a string, and the values are a sequence of ``(name, value)`` tuples with the form data. """ values = form.form_values() if extra_values: if hasattr(extra_values, 'items'): extra_values = extra_values.items() values.extend(extra_values) if url is None: if form.action: url = form.action else: url = form.base_url return self.request(form.method, url, data=values) @property def raw_html(self): """Bytes representation of the HTML content. (`learn more `_). """ if self._html: return self._html else: return tostring(self.element, encoding=self.encoding) @raw_html.setter def raw_html(self, html): """Property setter for raw_html. Type can be bytes.""" self._html = html @property def html(self): """Unicode representation of the HTML content.""" if self._html: return self.decode() else: return tostring(self.element, encoding='unicode') @html.setter def html(self, html): """Property setter for self.html""" if not isinstance(html, str): raise TypeError self._html = html self.decode() def encode(self, encoding=None, errors='xmlcharrefreplace'): """Returns the html encoded with specified encoding.""" return self.html.encode(encoding=encoding, errors=errors) def decode(self): """Decodes the html set to this object and returns used encoding and decoded html.""" self._encoding, html = self.decode_html(self._html, self._encoding, self.default_encoding) return html @staticmethod def decode_html(html_string, encoding=None, default_encoding='iso-8859-1'): """Decodes a html string into a unicode string. If explicit encoding is defined then it would use it otherwise it will decoding it using beautiful soups UnicodeDammit feature, otherwise it will use w3lib to decode the html. Returns a two tuple with (, ) :rtype: (str, str) :returns: (used-encoding, unicode-markup) """ tried = [encoding, default_encoding] try: logger.info("Trying UnicodeDammit Codec for decoding html.") try: import bs4 except ImportError: raise ImportError( "bs4 module is not installed. " "Install it using pip: $ pip install bs4" ) converted = bs4.UnicodeDammit(html_string, [encoding], is_html=True) if not converted.unicode_markup: tried += converted.tried_encodings logger.critical( "UnicodeDammit decoder failed to decode html!" "Encoding tried by default enc: [%s]" "Trying fallback..." % ','.join(tried) ) raise UnicodeDecodeError return converted.original_encoding, converted.unicode_markup except (UnicodeDecodeError, ImportError): # This method will definitely decode the html though # the result could be corrupt. But if you getting a # corrupt html output then you definitely have to # manually provide the encoding. try: import w3lib from w3lib.encoding import html_to_unicode except ImportError: raise ImportError( "w3lib module is not installed. " "Install it using pip: $ pip install w3lib" ) enc, unc = w3lib.encoding.html_to_unicode( None, html_body_str=html_string, default_encoding=default_encoding ) return enc, unc @property def encoding(self): """The encoding string to be used, extracted from the HTML and :class:`HTMLResponse ` headers. """ if self._encoding is None: self.decode() return self._encoding @encoding.setter def encoding(self, enc): """Property setter for self.encoding.""" self._encoding = enc @property def lxml(self): """Parses the decoded self.html contents after decoding it by itself decoding detector (default) or decoding it using provided self.default_encoding. """ if self._lxml is None: self._lxml = fromstring(self.html) return self._lxml @property def bs4(self): """BeautifulSoup object under the hood. Read more about beautiful_soup at https://www.crummy.com/software/BeautifulSoup/doc """ try: import bs4 except ImportError: raise ImportError( "bs4 module is not installed. " "Install it using pip: $ pip install bs4" ) if self._soup is None: self._soup = bs4.BeautifulSoup(self.raw_html, 'lxml') return self._soup @property def pq(self): """`PyQuery `_ representation of the :class:`Element ` or :class:`HTML `. """ try: import pyquery except ImportError: raise ImportError( "pyquery module is not installed. " "Install it using pip: $ pip install pyquery" ) if self._pq is None: self._pq = pyquery.PyQuery(self.lxml) return self._pq @property def text(self): """The text content of the :class:`Element ` or :class:`HTML `. """ return self.pq.text() @property def full_text(self): """The full text content (including links) of the :class:`Element ` or :class:`HTML `. """ return self.lxml.text_content() def find(self, selector="*", containing=None, clean=False, first=False, _encoding=None): """Given a CSS Selector, returns a list of :class:`Element ` objects or a single one. :param selector: CSS Selector to use. :param clean: Whether or not to sanitize the found HTML of ``