././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1643486962.4502711 pyexcel-io-0.6.6/0000755000175100001710000000000000000000000013141 5ustar00runnerdocker././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1643486950.0 pyexcel-io-0.6.6/CHANGELOG.rst0000644000175100001710000003446100000000000015172 0ustar00runnerdockerChange log ================================================================================ 0.6.6 - 31.1.2022 -------------------------------------------------------------------------------- **updated** #. `#112 `_: Log Empty Row Warning instead 'print' 0.6.5 - 08.10.2021 -------------------------------------------------------------------------------- **updated** #. `#109 `_: enable ods3 to have datetime 0.6.4 - 31.10.2020 -------------------------------------------------------------------------------- **updated** #. `#102 `_: skip columns from imported excel sheet. 0.6.3 - 12.10.2020 -------------------------------------------------------------------------------- **fixed** #. `#96 `_: regression: unknown file type shall trigger NoSupportingPluginFound **updated** #. extra dependencies uses 0.6.0 based plugins 0.6.2 - 7.10.2020 -------------------------------------------------------------------------------- **updated** #. `#94 `_: keep backward compatibility for pyexcel-xls 0.4.1 0.6.1 - 7.10.2020 -------------------------------------------------------------------------------- **removed** #. python 3.6 lower versions are no longer supported **updated** #. pyexcel-io plugin interface has been rewritten. PyInstaller user will be impacted. please read 'Packaging with Pyinstaller' in the documentation. #. new query set reader plugin. pyexcel<=0.6.4 has used intrusive way of getting query set source done. it is against the plugin interface. **fixed** #. `#74 `_: handle zip files which contain non-UTF-8 encoded files. **added** #. `#86 `_: allow trailing options, get_data(...keep_trailing_empty_cells=True). 0.5.20 - 17.7.2019 -------------------------------------------------------------------------------- **updated** #. `#70 `_: when the given file is a root directory, the error shall read it is not a file 0.5.19 - 14.7.2019 -------------------------------------------------------------------------------- **updated** #. `pyexcel#185 `_: handle stream conversion if file type(html) needs string content then bytes to handle 0.5.18 - 12.06.2019 -------------------------------------------------------------------------------- **updated** #. `#69 `_: Force file type(force_file_type) on write 0.5.17 - 04.04.2019 -------------------------------------------------------------------------------- **updated** #. `#68 `_: Raise IOError when the data file does not exist 0.5.16 - 19.03.2019 -------------------------------------------------------------------------------- **updated** #. `#67 `_: fix conversion issue for long type on python 2.7 for ods 0.5.15 - 16.03.2019 -------------------------------------------------------------------------------- **updated** #. `pyexcel-ods#33 `_: fix integer comparision error on i586 0.5.14 - 21.02.2019 -------------------------------------------------------------------------------- **updated** #. `#65 `_: add tests/__init__.py because python2.7 setup.py test needs it 0.5.13 - 12.02.2019 -------------------------------------------------------------------------------- **updated** #. `#63 `_: Version 0.5.12 prevents xslx and ods plugin from being loaded 0.5.12 - 9.02.2019 -------------------------------------------------------------------------------- **updated** #. `#60 `_: include tests in tar ball #. `#61 `_: enable python setup.py test 0.5.11 - 3.12.2018 -------------------------------------------------------------------------------- **updated** #. `#59 `_: Please use scan_plugins_regex, which lml 0.7 complains about 0.5.10 - 27.11.2018 -------------------------------------------------------------------------------- **added** #. `#57 `_, long type will not be written in ods. please use string type. And if the integer is equal or greater than 10 to the power of 16, it will not be written either in ods. In both situation, IntegerPrecisionLossError will be raised. And this version enables pyexcel-ods and pyexcel-ods3 to do so. 0.5.9.1 - 30.08.2018 -------------------------------------------------------------------------------- **updated** #. `#53 `_, upgrade lml dependency to at least 0.0.2 0.5.9 - 23.08.2018 -------------------------------------------------------------------------------- **added** #. `pyexcel#148 `_, support force_file_type 0.5.8 - 16.08.2018 -------------------------------------------------------------------------------- **added** #. `#49 `_, support additional options when detecting float values in csv format. default_float_nan, ignore_nan_text 0.5.7 - 02.05.2018 -------------------------------------------------------------------------------- **fixed** #. `#48 `_, turn off pep 0515 #. `#47 `_, csv reader cannot handle relative file names 0.5.6 - 11.01.2018 -------------------------------------------------------------------------------- **fixed** #. `#46 `_, expose `bulk_save` to developer 0.5.5 - 23.12.2017 -------------------------------------------------------------------------------- **fixed** #. Issue `#45 `_, csv reader throws exception because google app engine does not support mmap. People who are not working with google app engine, need not to take this update. Enjoy your Christmas break. 0.5.4 - 10.11.2017 -------------------------------------------------------------------------------- **updated** #. PR `#44 `_, use unicodewriter for csvz writers. 0.5.3 - 23.10.2017 -------------------------------------------------------------------------------- **updated** #. pyexcel `pyexcel#105 `_, remove gease from setup_requires, introduced by 0.5.2. #. remove python2.6 test support 0.5.2 - 20.10.2017 -------------------------------------------------------------------------------- **added** #. `pyexcel#103 `_, include LICENSE file in MANIFEST.in, meaning LICENSE file will appear in the released tar ball. 0.5.1 - 02.09.2017 -------------------------------------------------------------------------------- **Fixed** #. `pyexcel-ods#25 `_, Unwanted dependency on pyexcel. 0.5.0 - 30.08.2017 -------------------------------------------------------------------------------- **Added** #. Collect all data type conversion codes as service.py. **Updated** #. `#19 `_, use cString by default. For python, it will be a performance boost 0.4.4 - 08.08.2017 -------------------------------------------------------------------------------- **Updated** #. `#42 `_, raise exception if database table name does not match the sheet name 0.4.3 - 29.07.2017 -------------------------------------------------------------------------------- **Updated** #. `#41 `_, walk away gracefully when mmap is not available. 0.4.2 - 05.07.2017 -------------------------------------------------------------------------------- **Updated** #. `#37 `_, permanently fix the residue folder pyexcel by release all future releases in a clean clone. 0.4.1 - 29.06.2017 -------------------------------------------------------------------------------- **Updated** #. `#39 `_, raise exception when bulk save in django fails. Please `bulk_save=False` if you as the developer choose to save the records one by one if bulk_save cannot be used. However, exception in one-by-one save case will be raised as well. This change is made to raise exception in the first place so that you as the developer will be suprised when it was deployed in production. 0.4.0 - 19.06.2017 -------------------------------------------------------------------------------- **Updated** #. 'built-in' as the value to the parameter 'library' as parameter to invoke pyexcel-io's built-in csv, tsv, csvz, tsvz, django and sql won't work. It is renamed to 'pyexcel-io'. #. built-in csv, tsv, csvz, tsvz, django and sql are lazy loaded. #. pyexcel-io plugin interface has been updated. v0.3.x plugins won't work. #. `#32 `_, csv and csvz file handle are made sure to be closed. File close mechanism is enfored. #. iget_data function is introduced to cope with dangling file handle issue. **Removed** #. Removed plugin loading code and lml is used instead 0.3.4 - 18.05.2017 -------------------------------------------------------------------------------- **Updated** #. `#33 `_, handle mmap object differently given as file content. This issue has put in a priority to single sheet csv over multiple sheets in a single memory stream. The latter format is pyexcel own creation but is rarely used. In latter case, multiple_sheet=True should be passed along get_data. #. `#34 `_, treat mmap object as a file content. #. `#35 `_, encoding parameter take no effect when given along with file content #. use ZIP_DEFALTED to really do the compression 0.3.3 - 30.03.2017 -------------------------------------------------------------------------------- **Updated** #. `#31 `_, support pyinstaller 0.3.2 - 26.01.2017 -------------------------------------------------------------------------------- **Updated** #. `#29 `_, change skip_empty_rows to False by default 0.3.1 - 21.01.2017 -------------------------------------------------------------------------------- **Added** #. updated versions of extra packages **Updated** #. `#23 `_, provide helpful message when old pyexcel plugin exists #. restored previously available diagnosis message for missing libraries 0.3.0 - 22.12.2016 -------------------------------------------------------------------------------- **Added** #. lazy loading of plugins. for example, pyexcel-xls is not entirely loaded until xls format is used at its first attempted reading or writing. Since it is loaded, it will not be loaded in the second io action. #. `pyexcel-xls#11 `_, make case-insensitive for file type 0.2.6 - 21.12.2016 -------------------------------------------------------------------------------- **Updated** #. `#24 `__, pass on batch_size 0.2.5 - 20.12.2016 -------------------------------------------------------------------------------- **Updated** #. `#26 `__, performance issue with getting the number of columns. 0.2.4 - 24.11.2016 -------------------------------------------------------------------------------- **Updated** #. `#23 `__, Failed to convert long integer string in python 2 to its actual value 0.2.3 - 16.09.2016 -------------------------------------------------------------------------------- **Added** #. `#21 `__, choose subset from data base tables for export #. `#22 `__, custom renderer if given `row_renderer` as parameter. 0.2.2 - 31.08.2016 -------------------------------------------------------------------------------- **Added** #. support pagination. two pairs: start_row, row_limit and start_column, column_limit help you deal with large files. #. `skip_empty_rows=True` was introduced. To include empty rows, put it to False. **Updated** #. `#20 `__, pyexcel-io attempts to parse cell contents of 'infinity' as a float/int, crashes 0.2.1 - 11.07.2016 -------------------------------------------------------------------------------- **Added** #. csv format: handle utf-16 encoded csv files. Potentially being able to decode other formats if correct "encoding" is provided #. csv format: write utf-16 encoded files. Potentially other encoding is also supported #. support stdin as input stream and stdout as output stream **Updated** #. Attention, user of pyexcel-io! No longer io stream validation is performed in python 3. The guideline is: io.StringIO for csv, tsv only, otherwise BytesIO for xlsx, xls, ods. You can use RWManager.get_io to produce a correct stream type for you. #. `#15 `__, support foreign django/sql foreign key 0.2.0 - 01.06.2016 -------------------------------------------------------------------------------- **Added** #. autoload of pyexcel-io plugins #. auto detect `datetime`, `float` and `int`. Detection can be switched off by `auto_detect_datetime`, `auto_detect_float`, `auto_detect_int` 0.1.0 - 17.01.2016 -------------------------------------------------------------------------------- **Added** #. yield key word to return generator as content ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1643486950.0 pyexcel-io-0.6.6/CONTRIBUTORS.rst0000644000175100001710000000067700000000000015642 0ustar00runnerdocker 6 contributors ================================================================================ In alphabetical order: * `Craig Anderson `_ * `John Vandenberg `_ * `Stephen J. Fuhry `_ * `Stephen Rauch `_ * `Vincent Raspal `_ * `Víctor Antonio Hernández Monroy `_ ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1643486950.0 pyexcel-io-0.6.6/LICENSE0000644000175100001710000000307400000000000014152 0ustar00runnerdockerCopyright (c) 2015-2022 by Onni Software Ltd. and its contributors All rights reserved. Redistribution and use in source and binary forms of the software as well as documentation, with or without modification, are permitted provided that the following conditions are met: * Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. * Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. * Neither the name of 'pyexcel-io' nor the names of the contributors may not be used to endorse or promote products derived from this software without specific prior written permission. THIS SOFTWARE AND DOCUMENTATION IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE AND DOCUMENTATION, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1643486950.0 pyexcel-io-0.6.6/MANIFEST.in0000644000175100001710000000024600000000000014701 0ustar00runnerdockerinclude README.rst include LICENSE include CHANGELOG.rst include CONTRIBUTORS.rst recursive-include tests * recursive-include docs * include Makefile include test.sh ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1643486950.0 pyexcel-io-0.6.6/Makefile0000644000175100001710000000025700000000000014605 0ustar00runnerdockerall: test test: lint bash test.sh install_test: pip install -r tests/requirements.txt lint: bash lint.sh format: bash format.sh git-diff-check: git diff --exit-code ././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1643486962.4502711 pyexcel-io-0.6.6/PKG-INFO0000644000175100001710000006066700000000000014255 0ustar00runnerdockerMetadata-Version: 2.1 Name: pyexcel-io Version: 0.6.6 Summary: A python library to read and write structured data in csv, zipped csvformat and to/from databases Home-page: https://github.com/pyexcel/pyexcel-io Author: C.W. Author-email: info@pyexcel.org License: New BSD Download-URL: https://github.com/pyexcel/pyexcel-io/archive/0.6.6.tar.gz Keywords: python,API,tsv,tsvz,csv,csvz,django,sqlalchemy Platform: UNKNOWN Classifier: Topic :: Software Development :: Libraries Classifier: Programming Language :: Python Classifier: Intended Audience :: Developers Classifier: Programming Language :: Python :: 3 :: Only Classifier: Programming Language :: Python :: 3.6 Classifier: Programming Language :: Python :: 3.7 Classifier: Programming Language :: Python :: 3.8 Classifier: License :: OSI Approved :: BSD License Classifier: Programming Language :: Python :: Implementation :: PyPy Requires-Python: >=3.6 Provides-Extra: xls Provides-Extra: xlsx Provides-Extra: ods License-File: LICENSE ================================================================================ pyexcel-io - Let you focus on data, instead of file formats ================================================================================ .. image:: https://raw.githubusercontent.com/pyexcel/pyexcel.github.io/master/images/patreon.png :target: https://www.patreon.com/chfw .. image:: https://raw.githubusercontent.com/pyexcel/pyexcel-mobans/master/images/awesome-badge.svg :target: https://awesome-python.com/#specific-formats-processing .. image:: https://github.com/pyexcel/pyexcel-io/workflows/run_tests/badge.svg :target: http://github.com/pyexcel/pyexcel-io/actions .. image:: https://codecov.io/gh/pyexcel/pyexcel-io/branch/master/graph/badge.svg :target: https://codecov.io/gh/pyexcel/pyexcel-io .. image:: https://badge.fury.io/py/pyexcel-io.svg :target: https://pypi.org/project/pyexcel-io .. image:: https://anaconda.org/conda-forge/pyexcel-io/badges/version.svg :target: https://anaconda.org/conda-forge/pyexcel-io .. image:: https://pepy.tech/badge/pyexcel-io/month :target: https://pepy.tech/project/pyexcel-io .. image:: https://anaconda.org/conda-forge/pyexcel-io/badges/downloads.svg :target: https://anaconda.org/conda-forge/pyexcel-io .. image:: https://img.shields.io/gitter/room/gitterHQ/gitter.svg :target: https://gitter.im/pyexcel/Lobby .. image:: https://img.shields.io/static/v1?label=continuous%20templating&message=%E6%A8%A1%E7%89%88%E6%9B%B4%E6%96%B0&color=blue&style=flat-square :target: https://moban.readthedocs.io/en/latest/#at-scale-continous-templating-for-open-source-projects .. image:: https://img.shields.io/static/v1?label=coding%20style&message=black&color=black&style=flat-square :target: https://github.com/psf/black .. image:: https://readthedocs.org/projects/pyexcel-io/badge/?version=latest :target: http://pyexcel-io.readthedocs.org/en/latest/ Support the project ================================================================================ If your company has embedded pyexcel and its components into a revenue generating product, please support me on github, `patreon `_ or `bounty source `_ to maintain the project and develop it further. If you are an individual, you are welcome to support me too and for however long you feel like. As my backer, you will receive `early access to pyexcel related contents `_. And your issues will get prioritized if you would like to become my patreon as `pyexcel pro user`. With your financial support, I will be able to invest a little bit more time in coding, documentation and writing interesting posts. Known constraints ================== Fonts, colors and charts are not supported. Nor to read password protected xls, xlsx and ods files. Introduction ================================================================================ **pyexcel-io** provides **one** application programming interface(API) to read and write the data in excel format, import the data into and export the data from database. It provides support for csv(z) format, django database and sqlalchemy supported databases. Its supported file formats are extended to cover "xls", "xlsx", "ods" by the following extensions: .. _file-format-list: .. _a-map-of-plugins-and-file-formats: .. table:: A list of file formats supported by external plugins ======================== ======================= ================= Package name Supported file formats Dependencies ======================== ======================= ================= `pyexcel-io`_ csv, csvz [#f1]_, tsv, tsvz [#f2]_ `pyexcel-xls`_ xls, xlsx(read only), `xlrd`_, xlsm(read only) `xlwt`_ `pyexcel-xlsx`_ xlsx `openpyxl`_ `pyexcel-ods3`_ ods `pyexcel-ezodf`_, lxml `pyexcel-ods`_ ods `odfpy`_ ======================== ======================= ================= .. table:: Dedicated file reader and writers ======================== ======================= ================= Package name Supported file formats Dependencies ======================== ======================= ================= `pyexcel-xlsxw`_ xlsx(write only) `XlsxWriter`_ `pyexcel-libxlsxw`_ xlsx(write only) `libxlsxwriter`_ `pyexcel-xlsxr`_ xlsx(read only) lxml `pyexcel-xlsbr`_ xlsb(read only) pyxlsb `pyexcel-odsr`_ read only for ods, fods lxml `pyexcel-odsw`_ write only for ods loxun `pyexcel-htmlr`_ html(read only) lxml,html5lib `pyexcel-pdfr`_ pdf(read only) camelot ======================== ======================= ================= Plugin shopping guide ------------------------ Since 2020, all pyexcel-io plugins have dropped the support for python versions which are lower than 3.6. If you want to use any of those Python versions, please use pyexcel-io and its plugins versions that are lower than 0.6.0. Except csv files, xls, xlsx and ods files are a zip of a folder containing a lot of xml files The dedicated readers for excel files can stream read In order to manage the list of plugins installed, you need to use pip to add or remove a plugin. When you use virtualenv, you can have different plugins per virtual environment. In the situation where you have multiple plugins that does the same thing in your environment, you need to tell pyexcel which plugin to use per function call. For example, pyexcel-ods and pyexcel-odsr, and you want to get_array to use pyexcel-odsr. You need to append get_array(..., library='pyexcel-odsr'). .. _pyexcel-io: https://github.com/pyexcel/pyexcel-io .. _pyexcel-xls: https://github.com/pyexcel/pyexcel-xls .. _pyexcel-xlsx: https://github.com/pyexcel/pyexcel-xlsx .. _pyexcel-ods: https://github.com/pyexcel/pyexcel-ods .. _pyexcel-ods3: https://github.com/pyexcel/pyexcel-ods3 .. _pyexcel-odsr: https://github.com/pyexcel/pyexcel-odsr .. _pyexcel-odsw: https://github.com/pyexcel/pyexcel-odsw .. _pyexcel-pdfr: https://github.com/pyexcel/pyexcel-pdfr .. _pyexcel-xlsxw: https://github.com/pyexcel/pyexcel-xlsxw .. _pyexcel-libxlsxw: https://github.com/pyexcel/pyexcel-libxlsxw .. _pyexcel-xlsxr: https://github.com/pyexcel/pyexcel-xlsxr .. _pyexcel-xlsbr: https://github.com/pyexcel/pyexcel-xlsbr .. _pyexcel-htmlr: https://github.com/pyexcel/pyexcel-htmlr .. _xlrd: https://github.com/python-excel/xlrd .. _xlwt: https://github.com/python-excel/xlwt .. _openpyxl: https://bitbucket.org/openpyxl/openpyxl .. _XlsxWriter: https://github.com/jmcnamara/XlsxWriter .. _pyexcel-ezodf: https://github.com/pyexcel/pyexcel-ezodf .. _odfpy: https://github.com/eea/odfpy .. _libxlsxwriter: http://libxlsxwriter.github.io/getting_started.html .. rubric:: Footnotes .. [#f1] zipped csv file .. [#f2] zipped tsv file If you need to manipulate the data, you might do it yourself or use its brother library `pyexcel `__ . If you would like to extend it, you may use it to write your own extension to handle a specific file format. Installation ================================================================================ You can install pyexcel-io via pip: .. code-block:: bash $ pip install pyexcel-io or clone it and install it: .. code-block:: bash $ git clone https://github.com/pyexcel/pyexcel-io.git $ cd pyexcel-io $ python setup.py install Development guide ================================================================================ Development steps for code changes #. git clone https://github.com/pyexcel/pyexcel-io.git #. cd pyexcel-io Upgrade your setup tools and pip. They are needed for development and testing only: #. pip install --upgrade setuptools pip Then install relevant development requirements: #. pip install -r rnd_requirements.txt # if such a file exists #. pip install -r requirements.txt #. pip install -r tests/requirements.txt Once you have finished your changes, please provide test case(s), relevant documentation and update changelog.yml .. note:: As to rnd_requirements.txt, usually, it is created when a dependent library is not released. Once the dependecy is installed (will be released), the future version of the dependency in the requirements.txt will be valid. How to test your contribution ------------------------------ Although `nose` and `doctest` are both used in code testing, it is adviable that unit tests are put in tests. `doctest` is incorporated only to make sure the code examples in documentation remain valid across different development releases. On Linux/Unix systems, please launch your tests like this:: $ make On Windows, please issue this command:: > test.bat Before you commit ------------------------------ Please run:: $ make format so as to beautify your code otherwise your build may fail your unit test. License ================================================================================ New BSD License Change log ================================================================================ 0.6.6 - 31.1.2022 -------------------------------------------------------------------------------- **updated** #. `#112 `_: Log Empty Row Warning instead 'print' 0.6.5 - 08.10.2021 -------------------------------------------------------------------------------- **updated** #. `#109 `_: enable ods3 to have datetime 0.6.4 - 31.10.2020 -------------------------------------------------------------------------------- **updated** #. `#102 `_: skip columns from imported excel sheet. 0.6.3 - 12.10.2020 -------------------------------------------------------------------------------- **fixed** #. `#96 `_: regression: unknown file type shall trigger NoSupportingPluginFound **updated** #. extra dependencies uses 0.6.0 based plugins 0.6.2 - 7.10.2020 -------------------------------------------------------------------------------- **updated** #. `#94 `_: keep backward compatibility for pyexcel-xls 0.4.1 0.6.1 - 7.10.2020 -------------------------------------------------------------------------------- **removed** #. python 3.6 lower versions are no longer supported **updated** #. pyexcel-io plugin interface has been rewritten. PyInstaller user will be impacted. please read 'Packaging with Pyinstaller' in the documentation. #. new query set reader plugin. pyexcel<=0.6.4 has used intrusive way of getting query set source done. it is against the plugin interface. **fixed** #. `#74 `_: handle zip files which contain non-UTF-8 encoded files. **added** #. `#86 `_: allow trailing options, get_data(...keep_trailing_empty_cells=True). 0.5.20 - 17.7.2019 -------------------------------------------------------------------------------- **updated** #. `#70 `_: when the given file is a root directory, the error shall read it is not a file 0.5.19 - 14.7.2019 -------------------------------------------------------------------------------- **updated** #. `pyexcel#185 `_: handle stream conversion if file type(html) needs string content then bytes to handle 0.5.18 - 12.06.2019 -------------------------------------------------------------------------------- **updated** #. `#69 `_: Force file type(force_file_type) on write 0.5.17 - 04.04.2019 -------------------------------------------------------------------------------- **updated** #. `#68 `_: Raise IOError when the data file does not exist 0.5.16 - 19.03.2019 -------------------------------------------------------------------------------- **updated** #. `#67 `_: fix conversion issue for long type on python 2.7 for ods 0.5.15 - 16.03.2019 -------------------------------------------------------------------------------- **updated** #. `pyexcel-ods#33 `_: fix integer comparision error on i586 0.5.14 - 21.02.2019 -------------------------------------------------------------------------------- **updated** #. `#65 `_: add tests/__init__.py because python2.7 setup.py test needs it 0.5.13 - 12.02.2019 -------------------------------------------------------------------------------- **updated** #. `#63 `_: Version 0.5.12 prevents xslx and ods plugin from being loaded 0.5.12 - 9.02.2019 -------------------------------------------------------------------------------- **updated** #. `#60 `_: include tests in tar ball #. `#61 `_: enable python setup.py test 0.5.11 - 3.12.2018 -------------------------------------------------------------------------------- **updated** #. `#59 `_: Please use scan_plugins_regex, which lml 0.7 complains about 0.5.10 - 27.11.2018 -------------------------------------------------------------------------------- **added** #. `#57 `_, long type will not be written in ods. please use string type. And if the integer is equal or greater than 10 to the power of 16, it will not be written either in ods. In both situation, IntegerPrecisionLossError will be raised. And this version enables pyexcel-ods and pyexcel-ods3 to do so. 0.5.9.1 - 30.08.2018 -------------------------------------------------------------------------------- **updated** #. `#53 `_, upgrade lml dependency to at least 0.0.2 0.5.9 - 23.08.2018 -------------------------------------------------------------------------------- **added** #. `pyexcel#148 `_, support force_file_type 0.5.8 - 16.08.2018 -------------------------------------------------------------------------------- **added** #. `#49 `_, support additional options when detecting float values in csv format. default_float_nan, ignore_nan_text 0.5.7 - 02.05.2018 -------------------------------------------------------------------------------- **fixed** #. `#48 `_, turn off pep 0515 #. `#47 `_, csv reader cannot handle relative file names 0.5.6 - 11.01.2018 -------------------------------------------------------------------------------- **fixed** #. `#46 `_, expose `bulk_save` to developer 0.5.5 - 23.12.2017 -------------------------------------------------------------------------------- **fixed** #. Issue `#45 `_, csv reader throws exception because google app engine does not support mmap. People who are not working with google app engine, need not to take this update. Enjoy your Christmas break. 0.5.4 - 10.11.2017 -------------------------------------------------------------------------------- **updated** #. PR `#44 `_, use unicodewriter for csvz writers. 0.5.3 - 23.10.2017 -------------------------------------------------------------------------------- **updated** #. pyexcel `pyexcel#105 `_, remove gease from setup_requires, introduced by 0.5.2. #. remove python2.6 test support 0.5.2 - 20.10.2017 -------------------------------------------------------------------------------- **added** #. `pyexcel#103 `_, include LICENSE file in MANIFEST.in, meaning LICENSE file will appear in the released tar ball. 0.5.1 - 02.09.2017 -------------------------------------------------------------------------------- **Fixed** #. `pyexcel-ods#25 `_, Unwanted dependency on pyexcel. 0.5.0 - 30.08.2017 -------------------------------------------------------------------------------- **Added** #. Collect all data type conversion codes as service.py. **Updated** #. `#19 `_, use cString by default. For python, it will be a performance boost 0.4.4 - 08.08.2017 -------------------------------------------------------------------------------- **Updated** #. `#42 `_, raise exception if database table name does not match the sheet name 0.4.3 - 29.07.2017 -------------------------------------------------------------------------------- **Updated** #. `#41 `_, walk away gracefully when mmap is not available. 0.4.2 - 05.07.2017 -------------------------------------------------------------------------------- **Updated** #. `#37 `_, permanently fix the residue folder pyexcel by release all future releases in a clean clone. 0.4.1 - 29.06.2017 -------------------------------------------------------------------------------- **Updated** #. `#39 `_, raise exception when bulk save in django fails. Please `bulk_save=False` if you as the developer choose to save the records one by one if bulk_save cannot be used. However, exception in one-by-one save case will be raised as well. This change is made to raise exception in the first place so that you as the developer will be suprised when it was deployed in production. 0.4.0 - 19.06.2017 -------------------------------------------------------------------------------- **Updated** #. 'built-in' as the value to the parameter 'library' as parameter to invoke pyexcel-io's built-in csv, tsv, csvz, tsvz, django and sql won't work. It is renamed to 'pyexcel-io'. #. built-in csv, tsv, csvz, tsvz, django and sql are lazy loaded. #. pyexcel-io plugin interface has been updated. v0.3.x plugins won't work. #. `#32 `_, csv and csvz file handle are made sure to be closed. File close mechanism is enfored. #. iget_data function is introduced to cope with dangling file handle issue. **Removed** #. Removed plugin loading code and lml is used instead 0.3.4 - 18.05.2017 -------------------------------------------------------------------------------- **Updated** #. `#33 `_, handle mmap object differently given as file content. This issue has put in a priority to single sheet csv over multiple sheets in a single memory stream. The latter format is pyexcel own creation but is rarely used. In latter case, multiple_sheet=True should be passed along get_data. #. `#34 `_, treat mmap object as a file content. #. `#35 `_, encoding parameter take no effect when given along with file content #. use ZIP_DEFALTED to really do the compression 0.3.3 - 30.03.2017 -------------------------------------------------------------------------------- **Updated** #. `#31 `_, support pyinstaller 0.3.2 - 26.01.2017 -------------------------------------------------------------------------------- **Updated** #. `#29 `_, change skip_empty_rows to False by default 0.3.1 - 21.01.2017 -------------------------------------------------------------------------------- **Added** #. updated versions of extra packages **Updated** #. `#23 `_, provide helpful message when old pyexcel plugin exists #. restored previously available diagnosis message for missing libraries 0.3.0 - 22.12.2016 -------------------------------------------------------------------------------- **Added** #. lazy loading of plugins. for example, pyexcel-xls is not entirely loaded until xls format is used at its first attempted reading or writing. Since it is loaded, it will not be loaded in the second io action. #. `pyexcel-xls#11 `_, make case-insensitive for file type 0.2.6 - 21.12.2016 -------------------------------------------------------------------------------- **Updated** #. `#24 `__, pass on batch_size 0.2.5 - 20.12.2016 -------------------------------------------------------------------------------- **Updated** #. `#26 `__, performance issue with getting the number of columns. 0.2.4 - 24.11.2016 -------------------------------------------------------------------------------- **Updated** #. `#23 `__, Failed to convert long integer string in python 2 to its actual value 0.2.3 - 16.09.2016 -------------------------------------------------------------------------------- **Added** #. `#21 `__, choose subset from data base tables for export #. `#22 `__, custom renderer if given `row_renderer` as parameter. 0.2.2 - 31.08.2016 -------------------------------------------------------------------------------- **Added** #. support pagination. two pairs: start_row, row_limit and start_column, column_limit help you deal with large files. #. `skip_empty_rows=True` was introduced. To include empty rows, put it to False. **Updated** #. `#20 `__, pyexcel-io attempts to parse cell contents of 'infinity' as a float/int, crashes 0.2.1 - 11.07.2016 -------------------------------------------------------------------------------- **Added** #. csv format: handle utf-16 encoded csv files. Potentially being able to decode other formats if correct "encoding" is provided #. csv format: write utf-16 encoded files. Potentially other encoding is also supported #. support stdin as input stream and stdout as output stream **Updated** #. Attention, user of pyexcel-io! No longer io stream validation is performed in python 3. The guideline is: io.StringIO for csv, tsv only, otherwise BytesIO for xlsx, xls, ods. You can use RWManager.get_io to produce a correct stream type for you. #. `#15 `__, support foreign django/sql foreign key 0.2.0 - 01.06.2016 -------------------------------------------------------------------------------- **Added** #. autoload of pyexcel-io plugins #. auto detect `datetime`, `float` and `int`. Detection can be switched off by `auto_detect_datetime`, `auto_detect_float`, `auto_detect_int` 0.1.0 - 17.01.2016 -------------------------------------------------------------------------------- **Added** #. yield key word to return generator as content ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1643486950.0 pyexcel-io-0.6.6/README.rst0000644000175100001710000002224000000000000014630 0ustar00runnerdocker================================================================================ pyexcel-io - Let you focus on data, instead of file formats ================================================================================ .. image:: https://raw.githubusercontent.com/pyexcel/pyexcel.github.io/master/images/patreon.png :target: https://www.patreon.com/chfw .. image:: https://raw.githubusercontent.com/pyexcel/pyexcel-mobans/master/images/awesome-badge.svg :target: https://awesome-python.com/#specific-formats-processing .. image:: https://github.com/pyexcel/pyexcel-io/workflows/run_tests/badge.svg :target: http://github.com/pyexcel/pyexcel-io/actions .. image:: https://codecov.io/gh/pyexcel/pyexcel-io/branch/master/graph/badge.svg :target: https://codecov.io/gh/pyexcel/pyexcel-io .. image:: https://badge.fury.io/py/pyexcel-io.svg :target: https://pypi.org/project/pyexcel-io .. image:: https://anaconda.org/conda-forge/pyexcel-io/badges/version.svg :target: https://anaconda.org/conda-forge/pyexcel-io .. image:: https://pepy.tech/badge/pyexcel-io/month :target: https://pepy.tech/project/pyexcel-io .. image:: https://anaconda.org/conda-forge/pyexcel-io/badges/downloads.svg :target: https://anaconda.org/conda-forge/pyexcel-io .. image:: https://img.shields.io/gitter/room/gitterHQ/gitter.svg :target: https://gitter.im/pyexcel/Lobby .. image:: https://img.shields.io/static/v1?label=continuous%20templating&message=%E6%A8%A1%E7%89%88%E6%9B%B4%E6%96%B0&color=blue&style=flat-square :target: https://moban.readthedocs.io/en/latest/#at-scale-continous-templating-for-open-source-projects .. image:: https://img.shields.io/static/v1?label=coding%20style&message=black&color=black&style=flat-square :target: https://github.com/psf/black .. image:: https://readthedocs.org/projects/pyexcel-io/badge/?version=latest :target: http://pyexcel-io.readthedocs.org/en/latest/ Support the project ================================================================================ If your company has embedded pyexcel and its components into a revenue generating product, please support me on github, `patreon `_ or `bounty source `_ to maintain the project and develop it further. If you are an individual, you are welcome to support me too and for however long you feel like. As my backer, you will receive `early access to pyexcel related contents `_. And your issues will get prioritized if you would like to become my patreon as `pyexcel pro user`. With your financial support, I will be able to invest a little bit more time in coding, documentation and writing interesting posts. Known constraints ================== Fonts, colors and charts are not supported. Nor to read password protected xls, xlsx and ods files. Introduction ================================================================================ **pyexcel-io** provides **one** application programming interface(API) to read and write the data in excel format, import the data into and export the data from database. It provides support for csv(z) format, django database and sqlalchemy supported databases. Its supported file formats are extended to cover "xls", "xlsx", "ods" by the following extensions: .. _file-format-list: .. _a-map-of-plugins-and-file-formats: .. table:: A list of file formats supported by external plugins ======================== ======================= ================= Package name Supported file formats Dependencies ======================== ======================= ================= `pyexcel-io`_ csv, csvz [#f1]_, tsv, tsvz [#f2]_ `pyexcel-xls`_ xls, xlsx(read only), `xlrd`_, xlsm(read only) `xlwt`_ `pyexcel-xlsx`_ xlsx `openpyxl`_ `pyexcel-ods3`_ ods `pyexcel-ezodf`_, lxml `pyexcel-ods`_ ods `odfpy`_ ======================== ======================= ================= .. table:: Dedicated file reader and writers ======================== ======================= ================= Package name Supported file formats Dependencies ======================== ======================= ================= `pyexcel-xlsxw`_ xlsx(write only) `XlsxWriter`_ `pyexcel-libxlsxw`_ xlsx(write only) `libxlsxwriter`_ `pyexcel-xlsxr`_ xlsx(read only) lxml `pyexcel-xlsbr`_ xlsb(read only) pyxlsb `pyexcel-odsr`_ read only for ods, fods lxml `pyexcel-odsw`_ write only for ods loxun `pyexcel-htmlr`_ html(read only) lxml,html5lib `pyexcel-pdfr`_ pdf(read only) camelot ======================== ======================= ================= Plugin shopping guide ------------------------ Since 2020, all pyexcel-io plugins have dropped the support for python versions which are lower than 3.6. If you want to use any of those Python versions, please use pyexcel-io and its plugins versions that are lower than 0.6.0. Except csv files, xls, xlsx and ods files are a zip of a folder containing a lot of xml files The dedicated readers for excel files can stream read In order to manage the list of plugins installed, you need to use pip to add or remove a plugin. When you use virtualenv, you can have different plugins per virtual environment. In the situation where you have multiple plugins that does the same thing in your environment, you need to tell pyexcel which plugin to use per function call. For example, pyexcel-ods and pyexcel-odsr, and you want to get_array to use pyexcel-odsr. You need to append get_array(..., library='pyexcel-odsr'). .. _pyexcel-io: https://github.com/pyexcel/pyexcel-io .. _pyexcel-xls: https://github.com/pyexcel/pyexcel-xls .. _pyexcel-xlsx: https://github.com/pyexcel/pyexcel-xlsx .. _pyexcel-ods: https://github.com/pyexcel/pyexcel-ods .. _pyexcel-ods3: https://github.com/pyexcel/pyexcel-ods3 .. _pyexcel-odsr: https://github.com/pyexcel/pyexcel-odsr .. _pyexcel-odsw: https://github.com/pyexcel/pyexcel-odsw .. _pyexcel-pdfr: https://github.com/pyexcel/pyexcel-pdfr .. _pyexcel-xlsxw: https://github.com/pyexcel/pyexcel-xlsxw .. _pyexcel-libxlsxw: https://github.com/pyexcel/pyexcel-libxlsxw .. _pyexcel-xlsxr: https://github.com/pyexcel/pyexcel-xlsxr .. _pyexcel-xlsbr: https://github.com/pyexcel/pyexcel-xlsbr .. _pyexcel-htmlr: https://github.com/pyexcel/pyexcel-htmlr .. _xlrd: https://github.com/python-excel/xlrd .. _xlwt: https://github.com/python-excel/xlwt .. _openpyxl: https://bitbucket.org/openpyxl/openpyxl .. _XlsxWriter: https://github.com/jmcnamara/XlsxWriter .. _pyexcel-ezodf: https://github.com/pyexcel/pyexcel-ezodf .. _odfpy: https://github.com/eea/odfpy .. _libxlsxwriter: http://libxlsxwriter.github.io/getting_started.html .. rubric:: Footnotes .. [#f1] zipped csv file .. [#f2] zipped tsv file If you need to manipulate the data, you might do it yourself or use its brother library `pyexcel `__ . If you would like to extend it, you may use it to write your own extension to handle a specific file format. Installation ================================================================================ You can install pyexcel-io via pip: .. code-block:: bash $ pip install pyexcel-io or clone it and install it: .. code-block:: bash $ git clone https://github.com/pyexcel/pyexcel-io.git $ cd pyexcel-io $ python setup.py install Development guide ================================================================================ Development steps for code changes #. git clone https://github.com/pyexcel/pyexcel-io.git #. cd pyexcel-io Upgrade your setup tools and pip. They are needed for development and testing only: #. pip install --upgrade setuptools pip Then install relevant development requirements: #. pip install -r rnd_requirements.txt # if such a file exists #. pip install -r requirements.txt #. pip install -r tests/requirements.txt Once you have finished your changes, please provide test case(s), relevant documentation and update changelog.yml .. note:: As to rnd_requirements.txt, usually, it is created when a dependent library is not released. Once the dependecy is installed (will be released), the future version of the dependency in the requirements.txt will be valid. How to test your contribution ------------------------------ Although `nose` and `doctest` are both used in code testing, it is adviable that unit tests are put in tests. `doctest` is incorporated only to make sure the code examples in documentation remain valid across different development releases. On Linux/Unix systems, please launch your tests like this:: $ make On Windows, please issue this command:: > test.bat Before you commit ------------------------------ Please run:: $ make format so as to beautify your code otherwise your build may fail your unit test. License ================================================================================ New BSD License ././@PaxHeader0000000000000000000000000000003300000000000010211 xustar0027 mtime=1643486962.438271 pyexcel-io-0.6.6/docs/0000755000175100001710000000000000000000000014071 5ustar00runnerdocker././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1643486950.0 pyexcel-io-0.6.6/docs/Makefile0000644000175100001710000001520300000000000015532 0ustar00runnerdocker# Makefile for Sphinx documentation # # You can set these variables from the command line. SPHINXOPTS = SPHINXBUILD = sphinx-build PAPER = BUILDDIR = build # User-friendly check for sphinx-build ifeq ($(shell which $(SPHINXBUILD) >/dev/null 2>&1; echo $$?), 1) $(error The '$(SPHINXBUILD)' command was not found. Make sure you have Sphinx installed, then set the SPHINXBUILD environment variable to point to the full path of the '$(SPHINXBUILD)' executable. Alternatively you can add the directory with the executable to your PATH. If you don't have Sphinx installed, grab it from http://sphinx-doc.org/) endif # Internal variables. PAPEROPT_a4 = -D latex_paper_size=a4 PAPEROPT_letter = -D latex_paper_size=letter ALLSPHINXOPTS = -d $(BUILDDIR)/doctrees $(PAPEROPT_$(PAPER)) $(SPHINXOPTS) source # the i18n builder cannot share the environment and doctrees with the others I18NSPHINXOPTS = $(PAPEROPT_$(PAPER)) $(SPHINXOPTS) source .PHONY: help clean html dirhtml singlehtml pickle json htmlhelp qthelp devhelp epub latex latexpdf text man changes linkcheck doctest gettext help: @echo "Please use \`make ' where is one of" @echo " html to make standalone HTML files" @echo " dirhtml to make HTML files named index.html in directories" @echo " singlehtml to make a single large HTML file" @echo " pickle to make pickle files" @echo " json to make JSON files" @echo " htmlhelp to make HTML files and a HTML help project" @echo " qthelp to make HTML files and a qthelp project" @echo " devhelp to make HTML files and a Devhelp project" @echo " epub to make an epub" @echo " latex to make LaTeX files, you can set PAPER=a4 or PAPER=letter" @echo " latexpdf to make LaTeX files and run them through pdflatex" @echo " latexpdfja to make LaTeX files and run them through platex/dvipdfmx" @echo " text to make text files" @echo " man to make manual pages" @echo " texinfo to make Texinfo files" @echo " info to make Texinfo files and run them through makeinfo" @echo " gettext to make PO message catalogs" @echo " changes to make an overview of all changed/added/deprecated items" @echo " xml to make Docutils-native XML files" @echo " pseudoxml to make pseudoxml-XML files for display purposes" @echo " linkcheck to check all external links for integrity" @echo " doctest to run all doctests embedded in the documentation (if enabled)" clean: rm -rf $(BUILDDIR)/* html: $(SPHINXBUILD) -b html $(ALLSPHINXOPTS) $(BUILDDIR)/html @echo @echo "Build finished. The HTML pages are in $(BUILDDIR)/html." dirhtml: $(SPHINXBUILD) -b dirhtml $(ALLSPHINXOPTS) $(BUILDDIR)/dirhtml @echo @echo "Build finished. The HTML pages are in $(BUILDDIR)/dirhtml." singlehtml: $(SPHINXBUILD) -b singlehtml $(ALLSPHINXOPTS) $(BUILDDIR)/singlehtml @echo @echo "Build finished. The HTML page is in $(BUILDDIR)/singlehtml." pickle: $(SPHINXBUILD) -b pickle $(ALLSPHINXOPTS) $(BUILDDIR)/pickle @echo @echo "Build finished; now you can process the pickle files." json: $(SPHINXBUILD) -b json $(ALLSPHINXOPTS) $(BUILDDIR)/json @echo @echo "Build finished; now you can process the JSON files." htmlhelp: $(SPHINXBUILD) -b htmlhelp $(ALLSPHINXOPTS) $(BUILDDIR)/htmlhelp @echo @echo "Build finished; now you can run HTML Help Workshop with the" \ ".hhp project file in $(BUILDDIR)/htmlhelp." qthelp: $(SPHINXBUILD) -b qthelp $(ALLSPHINXOPTS) $(BUILDDIR)/qthelp @echo @echo "Build finished; now you can run "qcollectiongenerator" with the" \ ".qhcp project file in $(BUILDDIR)/qthelp, like this:" @echo "# qcollectiongenerator $(BUILDDIR)/qthelp/pyexcel_io.qhcp" @echo "To view the help file:" @echo "# assistant -collectionFile $(BUILDDIR)/qthelp/pyexcel_io.qhc" devhelp: $(SPHINXBUILD) -b devhelp $(ALLSPHINXOPTS) $(BUILDDIR)/devhelp @echo @echo "Build finished." @echo "To view the help file:" @echo "# mkdir -p $$HOME/.local/share/devhelp/pyexcel_io" @echo "# ln -s $(BUILDDIR)/devhelp $$HOME/.local/share/devhelp/pyexcel_io" @echo "# devhelp" epub: $(SPHINXBUILD) -b epub $(ALLSPHINXOPTS) $(BUILDDIR)/epub @echo @echo "Build finished. The epub file is in $(BUILDDIR)/epub." latex: $(SPHINXBUILD) -b latex $(ALLSPHINXOPTS) $(BUILDDIR)/latex @echo @echo "Build finished; the LaTeX files are in $(BUILDDIR)/latex." @echo "Run \`make' in that directory to run these through (pdf)latex" \ "(use \`make latexpdf' here to do that automatically)." latexpdf: $(SPHINXBUILD) -b latex $(ALLSPHINXOPTS) $(BUILDDIR)/latex @echo "Running LaTeX files through pdflatex..." $(MAKE) -C $(BUILDDIR)/latex all-pdf @echo "pdflatex finished; the PDF files are in $(BUILDDIR)/latex." latexpdfja: $(SPHINXBUILD) -b latex $(ALLSPHINXOPTS) $(BUILDDIR)/latex @echo "Running LaTeX files through platex and dvipdfmx..." $(MAKE) -C $(BUILDDIR)/latex all-pdf-ja @echo "pdflatex finished; the PDF files are in $(BUILDDIR)/latex." text: $(SPHINXBUILD) -b text $(ALLSPHINXOPTS) $(BUILDDIR)/text @echo @echo "Build finished. The text files are in $(BUILDDIR)/text." man: $(SPHINXBUILD) -b man $(ALLSPHINXOPTS) $(BUILDDIR)/man @echo @echo "Build finished. The manual pages are in $(BUILDDIR)/man." texinfo: $(SPHINXBUILD) -b texinfo $(ALLSPHINXOPTS) $(BUILDDIR)/texinfo @echo @echo "Build finished. The Texinfo files are in $(BUILDDIR)/texinfo." @echo "Run \`make' in that directory to run these through makeinfo" \ "(use \`make info' here to do that automatically)." info: $(SPHINXBUILD) -b texinfo $(ALLSPHINXOPTS) $(BUILDDIR)/texinfo @echo "Running Texinfo files through makeinfo..." make -C $(BUILDDIR)/texinfo info @echo "makeinfo finished; the Info files are in $(BUILDDIR)/texinfo." gettext: $(SPHINXBUILD) -b gettext $(I18NSPHINXOPTS) $(BUILDDIR)/locale @echo @echo "Build finished. The message catalogs are in $(BUILDDIR)/locale." changes: $(SPHINXBUILD) -b changes $(ALLSPHINXOPTS) $(BUILDDIR)/changes @echo @echo "The overview file is in $(BUILDDIR)/changes." linkcheck: $(SPHINXBUILD) -b linkcheck $(ALLSPHINXOPTS) $(BUILDDIR)/linkcheck @echo @echo "Link check complete; look for any errors in the above output " \ "or in $(BUILDDIR)/linkcheck/output.txt." doctest: $(SPHINXBUILD) -b doctest $(ALLSPHINXOPTS) $(BUILDDIR)/doctest @echo "Testing of doctests in the sources finished, look at the " \ "results in $(BUILDDIR)/doctest/output.txt." xml: $(SPHINXBUILD) -b xml $(ALLSPHINXOPTS) $(BUILDDIR)/xml @echo @echo "Build finished. The XML files are in $(BUILDDIR)/xml." pseudoxml: $(SPHINXBUILD) -b pseudoxml $(ALLSPHINXOPTS) $(BUILDDIR)/pseudoxml @echo @echo "Build finished. The pseudo-XML files are in $(BUILDDIR)/pseudoxml." ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1643486950.0 pyexcel-io-0.6.6/docs/make.bat0000644000175100001710000001451400000000000015503 0ustar00runnerdocker@ECHO OFF REM Command file for Sphinx documentation if "%SPHINXBUILD%" == "" ( set SPHINXBUILD=sphinx-build ) set BUILDDIR=build set ALLSPHINXOPTS=-d %BUILDDIR%/doctrees %SPHINXOPTS% source set I18NSPHINXOPTS=%SPHINXOPTS% source if NOT "%PAPER%" == "" ( set ALLSPHINXOPTS=-D latex_paper_size=%PAPER% %ALLSPHINXOPTS% set I18NSPHINXOPTS=-D latex_paper_size=%PAPER% %I18NSPHINXOPTS% ) if "%1" == "" goto help if "%1" == "help" ( :help echo.Please use `make ^` where ^ is one of echo. html to make standalone HTML files echo. dirhtml to make HTML files named index.html in directories echo. singlehtml to make a single large HTML file echo. pickle to make pickle files echo. json to make JSON files echo. htmlhelp to make HTML files and a HTML help project echo. qthelp to make HTML files and a qthelp project echo. devhelp to make HTML files and a Devhelp project echo. epub to make an epub echo. latex to make LaTeX files, you can set PAPER=a4 or PAPER=letter echo. text to make text files echo. man to make manual pages echo. texinfo to make Texinfo files echo. gettext to make PO message catalogs echo. changes to make an overview over all changed/added/deprecated items echo. xml to make Docutils-native XML files echo. pseudoxml to make pseudoxml-XML files for display purposes echo. linkcheck to check all external links for integrity echo. doctest to run all doctests embedded in the documentation if enabled goto end ) if "%1" == "clean" ( for /d %%i in (%BUILDDIR%\*) do rmdir /q /s %%i del /q /s %BUILDDIR%\* goto end ) %SPHINXBUILD% 2> nul if errorlevel 9009 ( echo. echo.The 'sphinx-build' command was not found. Make sure you have Sphinx echo.installed, then set the SPHINXBUILD environment variable to point echo.to the full path of the 'sphinx-build' executable. Alternatively you echo.may add the Sphinx directory to PATH. echo. echo.If you don't have Sphinx installed, grab it from echo.http://sphinx-doc.org/ exit /b 1 ) if "%1" == "html" ( %SPHINXBUILD% -b html %ALLSPHINXOPTS% %BUILDDIR%/html if errorlevel 1 exit /b 1 echo. echo.Build finished. The HTML pages are in %BUILDDIR%/html. goto end ) if "%1" == "dirhtml" ( %SPHINXBUILD% -b dirhtml %ALLSPHINXOPTS% %BUILDDIR%/dirhtml if errorlevel 1 exit /b 1 echo. echo.Build finished. The HTML pages are in %BUILDDIR%/dirhtml. goto end ) if "%1" == "singlehtml" ( %SPHINXBUILD% -b singlehtml %ALLSPHINXOPTS% %BUILDDIR%/singlehtml if errorlevel 1 exit /b 1 echo. echo.Build finished. The HTML pages are in %BUILDDIR%/singlehtml. goto end ) if "%1" == "pickle" ( %SPHINXBUILD% -b pickle %ALLSPHINXOPTS% %BUILDDIR%/pickle if errorlevel 1 exit /b 1 echo. echo.Build finished; now you can process the pickle files. goto end ) if "%1" == "json" ( %SPHINXBUILD% -b json %ALLSPHINXOPTS% %BUILDDIR%/json if errorlevel 1 exit /b 1 echo. echo.Build finished; now you can process the JSON files. goto end ) if "%1" == "htmlhelp" ( %SPHINXBUILD% -b htmlhelp %ALLSPHINXOPTS% %BUILDDIR%/htmlhelp if errorlevel 1 exit /b 1 echo. echo.Build finished; now you can run HTML Help Workshop with the ^ .hhp project file in %BUILDDIR%/htmlhelp. goto end ) if "%1" == "qthelp" ( %SPHINXBUILD% -b qthelp %ALLSPHINXOPTS% %BUILDDIR%/qthelp if errorlevel 1 exit /b 1 echo. echo.Build finished; now you can run "qcollectiongenerator" with the ^ .qhcp project file in %BUILDDIR%/qthelp, like this: echo.^> qcollectiongenerator %BUILDDIR%\qthelp\pyexcel_io.qhcp echo.To view the help file: echo.^> assistant -collectionFile %BUILDDIR%\qthelp\pyexcel_io.ghc goto end ) if "%1" == "devhelp" ( %SPHINXBUILD% -b devhelp %ALLSPHINXOPTS% %BUILDDIR%/devhelp if errorlevel 1 exit /b 1 echo. echo.Build finished. goto end ) if "%1" == "epub" ( %SPHINXBUILD% -b epub %ALLSPHINXOPTS% %BUILDDIR%/epub if errorlevel 1 exit /b 1 echo. echo.Build finished. The epub file is in %BUILDDIR%/epub. goto end ) if "%1" == "latex" ( %SPHINXBUILD% -b latex %ALLSPHINXOPTS% %BUILDDIR%/latex if errorlevel 1 exit /b 1 echo. echo.Build finished; the LaTeX files are in %BUILDDIR%/latex. goto end ) if "%1" == "latexpdf" ( %SPHINXBUILD% -b latex %ALLSPHINXOPTS% %BUILDDIR%/latex cd %BUILDDIR%/latex make all-pdf cd %BUILDDIR%/.. echo. echo.Build finished; the PDF files are in %BUILDDIR%/latex. goto end ) if "%1" == "latexpdfja" ( %SPHINXBUILD% -b latex %ALLSPHINXOPTS% %BUILDDIR%/latex cd %BUILDDIR%/latex make all-pdf-ja cd %BUILDDIR%/.. echo. echo.Build finished; the PDF files are in %BUILDDIR%/latex. goto end ) if "%1" == "text" ( %SPHINXBUILD% -b text %ALLSPHINXOPTS% %BUILDDIR%/text if errorlevel 1 exit /b 1 echo. echo.Build finished. The text files are in %BUILDDIR%/text. goto end ) if "%1" == "man" ( %SPHINXBUILD% -b man %ALLSPHINXOPTS% %BUILDDIR%/man if errorlevel 1 exit /b 1 echo. echo.Build finished. The manual pages are in %BUILDDIR%/man. goto end ) if "%1" == "texinfo" ( %SPHINXBUILD% -b texinfo %ALLSPHINXOPTS% %BUILDDIR%/texinfo if errorlevel 1 exit /b 1 echo. echo.Build finished. The Texinfo files are in %BUILDDIR%/texinfo. goto end ) if "%1" == "gettext" ( %SPHINXBUILD% -b gettext %I18NSPHINXOPTS% %BUILDDIR%/locale if errorlevel 1 exit /b 1 echo. echo.Build finished. The message catalogs are in %BUILDDIR%/locale. goto end ) if "%1" == "changes" ( %SPHINXBUILD% -b changes %ALLSPHINXOPTS% %BUILDDIR%/changes if errorlevel 1 exit /b 1 echo. echo.The overview file is in %BUILDDIR%/changes. goto end ) if "%1" == "linkcheck" ( %SPHINXBUILD% -b linkcheck %ALLSPHINXOPTS% %BUILDDIR%/linkcheck if errorlevel 1 exit /b 1 echo. echo.Link check complete; look for any errors in the above output ^ or in %BUILDDIR%/linkcheck/output.txt. goto end ) if "%1" == "doctest" ( %SPHINXBUILD% -b doctest %ALLSPHINXOPTS% %BUILDDIR%/doctest if errorlevel 1 exit /b 1 echo. echo.Testing of doctests in the sources finished, look at the ^ results in %BUILDDIR%/doctest/output.txt. goto end ) if "%1" == "xml" ( %SPHINXBUILD% -b xml %ALLSPHINXOPTS% %BUILDDIR%/xml if errorlevel 1 exit /b 1 echo. echo.Build finished. The XML files are in %BUILDDIR%/xml. goto end ) if "%1" == "pseudoxml" ( %SPHINXBUILD% -b pseudoxml %ALLSPHINXOPTS% %BUILDDIR%/pseudoxml if errorlevel 1 exit /b 1 echo. echo.Build finished. The pseudo-XML files are in %BUILDDIR%/pseudoxml. goto end ) :end ././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1643486962.4422712 pyexcel-io-0.6.6/docs/source/0000755000175100001710000000000000000000000015371 5ustar00runnerdocker././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1643486962.4422712 pyexcel-io-0.6.6/docs/source/_static/0000755000175100001710000000000000000000000017017 5ustar00runnerdocker././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1643486950.0 pyexcel-io-0.6.6/docs/source/_static/theme_overrides.css0000644000175100001710000000036600000000000022722 0ustar00runnerdocker/* override table width restrictions */ .wy-table-responsive table td, .wy-table-responsive table th { white-space: normal !important; } .wy-table-responsive { margin-bottom: 24px; max-width: 100%; overflow: visible !important; }././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1643486962.4422712 pyexcel-io-0.6.6/docs/source/api/0000755000175100001710000000000000000000000016142 5ustar00runnerdocker././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1643486950.0 pyexcel-io-0.6.6/docs/source/api/pyexcel_io.get_data.rst0000644000175100001710000000014600000000000022604 0ustar00runnerdockerpyexcel\_io.get\_data ===================== .. currentmodule:: pyexcel_io .. autofunction:: get_data././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1643486950.0 pyexcel-io-0.6.6/docs/source/api/pyexcel_io.iget_data.rst0000644000175100001710000000015100000000000022751 0ustar00runnerdockerpyexcel\_io.iget\_data ====================== .. currentmodule:: pyexcel_io .. autofunction:: iget_data././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1643486950.0 pyexcel-io-0.6.6/docs/source/api/pyexcel_io.save_data.rst0000644000175100001710000000015100000000000022757 0ustar00runnerdockerpyexcel\_io.save\_data ====================== .. currentmodule:: pyexcel_io .. autofunction:: save_data././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1643486950.0 pyexcel-io-0.6.6/docs/source/common_parameters.rst0000644000175100001710000000250500000000000021640 0ustar00runnerdockerCommon parameters ================================================================================ 'library' option is added -------------------------------------------------------------------------------- In order to have overlapping plugins co-exist, 'library' option is added to get_data and save_data. get_data only parameters -------------------------------------------------------------------------------- keep_trailing_empty_cells ******************************************************************************** default: False If turned on, the return data will contain trailing empty cells. auto_dectect_datetime ******************************************************************************** The datetime formats are: #. %Y-%m-%d #. %Y-%m-%d %H:%M:%S #. %Y-%m-%d %H:%M:%S.%f Any other datetime formats will be thrown as ValueError csv only parameters -------------------------------------------------------------------------------- pep_0515_off ******************************************************************************** This is related to `PEP 0515 `_, where '_' in numeric values are considered legal in python 3.6. This behavior is not consistent along with other python versions. PEP 0515 by default is suppressed. And this flag allows you to turn it on in python 3.6. ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1643486950.0 pyexcel-io-0.6.6/docs/source/conf.py0000644000175100001710000000630600000000000016675 0ustar00runnerdocker# -*- coding: utf-8 -*- DESCRIPTION = ( 'A python library to read and write structured data in csv, zipped csv ' + 'format and to/from databases' + '' ) # Configuration file for the Sphinx documentation builder. # # This file only contains a selection of the most common options. For a full # list see the documentation: # https://www.sphinx-doc.org/en/master/usage/configuration.html # -- Path setup -------------------------------------------------------------- # If extensions (or modules to document with autodoc) are in another directory, # add these directories to sys.path here. If the directory is relative to the # documentation root, use os.path.abspath to make it absolute, like shown here. # # import os # import sys # sys.path.insert(0, os.path.abspath('.')) # -- Project information ----------------------------------------------------- project = 'pyexcel-io' copyright = '2015-2022 Onni Software Ltd.' author = 'C.W.' # The short X.Y version version = '0.6.6' # The full version, including alpha/beta/rc tags release = '0.6.6' # -- General configuration --------------------------------------------------- # Add any Sphinx extension module names here, as strings. They can be # extensions coming with Sphinx (named 'sphinx.ext.*') or your custom # ones. extensions = [ 'sphinx.ext.autosummary', 'sphinx.ext.doctest', 'sphinx.ext.intersphinx', 'sphinx.ext.viewcode', 'sphinx.ext.autodoc',] # Add any paths that contain templates here, relative to this directory. templates_path = ['_templates'] # The language for content autogenerated by Sphinx. Refer to documentation # for a list of supported languages. # # This is also used if you do content translation via gettext catalogs. # Usually you set "language" from the command line for these cases. language = 'en' # List of patterns, relative to source directory, that match files and # directories to ignore when looking for source files. # This pattern also affects html_static_path and html_extra_path. exclude_patterns = [] # -- Options for HTML output ------------------------------------------------- # The theme to use for HTML and HTML Help pages. See the documentation for # a list of builtin themes. # html_theme = 'alabaster' # Add any paths that contain custom static files (such as style sheets) here, # relative to this directory. They are copied after the builtin static files, # so a file named "default.css" will overwrite the builtin "default.css". html_static_path = ['_static'] # -- Extension configuration ------------------------------------------------- # -- Options for intersphinx extension --------------------------------------- # Example configuration for intersphinx: refer to the Python standard library. intersphinx_mapping = {'https://docs.python.org/3/': None} # TODO: html_theme not configurable upstream html_theme = 'default' def setup(app): app.add_stylesheet('theme_overrides.css') # TODO: DESCRIPTION not configurable upstream texinfo_documents = [ ('index', 'pyexcel-io', 'pyexcel-io Documentation', 'Onni Software Ltd.', 'pyexcel-io', DESCRIPTION, 'Miscellaneous'), ] intersphinx_mapping.update({ 'pyexcel': ('http://pyexcel.readthedocs.io/en/latest/', None), }) master_doc = "index" ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1643486950.0 pyexcel-io-0.6.6/docs/source/csvz.rst0000644000175100001710000000623200000000000017113 0ustar00runnerdocker==================================== File formats: .csvz and .tsvz ==================================== .. _csvz: Introduction ------------- 'csvz' and 'tsvz' are newly invented excel file formats by pyexcel. Simply put, 'csvz' is the zipped content of one or more csv file(s). 'tsvz' is the twin brother of 'csvz'. They are similiar to the implementation of xlsx format, which is a zip of excel content in xml format. The obvious tangile benefit of zipped csv over normal csv is the reduced file size. However, the drawback is the need of unzipping software. Single Sheet ------------ When a single sheet is to be saved, the resulting csvz file will be a zip file that contains one csv file bearing the name of :class:`~pyexcel.Sheet`. >>> from pyexcel_io import save_data >>> data = [[1,2,3]] >>> save_data("myfile.csvz", data) >>> import zipfile >>> zip = zipfile.ZipFile("myfile.csvz", 'r') >>> zip.namelist() ['pyexcel_sheet1.csv'] >>> zip.close() And it can be read out as well and can be saved in any other supported format. >>> from pyexcel_io import get_data >>> data = get_data("myfile.csvz") >>> import json >>> json.dumps(data) '{"pyexcel_sheet1": [[1, 2, 3]]}' Multiple Sheet Book ------------------- When multiple sheets are to be saved as a book, the resulting csvz file will be a zip file that contains each sheet as a csv file named after corresponding sheet name. >>> from pyexcel_io._compact import OrderedDict >>> content = OrderedDict() >>> content.update({ ... 'Sheet 1': ... [ ... [1.0, 2.0, 3.0], ... [4.0, 5.0, 6.0], ... [7.0, 8.0, 9.0] ... ] ... }) >>> content.update({ ... 'Sheet 2': ... [ ... ['X', 'Y', 'Z'], ... [1.0, 2.0, 3.0], ... [4.0, 5.0, 6.0] ... ] ... }) >>> content.update({ ... 'Sheet 3': ... [ ... ['O', 'P', 'Q'], ... [3.0, 2.0, 1.0], ... [4.0, 3.0, 2.0] ... ] ... }) >>> save_data("mybook.csvz", content) >>> import zipfile >>> zip = zipfile.ZipFile("mybook.csvz", 'r') >>> zip.namelist() ['Sheet 1.csv', 'Sheet 2.csv', 'Sheet 3.csv'] >>> zip.close() The csvz book can be read back with two lines of code. And once it is read out, it can be saved in any other supported format. >>> book2 = get_data("mybook.csvz") >>> json.dumps(book2) '{"Sheet 1": [[1.0, 2.0, 3.0], [4.0, 5.0, 6.0], [7.0, 8.0, 9.0]], "Sheet 2": [["X", "Y", "Z"], [1.0, 2.0, 3.0], [4.0, 5.0, 6.0]], "Sheet 3": [["O", "P", "Q"], [3.0, 2.0, 1.0], [4.0, 3.0, 2.0]]}' Open csvz without pyexcel-io ---------------------------- All you need is a unzipping software. I would recommend 7zip which is open source and is available on all available OS platforms. On latest Windows platform (windows 8), zip file is supported so just give the "csvz" file a file extension as ".zip". The file can be opened by File Explorer. .. testcode:: :hide: >>> import os >>> os.unlink("myfile.csvz") >>> os.unlink("mybook.csvz") ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1643486950.0 pyexcel-io-0.6.6/docs/source/django.rst0000644000175100001710000001353400000000000017373 0ustar00runnerdockerWorking with django database ================================================================================ .. testcode:: :hide: >>> class Attributable: ... def __init__(self, adict): ... self.mydict = adict ... ... def __getattr__(self, field): ... return self.mydict[field] ... >>> class Objects: ... def __init__(self): ... self.objs = None ... ... def bulk_create(self, objs, batch_size): ... self.objs = objs ... self.batch_size = batch_size ... ... def all(self): ... return [Attributable(o) for o in self.objs] ... >>> class Field: ... def __init__(self, name): ... self.attname = name ... >>> class Meta: ... instance = 0 ... def __init__(self): ... self.model_name = "Sheet%d" % Meta.instance ... self.concrete_fields = [] ... Meta.instance = Meta.instance + 1 ... ... def update(self, data): ... for f in data: ... self.concrete_fields.append(Field(f)) This section shows the way to to write and read from django database. Becuase it is "heavy"" to include a django site here to show you. A mocked django model is used here to demonstate it:: >>> class FakeDjangoModel: ... def __init__(self): ... self.objects = Objects() ... self._meta = Meta() ... ... def __call__(self, **keywords): ... return keywords .. note:: You can visit `django-excel documentation `_ if you would prefer a real django model to be used in tutorial. Write data to a django model -------------------------------------------------------------------------------- Let's suppose we have a django model: >>> from pyexcel_io import save_data >>> from pyexcel_io.constants import DB_DJANGO, DEFAULT_SHEET_NAME >>> from pyexcel_io.database.common import DjangoModelImporter, DjangoModelImportAdapter >>> from pyexcel_io.database.common import DjangoModelExporter, DjangoModelExportAdapter >>> model = FakeDjangoModel() Suppose you have these data:: >>> data = [ ... ["X", "Y", "Z"], ... [1, 2, 3], ... [4, 5, 6] ... ] >>> importer = DjangoModelImporter() >>> adapter = DjangoModelImportAdapter(model) >>> adapter.column_names = data[0] >>> importer.append(adapter) >>> save_data(importer, {adapter.get_name(): data[1:]}, file_type=DB_DJANGO) >>> import pprint >>> pprint.pprint(model.objects.objs) [{'X': 1, 'Y': 2, 'Z': 3}, {'X': 4, 'Y': 5, 'Z': 6}] Read data from a django model -------------------------------------------------------------------------------- .. testcode:: :hide: >>> model._meta.update(["X", "Y", "Z"]) Continue from previous example, you can read this back:: >>> from pyexcel_io import get_data >>> exporter = DjangoModelExporter() >>> adapter = DjangoModelExportAdapter(model) >>> exporter.append(adapter) >>> data = get_data(exporter, file_type=DB_DJANGO) >>> data OrderedDict([('Sheet0', [['X', 'Y', 'Z'], [1, 2, 3], [4, 5, 6]])]) Read a sub set of the columns: >>> exporter = DjangoModelExporter() >>> adapter = DjangoModelExportAdapter(model, ['X']) >>> exporter.append(adapter) >>> data = get_data(exporter, file_type=DB_DJANGO) >>> data OrderedDict([('Sheet0', [['X'], [1], [4]])]) Write data into multiple models -------------------------------------------------------------------------------- Suppose you have the following data to be stored in the database: >>> data = { ... "Sheet1": [['X', 'Y', 'Z'], [1, 4, 7], [2, 5, 8], [3, 6, 9]], ... "Sheet2": [['A', 'B', 'C'], [1, 4, 7], [2, 5, 8], [3, 6, 9]] ... } And want to save them to two django models: >>> model1 = FakeDjangoModel() >>> model2 = FakeDjangoModel() In order to store a dictionary data structure, you need to do some transformation:: >>> importer = DjangoModelImporter() >>> adapter1 = DjangoModelImportAdapter(model1) >>> adapter1.column_names = data['Sheet1'][0] >>> adapter2 = DjangoModelImportAdapter(model2) >>> adapter2.column_names = data['Sheet2'][0] >>> importer.append(adapter1) >>> importer.append(adapter2) >>> to_store = { ... adapter1.get_name(): data['Sheet1'][1:], ... adapter2.get_name(): data['Sheet2'][1:] ... } >>> save_data(importer, to_store, file_type=DB_DJANGO) >>> pprint.pprint(model1.objects.objs) [{'X': 1, 'Y': 4, 'Z': 7}, {'X': 2, 'Y': 5, 'Z': 8}, {'X': 3, 'Y': 6, 'Z': 9}] >>> pprint.pprint(model2.objects.objs) [{'A': 1, 'B': 4, 'C': 7}, {'A': 2, 'B': 5, 'C': 8}, {'A': 3, 'B': 6, 'C': 9}] .. testcode: :hide: >>> model1._meta.update(["X", "Y", "Z"]) >>> model2._meta.update(["A", "B", "C"]) Read content from multiple tables -------------------------------------------------------------------------------- Here's what you need to do: >>> exporter = DjangoModelExporter() >>> adapter1 = DjangoModelExportAdapter(model1) >>> adapter2 = DjangoModelExportAdapter(model2) >>> exporter.append(adapter1) >>> exporter.append(adapter2) >>> data = get_data(exporter, file_type=DB_DJANGO) >>> data OrderedDict([('Sheet1', [['X', 'Y', 'Z'], [1, 4, 7], [2, 5, 8], [3, 6, 9]]), ('Sheet2', [['A', 'B', 'C'], [1, 4, 7], [2, 5, 8], [3, 6, 9]])]) What if we need only a subset of each model >>> exporter = DjangoModelExporter() >>> adapter1 = DjangoModelExportAdapter(model1, ['X']) >>> adapter2 = DjangoModelExportAdapter(model2, ['A']) >>> exporter.append(adapter1) >>> exporter.append(adapter2) >>> data = get_data(exporter, file_type=DB_DJANGO) >>> data OrderedDict([('Sheet1', [['X'], [1], [2], [3]]), ('Sheet2', [['A'], [1], [2], [3]])]) ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1643486950.0 pyexcel-io-0.6.6/docs/source/extendedcsv.rst0000644000175100001710000000474400000000000020450 0ustar00runnerdockerSaving multiple sheets as CSV format ================================================================================ .. testcode:: :hide: >>> from pyexcel_io._compact import OrderedDict, StringIO Write to multiple sibling csv files ------------------------------------------------------------------------------ Here's the sample code to write a dictionary to multiple sibling csv files:: >>> from pyexcel_io import save_data >>> data = OrderedDict() # from collections import OrderedDict >>> data.update({"Sheet 1": [[1, 2, 3], [4, 5, 6]]}) >>> data.update({"Sheet 2": [["row 1", "row 2", "row 3"]]}) >>> save_data("your_file.csv", data) Read from multiple sibling csv files -------------------------------------------------------------------------------- Here's the sample code:: >>> from pyexcel_io import get_data >>> data = get_data("your_file.csv") >>> import json >>> print(json.dumps(data)) {"Sheet 1": [[1, 2, 3], [4, 5, 6]], "Sheet 2": [["row 1", "row 2", "row 3"]]} Here is what you would get:: >>> import glob >>> list = glob.glob("your_file__*.csv") >>> json.dumps(sorted(list)) '["your_file__Sheet 1__0.csv", "your_file__Sheet 2__1.csv"]' Write multiple sibling csv files to memory -------------------------------------------------------------------------------- Here's the sample code to write a dictionary of named two dimensional array into memory:: >>> from pyexcel_io import save_data >>> data = OrderedDict() >>> data.update({"Sheet 1": [[1, 2, 3], [4, 5, 6]]}) >>> data.update({"Sheet 2": [[7, 8, 9], [10, 11, 12]]}) >>> io = StringIO() >>> save_data(io, data) >>> # do something with the io >>> # In reality, you might give it to your http response >>> # object for downloading Read multiple sibling csv files from memory -------------------------------------------------------------------------------- Continue from previous example:: >>> # This is just an illustration >>> # In reality, you might deal with csv file upload >>> # where you will read from requests.FILES['YOUR_XL_FILE'] >>> data = get_data(io, multiple_sheets=True) >>> print(json.dumps(data)) {"Sheet 1": [[1, 2, 3], [4, 5, 6]], "Sheet 2": [[7, 8, 9], [10, 11, 12]]} .. testcode:: :hide: >>> import os >>> if os.path.exists("your_file.csv"): ... os.unlink("your_file.csv") >>> os.unlink("your_file__Sheet 1__0.csv") >>> os.unlink("your_file__Sheet 2__1.csv") ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1643486950.0 pyexcel-io-0.6.6/docs/source/extensions.rst0000644000175100001710000001202300000000000020320 0ustar00runnerdockerExtend pyexcel-io for other excel or tabular formats ================================================================================ You are welcome to extend pyexcel-io to read and write more tabular formats. No. 1 rule, your plugin must have a prefix 'pyexcel_' in its module path. For example, `pyexcel-xls` has 'pyexcel_xls' as its module path. Otherwise, pyexcel-io will not load your plugin. On github, you will find two examples in `examples` folder. This section explains its implementations to help you write yours. .. note:: No longer, you will need to do explicit imports for pyexcel-io extensions. Instead, you install them and manage them via pip. Simple Reader for a yaml file -------------------------------------------------------------------------------- Suppose we have a yaml file, containing a dictionary where the values are two dimensional array. The task is to write a reader plugin to pyexcel-io so that we can use get_data() to read yaml file out. .. literalinclude:: ../../examples/test.yaml :language: yaml **Implement IReader** First, let's implement reader interface: 1. `content_array` attribute, is expected to be a list of `NamedContent` 2. `read_sheet` function, read sheet content by its index. 3. `close` function, to clean up any file handle .. literalinclude:: ../../examples/custom_yaml_reader.py :language: python :lines: 19-33 **Implement ISheet** `YourSingleSheet` makes this simple task complex in order to show case its inner workings. Two abstract functions require implementation: 1. `row_iterator`: should return a row: either content arary or content index as long as `column_iterator` can use it to return the cell value. 2. `column_iterator`: should iterate cell value from the given row. .. literalinclude:: ../../examples/custom_yaml_reader.py :language: python :lines: 8-16 **Plug in pyexcel-io** Last thing is to register with pyexcel-io about your new reader. `relative_plugin_class_path` meant reference from current module, how to refer to `YourReader`. `locations` meant the physical presence of the data source: "file", "memory" or "content". "file" means files on physical disk. "memory" means a file stream. "content" means a string buffer. `stream_type` meant the type of the stream: binary for BytesIO and text for StringIO. .. literalinclude:: ../../examples/custom_yaml_reader.py :language: python :lines: 36-41 Usually, this registration code was placed in __init__.py file at the top level of your extension source tree. You can take a look at any pyexcel plugins for reference. **Test your reader** Let's run the following code and see if it works. .. literalinclude:: ../../examples/custom_yaml_reader.py :language: python :lines: 43-45 You would see these in standard output: .. code-block:: bash $ python custom_yaml_reader.py OrderedDict([('sheet 1', [[1, 2, 3], [2, 3, 4]]), ('sheet 2', [['A', 'B', 'C']])]) A writer to write content in yaml -------------------------------------------------------------------------------- Now for the writer, let's write a pyexcel-io writer that write a dictionary of two dimentaional arrays back into a yaml file seen above. **Implement IWriter** Two abstract functions are required: 1. `create_sheet` creates a native sheet by sheet name, that understands how to code up the native sheet. Interestingly, it returns your sheet. 2. `close` function closes file handle if any. .. literalinclude:: ../../examples/custom_yaml_writer.py :language: python :lines: 18-30 **Implement ISheetWriter** It is imagined that you will have your own sheet writer. You simply need to figure out how to write a row. Row by row write action was already written by `ISheetWrier`. .. literalinclude:: ../../examples/custom_yaml_writer.py :language: python :lines: 7-14 **Plug in pyexcel-io** Like the reader plugin, we register a writer. .. literalinclude:: ../../examples/custom_yaml_writer.py :language: python :lines: 33-38 **Test It** Let's run the following code and please examine `mytest.yaml` yourself. .. literalinclude:: ../../examples/custom_yaml_writer.py :language: python :lines: 40-46 And you shall find a file named 'mytest.yaml': .. code-block:: bash $ cat mytest.yaml sheet 1: - - 1 - 3 - 4 - - 2 - 4 - 9 sheet 2: - - B - C - D Other pyexcel-io plugins ----------------------------------------------------------------------------- Get xls support .. code-block:: $ pip install pyexcel-xls Here's what is needed:: >>> from pyexcel_io import save_data >>> data = [[1,2,3]] >>> save_data("test.xls", data) And you can also get the data back:: >>> from pyexcel_io import get_data >>> data = get_data("test.xls") >>> data['pyexcel_sheet1'] [[1, 2, 3]] Other formats ----------------------------------------------------------------------------- As illustrated above, you can start to play with pyexcel-xlsx, pyexcel-ods and pyexcel-ods3 plugins. .. testcode:: :hide: >>> import os >>> os.unlink("test.xls") ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1643486950.0 pyexcel-io-0.6.6/docs/source/index.rst0000644000175100001710000002034700000000000017240 0ustar00runnerdocker.. pyexcel_io documentation master file, created by sphinx-quickstart on Fri May 08 09:33:05 2015. You can adapt this file completely to your liking, but it should at least contain the root `toctree` directive. `pyexcel-io` - Let you focus on data, instead of file formats ================================================================================ :Author: C.W. :Source code: http://github.com/pyexcel/pyexcel-io.git :Issues: http://github.com/pyexcel/pyexcel-io/issues :License: New BSD License :Released: |version| :Generated: |today| Introduction -------------------------------------------------------------------------------- **pyexcel-io** provides **one** application programming interface(API) to read and write data in different excel formats. It makes information processing involving excel files a simple task. The data in excel files can be turned into an ordered dictionary with least code. This library focuses on data processing using excel files as storage media hence fonts, colors and charts were not and will not be considered. It was created due to the lack of uniform programming interface to access data in different excel formats. A developer needs to use different methods of different libraries to read the same data in different excel formats, hence the resulting code is cluttered and unmaintainable. This is a challenge posed by users who do not know or care about the differences in excel file formats. Instead of educating the users about the specific excel format a data processing application supports, the library takes up the challenge and promises to support all known excel formats. All great work have done by individual library developers. This library unites only the data access API. With that said, **pyexcel-io** also bring something new on the table: :ref:`"csvz" and "tsvz"` format, new format names as of 2014. They are invented and supported by `pyexcel-io`_. Installation -------------------------------------------------------------------------------- You can install pyexcel-io via pip: .. code-block:: bash $ pip install pyexcel-io or clone it and install it: .. code-block:: bash $ git clone https://github.com/pyexcel/pyexcel-io.git $ cd pyexcel-io $ python setup.py install For individual excel file formats, please install them as you wish: .. _file-format-list: .. _a-map-of-plugins-and-file-formats: .. table:: A list of file formats supported by external plugins ======================== ======================= ================= Package name Supported file formats Dependencies ======================== ======================= ================= `pyexcel-io`_ csv, csvz [#f1]_, tsv, tsvz [#f2]_ `pyexcel-xls`_ xls, xlsx(read only), `xlrd`_, xlsm(read only) `xlwt`_ `pyexcel-xlsx`_ xlsx `openpyxl`_ `pyexcel-ods3`_ ods `pyexcel-ezodf`_, lxml `pyexcel-ods`_ ods `odfpy`_ ======================== ======================= ================= .. table:: Dedicated file reader and writers ======================== ======================= ================= Package name Supported file formats Dependencies ======================== ======================= ================= `pyexcel-xlsxw`_ xlsx(write only) `XlsxWriter`_ `pyexcel-libxlsxw`_ xlsx(write only) `libxlsxwriter`_ `pyexcel-xlsxr`_ xlsx(read only) lxml `pyexcel-xlsbr`_ xlsb(read only) pyxlsb `pyexcel-odsr`_ read only for ods, fods lxml `pyexcel-odsw`_ write only for ods loxun `pyexcel-htmlr`_ html(read only) lxml,html5lib `pyexcel-pdfr`_ pdf(read only) camelot ======================== ======================= ================= Plugin shopping guide ------------------------ Since 2020, all pyexcel-io plugins have dropped the support for python versions which are lower than 3.6. If you want to use any of those Python versions, please use pyexcel-io and its plugins versions that are lower than 0.6.0. Except csv files, xls, xlsx and ods files are a zip of a folder containing a lot of xml files The dedicated readers for excel files can stream read In order to manage the list of plugins installed, you need to use pip to add or remove a plugin. When you use virtualenv, you can have different plugins per virtual environment. In the situation where you have multiple plugins that does the same thing in your environment, you need to tell pyexcel which plugin to use per function call. For example, pyexcel-ods and pyexcel-odsr, and you want to get_array to use pyexcel-odsr. You need to append get_array(..., library='pyexcel-odsr'). .. _pyexcel-io: https://github.com/pyexcel/pyexcel-io .. _pyexcel-xls: https://github.com/pyexcel/pyexcel-xls .. _pyexcel-xlsx: https://github.com/pyexcel/pyexcel-xlsx .. _pyexcel-ods: https://github.com/pyexcel/pyexcel-ods .. _pyexcel-ods3: https://github.com/pyexcel/pyexcel-ods3 .. _pyexcel-odsr: https://github.com/pyexcel/pyexcel-odsr .. _pyexcel-odsw: https://github.com/pyexcel/pyexcel-odsw .. _pyexcel-pdfr: https://github.com/pyexcel/pyexcel-pdfr .. _pyexcel-xlsxw: https://github.com/pyexcel/pyexcel-xlsxw .. _pyexcel-libxlsxw: https://github.com/pyexcel/pyexcel-libxlsxw .. _pyexcel-xlsxr: https://github.com/pyexcel/pyexcel-xlsxr .. _pyexcel-xlsbr: https://github.com/pyexcel/pyexcel-xlsbr .. _pyexcel-htmlr: https://github.com/pyexcel/pyexcel-htmlr .. _xlrd: https://github.com/python-excel/xlrd .. _xlwt: https://github.com/python-excel/xlwt .. _openpyxl: https://bitbucket.org/openpyxl/openpyxl .. _XlsxWriter: https://github.com/jmcnamara/XlsxWriter .. _pyexcel-ezodf: https://github.com/pyexcel/pyexcel-ezodf .. _odfpy: https://github.com/eea/odfpy .. _libxlsxwriter: http://libxlsxwriter.github.io/getting_started.html .. rubric:: Footnotes .. [#f1] zipped csv file .. [#f2] zipped tsv file After that, you can start get and save data in the loaded format. There are two plugins for the same file format, e.g. pyexcel-ods3 and pyexcel-ods. If you want to choose one, please try pip uninstall the un-wanted one. And if you want to have both installed but wanted to use one of them for a function call(or file type) and the other for another function call(or file type), you can pass on "library" option to get_data and save_data, e.g. get_data(.., library='pyexcel-ods') .. note:: pyexcel-text is no longer a plugin of pyexcel-io but a direct plugin of pyexcel .. table:: Plugin compatibility table ============= ======= ======== ======= ======== ======== ======== `pyexcel-io`_ `xls`_ `xlsx`_ `ods`_ `ods3`_ `odsr`_ `xlsxw`_ ============= ======= ======== ======= ======== ======== ======== 0.6.0+ 0.5.0+ 0.5.0+ 0.5.4 0.5.3 0.5.0+ 0.5.0+ 0.5.10+ 0.5.0+ 0.5.0+ 0.5.4 0.5.3 0.5.0+ 0.5.0+ 0.5.1+ 0.5.0+ 0.5.0+ 0.5.0+ 0.5.0+ 0.5.0+ 0.5.0+ 0.4.x 0.4.x 0.4.x 0.4.x 0.4.x 0.4.x 0.4.x 0.3.0+ 0.3.0+ 0.3.0 0.3.0+ 0.3.0+ 0.3.0 0.3.0 0.2.2+ 0.2.2+ 0.2.2+ 0.2.1+ 0.2.1+ 0.0.1 0.2.0+ 0.2.0+ 0.2.0+ 0.2.0 0.2.0 0.0.1 ============= ======= ======== ======= ======== ======== ======== .. _pyexcel-io: https://github.com/pyexcel/pyexcel-io .. _xls: https://github.com/pyexcel/pyexcel-xls .. _xlsx: https://github.com/pyexcel/pyexcel-xlsx .. _xlsxw: https://github.com/pyexcel/pyexcel-xlsxw .. _odsr: https://github.com/pyexcel/pyexcel-odsr .. _ods: https://github.com/pyexcel/pyexcel-ods .. _ods3: https://github.com/pyexcel/pyexcel-ods3 .. toctree:: :caption: Migration Note :maxdepth: 2 pyinstaller .. toctree:: :caption: Tutorial :maxdepth: 2 plaincsv pagination renderer extendedcsv csvz sqlalchemy django extensions API -------------------------------------------------- .. toctree:: :maxdepth: 2 common_parameters .. currentmodule:: pyexcel_io .. autosummary:: :toctree: api/ iget_data get_data save_data Indices and tables ================== * :ref:`genindex` * :ref:`modindex` * :ref:`search` ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1643486950.0 pyexcel-io-0.6.6/docs/source/pagination.rst0000644000175100001710000000335700000000000020264 0ustar00runnerdockerRead partial data ================================================================================ When you are dealing with huge amount of data, obviously you would not like to fill up your memory with those data. Here is a the feature to support pagination of your data. .. testcode:: :hide: >>> import sys >>> if sys.version_info[0] < 3: ... from StringIO import StringIO ... else: ... from io import StringIO >>> from pyexcel_io._compact import OrderedDict Let's assume the following file is a huge csv file: .. code-block:: python >>> import datetime >>> from pyexcel_io import save_data >>> data = [ ... [1, 21, 31], ... [2, 22, 32], ... [3, 23, 33], ... [4, 24, 34], ... [5, 25, 35], ... [6, 26, 36] ... ] >>> save_data("your_file.csv", data) And let's pretend to read partial data: .. code-block:: python >>> from pyexcel_io import get_data >>> data = get_data("your_file.csv", start_row=2, row_limit=3) >>> data['your_file.csv'] [[3, 23, 33], [4, 24, 34], [5, 25, 35]] And you could as well do the same for columns: .. code-block:: python >>> data = get_data("your_file.csv", start_column=1, column_limit=2) >>> data['your_file.csv'] [[21, 31], [22, 32], [23, 33], [24, 34], [25, 35], [26, 36]] Obvious, you could do both at the same time: .. code-block:: python >>> data = get_data("your_file.csv", ... start_row=2, row_limit=3, ... start_column=1, column_limit=2) >>> data['your_file.csv'] [[23, 33], [24, 34], [25, 35]] The pagination support is available across all pyexcel-io plugins. .. testcode:: :hide: >>> import os >>> if os.path.exists("your_file.csv"): ... os.unlink("your_file.csv") ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1643486950.0 pyexcel-io-0.6.6/docs/source/plaincsv.rst0000644000175100001710000001371200000000000017746 0ustar00runnerdockerWorking with CSV format ================================================================================ Please note that csv reader load data in a lazy manner. It ignores excessive trailing cells that has None value. For example, the following csv content:: 1,2,,,,, 3,4,,,,, 5,,,,,,, would end up as:: 1,2 3,4 5, Write to a csv file -------------------------------------------------------------------------------- .. testcode:: :hide: >>> import sys >>> if sys.version_info[0] < 3: ... from StringIO import StringIO ... else: ... from io import StringIO >>> from pyexcel_io._compact import OrderedDict Here's the sample code to write an array to a csv file :: >>> import datetime >>> from pyexcel_io import save_data >>> data = [ ... [1, 2.0, 3.0], ... [ ... datetime.date(2016, 5, 4), ... datetime.datetime(2016, 5, 4, 17, 39, 12), ... datetime.datetime(2016, 5, 4, 17, 40, 12, 100) ... ] ... ] >>> save_data("your_file.csv", data) Let's verify the file content:: >>> with open("your_file.csv", "r") as csvfile: ... for line in csvfile.readlines(): ... print(line.strip()) 1,2.0,3.0 2016-05-04,2016-05-04 17:39:12,2016-05-04 17:40:12.000100 Change line endings ************************* By default, python csv module provides windows line ending '\r\n'. In order to change it, you can do: >>> save_data("your_file.csv", data, lineterminator='\n') Read from a csv file -------------------------------------------------------------------------------- And we can read the written csv file back as the following code:: >>> from pyexcel_io import get_data >>> import pprint >>> data = get_data("your_file.csv") >>> pprint.pprint(data['your_file.csv']) [[1, 2.0, 3.0], [datetime.date(2016, 5, 4), datetime.datetime(2016, 5, 4, 17, 39, 12), datetime.datetime(2016, 5, 4, 17, 40, 12, 100)]] As you can see, pyexcel-io not only reads the csv file back but also recognizes the data types: `int`, `float`, `date` and `datetime`. However, it does give your cpu some extra job. When you are handling a large csv file and the cpu budget is of your concern, you may switch off the type detection feature. For example, let's switch all off: >>> data = get_data("your_file.csv", auto_detect_float=False, auto_detect_datetime=False) >>> import json >>> json.dumps(data['your_file.csv']) '[[1, "2.0", "3.0"], ["2016-05-04", "2016-05-04 17:39:12", "2016-05-04 17:40:12.000100"]]' In addition to `auto_detect_float` and `auto_detect_datetime`, there is another flag named `auto_detect_int`, which becomes active only if `auto_detect_float` is `True`. Now, let's play a bit with `auto_detect_int`: >>> data = get_data("your_file.csv", auto_detect_int=False) >>> pprint.pprint(data['your_file.csv']) [[1.0, 2.0, 3.0], [datetime.date(2016, 5, 4), datetime.datetime(2016, 5, 4, 17, 39, 12), datetime.datetime(2016, 5, 4, 17, 40, 12, 100)]] As you see, all numeric data are identified as float type. If you looked a few paragraphs above, you would notice `auto_detect_int` affected [1, 2, ..] in the first row. Write a csv to memory -------------------------------------------------------------------------------- Here's the sample code to write a dictionary as a csv into memory:: >>> from pyexcel_io import save_data >>> data = [[1, 2, 3], [4, 5, 6]] >>> io = StringIO() >>> save_data(io, data) >>> # do something with the io >>> # In reality, you might give it to your http response >>> # object for downloading Read from a csv from memory -------------------------------------------------------------------------------- Continue from previous example: >>> # This is just an illustration >>> # In reality, you might deal with csv file upload >>> # where you will read from requests.FILES['YOUR_XL_FILE'] >>> import json >>> data = get_data(io) >>> print(json.dumps(data)) {"csv": [[1, 2, 3], [4, 5, 6]]} Encoding parameter -------------------------------------------------------------------------------- In general, if you would like to save your csv file into a custom encoding, you can specify 'encoding' parameter. Here is how you write verses of a finnish song, "Aurinko laskee länteen"[#f1]_ into a csv file .. code-block:: python >>> content = [[u'Aurinko laskee länteen', u'Näin sen ja ymmärsin sen', u'Poissa aika on rakkauden Kun aurinko laskee länteen']] >>> test_file = "test-utf16-encoding.csv" >>> save_data(test_file, content, encoding="utf-16", lineterminator="\n") In the reverse direction, if you would like to read your csv file with custom encoding back, you do the same to get_data: .. code-block:: python >>> custom_encoded_content = get_data(test_file, encoding="utf-16") >>> assert custom_encoded_content[test_file] == content .. [#f1] A finnish song that was entered in Eurovision in 1965. You can check out its lyrics at `diggiloo.net `_ Byte order mark (BOM) in csv file -------------------------------------------------------------------------------- By passing **encoding="utf-8-sig", You can write UTF-8 BOM header into your csv file. Here is an example to write a sentence of "Shui Dial Getou"[#f2] into a csv file: .. code-block:: python >>> content = [['löyly', 'löyly']] >>> test_file = "test-utf8-BOM.csv" >>> save_data(test_file, content, encoding="utf-8-sig", lineterminator="\n") When you read it back you will have to specify encoding too. .. code-block:: python >>> custom_encoded_content = get_data(test_file, encoding="utf-8-sig") >>> assert custom_encoded_content[test_file] == content .. [#f2] One of Su shi's most famous poem. Here is the `wiki link `_ .. testcode:: :hide: >>> import os >>> os.unlink("your_file.csv") >>> os.unlink("test-utf16-encoding.csv") >>> os.unlink(test_file) ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1643486950.0 pyexcel-io-0.6.6/docs/source/pyinstaller.rst0000644000175100001710000000551200000000000020474 0ustar00runnerdockerPackaging with PyInstaller ================================================================================ With pyexcel-io v0.6.0, the way to package it has been changed because plugin interface update. Built-in plugins for pyexcel-io --------------------------------- In order to package every built-in plugins of pyexcel-io, you need to specify:: --hidden-import pyexcel_io.readers.csv_in_file --hidden-import pyexcel_io.readers.csv_in_memory --hidden-import pyexcel_io.readers.csv_content --hidden-import pyexcel_io.readers.csvz --hidden-import pyexcel_io.writers.csv_in_file --hidden-import pyexcel_io.writers.csv_in_memory --hidden-import pyexcel_io.writers.csvz_writer --hidden-import pyexcel_io.database.importers.django --hidden-import pyexcel_io.database.importers.sqlalchemy --hidden-import pyexcel_io.database.exporters.django --hidden-import pyexcel_io.database.exporters.sqlalchemy With pyexcel-io v0.4.0, the way to package it has been changed because it uses lml for all plugins. Built-in plugins of pyexcel-io ------------------------------- In order to package every built-in plugins of pyexcel-io, you need to specify:: --hidden-import pyexcel_io.readers.csvr --hidden-import pyexcel_io.readers.csvz --hidden-import pyexcel_io.readers.tsv --hidden-import pyexcel_io.readers.tsvz --hidden-import pyexcel_io.writers.csvw --hidden-import pyexcel_io.writers.csvz --hidden-import pyexcel_io.writers.tsv --hidden-import pyexcel_io.writers.tsvz --hidden-import pyexcel_io.database.importers.django --hidden-import pyexcel_io.database.importers.sqlalchemy --hidden-import pyexcel_io.database.exporters.django --hidden-import pyexcel_io.database.exporters.sqlalchemy pyexcel-xlsx ---------------- In order to package pyexcel-xlsx, you need to specify:: --hidden-import pyexcel_xlsx --hidden-import pyexcel_xlsx.xlsxr --hidden-import pyexcel_xlsx.xlsxw pyexcel-xlsxw ---------------- In order to package pyexcel-xlsxw, you need to specify:: --hidden-import pyexcel_xlsxw --hidden-import pyexcel_xlsxw.xlsxw pyexcel-xls ---------------- In order to package pyexcel-xls, you need to specify:: --hidden-import pyexcel_xls --hidden-import pyexcel_xls.xlsr --hidden-import pyexcel_xls.xlsw pyexcel-ods ---------------- In order to package pyexcel-ods, you need to specify:: --hidden-import pyexcel_ods --hidden-import pyexcel_ods.odsr --hidden-import pyexcel_ods.odsw pyexcel-ods3 ---------------- In order to package pyexcel-ods3, you need to specify:: --hidden-import pyexcel_ods3 --hidden-import pyexcel_ods3.odsr --hidden-import pyexcel_ods3.odsw pyexcel-odsr ---------------- In order to package pyexcel-odsr, you need to specify:: --hidden-import pyexcel_odsr --hidden-import pyexcel_odsr.odsr ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1643486950.0 pyexcel-io-0.6.6/docs/source/renderer.rst0000644000175100001710000000254600000000000017740 0ustar00runnerdockerRendering(Formatting) the data ================================================================================ You might want to do custom rendering on your data obtained. `row_renderer` was added since version 0.2.3. Here is how you can use it. .. testcode:: :hide: >>> import sys >>> if sys.version_info[0] < 3: ... from StringIO import StringIO ... else: ... from io import StringIO >>> from pyexcel_io._compact import OrderedDict Let's assume the following file: .. code-block:: python >>> import datetime >>> from pyexcel_io import save_data >>> data = [ ... [1, 21, 31], ... [2, 22, 32], ... [3, 23, 33] ... ] >>> save_data("your_file.csv", data) And let's read them back: .. code-block:: python >>> from pyexcel_io import get_data >>> data = get_data("your_file.csv") >>> data['your_file.csv'] [[1, 21, 31], [2, 22, 32], [3, 23, 33]] And you may want use row_renderer to customize it to string: .. code-block:: python >>> def my_renderer(row): ... return [str(element) for element in row] >>> data = get_data("your_file.csv", row_renderer=my_renderer) >>> data['your_file.csv'] [['1', '21', '31'], ['2', '22', '32'], ['3', '23', '33']] .. testcode:: :hide: >>> import os >>> if os.path.exists("your_file.csv"): ... os.unlink("your_file.csv") ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1643486950.0 pyexcel-io-0.6.6/docs/source/sqlalchemy.rst0000644000175100001710000002244200000000000020271 0ustar00runnerdockerWorking with sqlalchemy ================================================================================ Suppose we have a pure sql database connection via sqlalchemy: >>> from sqlalchemy import create_engine >>> from sqlalchemy.ext.declarative import declarative_base >>> from sqlalchemy import Column , Integer, String, Float, Date >>> from sqlalchemy.orm import sessionmaker >>> engine=create_engine("sqlite:///sqlalchemy.db") >>> Base=declarative_base() >>> Session=sessionmaker(bind=engine) Write data to a database table -------------------------------------------------------------------------------- Assume we have the following database table:: >>> class Pyexcel(Base): ... __tablename__='pyexcel' ... id=Column(Integer, primary_key=True) ... name=Column(String) ... weight=Column(Float) ... birth=Column(Date) Let's clear the database and create previous table in the database:: >>> Base.metadata.create_all(engine) And suppose we have the following data structure to be saved:: >>> import datetime >>> data = [ ... ['birth', 'id', 'name', 'weight'], ... [datetime.date(2014, 11, 11), 0, 'Adam', 11.25], ... [datetime.date(2014, 11, 12), 1, 'Smith', 12.25] ... ] Here's the actual code to achieve it: >>> from pyexcel_io import save_data >>> from pyexcel_io.constants import DB_SQL, DEFAULT_SHEET_NAME >>> from pyexcel_io.database.common import SQLTableImporter, SQLTableImportAdapter >>> mysession = Session() >>> importer = SQLTableImporter(mysession) >>> adapter = SQLTableImportAdapter(Pyexcel) >>> adapter.column_names = data[0] >>> importer.append(adapter) >>> save_data(importer, {adapter.get_name(): data[1:]}, file_type=DB_SQL) Please note that, the data dict shall have table name as its key. Now let's verify the data: >>> from pyexcel_io.database.querysets import QuerysetsReader >>> query_sets=mysession.query(Pyexcel).all() >>> reader = QuerysetsReader(query_sets, data[0]) >>> results = reader.to_array() >>> import json >>> json.dumps(list(results)) '[["birth", "id", "name", "weight"], ["2014-11-11", 0, "Adam", 11.25], ["2014-11-12", 1, "Smith", 12.25]]' Read data from a database table -------------------------------------------------------------------------------- Let's use previous data for reading and see if we could get them via :meth:`~pyexcel_io.get_data` : >>> from pyexcel_io import get_data >>> from pyexcel_io.database.common import SQLTableExporter, SQLTableExportAdapter >>> exporter = SQLTableExporter(mysession) >>> adapter = SQLTableExportAdapter(Pyexcel) >>> exporter.append(adapter) >>> data = get_data(exporter, file_type=DB_SQL) >>> json.dumps(list(data['pyexcel'])) '[["birth", "id", "name", "weight"], ["2014-11-11", 0, "Adam", 11.25], ["2014-11-12", 1, "Smith", 12.25]]' Read a subset from the table: >>> exporter = SQLTableExporter(mysession) >>> adapter = SQLTableExportAdapter(Pyexcel, ['birth']) >>> exporter.append(adapter) >>> data = get_data(exporter, file_type=DB_SQL) >>> json.dumps(list(data['pyexcel'])) '[["birth"], ["2014-11-11"], ["2014-11-12"]]' Write data to multiple tables -------------------------------------------------------------------------------- Before we start, let's clear off previous table: >>> Base.metadata.drop_all(engine) Now suppose we have these more complex tables: >>> from sqlalchemy import ForeignKey, DateTime >>> from sqlalchemy.orm import relationship, backref >>> import sys >>> class Post(Base): ... __tablename__ = 'post' ... id = Column(Integer, primary_key=True) ... title = Column(String(80)) ... body = Column(String(100)) ... pub_date = Column(DateTime) ... ... category_id = Column(Integer, ForeignKey('category.id')) ... category = relationship('Category', ... backref=backref('posts', lazy='dynamic')) ... ... def __init__(self, title, body, category, pub_date=None): ... self.title = title ... self.body = body ... if pub_date is None: ... pub_date = datetime.utcnow() ... self.pub_date = pub_date ... self.category = category ... ... def __repr__(self): ... return '' % self.title ... >>> class Category(Base): ... __tablename__ = 'category' ... id = Column(Integer, primary_key=True) ... name = Column(String(50)) ... ... def __init__(self, name): ... self.name = name ... ... def __repr__(self): ... return '' % self.name ... def __str__(self): ... return self.__repr__() Let's clear the database and create previous table in the database: >>> Base.metadata.create_all(engine) Suppose we have these data: >>> data = { ... "Category":[ ... ["id", "name"], ... [1, "News"], ... [2, "Sports"] ... ], ... "Post":[ ... ["id", "title", "body", "pub_date", "category"], ... [1, "Title A", "formal", datetime.datetime(2015,1,20,23,28,29), "News"], ... [2, "Title B", "informal", datetime.datetime(2015,1,20,23,28,30), "Sports"] ... ] ... } Both table has gotten initialization functions: >>> def category_init_func(row): ... c = Category(row['name']) ... c.id = row['id'] ... return c and particularly **Post** has a foreign key to **Category**, so we need to query **Category** out and assign it to **Post** instance >>> def post_init_func(row): ... c = mysession.query(Category).filter_by(name=row['category']).first() ... p = Post(row['title'], row['body'], c, row['pub_date']) ... return p Here's the code to update both: >>> tables = { ... "Category": [Category, data['Category'][0], None, category_init_func], ... "Post": [Post, data['Post'][0], None, post_init_func] ... } >>> from pyexcel_io._compact import OrderedDict >>> importer = SQLTableImporter(mysession) >>> adapter1 = SQLTableImportAdapter(Category) >>> adapter1.column_names = data['Category'][0] >>> adapter1.row_initializer = category_init_func >>> importer.append(adapter1) >>> adapter2 = SQLTableImportAdapter(Post) >>> adapter2.column_names = data['Post'][0] >>> adapter2.row_initializer = post_init_func >>> importer.append(adapter2) >>> to_store = OrderedDict() >>> to_store.update({adapter1.get_name(): data['Category'][1:]}) >>> to_store.update({adapter2.get_name(): data['Post'][1:]}) >>> save_data(importer, to_store, file_type=DB_SQL) Let's verify what do we have in the database: >>> query_sets = mysession.query(Category).all() >>> reader = QuerysetsReader(query_sets, data['Category'][0]) >>> results = reader.to_array() >>> import json >>> json.dumps(list(results)) '[["id", "name"], [1, "News"], [2, "Sports"]]' >>> query_sets = mysession.query(Post).all() >>> reader = QuerysetsReader(query_sets, ["id", "title", "body", "pub_date"]) >>> results = reader.to_array() >>> json.dumps(list(results)) '[["id", "title", "body", "pub_date"], [1, "Title A", "formal", "2015-01-20T23:28:29"], [2, "Title B", "informal", "2015-01-20T23:28:30"]]' Skipping existing record ****************************** When you import data into a database that has data already, you can skip existing record if :class:`pyexcel_io.PyexcelSQLSkipRowException` is raised. Example can be found here in `test code `_. Update existing record *************************** When you import data into a database that has data already, you can update an existing record if you can query it from the database and set the data yourself and most importantly return it. You can find an example in `test skipping row `_ Read data from multiple tables ---------------------------------------------------------------------------------- Let's use previous data for reading and see if we could get them via :meth:`~pyexcel_io.get_data` : >>> exporter = SQLTableExporter(mysession) >>> adapter = SQLTableExportAdapter(Category) >>> exporter.append(adapter) >>> adapter = SQLTableExportAdapter(Post) >>> exporter.append(adapter) >>> data = get_data(exporter, file_type=DB_SQL) >>> json.dumps(data) '{"category": [["id", "name"], [1, "News"], [2, "Sports"]], "post": [["body", "category_id", "id", "pub_date", "title"], ["formal", 1, 1, "2015-01-20T23:28:29", "Title A"], ["informal", 2, 2, "2015-01-20T23:28:30", "Title B"]]}' What if we read a subset per each table >>> exporter = SQLTableExporter(mysession) >>> adapter = SQLTableExportAdapter(Category, ['name']) >>> exporter.append(adapter) >>> adapter = SQLTableExportAdapter(Post, ['title']) >>> exporter.append(adapter) >>> data = get_data(exporter, file_type=DB_SQL) >>> json.dumps(data) '{"category": [["name"], ["News"], ["Sports"]], "post": [["title"], ["Title A"], ["Title B"]]}' .. testcode:: :hide: >>> mysession.close() >>> import os >>> os.unlink('sqlalchemy.db') ././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1643486962.4422712 pyexcel-io-0.6.6/pyexcel_io/0000755000175100001710000000000000000000000015301 5ustar00runnerdocker././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1643486950.0 pyexcel-io-0.6.6/pyexcel_io/__init__.py0000644000175100001710000000131500000000000017412 0ustar00runnerdocker""" pyexcel_io ~~~~~~~~~~~~~~~~~~~ Uniform interface for reading/writing different excel file formats :copyright: (c) 2014-2022 by Onni Software Ltd. :license: New BSD License, see LICENSE for more details """ import logging import pyexcel_io.plugins as plugins from .io import get_data, iget_data, save_data # noqa from ._compact import NullHandler logging.getLogger(__name__).addHandler(NullHandler()) # noqa BLACK_LIST = [__name__, "pyexcel_webio", "pyexcel_text"] WHITE_LIST = [ "pyexcel_io.readers", "pyexcel_io.writers", "pyexcel_io.database", ] PREFIX_PATTERN = "^pyexcel_.*$" plugins.load_plugins( PREFIX_PATTERN, __path__, BLACK_LIST, WHITE_LIST # noqa: F821 ) ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1643486950.0 pyexcel-io-0.6.6/pyexcel_io/_compact.py0000644000175100001710000000164400000000000017445 0ustar00runnerdocker""" pyexcel_io._compact ~~~~~~~~~~~~~~~~~~~ Compatibles :copyright: (c) 2014-2022 by Onni Software Ltd. :license: New BSD License, see LICENSE for more details """ import sys import logging from io import BytesIO, StringIO # noqa: F401 from collections import OrderedDict # noqa: F401 try: from logging import NullHandler except ImportError: class NullHandler(logging.Handler): def emit(self, record): pass text_type = str irange = range PY2 = sys.version[0] == 2 def isstream(instance): """check if a instance is a stream""" try: import mmap i_am_not_mmap_obj = not isinstance(instance, mmap.mmap) except ImportError: # Python 2.6 or Google App Engine i_am_not_mmap_obj = True return hasattr(instance, "read") and i_am_not_mmap_obj def is_string(atype): """find out if a type is str or not""" return atype == str ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1643486950.0 pyexcel-io-0.6.6/pyexcel_io/book.py0000644000175100001710000001556100000000000016615 0ustar00runnerdocker""" pyexcel_io.book ~~~~~~~~~~~~~~~~~~~ The io interface to file extensions :copyright: (c) 2014-2022 by Onni Software Ltd. :license: New BSD License, see LICENSE for more details """ import warnings import pyexcel_io.manager as manager from pyexcel_io._compact import OrderedDict, isstream from .constants import MESSAGE_ERROR_03, MESSAGE_WRONG_IO_INSTANCE DEPRECATED_SINCE_0_6_0 = ( "Deprecated since v0.6.0! " + "Although backward compatibility is preserved, " + "it is recommended to upgrade to get new features." ) class RWInterface(object): """ The common methods for book reader and writer """ stream_type = None def __init__(self): warnings.warn(DEPRECATED_SINCE_0_6_0) self._file_type = None def open(self, file_name, **keywords): """open a file for read or write""" raise NotImplementedError("Please implement this method") def open_stream(self, file_stream, **keywords): """open a file stream for read or write""" raise NotImplementedError("Please implement this method") def open_content(self, file_stream, **keywords): """open a file content for read or write""" raise NotImplementedError("Please implement this method") def set_type(self, file_type): """ set the file type for the instance file type is needed when a third party library could handle more than one file type""" self._file_type = file_type def close(self): """ close the file handle if necessary """ pass # implement context manager def __enter__(self): return self def __exit__(self, a_type, value, traceback): self.close() class BookReader(RWInterface): """ Standard book reader """ def __init__(self): super(BookReader, self).__init__() self._file_name = None self._file_stream = None self._keywords = None self._native_book = None def open(self, file_name, **keywords): """ open a file with unlimited keywords keywords are passed on to individual readers """ self._file_name = file_name self._keywords = keywords def open_stream(self, file_stream, **keywords): """ open a file with unlimited keywords for reading keywords are passed on to individual readers """ if isstream(file_stream): from io import UnsupportedOperation try: file_stream.seek(0) except UnsupportedOperation: # python 3 file_stream = _convert_content_to_stream( file_stream.read(), self._file_type ) self._file_stream = file_stream self._keywords = keywords else: raise IOError(MESSAGE_WRONG_IO_INSTANCE) def open_content(self, file_content, **keywords): """ read file content as if it is a file stream with unlimited keywords for reading keywords are passed on to individual readers """ file_stream = _convert_content_to_stream(file_content, self._file_type) self.open_stream(file_stream, **keywords) def read_sheet_by_name(self, sheet_name): """ read a named sheet from a excel data book """ named_contents = [ content for content in self._native_book if content.name == sheet_name ] if len(named_contents) == 1: return {named_contents[0].name: self.read_sheet(named_contents[0])} else: raise ValueError("Cannot find sheet %s" % sheet_name) def read_sheet_by_index(self, sheet_index): """ read an indexed sheet from a excel data book """ try: sheet = self._native_book[sheet_index] return {sheet.name: self.read_sheet(sheet)} except IndexError: self.close() raise def read_all(self): """ read everything from a excel data book """ result = OrderedDict() for sheet in self._native_book: result[sheet.name] = self.read_sheet(sheet) return result def read_many(self, sheets): """ read everything from a excel data book """ result = OrderedDict() for sheet in sheets: if isinstance(sheet, int): result.update(self.read_sheet_by_index(sheet)) else: result.update(self.read_sheet_by_name(sheet)) return result def read_sheet(self, native_sheet): """ Return a context specific sheet from a native sheet """ raise NotImplementedError("Please implement this method") class BookWriter(RWInterface): """ Standard book writer """ def __init__(self): super(BookWriter, self).__init__() self._file_alike_object = None self._keywords = None def open(self, file_name, **keywords): """ open a file with unlimited keywords for writing keywords are passed on to individual writers """ self._file_alike_object = file_name self._keywords = keywords def open_stream(self, file_stream, **keywords): """ open a file stream with unlimited keywords for writing keywords are passed on to individual writers """ if not isstream(file_stream): raise IOError(MESSAGE_ERROR_03) self.open(file_stream, **keywords) def open_content(self, file_stream, **keywords): """open a file content for read or write""" raise Exception("Normal writer would not need this interface") def write(self, incoming_dict): """ write a dictionary into an excel file """ for sheet_name in incoming_dict: sheet_writer = self.create_sheet(sheet_name) if sheet_writer: sheet_writer.write_array(incoming_dict[sheet_name]) sheet_writer.close() else: raise Exception("Cannot create a sheet writer!") def create_sheet(self, sheet_name): """ implement this method for easy extension """ raise NotImplementedError("Please implement create_sheet()") def _convert_content_to_stream(file_content, file_type): stream = manager.get_io(file_type) target_content_type = manager.get_io_type(file_type) needs_encode = target_content_type == "bytes" and not isinstance( file_content, bytes ) needs_decode = target_content_type == "string" and isinstance( file_content, bytes ) if needs_encode: file_content = file_content.encode("utf-8") elif needs_decode: file_content = file_content.decode("utf-8") stream.write(file_content) stream.seek(0) return stream ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1643486950.0 pyexcel-io-0.6.6/pyexcel_io/constants.py0000644000175100001710000000416500000000000017675 0ustar00runnerdocker""" pyexcel_io.constants ~~~~~~~~~~~~~~~~~~~ Constants appeared in pyexcel :copyright: (c) 2014-2022 by Onni Software Ltd. :license: New BSD License """ # flake8: noqa DEFAULT_NAME = "pyexcel" DEFAULT_SHEET_NAME = "%s_sheet1" % DEFAULT_NAME DEFAULT_PLUGIN_NAME = "__%s_io_plugins__" % DEFAULT_NAME MESSAGE_INVALID_PARAMETERS = "Invalid parameters" MESSAGE_ERROR_02 = "No content, file name. Nothing is given" MESSAGE_ERROR_03 = "cannot handle unknown content" MESSAGE_WRONG_IO_INSTANCE = "Wrong io instance is passed for your file format." MESSAGE_FILE_NAME_SHOULD_BE_STRING = "file_name should be a string" MESSAGE_CANNOT_WRITE_STREAM_FORMATTER = ( "Cannot write content of file type %s to stream" ) MESSAGE_CANNOT_READ_STREAM_FORMATTER = ( "Cannot read content of file type %s from stream" ) MESSAGE_CANNOT_WRITE_FILE_TYPE_FORMATTER = ( "Cannot write content of file type %s to file %s" ) MESSAGE_CANNOT_READ_FILE_TYPE_FORMATTER = ( "Cannot read content of file type %s from file %s" ) MESSAGE_LOADING_FORMATTER = ( "The plugin for file type %s is not installed. Please install %s" ) MESSAGE_NOT_FILE_FORMATTER = "%s is not a file" MESSAGE_FILE_DOES_NOT_EXIST = "%s does not exist" MESSAGE_EMPTY_ARRAY = "One empty row is found" MESSAGE_IGNORE_ROW = "One row is ignored" MESSAGE_DB_EXCEPTION = """ Warning: Bulk insertion got below exception. Trying to do it one by one slowly.""" FILE_FORMAT_CSV = "csv" FILE_FORMAT_TSV = "tsv" FILE_FORMAT_CSVZ = "csvz" FILE_FORMAT_TSVZ = "tsvz" FILE_FORMAT_ODS = "ods" FILE_FORMAT_XLS = "xls" FILE_FORMAT_XLSX = "xlsx" FILE_FORMAT_XLSM = "xlsm" FILE_FORMAT_XLSB = "xlsb" FILE_FORMAT_HTML = "html" FILE_FORMAT_PDF = "pdf" DB_SQL = "sql" DB_DJANGO = "django" DB_QUERYSET = "queryset" KEYWORD_TSV_DIALECT = "excel-tab" KEYWORD_LINE_TERMINATOR = "lineterminator" SKIP_DATA = -1 TAKE_DATA = 0 STOP_ITERATION = 1 DEFAULT_MULTI_CSV_SEPARATOR = "__" SEPARATOR_FORMATTER = "---%s---" % DEFAULT_NAME + "%s" SEPARATOR_MATCHER = "---%s:(.*)---" % DEFAULT_NAME DEFAULT_CSV_STREAM_FILE_FORMATTER = "---%s:" % DEFAULT_NAME + "%s---%s" DEFAULT_CSV_NEWLINE = "\r\n" MAX_INTEGER = 999999999999999 ././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1643486962.4422712 pyexcel-io-0.6.6/pyexcel_io/database/0000755000175100001710000000000000000000000017045 5ustar00runnerdocker././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1643486950.0 pyexcel-io-0.6.6/pyexcel_io/database/__init__.py0000644000175100001710000000217700000000000021165 0ustar00runnerdocker""" pyexcel_io.database ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ database data importer and exporter :copyright: (c) 2014-2022 by Onni Software Ltd. :license: New BSD License, see LICENSE for more details """ from pyexcel_io.plugins import IOPluginInfoChainV2 from pyexcel_io.constants import DB_SQL, DB_DJANGO, DB_QUERYSET IOPluginInfoChainV2(__name__).add_a_reader( relative_plugin_class_path="exporters.queryset.QueryReader", locations=["file", "memory", "content"], file_types=[DB_QUERYSET], ).add_a_reader( relative_plugin_class_path="exporters.django.DjangoBookReader", locations=["file", "memory", "content"], file_types=[DB_DJANGO], ).add_a_writer( relative_plugin_class_path="importers.django.DjangoBookWriter", locations=["file", "content", "memory"], file_types=[DB_DJANGO], ).add_a_reader( relative_plugin_class_path="exporters.sqlalchemy.SQLBookReader", locations=["file", "memory", "content"], file_types=[DB_SQL], ).add_a_writer( relative_plugin_class_path="importers.sqlalchemy.SQLBookWriter", locations=["file", "content", "memory"], file_types=[DB_SQL], ) ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1643486950.0 pyexcel-io-0.6.6/pyexcel_io/database/common.py0000644000175100001710000001207700000000000020716 0ustar00runnerdocker""" pyexcel_io.database.common ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Common classes shared among database importers and exporters :copyright: (c) 2014-2022 by Onni Software Ltd. :license: New BSD License, see LICENSE for more details """ class DjangoModelExportAdapter(object): """django export parameter holder""" def __init__(self, model, export_columns=None): self.model = model self.export_columns = export_columns @property def name(self): """get database table name""" return self.get_name() def get_name(self): """get database table name""" return self.model._meta.model_name class DjangoModelImportAdapter(DjangoModelExportAdapter): """parameter holder for django data import""" class InOutParameter(object): """local class to manipulate variable io""" def __init__(self): self.output = None self.input = None def __init__(self, model): DjangoModelExportAdapter.__init__(self, model) self._column_names = self.InOutParameter() self._column_name_mapping_dict = self.InOutParameter() self._row_initializer = self.InOutParameter() self._process_parameters() @property def row_initializer(self): """contructor for a database table entry""" return self._row_initializer.output @property def column_names(self): """the desginated database column names""" return self._column_names.output @property def column_name_mapping_dict(self): """if not the same, a mapping dictionary is looked up""" return self._column_name_mapping_dict.output @row_initializer.setter def row_initializer(self, a_function): """set the contructor""" self._row_initializer.input = a_function self._process_parameters() @column_names.setter def column_names(self, column_names): """set the column names""" self._column_names.input = column_names self._process_parameters() @column_name_mapping_dict.setter def column_name_mapping_dict(self, mapping_dict): """set the mapping dict""" self._column_name_mapping_dict.input = mapping_dict self._process_parameters() def _process_parameters(self): if self._row_initializer.input is None: self._row_initializer.output = None else: self._row_initializer.output = self._row_initializer.input if isinstance(self._column_name_mapping_dict.input, list): self._column_names.output = self._column_name_mapping_dict.input self._column_name_mapping_dict.output = None elif isinstance(self._column_name_mapping_dict.input, dict): if self._column_names.input: self._column_names.output = [] indices = [] for index, name in enumerate(self._column_names.input): if name in self._column_name_mapping_dict.input: self._column_names.output.append( self._column_name_mapping_dict.input[name] ) indices.append(index) self._column_name_mapping_dict.output = indices if self._column_names.output is None: self._column_names.output = self._column_names.input class DjangoModelExporter(object): """public interface for django model export""" def __init__(self): self.adapters = [] def append(self, import_adapter): """store model parameter for more than one model""" self.adapters.append(import_adapter) class DjangoModelImporter(object): """public interface for django model import""" def __init__(self): self._adapters = {} def append(self, import_adapter): """store model parameter for more than one model""" self._adapters[import_adapter.get_name()] = import_adapter def get(self, name): """get a parameter out""" return self._adapters.get(name, None) class SQLTableExportAdapter(DjangoModelExportAdapter): """parameter holder for sql table data export""" def __init__(self, model, export_columns=None): DjangoModelExportAdapter.__init__(self, model, export_columns) self.table = model def get_name(self): return getattr(self.table, "__tablename__", None) class SQLTableImportAdapter(DjangoModelImportAdapter): """parameter holder for sqlalchemy table import""" def __init__(self, model): DjangoModelImportAdapter.__init__(self, model) self.table = model def get_name(self): return getattr(self.table, "__tablename__", None) class SQLTableExporter(DjangoModelExporter): """public interface for sql table export""" def __init__(self, session): DjangoModelExporter.__init__(self) self.session = session class SQLTableImporter(DjangoModelImporter): """public interface to do data import via sqlalchemy""" def __init__(self, session): DjangoModelImporter.__init__(self) self.session = session ././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1643486962.4422712 pyexcel-io-0.6.6/pyexcel_io/database/exporters/0000755000175100001710000000000000000000000021100 5ustar00runnerdocker././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1643486950.0 pyexcel-io-0.6.6/pyexcel_io/database/exporters/__init__.py0000644000175100001710000000000000000000000023177 0ustar00runnerdocker././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1643486950.0 pyexcel-io-0.6.6/pyexcel_io/database/exporters/django.py0000644000175100001710000000252200000000000022715 0ustar00runnerdocker""" pyexcel_io.database.django ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The lower level handler for django import and export :copyright: (c) 2014-2022 by Onni Software Ltd. :license: New BSD License, see LICENSE for more details """ from pyexcel_io.plugin_api import IReader from pyexcel_io.database.querysets import QuerysetsReader class DjangoModelReader(QuerysetsReader): """Read from django model""" def __init__(self, model, export_columns=None, **keywords): self.__model = model if export_columns: column_names = export_columns else: column_names = sorted( [field.attname for field in self.__model._meta.concrete_fields] ) QuerysetsReader.__init__( self, self.__model.objects.all(), column_names, **keywords ) class DjangoBookReader(IReader): """read django models""" def __init__(self, exporter, _, **keywords): self.exporter = exporter self.keywords = keywords self.content_array = self.exporter.adapters def read_sheet(self, native_sheet_index): native_sheet = self.content_array[native_sheet_index] reader = DjangoModelReader( native_sheet.model, export_columns=native_sheet.export_columns ) return reader def close(self): pass ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1643486950.0 pyexcel-io-0.6.6/pyexcel_io/database/exporters/queryset.py0000644000175100001710000000107500000000000023336 0ustar00runnerdockerfrom pyexcel_io.plugin_api import IReader from pyexcel_io.database.querysets import QuerysetsReader class QueryReader(IReader): def __init__(self, query_sets, _, column_names=None, **keywords): self.query_sets = query_sets self.column_names = column_names self.keywords = keywords self.content_array = [ QuerysetsReader( self.query_sets, self.column_names, **self.keywords ) ] def read_sheet(self, index): return self.content_array[index] def close(self): pass ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1643486950.0 pyexcel-io-0.6.6/pyexcel_io/database/exporters/sqlalchemy.py0000644000175100001710000000303100000000000023611 0ustar00runnerdocker""" pyexcel_io.database.sql ~~~~~~~~~~~~~~~~~~~~~~~~~ The lower level handler for database import and export :copyright: (c) 2014-2022 by Onni Software Ltd. :license: New BSD License, see LICENSE for more details """ from pyexcel_io.plugin_api import IReader from pyexcel_io.database.querysets import QuerysetsReader class SQLTableReader(QuerysetsReader): """Read a table""" def __init__(self, session, table, export_columns=None, **keywords): everything = session.query(table).all() column_names = None if export_columns: column_names = export_columns else: if len(everything) > 0: column_names = sorted( [ column for column in everything[0].__dict__ if column != "_sa_instance_state" ] ) QuerysetsReader.__init__(self, everything, column_names, **keywords) class SQLBookReader(IReader): """read a table via sqlalchemy""" def __init__(self, exporter, _, **keywords): self.__exporter = exporter self.content_array = self.__exporter.adapters self.keywords = keywords def read_sheet(self, native_sheet_index): native_sheet = self.content_array[native_sheet_index] reader = SQLTableReader( self.__exporter.session, native_sheet.table, native_sheet.export_columns, ) return reader def close(self): pass ././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1643486962.4462712 pyexcel-io-0.6.6/pyexcel_io/database/importers/0000755000175100001710000000000000000000000021071 5ustar00runnerdocker././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1643486950.0 pyexcel-io-0.6.6/pyexcel_io/database/importers/__init__.py0000644000175100001710000000000000000000000023170 0ustar00runnerdocker././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1643486950.0 pyexcel-io-0.6.6/pyexcel_io/database/importers/django.py0000644000175100001710000000541700000000000022714 0ustar00runnerdocker""" pyexcel_io.database.django ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The lower level handler for django import and export :copyright: (c) 2014-2022 by Onni Software Ltd. :license: New BSD License, see LICENSE for more details """ import logging import pyexcel_io.constants as constants from pyexcel_io.utils import is_empty_array, swap_empty_string_for_none from pyexcel_io.plugin_api import IWriter, ISheetWriter log = logging.getLogger(__name__) class DjangoModelWriter(ISheetWriter): """import data into a django model""" def __init__(self, importer, adapter, batch_size=None, bulk_save=True): self.batch_size = batch_size self.model = adapter.model self.column_names = adapter.column_names self.mapdict = adapter.column_name_mapping_dict self.initializer = adapter.row_initializer self.objs = [] self.bulk_save = bulk_save self.adapter = adapter def write_row(self, array): if is_empty_array(array): log.warning(constants.MESSAGE_EMPTY_ARRAY) else: new_array = swap_empty_string_for_none(array) if self.mapdict: another_new_array = [] for index, element in enumerate(new_array): if index in self.mapdict: another_new_array.append(element) new_array = another_new_array model_to_be_created = new_array if self.initializer is not None: model_to_be_created = self.initializer(new_array) if model_to_be_created: row = dict(zip(self.column_names, model_to_be_created)) self.objs.append(self.model(**row)) # else # skip the row def close(self): if self.bulk_save: self.model.objects.bulk_create( self.objs, batch_size=self.batch_size ) else: for an_object in self.objs: an_object.save() class DjangoBookWriter(IWriter): """write data into django models""" def __init__(self, exporter, _, **keywords): self.importer = exporter self._keywords = keywords def create_sheet(self, sheet_name): sheet_writer = None model = self.importer.get(sheet_name) if model: sheet_writer = DjangoModelWriter( self.importer, model, batch_size=self._keywords.get("batch_size", None), bulk_save=self._keywords.get("bulk_save", True), ) else: raise Exception( "Sheet: %s does not match any given models." % sheet_name + "Please be aware of case sensitivity." ) return sheet_writer def close(self): pass ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1643486950.0 pyexcel-io-0.6.6/pyexcel_io/database/importers/sqlalchemy.py0000644000175100001710000000620300000000000023606 0ustar00runnerdocker""" pyexcel_io.database.sql ~~~~~~~~~~~~~~~~~~~~~~~~~ The lower level handler for database import and export :copyright: (c) 2014-2022 by Onni Software Ltd. :license: New BSD License, see LICENSE for more details """ import logging import pyexcel_io.constants as constants from pyexcel_io.utils import is_empty_array, swap_empty_string_for_none from pyexcel_io.plugin_api import IWriter, ISheetWriter LOG = logging.getLogger(__name__) class PyexcelSQLSkipRowException(Exception): """ Raised this exception to skipping a row while data import """ pass class SQLTableWriter(ISheetWriter): """Write to a table""" def __init__( self, importer, adapter, auto_commit=True, bulk_size=1000, **keywords ): self.__auto_commit = auto_commit self.__count = 0 self.__bulk_size = bulk_size self.adapter = adapter self.importer = importer def write_row(self, array): if is_empty_array(array): LOG.warning(constants.MESSAGE_EMPTY_ARRAY) else: new_array = swap_empty_string_for_none(array) try: self._write_row(new_array) except PyexcelSQLSkipRowException: LOG.info(constants.MESSAGE_IGNORE_ROW) LOG.info(new_array) def _write_row(self, array): new_array = array if self.adapter.column_name_mapping_dict: another_new_array = [] for index, element in enumerate(new_array): if index in self.adapter.column_name_mapping_dict: another_new_array.append(element) new_array = another_new_array row = dict(zip(self.adapter.column_names, new_array)) obj = None if self.adapter.row_initializer: # allow initinalizer to return None # if skipping is needed obj = self.adapter.row_initializer(row) if obj is None: obj = self.adapter.table() for name in self.adapter.column_names: setattr(obj, name, row[name]) self.importer.session.add(obj) if self.__auto_commit and self.__bulk_size != float("inf"): self.__count += 1 if self.__count % self.__bulk_size == 0: self.importer.session.commit() def close(self): if self.__auto_commit: self.importer.session.commit() class SQLBookWriter(IWriter): """write data into database tables via sqlalchemy""" def __init__(self, file_content, _, auto_commit=True, **keywords): self.__importer = file_content self.__auto_commit = auto_commit def create_sheet(self, sheet_name): sheet_writer = None adapter = self.__importer.get(sheet_name) if adapter: sheet_writer = SQLTableWriter( self.__importer, adapter, auto_commit=self.__auto_commit ) else: raise Exception( "Sheet: %s does not match any given tables." % sheet_name + "Please be aware of case sensitivity." ) return sheet_writer def close(self): pass ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1643486950.0 pyexcel-io-0.6.6/pyexcel_io/database/querysets.py0000644000175100001710000000401400000000000021462 0ustar00runnerdocker""" pyexcel_io.database.querysets ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The lower level handler for querysets :copyright: (c) 2014-2022 by Onni Software Ltd. :license: New BSD License, see LICENSE for more details """ import datetime from itertools import chain from pyexcel_io.constants import DEFAULT_SHEET_NAME from pyexcel_io.plugin_api.abstract_sheet import ISheet class QuerysetsReader(ISheet): """turn querysets into an array""" def __init__(self, query_sets, column_names): self.name = DEFAULT_SHEET_NAME self.__column_names = column_names self.__query_sets = query_sets def row_iterator(self): return chain([self.__column_names], self.__query_sets) def to_array(self): """ Convert query sets into an array """ if len(self.__query_sets) == 0: yield [] for row in self.row_iterator(): row_values = [] for value in self.column_iterator(row): row_values.append(value) yield row_values def column_iterator(self, row): if self.__column_names is None: return if isinstance(row, list): for element in row: yield element else: for column in self.__column_names: if "__" in column: value = get_complex_attribute(row, column) else: value = get_simple_attribute(row, column) yield value def get_complex_attribute(row, attribute): """recursively get an attribute""" attributes = attribute.split("__") value = row try: for attributee in attributes: value = get_simple_attribute(value, attributee) except AttributeError: value = None return value def get_simple_attribute(row, attribute): """get dotted attribute""" value = getattr(row, attribute) if isinstance(value, (datetime.date, datetime.time)): value = value.isoformat() return value ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1643486950.0 pyexcel-io-0.6.6/pyexcel_io/exceptions.py0000644000175100001710000000235400000000000020040 0ustar00runnerdocker""" pyexcel_io.exceptions ~~~~~~~~~~~~~~~~~~~~~~ all possible exceptions :copyright: (c) 2014-2022 by Onni Software Ltd. :license: New BSD License, see LICENSE for more details """ class NoSupportingPluginFound(Exception): """raised when an known file extension is seen""" pass class SupportingPluginAvailableButNotInstalled(Exception): """raised when a known plugin is not installed""" pass class IntegerAccuracyLossError(Exception): """ When an interger is greater than 999999999999999, ods loses its accuracy. from pyexcel import Sheet, get_sheet s = Sheet() s[0,0] = 999999999999999 # 15 '9's print(s) s.save_as('abc.ods') b=get_sheet(file_name='abc.ods') b[0,0] == s[0,0] s = Sheet() s[0,0] = 9999999999999999 # 16 '9's print(s) s.save_as('abc.ods') b=get_sheet(file_name='abc.ods') b[0,0] != s[0,0] """ def __init__(self, message): custom_message = ( message + "\n" + "In order to keep its accuracy, please save as string. Then " + "convert to int, long or float after the value will be read back" ) super(IntegerAccuracyLossError, self).__init__(custom_message) ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1643486950.0 pyexcel-io-0.6.6/pyexcel_io/io.py0000644000175100001710000002322100000000000016262 0ustar00runnerdocker""" pyexcel_io.io ~~~~~~~~~~~~~~~~~~~ The io interface to file extensions :copyright: (c) 2014-2022 by Onni Software Ltd. :license: New BSD License, see LICENSE for more details """ import os import warnings from types import GeneratorType from pyexcel_io import constants from pyexcel_io.reader import Reader from pyexcel_io.writer import Writer from pyexcel_io.plugins import OLD_READERS, OLD_WRITERS from pyexcel_io._compact import isstream from pyexcel_io.exceptions import ( NoSupportingPluginFound, SupportingPluginAvailableButNotInstalled, ) def iget_data(afile, file_type=None, **keywords): """Get data from an excel file source The data has not gone into memory yet. If you use dedicated partial read plugins, such as pyexcel-xlsxr, pyexcel-odsr, you will notice the memory consumption drop when you work with big files. :param afile: a file name, a file stream or actual content :param sheet_name: the name of the sheet to be loaded :param sheet_index: the index of the sheet to be loaded :param sheets: a list of sheet to be loaded :param file_type: used only when filename is not a physical file name :param force_file_type: used only when filename refers to a physical file and it is intended to open it as forced file type. :param library: explicitly name a library for use. e.g. library='pyexcel-ods' :param auto_detect_float: defaults to True :param auto_detect_int: defaults to True :param auto_detect_datetime: defaults to True :param ignore_infinity: defaults to True :param ignore_nan_text: various forms of 'NaN', 'nan' are ignored :param default_float_nan: choose one form of 'NaN', 'nan' :param pep_0515_off: turn off pep 0515. default to True. :param keep_trailing_empty_cells: keep trailing columns. default to False :param keywords: any other library specific parameters :returns: an ordered dictionary """ data, reader = _get_data( afile, file_type=file_type, streaming=True, **keywords ) return data, reader def get_data(afile, file_type=None, streaming=None, **keywords): """Get data from an excel file source :param afile: a file name, a file stream or actual content :param sheet_name: the name of the sheet to be loaded :param sheet_index: the index of the sheet to be loaded :param sheets: a list of sheet to be loaded :param file_type: used only when filename is not a physial file name :param force_file_type: used only when filename refers to a physical file and it is intended to open it as forced file type. :param streaming: toggles the type of returned data. The values of the returned dictionary remain as generator if it is set to True. Default is False. :param library: explicitly name a library for use. e.g. library='pyexcel-ods' :param auto_detect_float: defaults to True :param auto_detect_int: defaults to True :param auto_detect_datetime: defaults to True :param ignore_infinity: defaults to True :param ignore_nan_text: various forms of 'NaN', 'nan' are ignored :param default_float_nan: choose one form of 'NaN', 'nan' :param pep_0515_off: turn off pep 0515. default to True. :param keep_trailing_empty_cells: keep trailing columns. default to False :param keywords: any other library specific parameters :returns: an ordered dictionary """ if streaming is not None and streaming is True: warnings.warn("Please use iget_data instead") data, _ = _get_data( afile, file_type=file_type, streaming=False, **keywords ) return data def _get_data(afile, file_type=None, **keywords): if isstream(afile): keywords.update( dict( file_stream=afile, file_type=file_type or constants.FILE_FORMAT_CSV, ) ) else: if afile is None or file_type is None: keywords.update(dict(file_name=afile, file_type=file_type)) else: keywords.update(dict(file_content=afile, file_type=file_type)) return load_data(**keywords) def save_data(afile, data, file_type=None, **keywords): """Save data to an excel file source Your data must be a dictionary :param filename: actual file name, a file stream or actual content :param data: a dictionary but an ordered dictionary is preferred :param file_type: used only when filename is not a physial file name :param force_file_type: used only when filename refers to a physical file and it is intended to open it as forced file type. :param library: explicitly name a library for use. e.g. library='pyexcel-ods' :param keywords: any other parameters that python csv module's `fmtparams `_ """ # noqa to_store = data is_list = isinstance(data, (list, GeneratorType)) if is_list: single_sheet_in_book = True to_store = {constants.DEFAULT_SHEET_NAME: data} else: keys = list(data.keys()) single_sheet_in_book = len(keys) == 1 no_file_type = isstream(afile) and file_type is None if no_file_type: file_type = constants.FILE_FORMAT_CSV if isstream(afile): keywords.update(dict(file_stream=afile, file_type=file_type)) else: keywords.update(dict(file_name=afile, file_type=file_type)) keywords["single_sheet_in_book"] = single_sheet_in_book with get_writer(**keywords) as writer: writer.write(to_store) def load_data( file_name=None, file_content=None, file_stream=None, file_type=None, force_file_type=None, sheet_name=None, sheet_index=None, sheets=None, library=None, streaming=False, **keywords ): """Load data from any supported excel formats :param filename: actual file name, a file stream or actual content :param file_type: used only when filename is not a physial file name :param force_file_type: used only when filename refers to a physical file and it is intended to open it as forced file type. :param sheet_name: the name of the sheet to be loaded :param sheet_index: the index of the sheet to be loaded :param keywords: any other parameters """ result = {} inputs = [file_name, file_content, file_stream] number_of_none_inputs = [x for x in inputs if x is not None] if len(number_of_none_inputs) != 1: raise IOError(constants.MESSAGE_ERROR_02) if file_type is None: if force_file_type: file_type = force_file_type else: try: file_type = file_name.split(".")[-1] except AttributeError: raise Exception(constants.MESSAGE_FILE_NAME_SHOULD_BE_STRING) try: reader = OLD_READERS.get_a_plugin(file_type, library) except (NoSupportingPluginFound, SupportingPluginAvailableButNotInstalled): reader = Reader(file_type, library) try: if file_name: reader.open(file_name, **keywords) elif file_content: reader.open_content(file_content, **keywords) elif file_stream: reader.open_stream(file_stream, **keywords) else: raise IOError("Unrecognized options") if sheet_name: result = reader.read_sheet_by_name(sheet_name) elif sheet_index is not None: result = reader.read_sheet_by_index(sheet_index) elif sheets is not None: result = reader.read_many(sheets) else: result = reader.read_all() if streaming is False: for key in result.keys(): result[key] = list(result[key]) reader.close() reader = None return result, reader except NoSupportingPluginFound: if file_name: if os.path.exists(file_name): if os.path.isfile(file_name): raise else: raise IOError( constants.MESSAGE_NOT_FILE_FORMATTER % file_name ) else: raise IOError( constants.MESSAGE_FILE_DOES_NOT_EXIST % file_name ) else: raise def get_writer( file_name=None, file_stream=None, file_type=None, library=None, force_file_type=None, **keywords ): """find a suitable writer""" inputs = [file_name, file_stream] number_of_none_inputs = [x for x in inputs if x is not None] if len(number_of_none_inputs) != 1: raise IOError(constants.MESSAGE_ERROR_02) file_type_given = True if file_type is None and file_name: if force_file_type: file_type = force_file_type else: try: file_type = file_name.split(".")[-1] except AttributeError: raise Exception(constants.MESSAGE_FILE_NAME_SHOULD_BE_STRING) file_type_given = False try: writer = OLD_WRITERS.get_a_plugin(file_type, library) except (NoSupportingPluginFound, SupportingPluginAvailableButNotInstalled): writer = Writer(file_type, library) if file_name: if file_type_given: writer.open_content(file_name, **keywords) else: writer.open(file_name, **keywords) elif file_stream: writer.open_stream(file_stream, **keywords) # else: is resolved by earlier raise statement return writer # backward compactibility store_data = save_data ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1643486950.0 pyexcel-io-0.6.6/pyexcel_io/manager.py0000644000175100001710000000346300000000000017273 0ustar00runnerdocker""" pyexcel_io.manager ~~~~~~~~~~~~~~~~~~~ Control file streams :copyright: (c) 2014-2022 by Onni Software Ltd. :license: New BSD License, see LICENSE for more details """ from pyexcel_io._compact import BytesIO, StringIO MIME_TYPES = {} FILE_TYPES = () TEXT_STREAM_TYPES = [] BINARY_STREAM_TYPES = [] def register_stream_type(file_type, stream_type): """ keep track of stream type for different file formats """ if stream_type == "text": TEXT_STREAM_TYPES.append(file_type) elif stream_type == "binary": BINARY_STREAM_TYPES.append(file_type) def get_io(file_type): """A utility function to help you generate a correct io stream :param file_type: a supported file type :returns: a appropriate io stream, None otherwise """ __file_type = None if file_type: __file_type = file_type.lower() if __file_type in TEXT_STREAM_TYPES: return StringIO() elif __file_type in BINARY_STREAM_TYPES: return BytesIO() else: return None def get_io_type(file_type): """A utility function to help you generate a correct io stream :param file_type: a supported file type :returns: a appropriate io stream, None otherwise """ __file_type = None if file_type: __file_type = file_type.lower() if __file_type in TEXT_STREAM_TYPES: return "string" elif __file_type in BINARY_STREAM_TYPES: return "bytes" else: return None def register_a_file_type(file_type, stream_type, mime_type): """ keep track of file format supports by this library """ global FILE_TYPES FILE_TYPES += (file_type,) stream_type = stream_type if mime_type is not None: MIME_TYPES[file_type] = mime_type register_stream_type(file_type, stream_type) ././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1643486962.4462712 pyexcel-io-0.6.6/pyexcel_io/plugin_api/0000755000175100001710000000000000000000000017430 5ustar00runnerdocker././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1643486950.0 pyexcel-io-0.6.6/pyexcel_io/plugin_api/__init__.py0000644000175100001710000000026300000000000021542 0ustar00runnerdockerfrom .abstract_sheet import ISheet, ISheetWriter, NamedContent # noqa: F401 from .abstract_reader import IReader # noqa: F401 from .abstract_writer import IWriter # noqa: F401 ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1643486950.0 pyexcel-io-0.6.6/pyexcel_io/plugin_api/abstract_reader.py0000644000175100001710000000103600000000000023127 0ustar00runnerdockerfrom .abstract_sheet import ISheet class IReader(object): """ content_array should be a list of NamedContent where: name is the sheet name, payload is the native sheet. """ def read_sheet(self, sheet_index) -> ISheet: raise NotImplementedError("Read the sheet by index") def sheet_names(self): return [content.name for content in self.content_array] def __len__(self): return len(self.content_array) def close(self): raise NotImplementedError("Close the file") ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1643486950.0 pyexcel-io-0.6.6/pyexcel_io/plugin_api/abstract_sheet.py0000644000175100001710000000144300000000000022777 0ustar00runnerdockerclass ISheet(object): def row_iterator(self): raise NotImplementedError("iterate each row") def column_iterator(self, row): raise NotImplementedError("iterate each column at a given row") class ISheetWriter(object): def write_row(self, data_row): raise NotImplementedError("How does your sheet write a row of data") def write_array(self, table): """ For standalone usage, write an array """ for row in table: self.write_row(row) def close(self): raise NotImplementedError("How would you close your file") class NamedContent(object): """ Helper class for content that does not have a name """ def __init__(self, name, payload): self.name = name self.payload = payload ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1643486950.0 pyexcel-io-0.6.6/pyexcel_io/plugin_api/abstract_writer.py0000644000175100001710000000104600000000000023202 0ustar00runnerdockerfrom .abstract_sheet import ISheetWriter class IWriter(object): def create_sheet(self, sheet_name) -> ISheetWriter: raise NotImplementedError("Please implement a native sheet writer") def write(self, incoming_dict): for sheet_name in incoming_dict: sheet_writer = self.create_sheet(sheet_name) if sheet_writer: sheet_writer.write_array(incoming_dict[sheet_name]) sheet_writer.close() else: raise Exception("Cannot create a sheet writer!") ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1643486950.0 pyexcel-io-0.6.6/pyexcel_io/plugins.py0000644000175100001710000002045600000000000017343 0ustar00runnerdocker""" pyexcel_io.plugins ~~~~~~~~~~~~~~~~~~~ factory for getting readers and writers :copyright: (c) 2014-2022 by Onni Software Ltd. :license: New BSD License, see LICENSE for more details """ import pyexcel_io.utils as ioutils import pyexcel_io.manager as manager import pyexcel_io.constants as constants import pyexcel_io.exceptions as exceptions from lml.loader import scan_plugins_regex from lml.plugin import PluginInfo, PluginManager, PluginInfoChain ERROR_MESSAGE_FORMATTER = "one of these plugins for %s data in '%s': %s" UPGRADE_MESSAGE = "Please upgrade the plugin '%s' according to \ plugin compactibility table." READER_PLUGIN = "pyexcel-io reader" READER_PLUGIN_V2 = "pyexcel-io v2 reader" WRITER_PLUGIN = "pyexcel-io writer" WRITER_PLUGIN_V2 = "pyexcel-io v2 writer" class IOPluginInfo(PluginInfo): """Pyexcel-io plugin info description""" def tags(self): for file_type in self.file_types: yield file_type class IOPluginInfoChain(PluginInfoChain): """provide custom functions to add a reader and a writer""" def add_a_reader( self, relative_plugin_class_path=None, file_types=None, stream_type=None, ): """add pyexcle-io reader plugin info""" a_plugin_info = IOPluginInfo( READER_PLUGIN, self._get_abs_path(relative_plugin_class_path), file_types=file_types, stream_type=stream_type, ) return self.add_a_plugin_instance(a_plugin_info) def add_a_writer( self, relative_plugin_class_path=None, file_types=None, stream_type=None, ): """add pyexcle-io writer plugin info""" a_plugin_info = IOPluginInfo( WRITER_PLUGIN, self._get_abs_path(relative_plugin_class_path), file_types=file_types, stream_type=stream_type, ) return self.add_a_plugin_instance(a_plugin_info) class IOPluginInfoChainV2(PluginInfoChain): """provide custom functions to add a reader and a writer""" def add_a_reader( self, relative_plugin_class_path=None, locations=(), file_types=None, stream_type=None, ): """add pyexcle-io reader plugin info""" a_plugin_info = IOPluginInfo( READER_PLUGIN_V2, self._get_abs_path(relative_plugin_class_path), file_types=[ f"{location}-{file_type}" for file_type in file_types for location in locations ], stream_type=stream_type, ) return self.add_a_plugin_instance(a_plugin_info) def add_a_writer( self, relative_plugin_class_path=None, locations=(), file_types=(), stream_type=None, ): """add pyexcle-io writer plugin info""" a_plugin_info = IOPluginInfo( WRITER_PLUGIN_V2, self._get_abs_path(relative_plugin_class_path), file_types=[ f"{location}-{file_type}" for file_type in file_types for location in locations ], stream_type=stream_type, ) return self.add_a_plugin_instance(a_plugin_info) class IOManager(PluginManager): """Manage pyexcel-io plugins""" def __init__(self, plugin_type, known_list): PluginManager.__init__(self, plugin_type) self.known_plugins = known_list self.action = "read" if self.plugin_name == WRITER_PLUGIN: self.action = "write" def load_me_later(self, plugin_info): PluginManager.load_me_later(self, plugin_info) _do_additional_registration(plugin_info) def register_a_plugin(self, cls, plugin_info): """for dynamically loaded plugin""" PluginManager.register_a_plugin(self, cls, plugin_info) _do_additional_registration(plugin_info) def get_a_plugin(self, file_type=None, library=None, **keywords): __file_type = file_type.lower() try: plugin = self.load_me_now(__file_type, library=library) except Exception: self.raise_exception(__file_type) handler = plugin() handler.set_type(__file_type) return handler def raise_exception(self, file_type): plugins = self.known_plugins.get(file_type, None) if plugins: message = "Please install " if len(plugins) > 1: message += ERROR_MESSAGE_FORMATTER % ( self.action, file_type, ",".join(plugins), ) else: message += plugins[0] raise exceptions.SupportingPluginAvailableButNotInstalled(message) else: raise exceptions.NoSupportingPluginFound( "No suitable library found for %s" % file_type ) def get_all_formats(self): """return all supported formats""" all_formats = set( list(self.registry.keys()) + list(self.known_plugins.keys()) ) all_formats = all_formats.difference( set([constants.DB_SQL, constants.DB_DJANGO]) ) return all_formats class NewIOManager(IOManager): def load_me_later(self, plugin_info): PluginManager.load_me_later(self, plugin_info) _do_additional_registration_for_new_plugins(plugin_info) def register_a_plugin(self, cls, plugin_info): """for dynamically loaded plugin""" PluginManager.register_a_plugin(self, cls, plugin_info) _do_additional_registration_for_new_plugins(plugin_info) def get_a_plugin( self, file_type=None, location=None, library=None, **keywords ): __file_type = file_type.lower() plugin = self.load_me_now(f"{location}-{__file_type}", library=library) return plugin def raise_exception(self, file_type): file_type = file_type.split("-")[1] plugins = self.known_plugins.get(file_type, None) if plugins: message = "Please install " if len(plugins) > 1: message += ERROR_MESSAGE_FORMATTER % ( self.action, file_type, ",".join(plugins), ) else: message += plugins[0] raise exceptions.SupportingPluginAvailableButNotInstalled(message) else: raise exceptions.NoSupportingPluginFound( "No suitable library found for %s" % file_type ) def get_all_formats(self): """return all supported formats""" all_formats = set( [x.split("-")[1] for x in self.registry.keys()] + list(self.known_plugins.keys()) ) return all_formats def _do_additional_registration(plugin_info): for file_type in plugin_info.tags(): manager.register_stream_type(file_type, plugin_info.stream_type) manager.register_a_file_type(file_type, plugin_info.stream_type, None) def _do_additional_registration_for_new_plugins(plugin_info): for file_type in plugin_info.tags(): manager.register_stream_type( file_type.split("-")[1], plugin_info.stream_type ) manager.register_a_file_type( file_type.split("-")[1], plugin_info.stream_type, None ) class AllReaders: def get_all_formats(self): return OLD_READERS.get_all_formats().union( NEW_READERS.get_all_formats() ) - set([constants.DB_SQL, constants.DB_DJANGO]) class AllWriters: def get_all_formats(self): return OLD_WRITERS.get_all_formats().union( NEW_WRITERS.get_all_formats() ) - set([constants.DB_SQL, constants.DB_DJANGO]) OLD_READERS = IOManager(READER_PLUGIN, ioutils.AVAILABLE_READERS) OLD_WRITERS = IOManager(WRITER_PLUGIN, ioutils.AVAILABLE_WRITERS) NEW_WRITERS = NewIOManager(WRITER_PLUGIN_V2, ioutils.AVAILABLE_WRITERS) NEW_READERS = NewIOManager(READER_PLUGIN_V2, ioutils.AVAILABLE_READERS) READERS = AllReaders() WRITERS = AllWriters() def load_plugins(plugin_name_patterns, path, black_list, white_list): """Try to discover all pyexcel-io plugins""" scan_plugins_regex( plugin_name_patterns=plugin_name_patterns, pyinstaller_path=path, black_list=black_list, white_list=white_list, ) ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1643486950.0 pyexcel-io-0.6.6/pyexcel_io/reader.py0000644000175100001710000001012400000000000017113 0ustar00runnerdockerfrom pyexcel_io.sheet import SheetReader from pyexcel_io.plugins import NEW_READERS from pyexcel_io._compact import OrderedDict def clean_keywords(keywords): sheet_keywords = {} native_sheet_keywords = {} args_list = [ "start_row", "row_limit", "start_column", "column_limit", "skip_column_func", "skip_row_func", "skip_empty_rows", "row_renderer", "keep_trailing_empty_cells", ] for arg in keywords: if arg in args_list: sheet_keywords[arg] = keywords[arg] else: native_sheet_keywords[arg] = keywords[arg] return sheet_keywords, native_sheet_keywords class Reader(object): def __init__(self, file_type, library=None): self.file_type = file_type self.library = library self.keywords = None # if you know which reader class to use, this attribute allows # you to set reader class externally. Since there is no # so call private field in Python, I am not going to create # useless setter and getter functions like Java. # in pyexcel, this attribute is mainly used for testing self.reader_class = None def open(self, file_name, **keywords): if self.reader_class is None: self.reader_class = NEW_READERS.get_a_plugin( self.file_type, location="file", library=self.library ) self.keywords, native_sheet_keywords = clean_keywords(keywords) self.reader = self.reader_class( file_name, self.file_type, **native_sheet_keywords ) return self.reader def open_content(self, file_content, **keywords): self.keywords, native_sheet_keywords = clean_keywords(keywords) if self.reader_class is None: self.reader_class = NEW_READERS.get_a_plugin( self.file_type, location="content", library=self.library ) self.reader = self.reader_class( file_content, self.file_type, **native_sheet_keywords ) return self.reader def open_stream(self, file_stream, **keywords): self.keywords, native_sheet_keywords = clean_keywords(keywords) if self.reader_class is None: self.reader_class = NEW_READERS.get_a_plugin( self.file_type, location="memory", library=self.library ) self.reader = self.reader_class( file_stream, self.file_type, **native_sheet_keywords ) return self.reader def read_sheet_by_name(self, sheet_name): """ read a named sheet from a excel data book """ sheet_names = self.reader.sheet_names() index = sheet_names.index(sheet_name) return self.read_sheet_by_index(index) def read_sheet_by_index(self, sheet_index): sheet_reader = self.reader.read_sheet(sheet_index) sheet_names = self.reader.sheet_names() sheet = EncapsulatedSheetReader(sheet_reader, **self.keywords) return {sheet_names[sheet_index]: sheet.to_array()} def read_all(self): """ read everything from a excel data book """ result = OrderedDict() for sheet_index in range(len(self.reader)): content_dict = self.read_sheet_by_index(sheet_index) result.update(content_dict) return result def read_many(self, sheets): """ read everything from a excel data book """ result = OrderedDict() for sheet in sheets: if isinstance(sheet, int): result.update(self.read_sheet_by_index(sheet)) else: result.update(self.read_sheet_by_name(sheet)) return result def close(self): return self.reader.close() def __enter__(self): return self def __exit__(self, a_type, value, traceback): self.close() class EncapsulatedSheetReader(SheetReader): def row_iterator(self): yield from self._native_sheet.row_iterator() def column_iterator(self, row): yield from self._native_sheet.column_iterator(row) ././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1643486962.4462712 pyexcel-io-0.6.6/pyexcel_io/readers/0000755000175100001710000000000000000000000016726 5ustar00runnerdocker././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1643486950.0 pyexcel-io-0.6.6/pyexcel_io/readers/__init__.py0000644000175100001710000000205100000000000021035 0ustar00runnerdocker""" pyexcel_io.readers ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ file readers :copyright: (c) 2014-2022 by Onni Software Ltd. :license: New BSD License, see LICENSE for more details """ from pyexcel_io.plugins import IOPluginInfoChainV2 IOPluginInfoChainV2(__name__).add_a_reader( relative_plugin_class_path="csv_in_file.FileReader", locations=["file"], file_types=["csv", "tsv"], stream_type="text", ).add_a_reader( relative_plugin_class_path="csv_content.ContentReader", locations=["content"], file_types=["csv", "tsv"], stream_type="text", ).add_a_reader( relative_plugin_class_path="csv_in_memory.MemoryReader", locations=["memory"], file_types=["csv", "tsv"], stream_type="text", ).add_a_reader( relative_plugin_class_path="csvz.FileReader", file_types=["csvz", "tsvz"], locations=["file", "memory"], stream_type="binary", ).add_a_reader( relative_plugin_class_path="csvz.ContentReader", file_types=["csvz", "tsvz"], locations=["content"], stream_type="binary", ) ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1643486950.0 pyexcel-io-0.6.6/pyexcel_io/readers/csv_content.py0000644000175100001710000000173600000000000021634 0ustar00runnerdockerimport mmap from pyexcel_io.book import _convert_content_to_stream from pyexcel_io.readers.csv_sheet import CSVMemoryMapIterator from pyexcel_io.readers.csv_in_memory import MemoryReader class ContentReader(MemoryReader): def __init__(self, file_content, file_type, **keywords): file_stream = ContentReader.convert_content_to_stream( file_content, file_type, **keywords ) super().__init__(file_stream, file_type, **keywords) @staticmethod def convert_content_to_stream(file_content, file_type, **keywords): encoding = keywords.get("encoding", "utf-8") if isinstance(file_content, mmap.mmap): # load from mmap file_stream = CSVMemoryMapIterator(file_content, encoding) else: if isinstance(file_content, bytes): file_content = file_content.decode(encoding) file_stream = _convert_content_to_stream(file_content, file_type) return file_stream ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1643486950.0 pyexcel-io-0.6.6/pyexcel_io/readers/csv_in_file.py0000644000175100001710000000412200000000000021557 0ustar00runnerdockerimport os import re import glob from pyexcel_io import constants from pyexcel_io.sheet import NamedContent from pyexcel_io.plugin_api import IReader from pyexcel_io.readers.csv_sheet import CSVFileReader DEFAULT_NEWLINE = "\r\n" class FileReader(IReader): def __init__(self, file_name, file_type, **keywords): """Load content from a file :params str filename: an accessible file path :returns: a book """ self.handles = [] self.keywords = keywords if file_type == constants.FILE_FORMAT_TSV: self.keywords["dialect"] = constants.KEYWORD_TSV_DIALECT self.__line_terminator = keywords.get( constants.KEYWORD_LINE_TERMINATOR, DEFAULT_NEWLINE ) names = os.path.splitext(file_name) filepattern = "%s%s*%s*%s" % ( names[0], constants.DEFAULT_MULTI_CSV_SEPARATOR, constants.DEFAULT_MULTI_CSV_SEPARATOR, names[1], ) filelist = glob.glob(filepattern) if len(filelist) == 0: file_parts = os.path.split(file_name) self.content_array = [NamedContent(file_parts[-1], file_name)] else: matcher = "%s%s(.*)%s(.*)%s" % ( names[0], constants.DEFAULT_MULTI_CSV_SEPARATOR, constants.DEFAULT_MULTI_CSV_SEPARATOR, names[1], ) tmp_file_list = [] for filen in filelist: result = re.match(matcher, filen) tmp_file_list.append((result.group(1), result.group(2), filen)) ret = [] for lsheetname, index, filen in sorted( tmp_file_list, key=lambda row: row[1] ): ret.append(NamedContent(lsheetname, filen)) self.content_array = ret def read_sheet(self, index): reader = CSVFileReader(self.content_array[index], **self.keywords) self.handles.append(reader) return reader def close(self): for reader in self.handles: reader.close() self.handles = [] ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1643486950.0 pyexcel-io-0.6.6/pyexcel_io/readers/csv_in_memory.py0000644000175100001710000000425600000000000022160 0ustar00runnerdockerimport re import pyexcel_io._compact as compact from pyexcel_io import constants from pyexcel_io.sheet import NamedContent from pyexcel_io.plugin_api import IReader from pyexcel_io.readers.csv_sheet import CSVinMemoryReader DEFAULT_SHEET_SEPARATOR_FORMATTER = f"---{constants.DEFAULT_NAME}---%s" class MemoryReader(IReader): def __init__( self, file_stream, file_type, multiple_sheets=False, **keywords ): """Load content from memory :params stream file_content: the actual file content in memory :returns: a book """ self.handles = [] self.keywords = keywords if file_type == constants.FILE_FORMAT_TSV: self.keywords["dialect"] = constants.KEYWORD_TSV_DIALECT self.file_type = file_type self.__load_from_memory_flag = True self.__line_terminator = keywords.get( constants.KEYWORD_LINE_TERMINATOR, constants.DEFAULT_CSV_NEWLINE ) separator = DEFAULT_SHEET_SEPARATOR_FORMATTER % self.__line_terminator if multiple_sheets: # will be slow for large files file_stream.seek(0) content = file_stream.read() sheets = content.split(separator) named_contents = [] for sheet in sheets: if sheet == "": # skip empty named sheet continue lines = sheet.split(self.__line_terminator) result = re.match(constants.SEPARATOR_MATCHER, lines[0]) new_content = "\n".join(lines[1:]) new_sheet = NamedContent( result.group(1), compact.StringIO(new_content) ) named_contents.append(new_sheet) self.content_array = named_contents else: if hasattr(file_stream, "seek"): file_stream.seek(0) self.content_array = [NamedContent(self.file_type, file_stream)] def read_sheet(self, index): reader = CSVinMemoryReader(self.content_array[index], **self.keywords) self.handles.append(reader) return reader def close(self): for reader in self.handles: reader.close() ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1643486950.0 pyexcel-io-0.6.6/pyexcel_io/readers/csv_sheet.py0000644000175100001710000001365000000000000021270 0ustar00runnerdocker""" pyexcel_io.readers.csv_sheet ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ csv file reader :copyright: (c) 2014-2022 by Onni Software Ltd. :license: New BSD License, see LICENSE for more details """ import csv import pyexcel_io.service as service import pyexcel_io._compact as compact import pyexcel_io.constants as constants from pyexcel_io.plugin_api import ISheet DEFAULT_SEPARATOR = "__" DEFAULT_SHEET_SEPARATOR_FORMATTER = "---%s---" % constants.DEFAULT_NAME + "%s" SEPARATOR_MATCHER = "---%s:(.*)---" % constants.DEFAULT_NAME DEFAULT_CSV_STREAM_FILE_FORMATTER = ( "---%s:" % constants.DEFAULT_NAME + "%s---%s" ) DEFAULT_NEWLINE = "\r\n" BOM_LITTLE_ENDIAN = b"\xff\xfe" BOM_BIG_ENDIAN = b"\xfe\ff" LITTLE_ENDIAN = 0 BIG_ENDIAN = 1 class CSVMemoryMapIterator(object): """ Wrapper class for mmap object mmap object does not handle encoding at all. This class provide the necessary transcoding for utf-8, utf-16 and utf-32 """ def __init__(self, mmap_obj, encoding): self.__mmap_obj = mmap_obj self.__encoding = encoding self.__count = 0 self.__endian = LITTLE_ENDIAN if encoding == "utf-8": # ..\r\x00\n # \x00\x.. self.__zeros_left_in_2_row = 0 elif encoding == "utf-16": # ..\r\x00\n # \x00\x.. self.__zeros_left_in_2_row = 1 elif encoding == "utf-32": # \r\x00\x00\x00\n # \x00\x00\x00\x.. self.__zeros_left_in_2_row = 3 elif encoding in ["utf-32-be", "utf-16-be"]: self.__zeros_left_in_2_row = 0 self.__endian = BIG_ENDIAN elif encoding == "utf-32-le": self.__zeros_left_in_2_row = 3 self.__endian = LITTLE_ENDIAN elif encoding == "utf-16-le": self.__zeros_left_in_2_row = 1 self.__endian = LITTLE_ENDIAN else: raise Exception("Encoding %s is not supported" % encoding) def __iter__(self): return self def __next__(self): line = self.__mmap_obj.readline() if self.__count == 0: utf_16_32 = ( self.__encoding == "utf-16" or self.__encoding == "utf-32" ) if utf_16_32: bom_header = line[:2] if bom_header == BOM_BIG_ENDIAN: self.__endian = BIG_ENDIAN elif self.__endian == LITTLE_ENDIAN: line = line[self.__zeros_left_in_2_row :] # noqa: E203 if self.__endian == LITTLE_ENDIAN: line = line.rstrip() line = line.decode(self.__encoding) self.__count += 1 if line == "": raise StopIteration return line def close(self): pass class CSVSheetReader(ISheet): """generic csv file reader""" def __init__( self, sheet, encoding="utf-8", auto_detect_float=True, ignore_infinity=True, auto_detect_int=True, auto_detect_datetime=True, pep_0515_off=True, ignore_nan_text=False, default_float_nan=None, **keywords ): self._native_sheet = sheet self._encoding = encoding self.__auto_detect_int = auto_detect_int self.__auto_detect_float = auto_detect_float self.__ignore_infinity = ignore_infinity self.__auto_detect_datetime = auto_detect_datetime self.__file_handle = None self.__pep_0515_off = pep_0515_off self.__ignore_nan_text = ignore_nan_text self.__default_float_nan = default_float_nan self._keywords = keywords def get_file_handle(self): """return me unicde reader for csv""" raise NotImplementedError("Please implement get_file_handle()") def row_iterator(self): self.__file_handle = self.get_file_handle() return csv.reader(self.__file_handle, **self._keywords) def column_iterator(self, row): for element in row: if element is not None and element != "": element = self.__convert_cell(element) yield element def __convert_cell(self, csv_cell_text): ret = None if self.__auto_detect_int: ret = service.detect_int_value(csv_cell_text, self.__pep_0515_off) if ret is None and self.__auto_detect_float: ret = service.detect_float_value( csv_cell_text, self.__pep_0515_off, ignore_nan_text=self.__ignore_nan_text, default_float_nan=self.__default_float_nan, ) shall_we_ignore_the_conversion = ( ret in [float("inf"), float("-inf")] ) and self.__ignore_infinity if shall_we_ignore_the_conversion: ret = None if ret is None and self.__auto_detect_datetime: ret = service.detect_date_value(csv_cell_text) if ret is None: ret = csv_cell_text return ret def close(self): if self.__file_handle: self.__file_handle.close() # else: means the generator has been run # yes, no run, no file open. class CSVFileReader(CSVSheetReader): """read csv from phyical file""" def get_file_handle(self): unicode_reader = open( self._native_sheet.payload, "r", encoding=self._encoding ) return unicode_reader class CSVinMemoryReader(CSVSheetReader): """read csv file from memory""" def get_file_handle(self): if isinstance(self._native_sheet.payload, compact.BytesIO): # please note that # if the end developer feed us bytesio in python3 # we will do the conversion to StriongIO but that # comes at a cost. content = self._native_sheet.payload.read() unicode_reader = compact.StringIO(content.decode(self._encoding)) else: unicode_reader = self._native_sheet.payload return unicode_reader ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1643486950.0 pyexcel-io-0.6.6/pyexcel_io/readers/csvz.py0000644000175100001710000000367200000000000020275 0ustar00runnerdocker""" pyexcel_io.fileformat.csvz ~~~~~~~~~~~~~~~~~~~~~~~~~~~ The lower level csvz file format handler. :copyright: (c) 2014-2022 by Onni Software Ltd. :license: New BSD License, see LICENSE for more details """ import zipfile from io import BytesIO import chardet from pyexcel_io import constants from pyexcel_io.sheet import NamedContent from pyexcel_io._compact import StringIO from pyexcel_io.readers.csv_sheet import CSVinMemoryReader from pyexcel_io.plugin_api.abstract_reader import IReader class FileReader(IReader): def __init__(self, file_alike_object, file_type, **keywords): self.content_array = [] try: self.zipfile = zipfile.ZipFile(file_alike_object, "r") sheets = [ NamedContent(_get_sheet_name(name), name) for name in self.zipfile.namelist() ] self.content_array = sheets self.keywords = keywords if file_type == constants.FILE_FORMAT_TSVZ: self.keywords["dialect"] = constants.KEYWORD_TSV_DIALECT except zipfile.BadZipfile: print("StringIO instance was passed by any chance?") raise def close(self): if self.zipfile: self.zipfile.close() def read_sheet(self, index): name = self.content_array[index].name content = self.zipfile.read(self.content_array[index].payload) encoding_guess = chardet.detect(content) sheet = StringIO(content.decode(encoding_guess["encoding"])) return CSVinMemoryReader(NamedContent(name, sheet), **self.keywords) class ContentReader(FileReader): def __init__(self, file_content, file_type, **keywords): io = BytesIO(file_content) super().__init__(io, file_type, **keywords) def _get_sheet_name(filename): len_of_a_dot = 1 len_of_csv_word = 3 name_len = len(filename) - len_of_a_dot - len_of_csv_word return filename[:name_len] ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1643486950.0 pyexcel-io-0.6.6/pyexcel_io/service.py0000644000175100001710000001337400000000000017323 0ustar00runnerdocker""" pyexcel_io.service ~~~~~~~~~~~~~~~~~~~ provide service code to downstream projects :copyright: (c) 2014-2022 by Onni Software Ltd. :license: New BSD License, see LICENSE for more details """ import re import math import datetime from pyexcel_io import constants, exceptions def has_no_digits_in_float(value): """check if a float value had zero value in digits""" return value == math.floor(value) def detect_date_value(cell_text): """ Read the date formats that were written by csv.writer """ ret = None try: if len(cell_text) == 10: ret = datetime.datetime.strptime(cell_text, "%Y-%m-%d") ret = ret.date() elif len(cell_text) == 19: ret = datetime.datetime.strptime(cell_text, "%Y-%m-%d %H:%M:%S") elif len(cell_text) > 19: ret = datetime.datetime.strptime( cell_text[0:26], "%Y-%m-%d %H:%M:%S.%f" ) except ValueError: pass return ret def detect_float_value( cell_text, pep_0515_off=True, ignore_nan_text=False, default_float_nan=None ): should_we_skip_it = ( cell_text.startswith("0") and cell_text.startswith("0.") is False ) if should_we_skip_it: # do not convert if a number starts with 0 # e.g. 014325 return None if pep_0515_off: pattern = "([0-9]+_)+[0-9]+.[0-9]*$" if re.match(pattern, cell_text): return None try: if ignore_nan_text: if cell_text.lower() == "nan": return None else: return float(cell_text) else: if cell_text.lower() == "nan": if cell_text == default_float_nan: return float("NaN") else: return None else: return float(cell_text) except ValueError: return None def detect_int_value(cell_text, pep_0515_off=True): if cell_text.startswith("0") and len(cell_text) > 1: return None if pep_0515_off: pattern = "([0-9]+_)+[0-9]+$" if re.match(pattern, cell_text): return None try: return int(cell_text) except ValueError: pattern = "([0-9]+,)*[0-9]+$" if re.match(pattern, cell_text): integer_string = cell_text.replace(",", "") return int(integer_string) else: return None def float_value(value): """convert a value to float""" ret = float(value) return ret def date_value(value): """convert to data value accroding ods specification""" ret = "invalid" try: # catch strptime exceptions only if len(value) == 10: ret = datetime.datetime.strptime(value, "%Y-%m-%d") ret = ret.date() elif len(value) == 19: ret = datetime.datetime.strptime(value, "%Y-%m-%dT%H:%M:%S") elif len(value) > 19: ret = datetime.datetime.strptime( value[0:26], "%Y-%m-%dT%H:%M:%S.%f" ) except ValueError: pass if ret == "invalid": raise Exception("Bad date value %s" % value) return ret def time_value(value): """convert to time value accroding the specification""" import re results = re.match(r"PT(\d+)H(\d+)M(\d+)S", value) if results and len(results.groups()) == 3: hour = int(results.group(1)) minute = int(results.group(2)) second = int(results.group(3)) if hour < 24: ret = datetime.time(hour, minute, second) else: ret = datetime.timedelta( hours=hour, minutes=minute, seconds=second ) else: ret = None return ret def boolean_value(value): """get bolean value""" if value == "true": ret = True elif value == "false": ret = False else: # needed for pyexcel-ods3 ret = value return ret ODS_FORMAT_CONVERSION = { "float": float, "date": datetime.date, "time": datetime.time, "timedelta": datetime.timedelta, "boolean": bool, "percentage": float, "currency": float, } ODS_WRITE_FORMAT_COVERSION = { float: "float", int: "float", str: "string", datetime.date: "date", datetime.time: "time", datetime.timedelta: "timedelta", datetime.datetime: "datetime", bool: "boolean", } VALUE_CONVERTERS = { "float": float_value, "date": date_value, "time": time_value, "timedelta": time_value, "boolean": boolean_value, "percentage": float_value, } def throw_exception(value): raise exceptions.IntegerAccuracyLossError("%s is too big" % value) def ods_float_value(value): if value > constants.MAX_INTEGER: raise exceptions.IntegerAccuracyLossError("%s is too big" % value) return value def ods_date_value(value): return value.strftime("%Y-%m-%d") def ods_time_value(value): return value.strftime("PT%HH%MM%SS") def ods_bool_value(value): """convert a boolean value to text""" if value is True: return "true" else: return "false" def ods_timedelta_value(cell): """convert a cell value to time delta""" hours = cell.days * 24 + cell.seconds // 3600 minutes = (cell.seconds // 60) % 60 seconds = cell.seconds % 60 return "PT%02dH%02dM%02dS" % (hours, minutes, seconds) ODS_VALUE_CONVERTERS = { "date": ods_date_value, "time": ods_time_value, "boolean": ods_bool_value, "timedelta": ods_timedelta_value, "float": ods_float_value, "long": ods_float_value, } VALUE_TOKEN = { "float": "value", "date": "date-value", "time": "time-value", "boolean": "boolean-value", "percentage": "value", "currency": "value", "timedelta": "time-value", } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1643486950.0 pyexcel-io-0.6.6/pyexcel_io/sheet.py0000644000175100001710000001166400000000000016773 0ustar00runnerdocker""" pyexcel_io.sheet ~~~~~~~~~~~~~~~~~~~ The io interface to file extensions :copyright: (c) 2014-2022 by Onni Software Ltd. :license: New BSD License, see LICENSE for more details """ import pyexcel_io.constants as constants from pyexcel_io.utils import _index_filter from pyexcel_io._compact import irange from pyexcel_io.plugin_api import NamedContent # noqa: F401 class SheetReader(object): """ Generic sheet reader """ def __init__( self, sheet, start_row=0, row_limit=-1, start_column=0, column_limit=-1, skip_row_func=None, skip_column_func=None, skip_empty_rows=False, row_renderer=None, keep_trailing_empty_cells=False, **deprecated_use_of_keywords_here ): self._native_sheet = sheet self._keywords = {} self._keywords.update(deprecated_use_of_keywords_here) self._start_row = start_row self._row_limit = row_limit self._start_column = start_column self._column_limit = column_limit self._skip_row = _index_filter self._skip_column = _index_filter self._skip_empty_rows = skip_empty_rows self._row_renderer = row_renderer self.keep_trailing_empty_cells = keep_trailing_empty_cells if skip_row_func: self._skip_row = skip_row_func if skip_column_func: self._skip_column = skip_column_func def to_array(self): """2 dimentional representation of the content""" for row_index, row in enumerate(self.row_iterator()): row_position = self._skip_row( row_index, self._start_row, self._row_limit ) if row_position == constants.SKIP_DATA: continue elif row_position == constants.STOP_ITERATION: break return_row = [] tmp_row = [] for column_index, cell_value in enumerate( self.column_iterator(row) ): column_position = self._skip_column( column_index, self._start_column, self._column_limit ) if column_position == constants.SKIP_DATA: continue elif column_position == constants.STOP_ITERATION: break if self.keep_trailing_empty_cells: return_row.append(cell_value) else: tmp_row.append(cell_value) if cell_value is not None and cell_value != "": return_row += tmp_row tmp_row = [] if self._skip_empty_rows and len(return_row) < 1: # we by-pass next yeild here # because it is an empty row continue if self._row_renderer: return_row = self._row_renderer(return_row) yield return_row def row_iterator(self): """ iterate each row override this function in the sitation where number_of_rows() is difficult or costly to implement """ return irange(self.number_of_rows()) def column_iterator(self, row): """ iterate each column of a given row override this function in the sitation where number_of_columns() is difficult or costly to implement """ for column in irange(self.number_of_columns()): yield self.cell_value(row, column) def number_of_rows(self): """ implement this method for easy extension """ raise Exception("Please implement number_of_rows()") def number_of_columns(self): """ implement this method for easy extension """ raise Exception("Please implement number_of_columns()") def cell_value(self, row, column): """ implement this method for easy extension """ raise Exception("Please implement cell_value()") def close(self): pass class SheetWriter(object): """ Generic sheet writer """ def __init__(self, native_book, native_sheet, name, **keywords): if name: sheet_name = name else: sheet_name = constants.DEFAULT_SHEET_NAME self._native_book = native_book self._native_sheet = native_sheet self._keywords = keywords self.set_sheet_name(sheet_name) def set_sheet_name(self, name): """ Set sheet name """ pass def write_row(self, array): """ write a row into the file """ raise NotImplementedError("Please implement write_row()") def write_array(self, table): """ For standalone usage, write an array """ for row in table: self.write_row(row) def close(self): """ This call actually save the file """ pass ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1643486950.0 pyexcel-io-0.6.6/pyexcel_io/utils.py0000644000175100001710000000455400000000000017023 0ustar00runnerdocker""" pyexcel_io.utils ~~~~~~~~~~~~~~~~~~~ utility functions :copyright: (c) 2014-2022 by Onni Software Ltd. :license: New BSD License, see LICENSE for more details """ import pyexcel_io.constants as constants XLS_PLUGIN = "pyexcel-xls" XLSX_PLUGIN = "pyexcel-xlsx" ODS_PLUGIN = "pyexcel-ods" ODS3_PLUGIN = "pyexcel-ods3" ODSR_PLUGIN = "pyexcel-odsr" ODSW_PLUGIN = "pyexcel-odsw" XLSXR_PLUGIN = "pyexcel-xlsxr" XLSXW_PLUGIN = "pyexcel-xlsxw" XLSBR_PLUGIN = "pyexcel-xlsbr" HTMLR_PLUGIN = "pyexcel-htmlr" PDFR_PLUGIN = "pyexcel-pdfr" IO_ITSELF = "pyexcel-io" AVAILABLE_NEW_READERS = {} AVAILABLE_READERS = { constants.FILE_FORMAT_CSV: [IO_ITSELF], constants.FILE_FORMAT_XLS: [XLS_PLUGIN], constants.FILE_FORMAT_XLSX: [XLS_PLUGIN, XLSX_PLUGIN], constants.FILE_FORMAT_XLSM: [XLS_PLUGIN, XLSX_PLUGIN], constants.FILE_FORMAT_ODS: [ODS_PLUGIN, ODS3_PLUGIN, ODSR_PLUGIN], constants.FILE_FORMAT_TSV: [IO_ITSELF], constants.FILE_FORMAT_CSVZ: [IO_ITSELF], constants.FILE_FORMAT_TSVZ: [IO_ITSELF], constants.FILE_FORMAT_XLSB: [XLSBR_PLUGIN], constants.FILE_FORMAT_HTML: [HTMLR_PLUGIN], constants.FILE_FORMAT_PDF: [PDFR_PLUGIN], } AVAILABLE_WRITERS = { constants.FILE_FORMAT_XLS: [XLS_PLUGIN], constants.FILE_FORMAT_XLSX: [XLSX_PLUGIN, XLSXW_PLUGIN], constants.FILE_FORMAT_XLSM: [XLSX_PLUGIN], constants.FILE_FORMAT_ODS: [ODS_PLUGIN, ODS3_PLUGIN, ODSW_PLUGIN], constants.FILE_FORMAT_CSV: [IO_ITSELF], constants.FILE_FORMAT_TSV: [IO_ITSELF], constants.FILE_FORMAT_CSVZ: [IO_ITSELF], constants.FILE_FORMAT_TSVZ: [IO_ITSELF], } def _index_filter(current_index, start, limit=-1): out_range = constants.SKIP_DATA if current_index >= start: out_range = constants.TAKE_DATA if limit > 0 and out_range == constants.TAKE_DATA: if current_index >= (start + limit): out_range = constants.STOP_ITERATION return out_range def is_empty_array(array): """ Check if an array is an array of '' or not """ empty_array = [element for element in array if element != ""] return len(empty_array) == 0 def swap_empty_string_for_none(array): """replace empty string fields with None""" def swap(value): """change empty string to None""" if value == "": return None else: return value return [swap(x) for x in array] ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1643486950.0 pyexcel-io-0.6.6/pyexcel_io/writer.py0000644000175100001710000000340400000000000017170 0ustar00runnerdockerfrom pyexcel_io.plugins import NEW_WRITERS class Writer(object): def __init__(self, file_type, library=None): self.file_type = file_type self.library = library self.keyboards = None # if you know which reader class to use, this attribute allows # you to set reader class externally. Since there is no # so call private field in Python, I am not going to create # useless setter and getter functions like Java. # in pyexcel, this attribute is mainly used for testing self.writer_class = None def open(self, file_name, **keywords): if self.writer_class is None: self.writer_class = NEW_WRITERS.get_a_plugin( self.file_type, library=self.library, location="file" ) self.writer = self.writer_class(file_name, self.file_type, **keywords) def open_content(self, file_stream, **keywords): if self.writer_class is None: self.writer_class = NEW_WRITERS.get_a_plugin( self.file_type, library=self.library, location="content" ) self.writer = self.writer_class( file_stream, self.file_type, **keywords ) def open_stream(self, file_stream, **keywords): if self.writer_class is None: self.writer_class = NEW_WRITERS.get_a_plugin( self.file_type, library=self.library, location="memory" ) self.writer = self.writer_class( file_stream, self.file_type, **keywords ) def write(self, incoming_dict): self.writer.write(incoming_dict) def close(self): self.writer.close() def __enter__(self): return self def __exit__(self, a_type, value, traceback): self.close() ././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1643486962.4462712 pyexcel-io-0.6.6/pyexcel_io/writers/0000755000175100001710000000000000000000000017000 5ustar00runnerdocker././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1643486950.0 pyexcel-io-0.6.6/pyexcel_io/writers/__init__.py0000644000175100001710000000142500000000000021113 0ustar00runnerdocker""" pyexcel_io.writers ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ file writers :copyright: (c) 2014-2022 by Onni Software Ltd. :license: New BSD License, see LICENSE for more details """ from pyexcel_io.plugins import IOPluginInfoChainV2 IOPluginInfoChainV2(__name__).add_a_writer( relative_plugin_class_path="csv_in_file.CsvFileWriter", locations=["file", "content"], file_types=["csv", "tsv"], stream_type="text", ).add_a_writer( relative_plugin_class_path="csv_in_memory.CsvMemoryWriter", locations=["memory"], file_types=["csv", "tsv"], stream_type="text", ).add_a_writer( relative_plugin_class_path="csvz_writer.CsvZipWriter", locations=["memory", "file", "content"], file_types=["csvz", "tsvz"], stream_type="binary", ) ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1643486950.0 pyexcel-io-0.6.6/pyexcel_io/writers/csv_in_file.py0000644000175100001710000000160200000000000021631 0ustar00runnerdockerfrom pyexcel_io import constants from pyexcel_io.plugin_api import IWriter from pyexcel_io.writers.csv_sheet import CSVFileWriter class CsvFileWriter(IWriter): def __init__(self, file_alike_object, file_type, **keywords): self._file_alike_object = file_alike_object self._keywords = keywords if file_type == constants.FILE_FORMAT_TSV: self._keywords["dialect"] = constants.KEYWORD_TSV_DIALECT self.__index = 0 self.handlers = [] def create_sheet(self, name): writer = CSVFileWriter( self._file_alike_object, name, sheet_index=self.__index, **self._keywords ) self.__index = self.__index + 1 self.handlers.append(writer) return writer def close(self): for writer in self.handlers: writer.close() self.handlers = [] ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1643486950.0 pyexcel-io-0.6.6/pyexcel_io/writers/csv_in_memory.py0000644000175100001710000000143600000000000022227 0ustar00runnerdockerfrom pyexcel_io import constants from pyexcel_io.plugin_api import IWriter from pyexcel_io.writers.csv_sheet import CSVMemoryWriter class CsvMemoryWriter(IWriter): def __init__(self, file_alike_object, file_type, **keywords): self._file_alike_object = file_alike_object self._keywords = keywords if file_type == constants.FILE_FORMAT_TSV: self._keywords["dialect"] = constants.KEYWORD_TSV_DIALECT self.__index = 0 def create_sheet(self, name): writer_class = CSVMemoryWriter writer = writer_class( self._file_alike_object, name, sheet_index=self.__index, **self._keywords ) self.__index = self.__index + 1 return writer def close(self): pass ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1643486950.0 pyexcel-io-0.6.6/pyexcel_io/writers/csv_sheet.py0000644000175100001710000000544400000000000021344 0ustar00runnerdocker""" pyexcel_io.writers.csv_sheet ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The lower level csv file format writer :copyright: (c) 2014-2022 by Onni Software Ltd. :license: New BSD License, see LICENSE for more details """ import csv import pyexcel_io.constants as constants from pyexcel_io.plugin_api import ISheetWriter class CSVFileWriter(ISheetWriter): """ csv file writer """ def __init__( self, filename, name, encoding="utf-8", single_sheet_in_book=False, sheet_index=None, **keywords ): self._encoding = encoding self._sheet_name = name if self._sheet_name is None or single_sheet_in_book: self._sheet_name = constants.DEFAULT_SHEET_NAME self._single_sheet_in_book = single_sheet_in_book self.__line_terminator = constants.DEFAULT_CSV_NEWLINE self._keywords = keywords if constants.KEYWORD_LINE_TERMINATOR in keywords: self.__line_terminator = keywords.get( constants.KEYWORD_LINE_TERMINATOR ) self._sheet_index = sheet_index self.file_handle = None self._native_book = filename self.writer = self.get_writer() def get_writer(self): if self._sheet_name != constants.DEFAULT_SHEET_NAME: names = self._native_book.split(".") file_name = "%s%s%s%s%s.%s" % ( names[0], constants.DEFAULT_MULTI_CSV_SEPARATOR, self._sheet_name, # sheet name constants.DEFAULT_MULTI_CSV_SEPARATOR, self._sheet_index, # sheet index names[1], ) else: file_name = self._native_book self.file_handle = open( file_name, "w", newline="", encoding=self._encoding ) return csv.writer(self.file_handle, **self._keywords) def write_row(self, array): """ write a row into the file """ self.writer.writerow(array) def close(self): self.file_handle.close() class CSVMemoryWriter(CSVFileWriter): """Write csv to a memory stream""" def get_writer(self): self.file_handle = self._native_book writer = csv.writer(self.file_handle, **self._keywords) if not self._single_sheet_in_book: writer.writerow( [ constants.DEFAULT_CSV_STREAM_FILE_FORMATTER % (self._sheet_name, "") ] ) return writer def close(self): if self._single_sheet_in_book: # on purpose, the this is not done # because the io stream can be used later pass else: self.writer.writerow([constants.SEPARATOR_FORMATTER % ""]) ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1643486950.0 pyexcel-io-0.6.6/pyexcel_io/writers/csvz_sheet.py0000644000175100001710000000174100000000000021532 0ustar00runnerdocker""" pyexcel_io.fileformat.csvz_sheet ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The lower level csvz file format handler. :copyright: (c) 2014-2022 by Onni Software Ltd. :license: New BSD License, see LICENSE for more details """ import csv from pyexcel_io._compact import StringIO from pyexcel_io.writers.csv_sheet import CSVFileWriter class CSVZipSheetWriter(CSVFileWriter): """handle the zipfile interface""" def __init__(self, zipfile, sheetname, file_extension, **keywords): self.file_extension = file_extension keywords["single_sheet_in_book"] = False self.content = StringIO() super().__init__(zipfile, sheetname, **keywords) def get_writer(self): return csv.writer(self.content, **self._keywords) def close(self): file_name = "%s.%s" % (self._sheet_name, self.file_extension) self.content.seek(0) self._native_book.writestr(file_name, self.content.read()) self.content.close() ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1643486950.0 pyexcel-io-0.6.6/pyexcel_io/writers/csvz_writer.py0000644000175100001710000000207700000000000021741 0ustar00runnerdockerimport zipfile from pyexcel_io import constants from pyexcel_io.plugin_api import IWriter from pyexcel_io.writers.csvz_sheet import CSVZipSheetWriter class CsvZipWriter(IWriter): """ csvz writer It is better to store csv files as a csvz as it saves your disk space. Pyexcel-io had the facility to unzip it for you or you could use any other unzip software. """ def __init__(self, file_name, file_type, **keywords): self._file_type = file_type self.zipfile = zipfile.ZipFile(file_name, "w", zipfile.ZIP_DEFLATED) self._keywords = keywords if file_type == constants.FILE_FORMAT_TSVZ: self._keywords["dialect"] = constants.KEYWORD_TSV_DIALECT def create_sheet(self, name): given_name = name if given_name is None: given_name = constants.DEFAULT_SHEET_NAME writer = CSVZipSheetWriter( self.zipfile, given_name, self._file_type[:3], **self._keywords ) return writer def close(self): if self.zipfile: self.zipfile.close() ././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1643486962.4422712 pyexcel-io-0.6.6/pyexcel_io.egg-info/0000755000175100001710000000000000000000000016773 5ustar00runnerdocker././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1643486962.0 pyexcel-io-0.6.6/pyexcel_io.egg-info/PKG-INFO0000644000175100001710000006066700000000000020107 0ustar00runnerdockerMetadata-Version: 2.1 Name: pyexcel-io Version: 0.6.6 Summary: A python library to read and write structured data in csv, zipped csvformat and to/from databases Home-page: https://github.com/pyexcel/pyexcel-io Author: C.W. Author-email: info@pyexcel.org License: New BSD Download-URL: https://github.com/pyexcel/pyexcel-io/archive/0.6.6.tar.gz Keywords: python,API,tsv,tsvz,csv,csvz,django,sqlalchemy Platform: UNKNOWN Classifier: Topic :: Software Development :: Libraries Classifier: Programming Language :: Python Classifier: Intended Audience :: Developers Classifier: Programming Language :: Python :: 3 :: Only Classifier: Programming Language :: Python :: 3.6 Classifier: Programming Language :: Python :: 3.7 Classifier: Programming Language :: Python :: 3.8 Classifier: License :: OSI Approved :: BSD License Classifier: Programming Language :: Python :: Implementation :: PyPy Requires-Python: >=3.6 Provides-Extra: xls Provides-Extra: xlsx Provides-Extra: ods License-File: LICENSE ================================================================================ pyexcel-io - Let you focus on data, instead of file formats ================================================================================ .. image:: https://raw.githubusercontent.com/pyexcel/pyexcel.github.io/master/images/patreon.png :target: https://www.patreon.com/chfw .. image:: https://raw.githubusercontent.com/pyexcel/pyexcel-mobans/master/images/awesome-badge.svg :target: https://awesome-python.com/#specific-formats-processing .. image:: https://github.com/pyexcel/pyexcel-io/workflows/run_tests/badge.svg :target: http://github.com/pyexcel/pyexcel-io/actions .. image:: https://codecov.io/gh/pyexcel/pyexcel-io/branch/master/graph/badge.svg :target: https://codecov.io/gh/pyexcel/pyexcel-io .. image:: https://badge.fury.io/py/pyexcel-io.svg :target: https://pypi.org/project/pyexcel-io .. image:: https://anaconda.org/conda-forge/pyexcel-io/badges/version.svg :target: https://anaconda.org/conda-forge/pyexcel-io .. image:: https://pepy.tech/badge/pyexcel-io/month :target: https://pepy.tech/project/pyexcel-io .. image:: https://anaconda.org/conda-forge/pyexcel-io/badges/downloads.svg :target: https://anaconda.org/conda-forge/pyexcel-io .. image:: https://img.shields.io/gitter/room/gitterHQ/gitter.svg :target: https://gitter.im/pyexcel/Lobby .. image:: https://img.shields.io/static/v1?label=continuous%20templating&message=%E6%A8%A1%E7%89%88%E6%9B%B4%E6%96%B0&color=blue&style=flat-square :target: https://moban.readthedocs.io/en/latest/#at-scale-continous-templating-for-open-source-projects .. image:: https://img.shields.io/static/v1?label=coding%20style&message=black&color=black&style=flat-square :target: https://github.com/psf/black .. image:: https://readthedocs.org/projects/pyexcel-io/badge/?version=latest :target: http://pyexcel-io.readthedocs.org/en/latest/ Support the project ================================================================================ If your company has embedded pyexcel and its components into a revenue generating product, please support me on github, `patreon `_ or `bounty source `_ to maintain the project and develop it further. If you are an individual, you are welcome to support me too and for however long you feel like. As my backer, you will receive `early access to pyexcel related contents `_. And your issues will get prioritized if you would like to become my patreon as `pyexcel pro user`. With your financial support, I will be able to invest a little bit more time in coding, documentation and writing interesting posts. Known constraints ================== Fonts, colors and charts are not supported. Nor to read password protected xls, xlsx and ods files. Introduction ================================================================================ **pyexcel-io** provides **one** application programming interface(API) to read and write the data in excel format, import the data into and export the data from database. It provides support for csv(z) format, django database and sqlalchemy supported databases. Its supported file formats are extended to cover "xls", "xlsx", "ods" by the following extensions: .. _file-format-list: .. _a-map-of-plugins-and-file-formats: .. table:: A list of file formats supported by external plugins ======================== ======================= ================= Package name Supported file formats Dependencies ======================== ======================= ================= `pyexcel-io`_ csv, csvz [#f1]_, tsv, tsvz [#f2]_ `pyexcel-xls`_ xls, xlsx(read only), `xlrd`_, xlsm(read only) `xlwt`_ `pyexcel-xlsx`_ xlsx `openpyxl`_ `pyexcel-ods3`_ ods `pyexcel-ezodf`_, lxml `pyexcel-ods`_ ods `odfpy`_ ======================== ======================= ================= .. table:: Dedicated file reader and writers ======================== ======================= ================= Package name Supported file formats Dependencies ======================== ======================= ================= `pyexcel-xlsxw`_ xlsx(write only) `XlsxWriter`_ `pyexcel-libxlsxw`_ xlsx(write only) `libxlsxwriter`_ `pyexcel-xlsxr`_ xlsx(read only) lxml `pyexcel-xlsbr`_ xlsb(read only) pyxlsb `pyexcel-odsr`_ read only for ods, fods lxml `pyexcel-odsw`_ write only for ods loxun `pyexcel-htmlr`_ html(read only) lxml,html5lib `pyexcel-pdfr`_ pdf(read only) camelot ======================== ======================= ================= Plugin shopping guide ------------------------ Since 2020, all pyexcel-io plugins have dropped the support for python versions which are lower than 3.6. If you want to use any of those Python versions, please use pyexcel-io and its plugins versions that are lower than 0.6.0. Except csv files, xls, xlsx and ods files are a zip of a folder containing a lot of xml files The dedicated readers for excel files can stream read In order to manage the list of plugins installed, you need to use pip to add or remove a plugin. When you use virtualenv, you can have different plugins per virtual environment. In the situation where you have multiple plugins that does the same thing in your environment, you need to tell pyexcel which plugin to use per function call. For example, pyexcel-ods and pyexcel-odsr, and you want to get_array to use pyexcel-odsr. You need to append get_array(..., library='pyexcel-odsr'). .. _pyexcel-io: https://github.com/pyexcel/pyexcel-io .. _pyexcel-xls: https://github.com/pyexcel/pyexcel-xls .. _pyexcel-xlsx: https://github.com/pyexcel/pyexcel-xlsx .. _pyexcel-ods: https://github.com/pyexcel/pyexcel-ods .. _pyexcel-ods3: https://github.com/pyexcel/pyexcel-ods3 .. _pyexcel-odsr: https://github.com/pyexcel/pyexcel-odsr .. _pyexcel-odsw: https://github.com/pyexcel/pyexcel-odsw .. _pyexcel-pdfr: https://github.com/pyexcel/pyexcel-pdfr .. _pyexcel-xlsxw: https://github.com/pyexcel/pyexcel-xlsxw .. _pyexcel-libxlsxw: https://github.com/pyexcel/pyexcel-libxlsxw .. _pyexcel-xlsxr: https://github.com/pyexcel/pyexcel-xlsxr .. _pyexcel-xlsbr: https://github.com/pyexcel/pyexcel-xlsbr .. _pyexcel-htmlr: https://github.com/pyexcel/pyexcel-htmlr .. _xlrd: https://github.com/python-excel/xlrd .. _xlwt: https://github.com/python-excel/xlwt .. _openpyxl: https://bitbucket.org/openpyxl/openpyxl .. _XlsxWriter: https://github.com/jmcnamara/XlsxWriter .. _pyexcel-ezodf: https://github.com/pyexcel/pyexcel-ezodf .. _odfpy: https://github.com/eea/odfpy .. _libxlsxwriter: http://libxlsxwriter.github.io/getting_started.html .. rubric:: Footnotes .. [#f1] zipped csv file .. [#f2] zipped tsv file If you need to manipulate the data, you might do it yourself or use its brother library `pyexcel `__ . If you would like to extend it, you may use it to write your own extension to handle a specific file format. Installation ================================================================================ You can install pyexcel-io via pip: .. code-block:: bash $ pip install pyexcel-io or clone it and install it: .. code-block:: bash $ git clone https://github.com/pyexcel/pyexcel-io.git $ cd pyexcel-io $ python setup.py install Development guide ================================================================================ Development steps for code changes #. git clone https://github.com/pyexcel/pyexcel-io.git #. cd pyexcel-io Upgrade your setup tools and pip. They are needed for development and testing only: #. pip install --upgrade setuptools pip Then install relevant development requirements: #. pip install -r rnd_requirements.txt # if such a file exists #. pip install -r requirements.txt #. pip install -r tests/requirements.txt Once you have finished your changes, please provide test case(s), relevant documentation and update changelog.yml .. note:: As to rnd_requirements.txt, usually, it is created when a dependent library is not released. Once the dependecy is installed (will be released), the future version of the dependency in the requirements.txt will be valid. How to test your contribution ------------------------------ Although `nose` and `doctest` are both used in code testing, it is adviable that unit tests are put in tests. `doctest` is incorporated only to make sure the code examples in documentation remain valid across different development releases. On Linux/Unix systems, please launch your tests like this:: $ make On Windows, please issue this command:: > test.bat Before you commit ------------------------------ Please run:: $ make format so as to beautify your code otherwise your build may fail your unit test. License ================================================================================ New BSD License Change log ================================================================================ 0.6.6 - 31.1.2022 -------------------------------------------------------------------------------- **updated** #. `#112 `_: Log Empty Row Warning instead 'print' 0.6.5 - 08.10.2021 -------------------------------------------------------------------------------- **updated** #. `#109 `_: enable ods3 to have datetime 0.6.4 - 31.10.2020 -------------------------------------------------------------------------------- **updated** #. `#102 `_: skip columns from imported excel sheet. 0.6.3 - 12.10.2020 -------------------------------------------------------------------------------- **fixed** #. `#96 `_: regression: unknown file type shall trigger NoSupportingPluginFound **updated** #. extra dependencies uses 0.6.0 based plugins 0.6.2 - 7.10.2020 -------------------------------------------------------------------------------- **updated** #. `#94 `_: keep backward compatibility for pyexcel-xls 0.4.1 0.6.1 - 7.10.2020 -------------------------------------------------------------------------------- **removed** #. python 3.6 lower versions are no longer supported **updated** #. pyexcel-io plugin interface has been rewritten. PyInstaller user will be impacted. please read 'Packaging with Pyinstaller' in the documentation. #. new query set reader plugin. pyexcel<=0.6.4 has used intrusive way of getting query set source done. it is against the plugin interface. **fixed** #. `#74 `_: handle zip files which contain non-UTF-8 encoded files. **added** #. `#86 `_: allow trailing options, get_data(...keep_trailing_empty_cells=True). 0.5.20 - 17.7.2019 -------------------------------------------------------------------------------- **updated** #. `#70 `_: when the given file is a root directory, the error shall read it is not a file 0.5.19 - 14.7.2019 -------------------------------------------------------------------------------- **updated** #. `pyexcel#185 `_: handle stream conversion if file type(html) needs string content then bytes to handle 0.5.18 - 12.06.2019 -------------------------------------------------------------------------------- **updated** #. `#69 `_: Force file type(force_file_type) on write 0.5.17 - 04.04.2019 -------------------------------------------------------------------------------- **updated** #. `#68 `_: Raise IOError when the data file does not exist 0.5.16 - 19.03.2019 -------------------------------------------------------------------------------- **updated** #. `#67 `_: fix conversion issue for long type on python 2.7 for ods 0.5.15 - 16.03.2019 -------------------------------------------------------------------------------- **updated** #. `pyexcel-ods#33 `_: fix integer comparision error on i586 0.5.14 - 21.02.2019 -------------------------------------------------------------------------------- **updated** #. `#65 `_: add tests/__init__.py because python2.7 setup.py test needs it 0.5.13 - 12.02.2019 -------------------------------------------------------------------------------- **updated** #. `#63 `_: Version 0.5.12 prevents xslx and ods plugin from being loaded 0.5.12 - 9.02.2019 -------------------------------------------------------------------------------- **updated** #. `#60 `_: include tests in tar ball #. `#61 `_: enable python setup.py test 0.5.11 - 3.12.2018 -------------------------------------------------------------------------------- **updated** #. `#59 `_: Please use scan_plugins_regex, which lml 0.7 complains about 0.5.10 - 27.11.2018 -------------------------------------------------------------------------------- **added** #. `#57 `_, long type will not be written in ods. please use string type. And if the integer is equal or greater than 10 to the power of 16, it will not be written either in ods. In both situation, IntegerPrecisionLossError will be raised. And this version enables pyexcel-ods and pyexcel-ods3 to do so. 0.5.9.1 - 30.08.2018 -------------------------------------------------------------------------------- **updated** #. `#53 `_, upgrade lml dependency to at least 0.0.2 0.5.9 - 23.08.2018 -------------------------------------------------------------------------------- **added** #. `pyexcel#148 `_, support force_file_type 0.5.8 - 16.08.2018 -------------------------------------------------------------------------------- **added** #. `#49 `_, support additional options when detecting float values in csv format. default_float_nan, ignore_nan_text 0.5.7 - 02.05.2018 -------------------------------------------------------------------------------- **fixed** #. `#48 `_, turn off pep 0515 #. `#47 `_, csv reader cannot handle relative file names 0.5.6 - 11.01.2018 -------------------------------------------------------------------------------- **fixed** #. `#46 `_, expose `bulk_save` to developer 0.5.5 - 23.12.2017 -------------------------------------------------------------------------------- **fixed** #. Issue `#45 `_, csv reader throws exception because google app engine does not support mmap. People who are not working with google app engine, need not to take this update. Enjoy your Christmas break. 0.5.4 - 10.11.2017 -------------------------------------------------------------------------------- **updated** #. PR `#44 `_, use unicodewriter for csvz writers. 0.5.3 - 23.10.2017 -------------------------------------------------------------------------------- **updated** #. pyexcel `pyexcel#105 `_, remove gease from setup_requires, introduced by 0.5.2. #. remove python2.6 test support 0.5.2 - 20.10.2017 -------------------------------------------------------------------------------- **added** #. `pyexcel#103 `_, include LICENSE file in MANIFEST.in, meaning LICENSE file will appear in the released tar ball. 0.5.1 - 02.09.2017 -------------------------------------------------------------------------------- **Fixed** #. `pyexcel-ods#25 `_, Unwanted dependency on pyexcel. 0.5.0 - 30.08.2017 -------------------------------------------------------------------------------- **Added** #. Collect all data type conversion codes as service.py. **Updated** #. `#19 `_, use cString by default. For python, it will be a performance boost 0.4.4 - 08.08.2017 -------------------------------------------------------------------------------- **Updated** #. `#42 `_, raise exception if database table name does not match the sheet name 0.4.3 - 29.07.2017 -------------------------------------------------------------------------------- **Updated** #. `#41 `_, walk away gracefully when mmap is not available. 0.4.2 - 05.07.2017 -------------------------------------------------------------------------------- **Updated** #. `#37 `_, permanently fix the residue folder pyexcel by release all future releases in a clean clone. 0.4.1 - 29.06.2017 -------------------------------------------------------------------------------- **Updated** #. `#39 `_, raise exception when bulk save in django fails. Please `bulk_save=False` if you as the developer choose to save the records one by one if bulk_save cannot be used. However, exception in one-by-one save case will be raised as well. This change is made to raise exception in the first place so that you as the developer will be suprised when it was deployed in production. 0.4.0 - 19.06.2017 -------------------------------------------------------------------------------- **Updated** #. 'built-in' as the value to the parameter 'library' as parameter to invoke pyexcel-io's built-in csv, tsv, csvz, tsvz, django and sql won't work. It is renamed to 'pyexcel-io'. #. built-in csv, tsv, csvz, tsvz, django and sql are lazy loaded. #. pyexcel-io plugin interface has been updated. v0.3.x plugins won't work. #. `#32 `_, csv and csvz file handle are made sure to be closed. File close mechanism is enfored. #. iget_data function is introduced to cope with dangling file handle issue. **Removed** #. Removed plugin loading code and lml is used instead 0.3.4 - 18.05.2017 -------------------------------------------------------------------------------- **Updated** #. `#33 `_, handle mmap object differently given as file content. This issue has put in a priority to single sheet csv over multiple sheets in a single memory stream. The latter format is pyexcel own creation but is rarely used. In latter case, multiple_sheet=True should be passed along get_data. #. `#34 `_, treat mmap object as a file content. #. `#35 `_, encoding parameter take no effect when given along with file content #. use ZIP_DEFALTED to really do the compression 0.3.3 - 30.03.2017 -------------------------------------------------------------------------------- **Updated** #. `#31 `_, support pyinstaller 0.3.2 - 26.01.2017 -------------------------------------------------------------------------------- **Updated** #. `#29 `_, change skip_empty_rows to False by default 0.3.1 - 21.01.2017 -------------------------------------------------------------------------------- **Added** #. updated versions of extra packages **Updated** #. `#23 `_, provide helpful message when old pyexcel plugin exists #. restored previously available diagnosis message for missing libraries 0.3.0 - 22.12.2016 -------------------------------------------------------------------------------- **Added** #. lazy loading of plugins. for example, pyexcel-xls is not entirely loaded until xls format is used at its first attempted reading or writing. Since it is loaded, it will not be loaded in the second io action. #. `pyexcel-xls#11 `_, make case-insensitive for file type 0.2.6 - 21.12.2016 -------------------------------------------------------------------------------- **Updated** #. `#24 `__, pass on batch_size 0.2.5 - 20.12.2016 -------------------------------------------------------------------------------- **Updated** #. `#26 `__, performance issue with getting the number of columns. 0.2.4 - 24.11.2016 -------------------------------------------------------------------------------- **Updated** #. `#23 `__, Failed to convert long integer string in python 2 to its actual value 0.2.3 - 16.09.2016 -------------------------------------------------------------------------------- **Added** #. `#21 `__, choose subset from data base tables for export #. `#22 `__, custom renderer if given `row_renderer` as parameter. 0.2.2 - 31.08.2016 -------------------------------------------------------------------------------- **Added** #. support pagination. two pairs: start_row, row_limit and start_column, column_limit help you deal with large files. #. `skip_empty_rows=True` was introduced. To include empty rows, put it to False. **Updated** #. `#20 `__, pyexcel-io attempts to parse cell contents of 'infinity' as a float/int, crashes 0.2.1 - 11.07.2016 -------------------------------------------------------------------------------- **Added** #. csv format: handle utf-16 encoded csv files. Potentially being able to decode other formats if correct "encoding" is provided #. csv format: write utf-16 encoded files. Potentially other encoding is also supported #. support stdin as input stream and stdout as output stream **Updated** #. Attention, user of pyexcel-io! No longer io stream validation is performed in python 3. The guideline is: io.StringIO for csv, tsv only, otherwise BytesIO for xlsx, xls, ods. You can use RWManager.get_io to produce a correct stream type for you. #. `#15 `__, support foreign django/sql foreign key 0.2.0 - 01.06.2016 -------------------------------------------------------------------------------- **Added** #. autoload of pyexcel-io plugins #. auto detect `datetime`, `float` and `int`. Detection can be switched off by `auto_detect_datetime`, `auto_detect_float`, `auto_detect_int` 0.1.0 - 17.01.2016 -------------------------------------------------------------------------------- **Added** #. yield key word to return generator as content ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1643486962.0 pyexcel-io-0.6.6/pyexcel_io.egg-info/SOURCES.txt0000644000175100001710000000477200000000000020671 0ustar00runnerdockerCHANGELOG.rst CONTRIBUTORS.rst LICENSE MANIFEST.in Makefile README.rst setup.cfg setup.py test.sh docs/Makefile docs/make.bat docs/source/common_parameters.rst docs/source/conf.py docs/source/csvz.rst docs/source/django.rst docs/source/extendedcsv.rst docs/source/extensions.rst docs/source/index.rst docs/source/pagination.rst docs/source/plaincsv.rst docs/source/pyinstaller.rst docs/source/renderer.rst docs/source/sqlalchemy.rst docs/source/_static/theme_overrides.css docs/source/api/pyexcel_io.get_data.rst docs/source/api/pyexcel_io.iget_data.rst docs/source/api/pyexcel_io.save_data.rst pyexcel_io/__init__.py pyexcel_io/_compact.py pyexcel_io/book.py pyexcel_io/constants.py pyexcel_io/exceptions.py pyexcel_io/io.py pyexcel_io/manager.py pyexcel_io/plugins.py pyexcel_io/reader.py pyexcel_io/service.py pyexcel_io/sheet.py pyexcel_io/utils.py pyexcel_io/writer.py pyexcel_io.egg-info/PKG-INFO pyexcel_io.egg-info/SOURCES.txt pyexcel_io.egg-info/dependency_links.txt pyexcel_io.egg-info/not-zip-safe pyexcel_io.egg-info/requires.txt pyexcel_io.egg-info/top_level.txt pyexcel_io/database/__init__.py pyexcel_io/database/common.py pyexcel_io/database/querysets.py pyexcel_io/database/exporters/__init__.py pyexcel_io/database/exporters/django.py pyexcel_io/database/exporters/queryset.py pyexcel_io/database/exporters/sqlalchemy.py pyexcel_io/database/importers/__init__.py pyexcel_io/database/importers/django.py pyexcel_io/database/importers/sqlalchemy.py pyexcel_io/plugin_api/__init__.py pyexcel_io/plugin_api/abstract_reader.py pyexcel_io/plugin_api/abstract_sheet.py pyexcel_io/plugin_api/abstract_writer.py pyexcel_io/readers/__init__.py pyexcel_io/readers/csv_content.py pyexcel_io/readers/csv_in_file.py pyexcel_io/readers/csv_in_memory.py pyexcel_io/readers/csv_sheet.py pyexcel_io/readers/csvz.py pyexcel_io/writers/__init__.py pyexcel_io/writers/csv_in_file.py pyexcel_io/writers/csv_in_memory.py pyexcel_io/writers/csv_sheet.py pyexcel_io/writers/csvz_sheet.py pyexcel_io/writers/csvz_writer.py tests/__init__.py tests/requirements.txt tests/test_base.py tests/test_book.py tests/test_csv_book.py tests/test_django_book.py tests/test_filter.py tests/test_io.py tests/test_issues.py tests/test_new_csv_book.py tests/test_new_csvz_book.py tests/test_plugin_api.py tests/test_pyexcel_integration.py tests/test_renderer.py tests/test_service.py tests/test_sheet.py tests/test_sql_book.py tests/fixtures/csv-encoding-utf16.csv tests/fixtures/force_file_type.txt tests/fixtures/issue20.csv tests/fixtures/issue23.csv tests/fixtures/test.csv././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1643486962.0 pyexcel-io-0.6.6/pyexcel_io.egg-info/dependency_links.txt0000644000175100001710000000000100000000000023041 0ustar00runnerdocker ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1643486962.0 pyexcel-io-0.6.6/pyexcel_io.egg-info/not-zip-safe0000644000175100001710000000000100000000000021221 0ustar00runnerdocker ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1643486962.0 pyexcel-io-0.6.6/pyexcel_io.egg-info/requires.txt0000644000175100001710000000013400000000000021371 0ustar00runnerdockerlml>=0.0.4 [ods] pyexcel-ods3>=0.6.0 [xls] pyexcel-xls>=0.6.0 [xlsx] pyexcel-xlsx>=0.6.0 ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1643486962.0 pyexcel-io-0.6.6/pyexcel_io.egg-info/top_level.txt0000644000175100001710000000001300000000000021517 0ustar00runnerdockerpyexcel_io ././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1643486962.4502711 pyexcel-io-0.6.6/setup.cfg0000644000175100001710000000015500000000000014763 0ustar00runnerdocker[metadata] description-file = README.rst [bdist_wheel] universal = 1 [egg_info] tag_build = tag_date = 0 ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1643486950.0 pyexcel-io-0.6.6/setup.py0000644000175100001710000001300300000000000014650 0ustar00runnerdocker#!/usr/bin/env python3 """ Template by pypi-mobans """ import os import sys import codecs import locale import platform from shutil import rmtree from setuptools import Command, setup, find_packages PY2 = sys.version_info[0] == 2 PY26 = PY2 and sys.version_info[1] < 7 PY33 = sys.version_info < (3, 4) # Work around mbcs bug in distutils. # http://bugs.python.org/issue10945 # This work around is only if a project supports Python < 3.4 # Work around for locale not being set try: lc = locale.getlocale() pf = platform.system() if pf != "Windows" and lc == (None, None): locale.setlocale(locale.LC_ALL, "C.UTF-8") except (ValueError, UnicodeError, locale.Error): locale.setlocale(locale.LC_ALL, "en_US.UTF-8") NAME = "pyexcel-io" AUTHOR = "C.W." VERSION = "0.6.6" EMAIL = "info@pyexcel.org" LICENSE = "New BSD" DESCRIPTION = ( "A python library to read and write structured data in csv, zipped csv" + "format and to/from databases" ) URL = "https://github.com/pyexcel/pyexcel-io" DOWNLOAD_URL = "%s/archive/0.6.6.tar.gz" % URL FILES = ["README.rst", "CHANGELOG.rst"] KEYWORDS = [ "python", "API", "tsv", "tsvz", "csv", "csvz", "django", "sqlalchemy", ] CLASSIFIERS = [ "Topic :: Software Development :: Libraries", "Programming Language :: Python", "Intended Audience :: Developers", "Programming Language :: Python :: 3 :: Only", "Programming Language :: Python :: 3.6", "Programming Language :: Python :: 3.7", "Programming Language :: Python :: 3.8", 'License :: OSI Approved :: BSD License', 'Programming Language :: Python :: Implementation :: PyPy' ] PYTHON_REQUIRES = ">=3.6" INSTALL_REQUIRES = [ "lml>=0.0.4", ] SETUP_COMMANDS = {} PACKAGES = find_packages(exclude=["ez_setup", "examples", "tests", "tests.*"]) EXTRAS_REQUIRE = { "xls": ['pyexcel-xls>=0.6.0'], "xlsx": ['pyexcel-xlsx>=0.6.0'], "ods": ['pyexcel-ods3>=0.6.0'], } # You do not need to read beyond this line PUBLISH_COMMAND = "{0} setup.py sdist bdist_wheel upload -r pypi".format(sys.executable) HERE = os.path.abspath(os.path.dirname(__file__)) GS_COMMAND = ("gease pyexcel-io v0.6.6 " + "Find 0.6.6 in changelog for more details") NO_GS_MESSAGE = ("Automatic github release is disabled. " + "Please install gease to enable it.") UPLOAD_FAILED_MSG = ( 'Upload failed. please run "%s" yourself.' % PUBLISH_COMMAND) class PublishCommand(Command): """Support setup.py upload.""" description = "Build and publish the package on github and pypi" user_options = [] @staticmethod def status(s): """Prints things in bold.""" print("\033[1m{0}\033[0m".format(s)) def initialize_options(self): pass def finalize_options(self): pass def run(self): try: self.status("Removing previous builds...") rmtree(os.path.join(HERE, "dist")) rmtree(os.path.join(HERE, "build")) rmtree(os.path.join(HERE, "pyexcel_io.egg-info")) except OSError: pass self.status("Building Source and Wheel (universal) distribution...") run_status = True if has_gease(): run_status = os.system(GS_COMMAND) == 0 else: self.status(NO_GS_MESSAGE) if run_status: if os.system(PUBLISH_COMMAND) != 0: self.status(UPLOAD_FAILED_MSG) sys.exit() SETUP_COMMANDS.update({ "publish": PublishCommand }) def has_gease(): """ test if github release command is installed visit http://github.com/moremoban/gease for more info """ try: import gease # noqa return True except ImportError: return False def read_files(*files): """Read files into setup""" text = "" for single_file in files: content = read(single_file) text = text + content + "\n" return text def read(afile): """Read a file into setup""" the_relative_file = os.path.join(HERE, afile) with codecs.open(the_relative_file, "r", "utf-8") as opened_file: content = filter_out_test_code(opened_file) content = "".join(list(content)) return content def filter_out_test_code(file_handle): found_test_code = False for line in file_handle.readlines(): if line.startswith(".. testcode:"): found_test_code = True continue if found_test_code is True: if line.startswith(" "): continue else: empty_line = line.strip() if len(empty_line) == 0: continue else: found_test_code = False yield line else: for keyword in ["|version|", "|today|"]: if keyword in line: break else: yield line if __name__ == "__main__": setup( test_suite="tests", name=NAME, author=AUTHOR, version=VERSION, author_email=EMAIL, description=DESCRIPTION, url=URL, download_url=DOWNLOAD_URL, long_description=read_files(*FILES), license=LICENSE, keywords=KEYWORDS, python_requires=PYTHON_REQUIRES, extras_require=EXTRAS_REQUIRE, tests_require=["nose"], install_requires=INSTALL_REQUIRES, packages=PACKAGES, include_package_data=True, zip_safe=False, classifiers=CLASSIFIERS, cmdclass=SETUP_COMMANDS ) ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1643486950.0 pyexcel-io-0.6.6/test.sh0000644000175100001710000000026100000000000014453 0ustar00runnerdocker#/bin/bash pip freeze nosetests --with-coverage --cover-package pyexcel_io --cover-package tests tests --with-doctest --doctest-extension=.rst README.rst docs/source pyexcel_io ././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1643486962.4502711 pyexcel-io-0.6.6/tests/0000755000175100001710000000000000000000000014303 5ustar00runnerdocker././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1643486950.0 pyexcel-io-0.6.6/tests/__init__.py0000644000175100001710000000011200000000000016406 0ustar00runnerdocker# needed for Python 2.7, python setup.py test to discover tests directory ././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1643486962.4502711 pyexcel-io-0.6.6/tests/fixtures/0000755000175100001710000000000000000000000016154 5ustar00runnerdocker././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1643486950.0 pyexcel-io-0.6.6/tests/fixtures/csv-encoding-utf16.csv0000644000175100001710000000013000000000000022205 0ustar00runnerdockerkkilhdt,Matkakirjoituksia,Matkatoimistot././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1643486950.0 pyexcel-io-0.6.6/tests/fixtures/force_file_type.txt0000644000175100001710000000000600000000000022047 0ustar00runnerdocker1,2,3 ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1643486950.0 pyexcel-io-0.6.6/tests/fixtures/issue20.csv0000644000175100001710000000002600000000000020161 0ustar00runnerdockerto,infinity,and,beyond././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1643486950.0 pyexcel-io-0.6.6/tests/fixtures/issue23.csv0000644000175100001710000000016700000000000020172 0ustar00runnerdocker"8204235414504252",inf "82042354145042521",-inf "820423541450425216",0 "820423541450425247",1 "8204235414504252490",1.1././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1643486950.0 pyexcel-io-0.6.6/tests/fixtures/test.csv0000644000175100001710000000002100000000000017641 0ustar00runnerdocker1,2,3 4,5,6 7,8,9././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1643486950.0 pyexcel-io-0.6.6/tests/requirements.txt0000644000175100001710000000025500000000000017571 0ustar00runnerdockernose mock;python_version<"3" codecov coverage flake8 black isort collective.checkdocs pygments moban moban_jinja2_github pyexcel pyexcel-xls==0.5.9 SQLAlchemy pyexcel-xlsxw ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1643486950.0 pyexcel-io-0.6.6/tests/test_base.py0000644000175100001710000000446300000000000016635 0ustar00runnerdockerfrom pyexcel_io.book import BookWriter from pyexcel_io.sheet import SheetReader, SheetWriter, NamedContent from pyexcel_io.utils import is_empty_array from nose.tools import raises @raises(NotImplementedError) def test_book_writer(): book = BookWriter() book.create_sheet("test") def test_is_empty_array(): a = ["", "", "", ""] assert is_empty_array(a) is True b = [1, "", "", ""] assert is_empty_array(b) is False class ArrayWriter(SheetWriter): def set_sheet_name(self, name): self._native_sheet.name = name def write_row(self, array): self._native_sheet.payload.append(array) class TestSheetReader: @raises(Exception) def test_abstractness(self): reader = SheetReader("test") reader.cell_value(1, 2) @raises(Exception) def test_number_of_columns(self): reader = SheetReader("test") reader.number_of_columns() @raises(Exception) def test_number_of_rows(self): reader = SheetReader("test") reader.number_of_rows() def test_to_array(self): name = "test" class B(SheetReader): @property def name(self): return self._native_sheet def to_array(self): pass b = B(name) b.to_array() assert b.name == name class TestSheetWriter: @raises(NotImplementedError) def test_abstractness(self): writer = SheetWriter("te", "st", "abstract") writer.write_row([]) def test_inheritance(self): class D(SheetWriter): def write_row(self, row): pass d = D("t", "e", "s") d.write_row([11, 11]) def test_writer(self): native_sheet = NamedContent("test", []) content = [[1, 2], [3, 4], [5, 6]] writer = ArrayWriter(None, native_sheet, "test") writer.write_row(content[0]) writer.write_array(content[1:]) assert native_sheet.payload == content def test_writer2(self): native_sheet = NamedContent("test", []) content = [[1, 2], [3, 4], [5, 6]] writer = ArrayWriter(None, native_sheet, None) writer.write_row(content[0]) writer.write_array(content[1:]) assert native_sheet.payload == content assert native_sheet.name == "pyexcel_sheet1" ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1643486950.0 pyexcel-io-0.6.6/tests/test_book.py0000644000175100001710000000206300000000000016647 0ustar00runnerdockerfrom pyexcel_io.book import ( BookReader, BookWriter, RWInterface, _convert_content_to_stream, ) from pyexcel_io._compact import BytesIO, StringIO from nose.tools import raises @raises(NotImplementedError) def test_rwinterface(): interface = RWInterface() interface.open(1) @raises(NotImplementedError) def test_rwinterface_2(): interface = RWInterface() interface.open_stream(1) @raises(NotImplementedError) def test_book_reader(): reader = BookReader() reader.read_sheet(1) @raises(IOError) def test_book_reader_open_stream(): reader = BookReader() reader.open_stream("a string") @raises(IOError) def test_book_writer(): writer = BookWriter() writer.open_stream("a string") def test_convert_to_bytes_stream(): file_content = b"test" stream = _convert_content_to_stream(file_content, "csv") assert isinstance(stream, StringIO) def test_convert_to_string_stream(): file_content = "test" stream = _convert_content_to_stream(file_content, "csvz") assert isinstance(stream, BytesIO) ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1643486950.0 pyexcel-io-0.6.6/tests/test_csv_book.py0000644000175100001710000001353400000000000017527 0ustar00runnerdockerimport os from textwrap import dedent from unittest import TestCase import pyexcel_io.manager as manager from pyexcel_io import get_data from pyexcel_io.sheet import NamedContent from pyexcel_io.reader import EncapsulatedSheetReader from pyexcel_io._compact import BytesIO, StringIO from pyexcel_io.readers.csv_sheet import ( CSVFileReader, CSVSheetReader, CSVinMemoryReader, ) from pyexcel_io.writers.csv_sheet import CSVFileWriter, CSVMemoryWriter from nose.tools import eq_, raises class TestReaders(TestCase): def setUp(self): self.file_type = "csv" self.test_file = "csv_book." + self.file_type self.data = [["1", "2", "3"], ["4", "5", "6"], ["7", "8", "9"]] self.expected_data = [[1, 2, 3], [4, 5, 6], [7, 8, 9]] with open(self.test_file, "w") as f: for row in self.data: f.write(",".join(row) + "\n") @raises(NotImplementedError) def test_sheet_reader(self): sheet = CSVSheetReader(self.test_file) sheet.get_file_handle() def test_sheet_file_reader(self): r = EncapsulatedSheetReader( CSVFileReader(NamedContent(self.file_type, self.test_file)) ) result = list(r.to_array()) self.assertEqual(result, self.expected_data) def test_sheet_memory_reader(self): io = manager.get_io(self.file_type) with open(self.test_file, "r") as f: io.write(f.read()) io.seek(0) r = EncapsulatedSheetReader( CSVinMemoryReader(NamedContent(self.file_type, io)) ) result = list(r.to_array()) self.assertEqual(result, self.expected_data) def tearDown(self): os.unlink(self.test_file) class TestWriter(TestCase): def setUp(self): self.file_type = "csv" self.test_file = "csv_book." + self.file_type self.data = [[1, 2, 3], [4, 5, 6], [7, 8, 9]] self.result = dedent( """ 1,2,3 4,5,6 7,8,9 """ ).strip("\n") def test_sheet_writer(self): w = CSVFileWriter(self.test_file, None) for row in self.data: w.write_row(row) w.close() with open(self.test_file, "r") as f: content = f.read().replace("\r", "") self.assertEqual(content.strip("\n"), self.result) def tearDown(self): os.unlink(self.test_file) class TestMemoryWriter(TestCase): def setUp(self): self.file_type = "csv" self.test_file = "csv_book." + self.file_type self.data = [[1, 2, 3], [4, 5, 6], [7, 8, 9]] self.result = dedent( """ 1,2,3 4,5,6 7,8,9 """ ).strip("\n") def test_sheet_writer_to_memory(self): io = manager.get_io(self.file_type) w = CSVMemoryWriter(io, None, single_sheet_in_book=True) for row in self.data: w.write_row(row) w.close() content = io.getvalue().replace("\r", "") self.assertEqual(content.strip("\n"), self.result) class TestNonUniformCSV(TestCase): def setUp(self): self.file_type = "csv" self.test_file = "csv_book." + self.file_type self.data = [["1"], ["4", "5", "6", "", ""], ["", "7"]] with open(self.test_file, "w") as f: for row in self.data: f.write(",".join(row) + "\n") def test_sheet_file_reader(self): r = EncapsulatedSheetReader( CSVFileReader(NamedContent(self.file_type, self.test_file)) ) result = list(r.to_array()) self.assertEqual(result, [[1], [4, 5, 6], ["", 7]]) def test_sheet_file_reader_with_trailing_empty_cells(self): r = EncapsulatedSheetReader( CSVFileReader(NamedContent(self.file_type, self.test_file)), keep_trailing_empty_cells=True, ) result = list(r.to_array()) self.assertEqual(result, [[1], [4, 5, 6, "", ""], ["", 7]]) def test_get_data_with_trailing_empty_cells(self): result = get_data(self.test_file, keep_trailing_empty_cells=True) self.assertEqual( result[self.test_file], [[1], [4, 5, 6, "", ""], ["", 7]] ) def tearDown(self): os.unlink(self.test_file) def test_utf16_decoding(): test_file = os.path.join("tests", "fixtures", "csv-encoding-utf16.csv") reader = EncapsulatedSheetReader( CSVFileReader(NamedContent("csv", test_file), encoding="utf-16") ) content = list(reader.to_array()) expected = [["Äkkilähdöt", "Matkakirjoituksia", "Matkatoimistot"]] eq_(content, expected) def test_utf16_encoding(): content = [[u"Äkkilähdöt", u"Matkakirjoituksia", u"Matkatoimistot"]] test_file = "test-utf16-encoding.csv" writer = CSVFileWriter( test_file, None, encoding="utf-16", lineterminator="\n" ) writer.write_array(content) writer.close() with open(test_file, "rb") as f: actual = f.read().decode("utf-16") eq_(actual, "Äkkilähdöt,Matkakirjoituksia,Matkatoimistot\n") os.unlink(test_file) def test_utf16_memory_decoding(): test_content = u"Äkkilähdöt,Matkakirjoituksia,Matkatoimistot" test_content = BytesIO(test_content.encode("utf-16")) reader = EncapsulatedSheetReader( CSVinMemoryReader(NamedContent("csv", test_content), encoding="utf-16") ) content = list(reader.to_array()) expected = [["Äkkilähdöt", "Matkakirjoituksia", "Matkatoimistot"]] eq_(content, expected) def test_utf16_memory_encoding(): content = [[u"Äkkilähdöt", u"Matkakirjoituksia", u"Matkatoimistot"]] io = StringIO() writer = CSVMemoryWriter( io, None, lineterminator="\n", single_sheet_in_book=True, encoding="utf-16", ) writer.write_array(content) actual = io.getvalue() eq_(actual, u"Äkkilähdöt,Matkakirjoituksia,Matkatoimistot\n") ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1643486950.0 pyexcel-io-0.6.6/tests/test_django_book.py0000644000175100001710000003435200000000000020177 0ustar00runnerdockerfrom pyexcel_io import save_data from pyexcel_io.reader import EncapsulatedSheetReader from pyexcel_io._compact import OrderedDict from pyexcel_io.constants import DB_DJANGO from pyexcel_io.database.common import ( DjangoModelExporter, DjangoModelImporter, DjangoModelExportAdapter, DjangoModelImportAdapter, ) from pyexcel_io.database.exporters.django import ( DjangoBookReader, DjangoModelReader, ) from pyexcel_io.database.importers.django import ( DjangoBookWriter, DjangoModelWriter, ) from nose.tools import eq_, raises class Package: def __init__(self, raiseException=False, **keywords): self.keywords = keywords self.raiseException = raiseException def get_content(self): return self.keywords def save(self): if self.raiseException: raise Exception("test exception") else: pass class Attributable: def __init__(self, adict): self.mydict = adict def __getattr__(self, field): return self.mydict[field] class Objects: def __init__(self): self.objs = [] def bulk_create(self, objs, batch_size): self.objs = [o.get_content() for o in objs] self.batch_size = batch_size def all(self): return [Attributable(o) for o in self.objs] class Field: def __init__(self, name): self.attname = name class Meta: instance = 1 def __init__(self): self.model_name = "Sheet%d" % Meta.instance self.concrete_fields = [] Meta.instance = Meta.instance + 1 def update(self, data): for f in data: self.concrete_fields.append(Field(f)) class FakeDjangoModel: def __init__(self): self.objects = Objects() self._meta = Meta() def __call__(self, **keywords): return Package(**keywords) class ExceptionObjects(Objects): def bulk_create(self, objs, batch_size): raise Exception("faked exception") class FakeExceptionDjangoModel(FakeDjangoModel): def __init__(self, raiseException=False): self.objects = ExceptionObjects() self._meta = Meta() self.raiseException = raiseException def __call__(self, **keywords): return Package(raiseException=self.raiseException, **keywords) class TestException: def setUp(self): self.data = [["X", "Y", "Z"], [1, 2, 3], [4, 5, 6]] self.result = [{"Y": 2, "X": 1, "Z": 3}, {"Y": 5, "X": 4, "Z": 6}] @raises(Exception) def test_bulk_save_to_django_model_with_exception(self): model = FakeExceptionDjangoModel() adapter = DjangoModelImportAdapter(model) adapter.column_names = self.data[0] writer = DjangoModelWriter(None, adapter) writer.write_array(self.data[1:]) writer.close() @raises(Exception) def test_single_save_to_django_model_with_exception(self): model = FakeExceptionDjangoModel(raiseException=True) adapter = DjangoModelImportAdapter(model) adapter.column_names = self.data[0] writer = DjangoModelWriter(None, adapter) writer.write_array(self.data[1:]) writer.close() class TestSheet: def setUp(self): self.data = [["X", "Y", "Z"], [1, 2, 3], [4, 5, 6]] self.result = [{"Y": 2, "X": 1, "Z": 3}, {"Y": 5, "X": 4, "Z": 6}] def test_sheet_save_to_django_model(self): model = FakeDjangoModel() adapter = DjangoModelImportAdapter(model) adapter.column_names = self.data[0] writer = DjangoModelWriter(None, adapter) writer.write_array(self.data[1:]) writer.close() eq_(model.objects.objs, self.result) def test_sheet_save_to_django_model_with_empty_array(self): model = FakeDjangoModel() data = [["X", "Y", "Z"], ["", "", ""], [1, 2, 3], [4, 5, 6]] adapter = DjangoModelImportAdapter(model) adapter.column_names = self.data[0] writer = DjangoModelWriter(None, adapter) writer.write_array(data[1:]) writer.close() assert model.objects.objs == self.result def test_sheet_save_to_django_model_3(self): model = FakeDjangoModel() def wrapper(row): row[0] = row[0] + 1 return row adapter = DjangoModelImportAdapter(model) adapter.column_names = self.data[0] adapter.row_initializer = wrapper writer = DjangoModelWriter(None, adapter) writer.write_array(self.data[1:]) writer.close() eq_( model.objects.objs, [ {"Y": 2, "X": 2, "Z": 3}, {"Y": 5, "X": 5, "Z": 6}, ], ) def test_sheet_save_to_django_model_skip_me(self): model = FakeDjangoModel() def wrapper(row): if row[0] == 4: return None else: return row adapter = DjangoModelImportAdapter(model) adapter.column_names = self.data[0] adapter.row_initializer = wrapper writer = DjangoModelWriter(None, adapter) writer.write_array(self.data[1:]) writer.close() eq_(model.objects.objs, [{"Y": 2, "X": 1, "Z": 3}]) def test_load_sheet_from_django_model(self): model = FakeDjangoModel() importer = DjangoModelImporter() adapter = DjangoModelImportAdapter(model) adapter.column_names = self.data[0] importer.append(adapter) save_data( importer, {adapter.get_name(): self.data[1:]}, file_type=DB_DJANGO ) assert model.objects.objs == self.result model._meta.update(["X", "Y", "Z"]) reader = DjangoModelReader(model) data = reader.to_array() assert list(data) == self.data def test_load_and_format_sheet_from_django_model(self): model = FakeDjangoModel() importer = DjangoModelImporter() adapter = DjangoModelImportAdapter(model) adapter.column_names = self.data[0] importer.append(adapter) save_data( importer, {adapter.get_name(): self.data[1:]}, file_type=DB_DJANGO ) assert model.objects.objs == self.result model._meta.update(["X", "Y", "Z"]) def row_renderer(row): return [str(element) for element in row] # the key point of this test case reader = EncapsulatedSheetReader( DjangoModelReader(model), row_renderer=row_renderer ) data = reader.to_array() expected = [["X", "Y", "Z"], ["1", "2", "3"], ["4", "5", "6"]] eq_(list(data), expected) def test_mapping_array(self): data2 = [["A", "B", "C"], [1, 2, 3], [4, 5, 6]] model = FakeDjangoModel() adapter = DjangoModelImportAdapter(model) adapter.column_names = data2[0] adapter.column_name_mapping_dict = ["X", "Y", "Z"] writer = DjangoModelWriter(None, adapter) writer.write_array(data2[1:]) writer.close() eq_(model.objects.objs, self.result) def test_mapping_dict(self): data2 = [["A", "B", "C"], [1, 2, 3], [4, 5, 6]] mapdict = {"C": "Z", "A": "X", "B": "Y"} model = FakeDjangoModel() adapter = DjangoModelImportAdapter(model) adapter.column_names = data2[0] adapter.column_name_mapping_dict = mapdict writer = DjangoModelWriter(None, adapter) writer.write_array(data2[1:]) writer.close() eq_(model.objects.objs, self.result) def test_jumping_columns(self): data2 = [["D", "A", "B", "C"], [1, 1, 2, 3], [10, 4, 5, 6]] mapdict = {"C": "Z", "A": "X", "B": "Y"} model = FakeDjangoModel() adapter = DjangoModelImportAdapter(model) adapter.column_names = data2[0] adapter.column_name_mapping_dict = mapdict writer = DjangoModelWriter(None, adapter) writer.write_array(data2[1:]) writer.close() eq_(model.objects.objs, self.result) def test_empty_model(self): model = FakeDjangoModel() reader = DjangoModelReader(model) data = reader.to_array() eq_(list(data), [[], []]) class TestMultipleModels: def setUp(self): self.content = OrderedDict() self.content.update( {"Sheet1": [[u"X", u"Y", u"Z"], [1, 4, 7], [2, 5, 8], [3, 6, 9]]} ) self.content.update( {"Sheet2": [[u"A", u"B", u"C"], [1, 4, 7], [2, 5, 8], [3, 6, 9]]} ) self.result1 = [ {"Y": 4, "X": 1, "Z": 7}, {"Y": 5, "X": 2, "Z": 8}, {"Y": 6, "X": 3, "Z": 9}, ] self.result2 = [ {"B": 4, "A": 1, "C": 7}, {"B": 5, "A": 2, "C": 8}, {"B": 6, "A": 3, "C": 9}, ] def test_save_to_more_models(self): sample_size = 10 model1 = FakeDjangoModel() model2 = FakeDjangoModel() importer = DjangoModelImporter() adapter1 = DjangoModelImportAdapter(model1) adapter1.column_names = self.content["Sheet1"][0] adapter2 = DjangoModelImportAdapter(model2) adapter2.column_names = self.content["Sheet2"][0] importer.append(adapter1) importer.append(adapter2) to_store = { adapter1.get_name(): self.content["Sheet1"][1:], adapter2.get_name(): self.content["Sheet2"][1:], } writer = DjangoBookWriter(importer, "django", batch_size=sample_size) writer.write(to_store) writer.close() assert model1.objects.objs == self.result1 assert model2.objects.objs == self.result2 assert model1.objects.batch_size == sample_size def test_bulk_save_flag(self): sample_size = 10 model1 = FakeDjangoModel() model2 = FakeDjangoModel() importer = DjangoModelImporter() adapter1 = DjangoModelImportAdapter(model1) adapter1.column_names = self.content["Sheet1"][0] adapter2 = DjangoModelImportAdapter(model2) adapter2.column_names = self.content["Sheet2"][0] importer.append(adapter1) importer.append(adapter2) to_store = { adapter1.get_name(): self.content["Sheet1"][1:], adapter2.get_name(): self.content["Sheet2"][1:], } writer = DjangoBookWriter( importer, "django", batch_size=sample_size, bulk_save=False ) writer.write(to_store) writer.close() assert model1.objects.objs == [] def test_reading_from_more_models(self): model1 = FakeDjangoModel() model2 = FakeDjangoModel() importer = DjangoModelImporter() adapter1 = DjangoModelImportAdapter(model1) adapter1.column_names = self.content["Sheet1"][0] adapter2 = DjangoModelImportAdapter(model2) adapter2.column_names = self.content["Sheet2"][0] importer.append(adapter1) importer.append(adapter2) to_store = { adapter1.get_name(): self.content["Sheet1"][1:], adapter2.get_name(): self.content["Sheet2"][1:], } save_data(importer, to_store, file_type=DB_DJANGO) assert model1.objects.objs == self.result1 assert model2.objects.objs == self.result2 model1._meta.model_name = "Sheet1" model2._meta.model_name = "Sheet2" model1._meta.update(["X", "Y", "Z"]) model2._meta.update(["A", "B", "C"]) exporter = DjangoModelExporter() adapter1 = DjangoModelExportAdapter(model1) adapter2 = DjangoModelExportAdapter(model2) exporter.append(adapter1) exporter.append(adapter2) reader = DjangoBookReader(exporter, "django") result = read_all(reader) for key in result: result[key] = list(result[key]) eq_(result, self.content) @raises(Exception) def test_special_case_where_only_one_model_used(self): model1 = FakeDjangoModel() importer = DjangoModelImporter() adapter = DjangoModelImportAdapter(model1) adapter.column_names = self.content["Sheet1"][0] importer.append(adapter) to_store = { adapter.get_name(): self.content["Sheet1"][1:], "Sheet2": self.content["Sheet2"][1:], } save_data(importer, to_store, file_type=DB_DJANGO) class TestFilter: def setUp(self): self.data = [["X", "Y", "Z"], [1, 2, 3], [4, 5, 6]] self.result = [{"Y": 2, "X": 1, "Z": 3}, {"Y": 5, "X": 4, "Z": 6}] self.model = FakeDjangoModel() importer = DjangoModelImporter() adapter = DjangoModelImportAdapter(self.model) adapter.column_names = self.data[0] importer.append(adapter) save_data( importer, {adapter.get_name(): self.data[1:]}, file_type=DB_DJANGO ) assert self.model.objects.objs == self.result self.model._meta.update(["X", "Y", "Z"]) def test_load_sheet_from_django_model_with_filter(self): reader = EncapsulatedSheetReader( DjangoModelReader(self.model), start_row=0, row_limit=2 ) data = reader.to_array() expected = [["X", "Y", "Z"], [1, 2, 3]] eq_(list(data), expected) def test_load_sheet_from_django_model_with_filter_1(self): reader = EncapsulatedSheetReader( DjangoModelReader(self.model), start_row=1, row_limit=3 ) data = reader.to_array() expected = [[1, 2, 3], [4, 5, 6]] eq_(list(data), expected) def test_load_sheet_from_django_model_with_filter_2(self): reader = EncapsulatedSheetReader( DjangoModelReader(self.model), start_column=1 ) data = reader.to_array() expected = [["Y", "Z"], [2, 3], [5, 6]] eq_(list(data), expected) def test_load_sheet_from_django_model_with_filter_3(self): reader = EncapsulatedSheetReader( DjangoModelReader(self.model), start_column=1, column_limit=1 ) data = reader.to_array() expected = [["Y"], [2], [5]] eq_(list(data), expected) def test_django_model_import_adapter(): adapter = DjangoModelImportAdapter(FakeDjangoModel) adapter.column_names = ["a"] adapter.row_initializer = "abc" eq_(adapter.row_initializer, "abc") def read_all(reader): result = OrderedDict() for index, sheet in enumerate(reader.content_array): result.update({sheet.name: reader.read_sheet(index).to_array()}) return result ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1643486950.0 pyexcel-io-0.6.6/tests/test_filter.py0000644000175100001710000000477200000000000017213 0ustar00runnerdockerimport os import pyexcel_io.constants as constants from pyexcel_io import get_data, save_data from pyexcel_io.utils import _index_filter from nose.tools import eq_ def test_index_filter(): current_index, start, limit, expected = (0, 1, -1, constants.SKIP_DATA) eq_(_index_filter(current_index, start, limit), expected) current_index, start, limit, expected = (2, 1, -1, constants.TAKE_DATA) eq_(_index_filter(current_index, start, limit), expected) current_index, start, limit, expected = (2, 1, 10, constants.TAKE_DATA) eq_(_index_filter(current_index, start, limit), expected) current_index, start, limit, expected = ( 100, 1, 10, constants.STOP_ITERATION, ) eq_(_index_filter(current_index, start, limit), expected) class TestFilter: def setUp(self): self.test_file = "test_filter.csv" sample = [ [1, 21, 31], [2, 22, 32], [3, 23, 33], [4, 24, 34], [5, 25, 35], [6, 26, 36], ] save_data(self.test_file, sample) def test_filter_row(self): filtered_data = get_data(self.test_file, start_row=3) expected = [[4, 24, 34], [5, 25, 35], [6, 26, 36]] eq_(filtered_data[self.test_file], expected) def test_filter_row_2(self): filtered_data = get_data(self.test_file, start_row=3, row_limit=1) expected = [[4, 24, 34]] eq_(filtered_data[self.test_file], expected) def test_filter_column(self): filtered_data = get_data(self.test_file, start_column=1) expected = [[21, 31], [22, 32], [23, 33], [24, 34], [25, 35], [26, 36]] eq_(filtered_data[self.test_file], expected) def test_filter_column_2(self): filtered_data = get_data( self.test_file, start_column=1, column_limit=1 ) expected = [[21], [22], [23], [24], [25], [26]] eq_(filtered_data[self.test_file], expected) def test_filter_both_ways(self): filtered_data = get_data(self.test_file, start_column=1, start_row=3) expected = [[24, 34], [25, 35], [26, 36]] eq_(filtered_data[self.test_file], expected) def test_filter_both_ways_2(self): filtered_data = get_data( self.test_file, start_column=1, column_limit=1, start_row=3, row_limit=1, ) expected = [[24]] eq_(filtered_data[self.test_file], expected) def tearDown(self): os.unlink(self.test_file) ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1643486950.0 pyexcel-io-0.6.6/tests/test_io.py0000644000175100001710000002127700000000000016334 0ustar00runnerdockerimport os import types from zipfile import BadZipfile from unittest import TestCase import pyexcel_io.manager as manager import pyexcel_io.exceptions as exceptions from pyexcel_io import get_data, iget_data, save_data from pyexcel_io.io import load_data, get_writer from pyexcel_io._compact import BytesIO, StringIO, OrderedDict, is_string from nose.tools import eq_, raises @raises(IOError) def test_directory_name_as_file(): get_data("/") def test_force_file_type(): test_file = "force_file_type.txt" data = get_data( os.path.join("tests", "fixtures", test_file), force_file_type="csv" ) expected = [[1, 2, 3]] eq_(expected, data[test_file]) def test_force_file_type_on_write(): test_file = "force_file_type_on_write.txt" save_data(test_file, {"sheet 1": [[1, 2]]}, force_file_type="csv") data = get_data(test_file, force_file_type="csv") expected = [[1, 2]] eq_(expected, data[test_file]) os.unlink(test_file) @raises(IOError) def test_invalid_file(): load_data("/something/does/not/exist") @raises(IOError) def test_no_valid_parameters(): load_data() @raises(IOError) def test_no_valid_parameters_2(): get_writer() @raises(IOError) def test_none_type_load_data(): get_data(None) @raises(Exception) def test_wrong_parameter_to_get_data(): get_data(1) @raises(Exception) def test_wrong_parameter_to_get_writer(): get_writer(1) # @raises(Exception) # def test_wrong_parameter_to_get_writer2(): # get_writer(1, file_type="csv") def test_load_ods_data(): msg = "Please install one of these plugins for read data in 'ods': " msg += "pyexcel-ods,pyexcel-ods3,pyexcel-odsr" try: get_data("test.ods") except exceptions.SupportingPluginAvailableButNotInstalled as e: eq_(str(e), msg) def test_load_ods_data_from_memory(): io = BytesIO() msg = "Please install one of these plugins for read data in 'ods': " msg += "pyexcel-ods,pyexcel-ods3,pyexcel-odsr" try: get_data(io, file_type="ods") except exceptions.SupportingPluginAvailableButNotInstalled as e: eq_(str(e), msg) def test_write_xlsx_data_to_memory(): data = {"Sheet": [[1]]} io = BytesIO() msg = "Please install one of these plugins for write data in 'xlsx': " msg += "pyexcel-xlsx,pyexcel-xlsxw" try: save_data(io, data, file_type="xlsx") except exceptions.SupportingPluginAvailableButNotInstalled as e: eq_(str(e), msg) @raises(IOError) def test_load_unknown_data(): get_data("test.unknown") @raises(exceptions.NoSupportingPluginFound) def test_load_unknown_data_from_memory(): io = BytesIO() get_data(io, file_type="unknown") @raises(BadZipfile) def test_load_csvz_data_from_memory(): io = StringIO() get_data(io, file_type="csvz") @raises(IOError) def test_write_xlsx_data(): get_data("test.xlsx") @raises(Exception) def test_writer_csvz_data_from_memory(): io = StringIO() writer = get_writer(io, file_type="csvz") writer.write({"adb": [[2, 3]]}) @raises(exceptions.NoSupportingPluginFound) def test_writer_xlsm_data_from_memory2(): io = BytesIO() get_writer(io, file_type="xlsms") @raises(exceptions.NoSupportingPluginFound) def test_writer_unknown_data_from_memory2(): io = BytesIO() # mock it manager.register_stream_type("unknown1", "text") get_writer(io, file_type="unknown1") def test_get_io(): io = manager.get_io("hello") assert io is None def test_get_io_type(): t = manager.get_io_type("hello") assert t is None t = manager.get_io_type("csv") eq_(t, "string") t = manager.get_io_type("xls") eq_(t, "bytes") def test_default_csv_format(): data = [["1", "2", "3"]] io = manager.get_io("csv") # test default format for saving is 'csv' save_data(io, data) io.seek(0) # test default format for reading is 'csv' result = get_data(io) assert result["csv"] == [[1, 2, 3]] def test_case_insentivity(): data = [["1", "2", "3"]] io = manager.get_io("CSV") # test default format for saving is 'csv' save_data(io, data) io.seek(0) # test default format for reading is 'csv' result = get_data(io) assert result["csv"] == [[1, 2, 3]] def test_file_handle_as_input(): test_file = "file_handle.csv" with open(test_file, "w") as f: f.write("1,2,3") with open(test_file, "r") as f: data = get_data(f, "csv") eq_(data["csv"], [[1, 2, 3]]) os.unlink("file_handle.csv") def test_file_type_case_insensitivity(): test_file = "file_handle.CSv" with open(test_file, "w") as f: f.write("1,2,3") with open(test_file, "r") as f: data = get_data(f, "csv") eq_(data["csv"], [[1, 2, 3]]) os.unlink("file_handle.CSv") def test_file_handle_as_output(): test_file = "file_handle.csv" with open(test_file, "w") as f: save_data(f, [[1, 2, 3]], "csv", lineterminator="\n") with open(test_file, "r") as f: content = f.read() eq_(content, "1,2,3\n") os.unlink("file_handle.csv") def test_binary_file_content(): data = [["1", "2", "3"]] io = manager.get_io("csvz") save_data(io, data, "csvz") result = get_data(io.getvalue(), "csvz") eq_(result["pyexcel_sheet1"], [[1, 2, 3]]) def test_text_file_content(): data = [["1", "2", "3"]] io = manager.get_io("csv") save_data(io, data, "csv") result = get_data(io.getvalue(), "csv") eq_(result["csv"], [[1, 2, 3]]) def test_library_parameter(): data = [["1", "2", "3"]] io = manager.get_io("csv") save_data(io, data, "csv", library="pyexcel-io") result = get_data(io.getvalue(), "csv", library="pyexcel-io") eq_(result["csv"], [[1, 2, 3]]) @raises(Exception) def test_library_parameter_error_situation(): data = [["1", "2", "3"]] io = manager.get_io("csv") save_data(io, data, "csv", library="doesnot-exist") def test_conversion_from_bytes_to_text(): test_file = "conversion.csv" data = [["1", "2", "3"]] save_data(test_file, data) with open(test_file, "rb") as f: content = f.read() result = get_data(content, "csv") assert result["csv"] == [[1, 2, 3]] os.unlink(test_file) def test_is_string(): assert is_string(type("a")) is True def test_generator_is_obtained(): data, reader = iget_data(os.path.join("tests", "fixtures", "test.csv")) assert isinstance(data["test.csv"], types.GeneratorType) reader.close() def test_generator_can_be_written(): test_filename = "generator.csv" test_fixture = os.path.join("tests", "fixtures", "test.csv") data = get_data(test_fixture, streaming=True) save_data(test_filename, data) assert os.path.exists(test_filename) data2 = get_data(test_filename) expected = get_data(test_fixture) assert data2[test_filename] == expected["test.csv"] os.unlink(test_filename) class TestReadMultipleSheets(TestCase): file_type = "csv" delimiter = "," def setUp(self): self.test_file_formatter = "csv_multiple__%s__%s." + self.file_type self.merged_book_file = "csv_multiple." + self.file_type self.data = [["1", "2", "3"], ["4", "5", "6"], ["7", "8", "9"]] self.expected_data = [[1, 2, 3], [4, 5, 6], [7, 8, 9]] self.sheets = OrderedDict() self.sheets.update({"sheet1": self.data}) self.sheets.update({"sheet2": self.data}) self.sheets.update({"sheet3": self.data}) self.expected_sheets = OrderedDict() self.expected_sheets.update({"sheet1": self.expected_data}) self.expected_sheets.update({"sheet2": self.expected_data}) self.expected_sheets.update({"sheet3": self.expected_data}) index = 0 for key, value in self.sheets.items(): file_name = self.test_file_formatter % (key, index) with open(file_name, "w") as f: for row in value: f.write(self.delimiter.join(row) + "\n") index = index + 1 def test_sheet_name(self): sheets = get_data(self.merged_book_file, sheet_name="sheet1") eq_(sheets["sheet1"], self.expected_sheets["sheet1"]) def test_sheet_index(self): sheets = get_data(self.merged_book_file, sheet_index=1) eq_(sheets["sheet2"], self.expected_sheets["sheet2"]) def test_read_many(self): sheets = get_data(self.merged_book_file, sheets=["sheet1", 2]) eq_(sheets["sheet1"], self.expected_sheets["sheet1"]) eq_(sheets["sheet3"], self.expected_sheets["sheet3"]) assert "sheet2" not in sheets def tearDown(self): index = 0 for key, value in self.sheets.items(): file_name = self.test_file_formatter % (key, index) os.unlink(file_name) index = index + 1 ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1643486950.0 pyexcel-io-0.6.6/tests/test_issues.py0000644000175100001710000001025200000000000017227 0ustar00runnerdockerimport os import pyexcel as p from pyexcel_io import get_data, save_data from pyexcel_io.exceptions import NoSupportingPluginFound from nose import SkipTest from nose.tools import eq_, raises IN_TRAVIS = "TRAVIS" in os.environ def test_issue_8(): test_file = "test_issue_8.csv" data = [[1, 2], [], [], [], [3, 4]] save_data(test_file, data) written_data = get_data(test_file, skip_empty_rows=False) eq_(data, written_data[test_file]) os.unlink(test_file) def test_issue_20(): test_file = get_fixture("issue20.csv") data = get_data(test_file) expected = [[u"to", u"infinity", u"and", u"beyond"]] eq_(data["issue20.csv"], expected) def test_issue_23(): test_file = get_fixture("issue23.csv") data = get_data(test_file) expected = [ [8204235414504252, u"inf"], [82042354145042521, u"-inf"], [820423541450425216, 0], [820423541450425247, 1], [8204235414504252490, 1.1], ] eq_(data["issue23.csv"], expected) def test_issue_33_34(): import mmap test_file = get_fixture("issue20.csv") with open(test_file, "r+b") as f: memory_mapped_file = mmap.mmap(f.fileno(), 0, access=mmap.ACCESS_READ) data = get_data(memory_mapped_file, file_type="csv") expected = [[u"to", u"infinity", u"and", u"beyond"]] eq_(data["csv"], expected) def test_issue_30_utf8_BOM_header(): content = [["人有悲歡離合", "月有陰晴圓缺"]] test_file = "test-utf8-BOM.csv" save_data(test_file, content, encoding="utf-8-sig", lineterminator="\n") custom_encoded_content = get_data(test_file, encoding="utf-8-sig") assert custom_encoded_content[test_file] == content with open(test_file, "rb") as f: content = f.read() assert content[0:3] == b"\xef\xbb\xbf" os.unlink(test_file) def test_issue_33_34_utf32_encoded_file(): check_mmap_encoding("utf-32") def test_issue_33_34_utf32be_encoded_file(): check_mmap_encoding("utf-32-be") def test_issue_33_34_utf32le_encoded_file(): check_mmap_encoding("utf-32-le") def test_issue_33_34_utf16_encoded_file(): check_mmap_encoding("utf-16") def test_issue_33_34_utf16be_encoded_file(): check_mmap_encoding("utf-16-be") def test_issue_33_34_utf16le_encoded_file(): check_mmap_encoding("utf-16-le") def test_issue_33_34_utf8_encoded_file(): check_mmap_encoding("utf-8") def check_mmap_encoding(encoding): import mmap content = [ [u"Äkkilähdöt", u"Matkakirjoituksia", u"Matkatoimistot"], [u"Äkkilähdöt", u"Matkakirjoituksia", u"Matkatoimistot"], ] test_file = "test-%s-encoding-in-mmap-file.csv" % encoding save_data(test_file, content, encoding=encoding) with open(test_file, "r+b") as f: memory_mapped_file = mmap.mmap(f.fileno(), 0, access=mmap.ACCESS_READ) data = get_data(memory_mapped_file, file_type="csv", encoding=encoding) eq_(data["csv"], content) memory_mapped_file.close() os.unlink(test_file) def test_issue_35_encoding_for_file_content(): encoding = "utf-16" content = [ [u"Äkkilähdöt", u"Matkakirjoituksia", u"Matkatoimistot"], [u"Äkkilähdöt", u"Matkakirjoituksia", u"Matkatoimistot"], ] test_file = "test-%s-encoding-in-mmap-file.csv" % encoding save_data(test_file, content, encoding=encoding) with open(test_file, "r+b") as f: csv_content = f.read() data = get_data(csv_content, file_type="csv", encoding=encoding) eq_(data["csv"], content) os.unlink(test_file) def test_issue_43(): # if not IN_TRAVIS: # raise SkipTest() p.get_book( url="https://github.com/pyexcel/pyexcel-xls/raw/master/tests/fixtures/file_with_an_empty_sheet.xls" ) # flake8: noqa def test_pyexcel_issue_138(): array = [["123_122", "123_1.", "123_1.0"]] save_data("test.csv", array) data = get_data("test.csv") expected = [["123_122", "123_1.", "123_1.0"]] eq_(data["test.csv"], expected) os.unlink("test.csv") @raises(NoSupportingPluginFound) def test_issue_96(): get_data("foo-bar-data", file_type="Idonotexist") def get_fixture(file_name): return os.path.join("tests", "fixtures", file_name) ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1643486950.0 pyexcel-io-0.6.6/tests/test_new_csv_book.py0000644000175100001710000002625300000000000020402 0ustar00runnerdockerimport os from textwrap import dedent from unittest import TestCase import pyexcel_io.manager as manager from pyexcel_io.reader import Reader from pyexcel_io.writer import Writer from pyexcel_io._compact import OrderedDict from nose.tools import raises class TestCSVReaders(TestCase): file_type = "csv" delimiter = "," def reader_class(self): return Reader(self.file_type) def setUp(self): self.test_file = "csv_book." + self.file_type self.data = [["1", "2", "3"], ["4", "5", "6"], ["7", "8", "9"]] self.expected_data = [[1, 2, 3], [4, 5, 6], [7, 8, 9]] with open(self.test_file, "w") as f: for row in self.data: f.write(self.delimiter.join(row) + "\n") def test_book_reader(self): b = self.reader_class() b.open(self.test_file) sheets = b.read_all() self.assertEqual(list(sheets[self.test_file]), self.expected_data) def test_book_reader_from_memory_source(self): io = manager.get_io(self.file_type) with open(self.test_file, "r") as f: io.write(f.read()) io.seek(0) b = self.reader_class() b.open_stream(io) sheets = b.read_all() self.assertEqual(list(sheets[self.file_type]), self.expected_data) def tearDown(self): os.unlink(self.test_file) class TestNewCSVReaders(TestCase): file_type = "csv" delimiter = "," def setUp(self): self.test_file = "csv_book." + self.file_type self.data = [["1", "2", "3"], ["4", "5", "6"], ["7", "8", "9"]] self.expected_data = [[1, 2, 3], [4, 5, 6], [7, 8, 9]] with open(self.test_file, "w") as f: for row in self.data: f.write(self.delimiter.join(row) + "\n") def test_book_reader(self): b = Reader(self.file_type) b.open(self.test_file) sheets = b.read_all() self.assertEqual(list(sheets[self.test_file]), self.expected_data) def test_book_reader_from_memory_source(self): io = manager.get_io(self.file_type) with open(self.test_file, "r") as f: io.write(f.read()) io.seek(0) b = Reader(self.file_type) b.open_stream(io) sheets = b.read_all() self.assertEqual(list(sheets[self.file_type]), self.expected_data) def tearDown(self): os.unlink(self.test_file) class TestTSVReaders(TestNewCSVReaders): file_type = "tsv" delimiter = "\t" class TestReadMultipleSheets(TestCase): file_type = "csv" delimiter = "," def reader_class(self): return Reader(self.file_type) def setUp(self): self.test_file_formatter = "csv_multiple__%s__%s." + self.file_type self.merged_book_file = "csv_multiple." + self.file_type self.data = [["1", "2", "3"], ["4", "5", "6"], ["7", "8", "9"]] self.expected_data = [[1, 2, 3], [4, 5, 6], [7, 8, 9]] self.sheets = OrderedDict() self.sheets.update({"sheet1": self.data}) self.sheets.update({"sheet2": self.data}) self.sheets.update({"sheet3": self.data}) self.expected_sheets = OrderedDict() self.expected_sheets.update({"sheet1": self.expected_data}) self.expected_sheets.update({"sheet2": self.expected_data}) self.expected_sheets.update({"sheet3": self.expected_data}) index = 0 for key, value in self.sheets.items(): file_name = self.test_file_formatter % (key, index) with open(file_name, "w") as f: for row in value: f.write(self.delimiter.join(row) + "\n") index = index + 1 def test_multiple_sheet(self): b = self.reader_class() b.open(self.merged_book_file) sheets = b.read_all() for key in sheets: sheets[key] = list(sheets[key]) self.assertEqual(sheets, self.expected_sheets) def test_read_one_from_many_by_name(self): b = self.reader_class() b.open(self.merged_book_file) sheets = b.read_sheet_by_name("sheet1") self.assertEqual( list(sheets["sheet1"]), self.expected_sheets["sheet1"] ) @raises(ValueError) def test_read_one_from_many_by_non_existent_name(self): b = self.reader_class() b.open(self.merged_book_file) b.read_sheet_by_name("notknown") def test_read_one_from_many_by_index(self): b = self.reader_class() b.open(self.merged_book_file) sheets = b.read_sheet_by_index(1) self.assertEqual( list(sheets["sheet2"]), self.expected_sheets["sheet2"] ) @raises(IndexError) def test_read_one_from_many_by_wrong_index(self): b = self.reader_class() b.open(self.merged_book_file) b.read_sheet_by_index(90) def tearDown(self): index = 0 for key, value in self.sheets.items(): file_name = self.test_file_formatter % (key, index) os.unlink(file_name) index = index + 1 class TestTSVBookReaders(TestReadMultipleSheets): file_type = "tsv" delimiter = "\t" class TestWriteMultipleSheets(TestCase): file_type = "csv" result1 = dedent( """ 1,2,3 4,5,6 7,8,9 """ ).strip("\n") result2 = dedent( """ 1,2,3 4,5,6 7,8,1000 """ ).strip("\n") result3 = dedent( """ 1,2,3 4,5,6888 7,8,9 """ ).strip("\n") merged = dedent( """\ ---pyexcel:sheet1--- 1,2,3 4,5,6 7,8,9 ---pyexcel--- ---pyexcel:sheet2--- 1,2,3 4,5,6 7,8,1000 ---pyexcel--- ---pyexcel:sheet3--- 1,2,3 4,5,6888 7,8,9 ---pyexcel--- """ ) def writer_class(self): return Writer(self.file_type) def reader_class(self): return Reader(self.file_type) def setUp(self): self.test_file_formatter = "csv_multiple__%s__%s." + self.file_type self.merged_book_file = "csv_multiple." + self.file_type self.data1 = [["1", "2", "3"], ["4", "5", "6"], ["7", "8", "9"]] self.data2 = [["1", "2", "3"], ["4", "5", "6"], ["7", "8", "1000"]] self.data3 = [["1", "2", "3"], ["4", "5", "6888"], ["7", "8", "9"]] self.expected_data1 = [[1, 2, 3], [4, 5, 6], [7, 8, 9]] self.expected_data2 = [[1, 2, 3], [4, 5, 6], [7, 8, 1000]] self.expected_data3 = [[1, 2, 3], [4, 5, 6888], [7, 8, 9]] self.sheets = OrderedDict() self.sheets.update({"sheet1": self.data1}) self.sheets.update({"sheet2": self.data2}) self.sheets.update({"sheet3": self.data3}) self.expected_sheets = OrderedDict() self.expected_sheets.update({"sheet1": self.expected_data1}) self.expected_sheets.update({"sheet2": self.expected_data2}) self.expected_sheets.update({"sheet3": self.expected_data3}) self.result_dict = OrderedDict() self.result_dict.update({"sheet1": self.result1}) self.result_dict.update({"sheet2": self.result2}) self.result_dict.update({"sheet3": self.result3}) def test_multiple_sheet(self): w = self.writer_class() w.open(self.merged_book_file) w.write(self.sheets) w.close() index = 0 for key, value in self.sheets.items(): file_name = self.test_file_formatter % (key, index) with open(file_name, "r") as f: content = f.read().replace("\r", "") assert content.strip("\n") == self.result_dict[key] index = index + 1 self.delete_files() def test_multiple_sheet_into_memory(self): io = manager.get_io(self.file_type) w = self.writer_class() w.open_stream(io, lineterminator="\n") w.write(self.sheets) w.close() content = io.getvalue() self.assertEqual(content, self.merged) def test_multiple_sheet_into_memory_2(self): """Write csv book into a single stream""" io = manager.get_io(self.file_type) w = self.writer_class() w.open_stream(io, lineterminator="\n") w.write(self.sheets) w.close() reader = self.reader_class() reader.open_stream(io, lineterminator="\n", multiple_sheets=True) sheets = reader.read_all() for sheet in sheets: sheets[sheet] = list(sheets[sheet]) self.assertEqual(sheets, self.expected_sheets) def delete_files(self): index = 0 for key, value in self.sheets.items(): file_name = self.test_file_formatter % (key, index) os.unlink(file_name) index = index + 1 class TestTSVWriteMultipleSheets(TestWriteMultipleSheets): file_type = "tsv" result1 = dedent( """ 1\t2\t3 4\t5\t6 7\t8\t9 """ ).strip("\n") result2 = dedent( """ 1\t2\t3 4\t5\t6 7\t8\t1000 """ ).strip("\n") result3 = dedent( """ 1\t2\t3 4\t5\t6888 7\t8\t9 """ ).strip("\n") merged = dedent( """\ ---pyexcel:sheet1--- 1\t2\t3 4\t5\t6 7\t8\t9 ---pyexcel--- ---pyexcel:sheet2--- 1\t2\t3 4\t5\t6 7\t8\t1000 ---pyexcel--- ---pyexcel:sheet3--- 1\t2\t3 4\t5\t6888 7\t8\t9 ---pyexcel--- """ ) class TestWriter(TestCase): file_type = "csv" result = dedent( """ 1,2,3 4,5,6 7,8,9 """ ).strip("\n") def writer_class(self): return Writer(self.file_type) def setUp(self): self.test_file = "csv_book." + self.file_type self.data = [["1", "2", "3"], ["4", "5", "6"], ["7", "8", "9"]] def test_book_writer(self): w = self.writer_class() w.open(self.test_file) w.write({None: self.data}) w.close() with open(self.test_file, "r") as f: content = f.read().replace("\r", "") self.assertEqual(content.strip("\n"), self.result) def tearDown(self): os.unlink(self.test_file) class TestTSVWriters(TestWriter): file_type = "tsv" result = dedent( """ 1\t2\t3 4\t5\t6 7\t8\t9 """ ).strip("\n") class TestMemoryWriter(TestCase): file_type = "csv" result = dedent( """ 1,2,3 4,5,6 7,8,9 """ ).strip("\n") def writer_class(self): return Writer(self.file_type) def setUp(self): self.test_file = "csv_book." + self.file_type self.data = [["1", "2", "3"], ["4", "5", "6"], ["7", "8", "9"]] def test_book_writer_to_memroy(self): io = manager.get_io(self.file_type) w = self.writer_class() w.open_stream(io, single_sheet_in_book=True) w.write({self.file_type: self.data}) w.close() content = io.getvalue().replace("\r", "") assert content.strip("\n") == self.result class TestTSVMemoryWriter(TestMemoryWriter): file_type = "tsv" result = dedent( """ 1\t2\t3 4\t5\t6 7\t8\t9 """ ).strip("\n") ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1643486950.0 pyexcel-io-0.6.6/tests/test_new_csvz_book.py0000644000175100001710000001061300000000000020565 0ustar00runnerdocker# -*- coding: utf-8 -*- import os import zipfile from unittest import TestCase import pyexcel_io.manager as manager from pyexcel_io import save_data from pyexcel_io.reader import Reader from pyexcel_io.writer import Writer from pyexcel_io._compact import OrderedDict from nose.tools import raises class TestCSVZ(TestCase): file_type = "csvz" result = u"中,文,1,2,3" def writer_class(self): return Writer(self.file_type) def reader_class(self): return Reader(self.file_type) def setUp(self): self.file = "csvz." + self.file_type def test_writing(self): data = [[u"中", u"文", 1, 2, 3]] file_name = "pyexcel_sheet1." + self.file_type[0:3] zipbook = self.writer_class() zipbook.open(self.file) zipbook.write({None: data}) zipbook.close() zip = zipfile.ZipFile(self.file, "r") self.assertEqual(zip.namelist(), [file_name]) content = zip.read(file_name) content = content.decode("utf-8") self.assertEqual(content.replace("\r", "").strip("\n"), self.result) zip.close() def test_reading(self): data = [[u"中", u"文", 1, 2, 3]] zipbook = self.writer_class() zipbook.open(self.file) zipbook.write({None: data}) zipbook.close() zipreader = self.reader_class() zipreader.open(self.file) data = zipreader.read_all() self.assertEqual(list(data["pyexcel_sheet1"]), [[u"中", u"文", 1, 2, 3]]) zipreader.close() def test_reading_utf32(self): zip = zipfile.ZipFile(self.file, "w") zip.writestr("something.ext", self.result.encode("utf-32")) zip.close() zipreader = self.reader_class() zipreader.open(self.file) data = zipreader.read_all() self.assertEqual(list(data["something"]), [[u"中", u"文", 1, 2, 3]]) zipreader.close() def tearDown(self): os.unlink(self.file) class TestTSVZ(TestCSVZ): file_type = "tsvz" result = u"中\t文\t1\t2\t3" def test_reading_from_memory(): data = [[1, 2, 3]] io = manager.get_io("csvz") zipbook = Writer("csvz") zipbook.open_stream(io) zipbook.write({None: data}) zipbook.close() zipreader = Reader("csvz") zipreader.open_stream(io) data = zipreader.read_all() assert list(data["pyexcel_sheet1"]) == [[1, 2, 3]] def test_reading_from_memory_tsvz(): data = [[1, 2, 3]] io = manager.get_io("tsvz") zipbook = Writer("tsvz") zipbook.open_stream(io) zipbook.write({None: data}) zipbook.close() zipreader = Reader("tsvz") zipreader.open_stream(io) data = zipreader.read_all() assert list(data["pyexcel_sheet1"]) == [[1, 2, 3]] class TestMultipleSheet(TestCase): file_name = "mybook.csvz" def reader_class(self): return Reader("csvz") def setUp(self): self.content = OrderedDict() self.content.update( {"Sheet 1": [[1.0, 2.0, 3.0], [4.0, 5.0, 6.0], [7.0, 8.0, 9.0]]} ) self.content.update( {"Sheet 2": [["X", "Y", "Z"], [1.0, 2.0, 3.0], [4.0, 5.0, 6.0]]} ) self.content.update( {"Sheet 3": [["O", "P", "Q"], [3.0, 2.0, 1.0], [4.0, 3.0, 2.0]]} ) save_data(self.file_name, self.content) def test_read_one_from_many_by_name(self): reader = self.reader_class() reader.open(self.file_name) sheets = reader.read_sheet_by_name("Sheet 1") self.assertEqual( list(sheets["Sheet 1"]), [[1, 2, 3], [4, 5, 6], [7, 8, 9]] ) @raises(ValueError) def test_read_one_from_many_by_unknown_name(self): reader = self.reader_class() reader.open(self.file_name) reader.read_sheet_by_name("Sheet X") def test_read_one_from_many_by_index(self): reader = self.reader_class() reader.open(self.file_name) sheets = reader.read_sheet_by_index(0) self.assertEqual( list(sheets["Sheet 1"]), [[1, 2, 3], [4, 5, 6], [7, 8, 9]] ) @raises(IndexError) def test_read_one_from_many_by_unknown_index(self): reader = self.reader_class() reader.open(self.file_name) reader.read_sheet_by_index(9999) def tearDown(self): os.unlink(self.file_name) class TestMultipleTSVSheet(TestMultipleSheet): file_name = "mybook.tsvz" def reader_class(self): return Reader("tsvz") ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1643486950.0 pyexcel-io-0.6.6/tests/test_plugin_api.py0000644000175100001710000000251200000000000020043 0ustar00runnerdockerfrom pyexcel_io.plugin_api import ISheet, IReader, IWriter, ISheetWriter from nose.tools import raises class TestISheet: def setUp(self): self.isheet = ISheet() @raises(NotImplementedError) def test_row_iterator(self): self.isheet.row_iterator() @raises(NotImplementedError) def test_column_iterator(self): self.isheet.column_iterator(1) class TestISheetWriter: def setUp(self): self.isheet_writer = ISheetWriter() @raises(NotImplementedError) def test_write_row(self): self.isheet_writer.write_row([1, 2]) @raises(NotImplementedError) def test_close(self): self.isheet_writer.close() class TestIReader: def setUp(self): self.ireader = IReader() @raises(NotImplementedError) def test_read_sheet(self): self.ireader.read_sheet(1) @raises(NotImplementedError) def test_close(self): self.ireader.close() class TestIWriter: def setUp(self): self.iwriter = IWriter() @raises(NotImplementedError) def test_create_sheet(self): self.iwriter.create_sheet("a name") @raises(Exception) def test_empty_writer(): class TestWriter(IWriter): def create_sheet(self, sheet_name): return None test_writer = TestWriter() test_writer.write({"sheet 1": [[1, 2]]}) ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1643486950.0 pyexcel-io-0.6.6/tests/test_pyexcel_integration.py0000644000175100001710000001245700000000000022001 0ustar00runnerdockerimport os import datetime from textwrap import dedent from unittest import TestCase import pyexcel as pe from pyexcel_io._compact import text_type class TestDateFormat(TestCase): def setUp(self): self.excel_filename = "testdateformat.csv" self.data = [ [ "01/06/2016", datetime.date(2014, 12, 25), datetime.datetime(2014, 12, 25, 11, 11, 11), datetime.datetime(2014, 12, 25, 11, 11, 11, 10), ] ] pe.save_as(dest_file_name=self.excel_filename, array=self.data) def test_auto_detect_float(self): sheet = pe.get_sheet(file_name=self.excel_filename) self.assertEqual(sheet.to_array(), self.data) def test_auto_detect_float_false(self): expected = [ [ "01/06/2016", "2014-12-25", "2014-12-25 11:11:11", "2014-12-25 11:11:11.000010", ] ] sheet = pe.get_sheet( file_name=self.excel_filename, auto_detect_datetime=False ) self.assertEqual(sheet.to_array(), expected) def tearDown(self): os.unlink(self.excel_filename) class TestAutoDetectInt(TestCase): def setUp(self): self.content = [[1, 2, 3.1]] self.test_file = "test_auto_detect_init.csv" pe.save_as(array=self.content, dest_file_name=self.test_file) def test_auto_detect_int(self): sheet = pe.get_sheet(file_name=self.test_file) expected = dedent( """ test_auto_detect_init.csv: +---+---+-----+ | 1 | 2 | 3.1 | +---+---+-----+""" ).strip() self.assertEqual(str(sheet), expected) def test_get_book_auto_detect_int(self): book = pe.get_book(file_name=self.test_file) expected = dedent( """ test_auto_detect_init.csv: +---+---+-----+ | 1 | 2 | 3.1 | +---+---+-----+""" ).strip() self.assertEqual(str(book), expected) def test_auto_detect_int_false(self): sheet = pe.get_sheet(file_name=self.test_file, auto_detect_int=False) expected = dedent( """ test_auto_detect_init.csv: +-----+-----+-----+ | 1.0 | 2.0 | 3.1 | +-----+-----+-----+""" ).strip() self.assertEqual(str(sheet), expected) def test_get_book_auto_detect_int_false(self): book = pe.get_book(file_name=self.test_file, auto_detect_int=False) expected = dedent( """ test_auto_detect_init.csv: +-----+-----+-----+ | 1.0 | 2.0 | 3.1 | +-----+-----+-----+""" ).strip() self.assertEqual(str(book), expected) def tearDown(self): os.unlink(self.test_file) class TestAutoDetectFloat(TestCase): """ csv format would keep all as text 1, 2.0, 3.1 without auto-detection, they all stay as text format after reading out """ def setUp(self): self.content = [[1, 2.0, 3.1]] self.test_file = "test_auto_detect_init.csv" pe.save_as(array=self.content, dest_file_name=self.test_file) def test_auto_detect_float_false(self): sheet = pe.get_sheet(file_name=self.test_file, auto_detect_float=False) self.assertEqual(sheet.to_array(), [[1, "2.0", "3.1"]]) expected = dedent( """ test_auto_detect_init.csv: +---+-----+-----+ | 1 | 2.0 | 3.1 | +---+-----+-----+""" ).strip() self.assertEqual(str(sheet), expected) def test_get_book_auto_detect_float_false(self): book = pe.get_book(file_name=self.test_file, auto_detect_float=False) self.assertEqual(book[0].to_array(), [[1, "2.0", "3.1"]]) expected = dedent( """ test_auto_detect_init.csv: +---+-----+-----+ | 1 | 2.0 | 3.1 | +---+-----+-----+""" ).strip() self.assertEqual(str(book), expected) def tearDown(self): os.unlink(self.test_file) class TestSpecialStrings(TestCase): """ csv format would keep all as text 1, 2.0, 3.1 without auto-detection, they all stay as text format after reading out """ def setUp(self): self.content = [["01", 1, 2.0, 3.1, "NaN", "nan"]] self.test_file = "test_auto_detect_init.csv" pe.save_as(array=self.content, dest_file_name=self.test_file) def test_auto_detect_float_true(self): sheet = pe.get_sheet(file_name=self.test_file) self.assertEqual(sheet.to_array(), [["01", 1, 2, 3.1, "NaN", "nan"]]) def test_auto_detect_float_false(self): sheet = pe.get_sheet(file_name=self.test_file, auto_detect_float=False) self.assertEqual( sheet.to_array(), [["01", 1, "2.0", "3.1", "NaN", "nan"]] ) def test_auto_detect_float_ignore_nan_text(self): sheet = pe.get_sheet(file_name=self.test_file, ignore_nan_text=True) self.assertEqual(sheet.to_array(), [["01", 1, 2.0, 3.1, "NaN", "nan"]]) def test_auto_detect_float_default_float_nan(self): sheet = pe.get_sheet(file_name=self.test_file, default_float_nan="nan") result = sheet.to_array() assert isinstance(result[0][5], float) assert isinstance(result[0][4], text_type) def tearDown(self): os.unlink(self.test_file) ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1643486950.0 pyexcel-io-0.6.6/tests/test_renderer.py0000644000175100001710000000121100000000000017515 0ustar00runnerdockerimport os from pyexcel_io import get_data, save_data from nose.tools import eq_ class TestRenderer: def setUp(self): self.test_file = "test_filter.csv" sample = [[1, 21, 31], [2, 22, 32]] save_data(self.test_file, sample) def test_filter_row(self): def custom_row_renderer(row): return [str(element) for element in row] custom_data = get_data( self.test_file, row_renderer=custom_row_renderer ) expected = [["1", "21", "31"], ["2", "22", "32"]] eq_(custom_data[self.test_file], expected) def tearDown(self): os.unlink(self.test_file) ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1643486950.0 pyexcel-io-0.6.6/tests/test_service.py0000644000175100001710000000757000000000000017365 0ustar00runnerdockerfrom datetime import time, datetime, timedelta from pyexcel_io.service import ( date_value, time_value, float_value, boolean_value, ods_bool_value, ods_date_value, ods_time_value, ods_float_value, throw_exception, detect_int_value, detect_float_value, ods_timedelta_value, ) from pyexcel_io.exceptions import IntegerAccuracyLossError from nose.tools import eq_, raises def test_date_util_parse(): value = "2015-08-17T19:20:00" d = date_value(value) assert d.strftime("%Y-%m-%dT%H:%M:%S") == "2015-08-17T19:20:00" value = "2015-08-17" d = date_value(value) assert d.strftime("%Y-%m-%d") == "2015-08-17" value = "2015-08-17T19:20:59.999999" d = date_value(value) assert d.strftime("%Y-%m-%dT%H:%M:%S") == "2015-08-17T19:20:59" value = "2015-08-17T19:20:59.99999" d = date_value(value) assert d.strftime("%Y-%m-%dT%H:%M:%S") == "2015-08-17T19:20:59" value = "2015-08-17T19:20:59.999999999999999" d = date_value(value) assert d.strftime("%Y-%m-%dT%H:%M:%S") == "2015-08-17T19:20:59" def test_issue_8_1(): # https://github.com/pyexcel/pyexcel-ods3/issues/8 result = time_value("PT1111") eq_(result, None) @raises(Exception) def test_invalid_date(): value = "2015-08-" date_value(value) @raises(Exception) def test_fake_date_time_10(): date_value("1234567890") @raises(Exception) def test_fake_date_time_19(): date_value("1234567890123456789") @raises(Exception) def test_fake_date_time_20(): date_value("12345678901234567890") def test_issue_1_error(): result = time_value("PT1111") eq_(result, None) def test_detect_int_value(): result = detect_int_value("123") eq_(result, 123) def test_detect_float_value(): result = detect_float_value("123.1") eq_(result, 123.1) def test_suppression_of_pep_0515_int(): result = detect_int_value("123_123") eq_(result, None) def test_suppression_of_pep_0515_float(): result = detect_float_value("123_123.") eq_(result, None) result = detect_float_value("123_123.1") eq_(result, None) def test_detect_float_value_on_nan(): result = detect_float_value("NaN", ignore_nan_text=True) eq_(result, None) def test_detect_float_value_on_custom_nan_text(): result = detect_float_value("NaN", default_float_nan="nan") eq_(result, None) def test_detect_float_value_on_custom_nan_text2(): result = detect_float_value("nan", default_float_nan="nan") eq_(str(result), "nan") @raises(IntegerAccuracyLossError) def test_big_int_value(): ods_float_value(1000000000000000) @raises(IntegerAccuracyLossError) def test_throw_exception(): throw_exception(1000000000000000) def test_boolean_value(): fixture = ["true", "false", 1] expected = [True, False, 1] actual = [boolean_value(element) for element in fixture] eq_(actual, expected) def test_time_delta_presentation(): a = datetime(2020, 12, 12, 12, 12, 12) b = datetime(2020, 11, 12, 12, 12, 11) delta = a - b value = ods_timedelta_value(delta) eq_(value, "PT720H00M01S") def test_ods_bool_to_string(): fixture = [True, False] expected = ["true", "false"] actual = [ods_bool_value(element) for element in fixture] eq_(actual, expected) def test_ods_time_value(): test = datetime(2020, 10, 6, 11, 11, 11) actual = ods_time_value(test) eq_(actual, "PT11H11M11S") def test_ods_date_value(): test = datetime(2020, 10, 6, 11, 11, 11) actual = ods_date_value(test) eq_(actual, "2020-10-06") def test_time_value_returns_time_delta(): test_time_value = "PT720H00M01S" delta = time_value(test_time_value) eq_(delta, timedelta(days=30, seconds=1)) def test_time_value(): test_time_value = "PT23H00M01S" delta = time_value(test_time_value) eq_(delta, time(23, 0, 1)) def test_float_value(): a = float_value("1.2") eq_(a, 1.2) ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1643486950.0 pyexcel-io-0.6.6/tests/test_sheet.py0000644000175100001710000000264700000000000017035 0ustar00runnerdockerimport pyexcel_io.constants as constants from pyexcel_io.sheet import SheetReader, SheetWriter from nose.tools import eq_ class MyWriter(SheetWriter): def set_size(self, size): self.native_book = size class MyReader(SheetReader): def number_of_rows(self): return len(self._native_sheet) def number_of_columns(self): return len(self._native_sheet[0]) def cell_value(self, row, column): return self._native_sheet[row][column] def test_write_empty_array(): test_string = "somebook" writer = MyWriter(test_string, "somesheet", "somename") writer.write_array([]) eq_(test_string, writer._native_book) def take_second_column(current_index, start, limit=-1): decision = constants.SKIP_DATA if current_index == 1: decision = constants.TAKE_DATA elif current_index > 1: decision = constants.STOP_ITERATION return decision def test_custom_skip_row_func(): take_second_row = take_second_column array = [[1, 2, 3], [4, 5, 6]] reader = MyReader(array, skip_row_func=take_second_row) actual = list(reader.to_array()) expected = [[4, 5, 6]] eq_(expected, actual) reader.close() def test_custom_skip_column_func(): array = [[1, 2, 3], [4, 5, 6]] reader = MyReader(array, skip_column_func=take_second_column) actual = list(reader.to_array()) expected = [[2], [5]] eq_(expected, actual) reader.close() ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1643486950.0 pyexcel-io-0.6.6/tests/test_sql_book.py0000644000175100001710000004532600000000000017537 0ustar00runnerdockerimport sys import json import datetime import platform from sqlalchemy import ( Date, Float, Column, String, Integer, DateTime, ForeignKey, create_engine, ) from sqlalchemy.orm import backref, relationship, sessionmaker from pyexcel_io.reader import EncapsulatedSheetReader from pyexcel_io._compact import OrderedDict from pyexcel_io.database.common import ( SQLTableExporter, SQLTableImporter, SQLTableExportAdapter, SQLTableImportAdapter, ) from sqlalchemy.ext.declarative import declarative_base from pyexcel_io.database.querysets import QuerysetsReader from pyexcel_io.database.exporters.queryset import QueryReader from pyexcel_io.database.exporters.sqlalchemy import ( SQLBookReader, SQLTableReader, ) from pyexcel_io.database.importers.sqlalchemy import ( SQLBookWriter, SQLTableWriter, PyexcelSQLSkipRowException, ) from nose.tools import eq_, raises PY3 = sys.version_info[0] == 3 PY36 = PY3 and sys.version_info[1] == 6 engine = None if platform.python_implementation() == "PyPy": engine = create_engine("sqlite:///tmp.db") else: engine = create_engine("sqlite://") Base = declarative_base() class Pyexcel(Base): __tablename__ = "pyexcel" id = Column(Integer, primary_key=True) name = Column(String, unique=True) weight = Column(Float) birth = Column(Date) class Post(Base): __tablename__ = "post" id = Column(Integer, primary_key=True) title = Column(String(80)) body = Column(String(100)) pub_date = Column(DateTime) category_id = Column(Integer, ForeignKey("category.id")) category = relationship( "Category", backref=backref("posts", lazy="dynamic") ) def __init__(self, title, body, category, pub_date=None): self.title = title self.body = body if pub_date is None: pub_date = datetime.utcnow() self.pub_date = pub_date self.category = category def __repr__(self): return "" % self.title class Category(Base): __tablename__ = "category" id = Column(Integer, primary_key=True) name = Column(String(50)) def __init__(self, name): self.name = name def __repr__(self): return "" % self.name def __str__(self): return self.__repr__() Session = sessionmaker(bind=engine) class TestSingleRead: def setUp(self): Base.metadata.drop_all(engine) Base.metadata.create_all(engine) p1 = Pyexcel( id=0, name="Adam", weight=11.25, birth=datetime.date(2014, 11, 11) ) self.session = Session() self.session.add(p1) p1 = Pyexcel( id=1, name="Smith", weight=12.25, birth=datetime.date(2014, 11, 12) ) self.session.add(p1) self.session.commit() self.session.close() def test_sql(self): mysession = Session() sheet = SQLTableReader(mysession, Pyexcel) data = sheet.to_array() content = [ ["birth", "id", "name", "weight"], ["2014-11-11", 0, "Adam", 11.25], ["2014-11-12", 1, "Smith", 12.25], ] # 'pyexcel' here is the table name eq_(list(data), content) mysession.close() def test_sql_formating(self): mysession = Session() def custom_renderer(row): return [str(element) for element in row] # the key for this test case sheet = EncapsulatedSheetReader( SQLTableReader(mysession, Pyexcel), row_renderer=custom_renderer ) data = sheet.to_array() content = [ ["birth", "id", "name", "weight"], ["2014-11-11", "0", "Adam", "11.25"], ["2014-11-12", "1", "Smith", "12.25"], ] eq_(list(data), content) mysession.close() def test_sql_filter(self): mysession = Session() sheet = EncapsulatedSheetReader( SQLTableReader(mysession, Pyexcel), start_row=1 ) data = sheet.to_array() content = [ ["2014-11-11", 0, "Adam", 11.25], ["2014-11-12", 1, "Smith", 12.25], ] # 'pyexcel'' here is the table name assert list(data) == content mysession.close() def test_sql_filter_1(self): mysession = Session() sheet = EncapsulatedSheetReader( SQLTableReader(mysession, Pyexcel), start_row=1, row_limit=1 ) data = sheet.to_array() content = [["2014-11-11", 0, "Adam", 11.25]] # 'pyexcel'' here is the table name assert list(data) == content mysession.close() def test_sql_filter_2(self): mysession = Session() sheet = EncapsulatedSheetReader( SQLTableReader(mysession, Pyexcel), start_column=1 ) data = sheet.to_array() content = [ ["id", "name", "weight"], [0, "Adam", 11.25], [1, "Smith", 12.25], ] # 'pyexcel'' here is the table name assert list(data) == content mysession.close() def test_sql_filter_3(self): mysession = Session() sheet = EncapsulatedSheetReader( SQLTableReader(mysession, Pyexcel), start_column=1, column_limit=1 ) data = sheet.to_array() content = [["id"], [0], [1]] # 'pyexcel'' here is the table name assert list(data) == content mysession.close() class TestSingleWrite: def setUp(self): Base.metadata.drop_all(engine) Base.metadata.create_all(engine) self.data = [ ["birth", "id", "name", "weight"], [datetime.date(2014, 11, 11), 0, "Adam", 11.25], [datetime.date(2014, 11, 12), 1, "Smith", 12.25], ] self.results = [ ["birth", "id", "name", "weight"], ["2014-11-11", 0, "Adam", 11.25], ["2014-11-12", 1, "Smith", 12.25], ] def test_one_table(self): mysession = Session() importer = SQLTableImporter(mysession) adapter = SQLTableImportAdapter(Pyexcel) adapter.column_names = self.data[0] writer = SQLTableWriter(importer, adapter) writer.write_array(self.data[1:]) writer.close() query_sets = mysession.query(Pyexcel).all() reader = QuerysetsReader(query_sets, self.data[0]) results = reader.to_array() assert list(results) == self.results query_sets = mysession.query(Pyexcel).all() query_reader = QueryReader(query_sets, None, column_names=self.data[0]) result = read_all(query_reader) for key in result: result[key] = list(result[key]) eq_(result, {"pyexcel_sheet1": self.results}) query_reader.close() mysession.close() def test_update_existing_row(self): mysession = Session() # write existing data importer = SQLTableImporter(mysession) adapter = SQLTableImportAdapter(Pyexcel) adapter.column_names = self.data[0] writer = SQLTableWriter(importer, adapter) writer.write_array(self.data[1:]) writer.close() query_sets = mysession.query(Pyexcel).all() results = QuerysetsReader(query_sets, self.data[0]).to_array() assert list(results) == self.results # update data using custom initializer update_data = [ ["birth", "id", "name", "weight"], [datetime.date(2014, 11, 11), 0, "Adam_E", 12.25], [datetime.date(2014, 11, 12), 1, "Smith_E", 11.25], ] updated_results = [ ["birth", "id", "name", "weight"], ["2014-11-11", 0, "Adam_E", 12.25], ["2014-11-12", 1, "Smith_E", 11.25], ] def row_updater(row): an_instance = mysession.query(Pyexcel).get(row["id"]) if an_instance is None: an_instance = Pyexcel() for name in row.keys(): setattr(an_instance, name, row[name]) return an_instance importer = SQLTableImporter(mysession) adapter = SQLTableImportAdapter(Pyexcel) adapter.column_names = update_data[0] adapter.row_initializer = row_updater writer = SQLTableWriter(importer, adapter) writer.write_array(update_data[1:]) writer.close() query_sets = mysession.query(Pyexcel).all() results = QuerysetsReader(query_sets, self.data[0]).to_array() assert list(results) == updated_results mysession.close() def test_skipping_rows_if_data_exist(self): mysession = Session() # write existing data importer = SQLTableImporter(mysession) adapter = SQLTableImportAdapter(Pyexcel) adapter.column_names = self.data[0] writer = SQLTableWriter(importer, adapter) writer.write_array(self.data[1:]) writer.close() query_sets = mysession.query(Pyexcel).all() results = QuerysetsReader(query_sets, self.data[0]).to_array() assert list(results) == self.results # update data using custom initializer update_data = [ ["birth", "id", "name", "weight"], [datetime.date(2014, 11, 11), 0, "Adam_E", 12.25], [datetime.date(2014, 11, 12), 1, "Smith_E", 11.25], ] def row_updater(row): an_instance = mysession.query(Pyexcel).get(row["id"]) if an_instance is not None: raise PyexcelSQLSkipRowException() importer = SQLTableImporter(mysession) adapter = SQLTableImportAdapter(Pyexcel) adapter.column_names = update_data[0] adapter.row_initializer = row_updater writer = SQLTableWriter(importer, adapter) writer.write_array(update_data[1:]) writer.close() query_sets = mysession.query(Pyexcel).all() results = QuerysetsReader(query_sets, self.data[0]).to_array() assert list(results) == self.results mysession.close() def test_one_table_with_empty_rows(self): mysession = Session() data = [ ["birth", "id", "name", "weight"], ["", "", ""], [datetime.date(2014, 11, 11), 0, "Adam", 11.25], [datetime.date(2014, 11, 12), 1, "Smith", 12.25], ] importer = SQLTableImporter(mysession) adapter = SQLTableImportAdapter(Pyexcel) adapter.column_names = data[0] writer = SQLTableWriter(importer, adapter) writer.write_array(data[1:]) writer.close() query_sets = mysession.query(Pyexcel).all() results = QuerysetsReader(query_sets, data[0]).to_array() assert list(results) == self.results mysession.close() def test_one_table_with_empty_string_in_unique_field(self): mysession = Session() data = [ ["birth", "id", "name", "weight"], [datetime.date(2014, 11, 11), 0, "", 11.25], [datetime.date(2014, 11, 12), 1, "", 12.25], ] importer = SQLTableImporter(mysession) adapter = SQLTableImportAdapter(Pyexcel) adapter.column_names = data[0] writer = SQLTableWriter(importer, adapter) writer.write_array(data[1:]) writer.close() query_sets = mysession.query(Pyexcel).all() results = QuerysetsReader(query_sets, data[0]).to_array() assert list(results) == [ ["birth", "id", "name", "weight"], ["2014-11-11", 0, None, 11.25], ["2014-11-12", 1, None, 12.25], ] mysession.close() def test_one_table_using_mapdict_as_array(self): mysession = Session() self.data = [ ["Birth Date", "Id", "Name", "Weight"], [datetime.date(2014, 11, 11), 0, "Adam", 11.25], [datetime.date(2014, 11, 12), 1, "Smith", 12.25], ] mapdict = ["birth", "id", "name", "weight"] importer = SQLTableImporter(mysession) adapter = SQLTableImportAdapter(Pyexcel) adapter.column_names = self.data[0] adapter.column_name_mapping_dict = mapdict writer = SQLTableWriter(importer, adapter) writer.write_array(self.data[1:]) writer.close() query_sets = mysession.query(Pyexcel).all() results = QuerysetsReader(query_sets, mapdict).to_array() assert list(results) == self.results mysession.close() def test_one_table_using_mapdict_as_dict(self): mysession = Session() self.data = [ ["Birth Date", "Id", "Name", "Weight"], [datetime.date(2014, 11, 11), 0, "Adam", 11.25], [datetime.date(2014, 11, 12), 1, "Smith", 12.25], ] mapdict = { "Birth Date": "birth", "Id": "id", "Name": "name", "Weight": "weight", } importer = SQLTableImporter(mysession) adapter = SQLTableImportAdapter(Pyexcel) adapter.column_names = self.data[0] adapter.column_name_mapping_dict = mapdict writer = SQLTableWriter(importer, adapter) writer.write_array(self.data[1:]) writer.close() query_sets = mysession.query(Pyexcel).all() results = QuerysetsReader( query_sets, ["birth", "id", "name", "weight"] ).to_array() assert list(results) == self.results mysession.close() class TestMultipleRead: def setUp(self): Base.metadata.drop_all(engine) Base.metadata.create_all(engine) self.session = Session() data = { "Category": [["id", "name"], [1, "News"], [2, "Sports"]], "Post": [ ["id", "title", "body", "pub_date", "category"], [ 1, "Title A", "formal", datetime.datetime(2015, 1, 20, 23, 28, 29), "News", ], [ 2, "Title B", "informal", datetime.datetime(2015, 1, 20, 23, 28, 30), "Sports", ], ], } def category_init_func(row): c = Category(row["name"]) c.id = row["id"] return c def post_init_func(row): c = ( self.session.query(Category) .filter_by(name=row["category"]) .first() ) p = Post(row["title"], row["body"], c, row["pub_date"]) return p importer = SQLTableImporter(self.session) category_adapter = SQLTableImportAdapter(Category) category_adapter.column_names = data["Category"][0] category_adapter.row_initializer = category_init_func importer.append(category_adapter) post_adapter = SQLTableImportAdapter(Post) post_adapter.column_names = data["Post"][0] post_adapter.row_initializer = post_init_func importer.append(post_adapter) writer = SQLBookWriter(importer, "sql") to_store = OrderedDict() to_store.update({category_adapter.get_name(): data["Category"][1:]}) to_store.update({post_adapter.get_name(): data["Post"][1:]}) writer.write(to_store) writer.close() def test_read(self): exporter = SQLTableExporter(self.session) category_adapter = SQLTableExportAdapter(Category) exporter.append(category_adapter) post_adapter = SQLTableExportAdapter(Post) exporter.append(post_adapter) reader = SQLBookReader(exporter, "sql") result = read_all(reader) for key in result: result[key] = list(result[key]) assert json.dumps(result) == ( '{"category": [["id", "name"], [1, "News"], [2, "Sports"]], ' + '"post": [["body", "category_id", "id", "pub_date", "title"], ' + '["formal", 1, 1, "2015-01-20T23:28:29", "Title A"], ' + '["informal", 2, 2, "2015-01-20T23:28:30", "Title B"]]}' ) def test_foreign_key(self): all_posts = self.session.query(Post).all() column_names = ["category__name", "title"] data = list(QuerysetsReader(all_posts, column_names).to_array()) eq_( json.dumps(data), '[["category__name", "title"], ["News", "Title A"],' + ' ["Sports", "Title B"]]', ) def tearDown(self): self.session.close() class TestZeroRead: def setUp(self): Base.metadata.drop_all(engine) Base.metadata.create_all(engine) def test_sql(self): mysession = Session() sheet = SQLTableReader(mysession, Pyexcel) data = sheet.to_array() content = [[], []] # 'pyexcel' here is the table name assert list(data) == content mysession.close() class TestNoAutoCommit: def setUp(self): Base.metadata.drop_all(engine) Base.metadata.create_all(engine) self.data = [ ["birth", "id", "name", "weight"], [datetime.date(2014, 11, 11), 0, "Adam", 11.25], [datetime.date(2014, 11, 12), 1, "Smith", 12.25], ] self.results = [ ["birth", "id", "name", "weight"], ["2014-11-11", 0, "Adam", 11.25], ["2014-11-12", 1, "Smith", 12.25], ] def test_one_table(self): if PY36: # skip the test # beause python 3.6 sqlite give segmentation fault return mysession = Session() importer = SQLTableImporter(mysession) adapter = SQLTableImportAdapter(Pyexcel) adapter.column_names = self.data[0] writer = SQLTableWriter(importer, adapter, auto_commit=False) writer.write_array(self.data[1:]) writer.close() mysession.close() mysession2 = Session() query_sets = mysession2.query(Pyexcel).all() eq_(len(query_sets), 0) mysession2.close() @raises(TypeError) def test_not_implemented_method(): reader = SQLBookReader() reader.open("afile") @raises(TypeError) def test_not_implemented_method_2(): reader = SQLBookReader() reader.open_stream("afile") def test_sql_table_import_adapter(): adapter = SQLTableImportAdapter(Pyexcel) adapter.column_names = ["a"] adapter.row_initializer = "abc" eq_(adapter.row_initializer, "abc") @raises(Exception) def test_unknown_sheet(): importer = SQLTableImporter(None) category_adapter = SQLTableImportAdapter(Category) category_adapter.column_names = [""] importer.append(category_adapter) writer = SQLBookWriter(importer, "sql") to_store = OrderedDict() to_store.update({"you do not see me": [[]]}) writer.write(to_store) def read_all(reader): result = OrderedDict() for index, sheet in enumerate(reader.content_array): result.update({sheet.name: reader.read_sheet(index).to_array()}) return result