The Bits That Python's zipfile Forgot

Summary: Portable and powerful tools for zipping and unzipping zipfiles Version: 1.3, October 28, 2021 (

changes

) License: Provided freely, but with no warranties of any kind Author: © M. Lutz (

learning-python.com

) 2017-2021 Run with: Python 3.X or 2.X on

Install: Unzip the

, no third-party libraries are required Web page:

learning-python.com/ziptools.html

This source-code package wraps Python's zipfile module to simplify common

usage

, and extends it with crucial extra utility—including support for adding entire folders, propagating modtimes and permissions, archiving symlinks on Unix and Windows, omitting cruft files, using long paths on Windows, and neutralizing the DST and timezone problems of zip modtimes.

In addition, ziptools is dual-mode software, providing:

Command-line scripts for general use (the zip-*.py files

here

)

Library tools for use in programs (module ziptools.py

here

)

Both foster flexible and cross-platform content management using zipfiles, above and beyond Python's standard library.

This

file is ziptools' first-level user guide; its source-code files

here

and

here

provide additional and lower-level details. If you have used ziptools in the past, see

Versions

at the end of this file for recent updates. Contents of this guide:

Quickstart

General Command Formats

Command-Line Examples

Program-Library Examples

Related-Tool Examples

Long Paths on Windows

Permissions Propagation

UTC Timestamps

Nonportable Filenames

Version 1.3: Oct-28-2021

Version 1.2: Apr-11-2020

Version 1.1: Apr-2-2020

Version 1.0: Jun-12-2017

Future Ideas

Quickstart

This section presents command formats and examples, as a quick introduction and reference. Throughout this guide:

$ stands for your system's command prompt

py is your installed Python's name (e.g., py -3, python3)

mycontent and myunzipdir may be relative or absolute pathnames

myarchive.zip may be located at a relative or absolute path

$Z is the path to the unzipped ziptools/ folder on your device

. means the current folder as create source or extract destination

* in zip sources is expanded (globbed) on both Unix and Windows

General Command Formats

This package provides both scripts and library calls that create (zip) and extract (unzip) zipfiles. Its scripts are run by Python command lines, and its library calls are run by other programs using a Python module. This section summarizes both usage modes.

In the following command-line formats, items enclosed in [] are optional, and ... means zero or more:

[py] zip-create.py [zipfilesource [source...] [-skipcruft] [-atlinks] [-zip@path] [-nocompress]] [py] zip-extract.py [zipfile [unzipto] [-nofixlinks] [-permissions] [-nomangle]] [py] zip-list.py [zipfile] Where:

zipfile is the pathname of a zipfile (a .zip is appended if absent)

source is the pathname of a file, folder, or symlink to be added to a zipfile

unzipto is the pathname of a folder in which to unzip (auto-created, . is here and default)

-skipcruft prevents system "cruft" files from being added to a zipfile

-atlinks saves items referenced by symlinks instead of links themselves

-zip@path gives an alternate path at which to zip all sources (. is top-level)

-nocompress disables standard item compression, for faster zip and unzip of large archives

-nofixlinks disables adjustment of symlink path separators for target platforms

-permissions enables propagation of Unix-style permissions on unzips

-nomangle disables replacement of nonportable characters with "_" on Windows

In program-library mode, most function arguments correspond to command-line switches:

createzipfile(zipname, [addnames], storedirs=True, cruftpatts={}, atlinks=False, trace=print, zipat=None, nocompress=False) extractzipfile(zipname, pathto='.', nofixlinks=False, trace=print, permissions=False, nomangle=False)

Command-Line Examples

Run the following commands in a console, script, or supporting IDE:

$ py $Z/zip-create.py myarchive.zip mycontent -skipcruft # Store all of folder mycontent in new zipfile myarchive.zip $ py $Z/zip-extract.py myarchive.zip myunzipdir # Extract the contents of myarchive.zip to folder myunzipdir $ py $Z/zip-extract.py myarchive.zip . -permissions # Ditto, but extract to '.' and propagate Unix-style permissions $ py $Z/zip-create.py ../myarchive.zip * -skipcruft # Store folder contents as top-level items, on Unix or Windows $ py $Z/zip-create.py mysourcecode.zip page.js gui.pyw *.py *.c # Store individual files as top-level items, on Unix or Windows $ py $Z/zip-create.py mysourcecode.zip src/latest/*.py -zip@. # Store individual files as top-level items, on Unix or Windows $ py $Z/zip-extract.py mysourcecode.zip code -nomangle # Extract to a folder, no "_" replacements (run fixer script) $ py $Z/zip-list.py myarchive.zip # List the contents of zipfile myarchive.zip $ py $Z/zip-create.py $ py $Z/zip-extract.py # Interactive mode: input parameters at prompts (see ahead)

Program-Library Examples

Run the following Python code samples in a script or at the >>> interactive prompt; they correspond to command lines above:

$ export PYTHONPATH=$Z:$PYTHONPATH $ py >>> import ziptools, glob >>> cruftdflt = ziptools.cruft_skip_keep >>> ziptools.createzipfile('myarchive.zip', ['mycontent'], cruftpatts=cruftdflt) >>> ziptools.extractzipfile('myarchive.zip', pathto='myunzipdir') >>> ziptools.extractzipfile('myarchive.zip', pathto='myunzipdir', permissions=True) >>> ziptools.createzipfile('../myarchive.zip', glob.glob('*'), cruftpatts=cruftdflt)

Related-Tool Examples

You may find some Mergeall scripts, available

here

, useful for verifying this package's results. In the following example command lines, $C is your source-code install folder:

$ py $C/mergeall/mergeall.py mycontent myunzipdir/mycontent -report -skipcruft # Verify result modtimes using Mergeall $ py $C/mergeall/diffall.py mycontent myunzipdir/mycontent -skipcruft # Verify result content using Mergeall

Features

This section explains why you may want to use ziptools, and describes its features. In brief, Python's standard-library

zipfile

module does great low-level work, but the ziptools package adds both much-needed features and higher-level access points, and documents some largely undocumented dark corners of Python's zip support along the way. Among its augmentations, ziptools:

Adds entire

folder trees

to zipfiles automatically

Propagates original

modtimes

for files, folders, and links

Can either include or skip system

"cruft" files

on request

Propagates

symlinks

to files and folders on Unix and Windows

Supports

long paths

to items on all Windows in all Pythons

Propagates Unix file-access

permissions

for all items on request

Fixes DST and timezone skew in modtimes with

UTC timestamps

Makes

nonportable-filename

mods explicit, optional, and avoidable

Although Python's

shutil

module has basic zipfile wrappers that add folders (make_archive) and extract all items (unpack_archive), they don't do anything about modtimes, permissions, symlinks, system cruft, or long Windows paths supported here, and don't address the limits of zip's local times or call out the perils of name mangling in Python's zipfile module. The next sections describe how ziptools does better on all these fronts.

Folder Trees

With ziptools, folder trees are added to zipfiles as a whole automatically with extra code—a sorely missed feature of Python's standard modules. Folder trees are also automatically extracted from zipfiles in full, and no user steps are required to enable folder-tree zips and unzips.

For folder-tree implementation details, see the create functions in

ziptools.py

Modtime Propagation

With ziptools, latest-modification times (a.k.a. modtimes) are automatically propagated to and from zipfiles for all items—files, folders, and symlinks. This is another glaring omission in Python's standard module, which stores modtimes in zips but ignores them on unzips. modtimes are crucial when unzipped results are used with tools that rely on file-modification timestamps—including the Mergeall program's incremental

backups

, and most source-control systems' change detection.

No user actions are necessary to enable modtime propagation. As of 1.2, this feature has grown even more useful with 1.2's UTC timestamps described

ahead

; this neutralizes the limits inherent in the zip standard's

local time

. For the implementation of basic modtime propagation, see the extract function in

ziptools.py

Cruft-File Skipping

With ziptools, platform-specific metadata items (a.k.a. cruft) can be automatically omitted from cross-platform zipfile archives. When enabled in creates, this ensures that these items won't wind up as clutter in contexts where they serve no purpose.

This feature is optional and configurable, and works for files, folders, and symlinks. To omit cruft items in zips, use the -skipcruft command-line switch or corresponding function argument. As shipped, this avoids propagating .DS_Store Finder

files

and ._* Apple-double resource

forks

on macOS; Desktop.ini files on

Windows

; .Trash* and other recycle

bins

; and most other .* files hidden per Unix convention which don't belong in zipped content.

For more control, you can also define what to skip in your use case. Cruft is identified with skip and keep filename patterns—either a custom set, or a provided general-purpose default used automatically by the command-line scripts. See

zip-create.py

and

ziptools.py

for more details, and

zipcruft.py

for the default-but-changeable cruft patterns shipped.

Symlinks Propagation

With ziptools, symbolic links (a.k.a. symlinks) to both files and folders are supported and propagated on both Unix and Windows. By default, symlinks are always copied verbatim to and from zipfiles, and their path separators are made portable along the way. No user action is required to enable this behavior.

Optionally, clients may also elect to instead copy the items which links reference with -atlinks and can opt out of link portability transforms with -nofixlinks (or their corresponding function arguments). In more detail:

When -atlinks is omitted, ziptools adds links themselves to the zip, and hence copies them verbatim. When -atlinks is used, ziptools instead follows links, and adds their targets to the zip in full.

When links are copied verbatim, they are by default also made portable between Unix and Windows by automatically adjusting link paths on unzips for the hosting platform's path separators— any \ are changed to / on Unix, and any / are morphed to \ on Windows. Simply zip and unzip to transport symlinks across platforms. Because \ is rare but valid in Unix

filenames

, the -nofixlinks switch suppresses these path-separator adjustments if needed.

When links are followed to copy their targets, links which are recursive (a.k.a. cyclic) are automatically detected and copied verbatim to avoid loops, on platforms that support

inode

-like identifiers. This recursion detection works on all Unix, and on Windows using Pythons 3.2 and later.

Subtly, /-to-\ adjustment is required when unzipping symlinks on Windows, even though most Windows APIs allow the two to be used interchangeably. On Windows 11 in Python 3.X: >>> os.symlink(r'temp\afile.txt', 'alink-n-w') # Windows path, like mklink >>> os.symlink(r'temp/afile.txt', 'alink-n-u') # Unix path (e.g., zipped) >>> os.readlink('alink-n-w') # read link path, \\ is \ 'temp\\afile.txt' >>> os.readlink('alink-n-u') # Unix path created okay 'temp/afile.txt' >>> open('alink-n-w').read() # follow link path 'hello nested' >>> open('alink-n-u').read() # Unix path fails (Explorer too) OSError: [Errno 22] Invalid argument: 'alink-n-u' Hence, all path-separators in symlink paths are by default converted to the unzip platform's flavor for portability. This generally makes symlinks portable, though link paths must also exist on the unzip host (e.g., absolute paths may not work), and platform-specific syntax must be recognized on the unzip target (e.g., Windows drive letters will fail on Unix). Scope: symlinks that mind these rules do survive round trips between Unix and Windows, but their support is uneven across platforms and Pythons. Windows symlinks, for example, do not retain their original modtimes on unzips in 3.X, and their permissions are largely moot. Moreover, unzips require admin permissions to create symlinks on Windows (until Windows 10/11); Android may not support symlink creation at all, and leave stub files on unzips; and Python's symlink support varies by version. See

Portability

ahead for more fine-grained details on symlink interoperability.

For a demo of ziptools' symlinks support, see the console log

here

. For symlink implementation details, see

zipsymlinks.py

. See also

zip-create.py

for more on -atlinks;

zip-extract.py

for more on -nofixlinks; and

ziptools.py

for more on both usage options.

Long Paths on Windows

With ziptools, filesystem pathnames of all items on Windows are allowed to exceed their normal 260/248-character length limit on that platform. This is accomplished by automatically prefixing paths with \\?\ whenever they are passed to tools in the underlying Python zipfile module, and other file-related interfaces.

No user action is required for this feature. On all versions of Windows, it supports files, folders, and symlinks at long Windows paths both when adding to and extracting from zip archives. Among other things, this is useful for unzipping and rezipping long-path items originally zipped on more-flexible Unix hosts, where this feature is unneeded and unused.

For implementation details, see

ziplongpaths.py

Windows path-limit install option in Python 3.6+:

on recent Windows 10 systems, the standard python.org

installers

for Python 3.6 and later include an

option

to automatically remove the Windows platform's former path-length limit. ziptools, and its

Mergeall

cousin, instead use the manual but more-inclusive technique described here to lift the limit for users of all Pythons and all Windows. While useful, the Python 3.6+ enhancement is optional and easy to miss; won't apply to frozen executables; and doesn't help those using Windows 7 and 8, or Pythons 2.X through 3.5—still-substantial audiences all. For more details, see this online

usage note

Permissions Propagation (New in 1.1)

With ziptools, Unix-style permissions for all items—files, folders, and symlinks—can be propagated both to and from zipfiles, and hence survive a zip and unzip. Specifically:

Unix permissions are always propagated to zipfiles on creates (zips). This has always been the case for files and folders, but includes permissions of symlinks as of 1.1.

Unix permissions are optionally propagated from zipfiles on extracts (unzips) in 1.1, when explicitly requested by using the new -permissions command-line argument, or its function-call equivalent.

Due to interoperability

issues

, this new extract option should generally be used only when unzipping from zipfiles known to have originated on Unix, and when unzipping back to Unix. Most use cases that require permissions to survive trips to/from zips probably satisfy this rule, but propagation isn't enabled by default because origin cannot reliably be determined. Propagation is generally harmless where Unix-style permissions are not supported: it doesn't abort ziptools zips or unzips, though results may lose permissions per the following note.

Scope: not all filesystems and platforms support Unix-style permissions. On exFAT, for instance, permission updates silently change

nothing

, and the extract option has no effect. It's okay to copy a zipfile to and from an exFAT drive as a whole, but don't unzip on exFAT if you care about retaining Unix permissions. Unix-style permissions also don't fully translate to Windows, and may not work at all on Android. See

Portability

ahead for full details, and don't send permissions on round trips to platforms that don't support them.

For a demo of 1.1 permissions propagation, see the screenshot

here

and the console logs

here

and

here

. For implementation details, see

ziptools.py

and

zipsymlinks.py

UTC Timestamps (New in 1.2)

With ziptools, the modtimes of files, folders, and symlinks are automatically made immune to changes in both timezone and DST (Daylight Savings Time). This is achieved by storing UTC (a.k.a. GMT) timestamps in one of the "extra fields" defined by the zipfile

standard

. This feature is always applied, and no user action is required to enable it. In more detail:

On zips, ziptools adds an extended-timestamp extra field (code 0x5455) to the zipfile's central-directory entry of each stored item, per the normal extensions mechanism of zip. These fields embed UTC timestamps for modtimes obtained from the filesystem.

On unzips, ziptools tries to fetch these timestamps from the same location. When present, they are used for propagated modtime directly, instead of trying to convert the main MS-DOS "local time" of items to UTC. When absent, local time is used instead.

This is a full fix to zip's local-time issues: because

UTC

timestamps are relative to a fixed point, they are agnostic to both timezone and DST changes. By contrast, local time is scaled for both, and may be better used for display. The zip standard's use of local time for file storage, however, makes it difficult to adjust for DST, and impossible to adjust for timezones—especially given zip's lack of timezone information (if you fly three timezones away, zip local times won't change with you). UTC is the only way to keep zipped modtimes comparable in all contexts with the original

data

Prior to 1.2, ziptools used the zipfile's main local time, and deferred to Python's library tools

here

and

here

to translate UTC time to and from local time and accommodate DST changes. That scheme's modtime results could vary from those of some other zip tools after DST changes, however, and did nothing about timezone changes. The new UTC timestamp scheme in 1.2 eliminates both DST and timezone modtime skew in a single step.

Interoperability: not all zip tools use or record the extra field added by ziptools 1.2, but the absence of support is harmless. When processing zipfiles with the new field, tools that don't recognize it will simply skip it, per zip standard. When processing zipfiles without the new field, ziptools will fall back on its original and subpar local-time scheme. More positively, ziptools will use UTC-timestamp modtimes recorded by other tools in the same fashion as its own.

For a demo of 1.2 UTC timestamps in action, see the console log

here

, and the screenshot

here

. For implementation details, see

zipmodtimeutc.py

, and its

ziptools.py

and

zipsymlinks.py

clients.

Nonportable Filenames (New in 1.3)

Filenames containing any of the characters / \ | < > ? * : " are nonportable, and cannot be saved on some filesystems. In ziptools, unzips (extracts) by default automatically change names of files, folders, and symlinks as needed so they can be saved. These changes, known as mangling, replace all nonportable filename characters with _ on unzip failures, and retry the failed save. As of version 1.3, ziptools mangling is explicit and optional: mangles are reported, and -nomangle disables its changes altogether.

Automatic filename mangling is done only on Windows, and is run for all filesystems on Windows because these rules are imposed across the platform. Mangling is not attempted for save failures on Android, or external drives written on other platforms. Android 11 shared storage emulates Windows' rules, but has a bug discussed ahead which precludes mangling. FAT32 and exFAT drives impose the same rules too, but Linux writes to such drives are allowed to fail with messages and skips to avoid perpetual diffs, and macOS silently munges nonportables to/from Unicode privates on such drives as noted ahead.

While auto-mangling may be helpful in some contexts, it can also create multiple problems later in content-management life cycles. Users are advised to instead run a new provided fixer script,

fix-nonportable-filenames.py

, to analyze and replace nonportable filename characters, before propagating content from Unix (e.g., macOS and Linux) to Windows, Android 11 shared storage, and external drives using filesystems which restrict filenames.

This fixer script sidesteps multiple interoperability traps. By fixing filenames at their source, it avoids potential overwrites and back-sync errors inherent in mangling; prevents recurring diffs on Linux and later network-sync errors on macOS; and can be useful for content transferred by ziptools unzips, Mergeall

syncs

, and other techniques (e.g., explorer copies to FAT32, exFAT, and BDR drives).

Run this script from a command line on Unix to display or repair all nonportable names in an entire folder tree:

$ python3 fix-nonportable-filenames.py folderrootpath 1 # list items only $ python3 fix-nonportable-filenames.py folderrootpath # list and rename items See the script's in-file

docs

for more details. If you choose to let auto-mangling happen anyhow, the next two sections provide more details about ziptools' policies for it, and explain why it's not performed on the similarly constrained Android. Mangling Transparency

Prior to version 1.3, filename mangling was performed in ziptools by Python's underlying

zipfile

module for all filenames, and on Windows alone. The filename changes made by that module both are silent, and have the potential to damage content in two ways:

Mangling can lead to data loss: mangled names may overwrite or be overwritten by other files whose names are coincidentally the same

Mangling can break syncs back the source: mangled names will no longer match, unless all source names are also mangled before syncs occur

Given these perils, filename mangling should never be mandatory and silent. To do better, ziptools 1.3 wrests control of mangling away from the Python module, and reimplements it to provide better transparency. It:

Explicitly reports and tallies all mangles and skips in run output, like

this

Makes mangling an optional feature, even on Windows: -nomangle turns it off

completely

Provides the fixer

script

noted earlier to inspect and change nonportable names ahead of time

The first two of these require forcibly disabling zipfile mangling, catching save errors, manually mangling, and rerunning extracts, but this allows both user disables and reporting. Check the

output

after an unzip for details about any filename changes made; mangles are labeled as "--Name mangled" and skip messages as "**SKIP". Tallies for mangles and skips are also displayed at the end for runs in which they are nonzero.

The fixer script is generally recommended over ziptools' auto-mangling as noted, but auto-mangling is enabled by default because its issues are rare. Indeed, nonportable filenames themselves are generally atypical; they may crop up most for trivial web-page saves on Unix, and are best avoided as a rule. ziptools policies provide a safer alternative when these files creep into your content anyhow.

It's worth adding that the native unzip extractor run by Windows 10's own file explorer simply discards files with nonportable names silently. This is roughly the same as ziptools -nomangle option, though ziptools also reports and tallies the skips, instead of quietly dropping your content.

Android Shared Storage

Beyond its lack of transparency, Python's underlying

zipfile

module also mangles on Windows only. This is insufficient:

shared storage

on some versions of Android, for example, is implemented with a filesystem driver that emulates FAT32, and disallows nonportable characters similarly. In principle, mangling should be applied on this platform too, to enable unzip saves.

In fact, Android, being Android, implements filename rules which vary by storage type, version, and perhaps even vendor. Two Android 11 devices tested disallow Windows illegal characters in shared storage, because it uses a FUSE driver; but an Android 10 device does not, because its shared storage uses SDCardFS. Moreover, no tested device disallows Windows illegals in app-specific or app-private storage, which generally use Linux's ext4 filesystem. On Android 11 in Termux:

$ cd /sdcard # shared storage (fails on 11, not 10) $ echo xxx > test\?.txt bash: test?.txt: Operation not permitted $ echo xxx > test\|.txt bash: test|.txt: Operation not permitted $ cd /sdcard/Android/data/com.termux # app-specific storage $ echo xxx > test\?.txt $ echo xxx > test\|.txt $ cd /data/data/com.termux # app-private storage $ echo xxx > test\?.txt $ echo xxx > test\|.txt To handle such convolution and be universal, mangling might be attempted on all platforms when extracts fail. Supporting Android shared storage this way was explored as part of this update, until it encountered a showstopper bug which rules out mangling on the platform altogether.

In short, there's nothing that ziptools can do for this use case because Android 11 is broken: its FUSE-based shared storage disallows nonportable characters in filenames as it should, but, unlike Windows, silently allows them in folder names, leaving crippled folders. A nonportable folders path will be silently created, but no files can be stored inside it afterwards—even if they are fully legal. Because no error is raised for such folders' creation, there is nothing for ziptools to catch and respond to. In Termux on Android 11 again:

$ cd /sdcard $ echo spam > file\|name.txt # illegal files fail in shared bash: file|name.txt: Operation not permitted $ $ mkdir dir\|name # but illegal folders work $ cd dir\|name $ pwd /sdcard/dir|name $ $ echo spam > file\|name.txt # but they don't accept files bash: file|name.txt: Operation not permitted $ $ echo spam > filename.txt # even if they are legal: bug! bash: filename.txt: Operation not permitted $ $ mkdir sub\|name # but broken subfolders are okay... $ ls 'sub|name' This isn't a problem on Windows, because invalid pathnames cannot be created, and raise errors as expected. Because Android 11 silently creates invalid pathnames, however, mangling must be limited to Windows alone. Else, mangling an item's path on Android would create a doppelgänger path, which parallels the nonportable and useless path made by Android. Moreover, it would be dangerous to simply remove or rename the bogus path built by Android, because it may have been created outside the scope of an unzip, and hence used arbitrarily.

To do better, ziptools would have to know if filename mangling is required before any folders are created. The third-party

psutil

library offers filesystem queries that might help on this front. Unfortunately, it does not support Android, per both documentation and testing: on unrooted Androids 10 and 11, the latest psutil 5.8 installed with pip in Termux yielded a permission-error exception on first import, for a failed access to /proc/stat. Later calls for disk-partitions info similarly failed on /proc/filesystems. This library is clearly unusable on Android today.

Even if psutil worked, however, it's a long shot: identifying Android version and filesystem combinations that require mangling seems nearly nondeterministic, and doing so ahead of time for Android would require duplicating zipfile's extract method in full, because it hardcodes a test for Windows alone. This is a mod too far—especially given that name mangling is on thin data-integrity ice in the first place.

Hence, failing nonportable filenames on Android are simply skipped with messages and tallies. For ziptools users hoping to unzip Unix content on Android, this leaves three work-arounds:

Change filenames in content before it is propagated You can still use Android 11 shared storage, as long as your filenames are portable. The included fixer

script

, borrowed from the Mergeall system, both reports and changes nonportable characters, and can be run before transfers by users concerned about data-loss potential. Use this to both analyze your content, and weed out any filename interoperability issues before they can cause cross-platform nightmares. Use app-specific or app-private storage on Android instead Android's app-specific and app-private storage use Linux filesystems, and so support most filename characters without mangling. They are also generally much faster than shared storage in Android 11. Regrettably, these storage categories also have restricted access, and content in both is by default automatically deleted on app uninstall; using either for general storage is limited and perilous. Accept a minor data loss of nonportably named content Nonportable filename characters often arise in web-page clippings saved by Unix browsers. Especially if Android devices are used in read-only mode, loss of such content on transfers may be an acceptable sacrifice. To see what may be lost, run the fixer

script

's list-only mode on folders in your content tree before copies; some intended uses of nonportable characters may require explicit handling, not automatic mangling. Of these, the first is again recommended. Before transferring content from Unix to Windows, Android shared storage, or other similarly limited contexts, run this script to sanitize it of characters that pose an interoperability hurdle. By doing this at content's point of origin, it both avoids ziptools skips on Android, and sidesteps potential overwrites and later back-sync issues inherent in ziptools' auto-mangling on Windows.

Bonus: running the fixer

script

also avoids similar issues in other contexts. Linux, for example, reports errors and refuses to copy nonportable filenames to Windows-filesystem drives (e.g., exFAT) in both file explorers and command lines. Worse, macOS breaks syncs by silently mangling nonportable filename characters to and from Unicode private characters (instead of "_") on FAT32 and exFAT drives, which fails when files are used outside macOS's scope; read the sordid details

here

. By fixing filenames at their source, you remove such variables—and unpleasant surprises—from your content copies.

Mangling Summary

To summarize ziptools' policies for nonportable filenames by platform:

When unzipping on Windows ziptools mangles unless the -nomangle option is used, so files can be saved. If mangling is used, mangles are reported and tallied in program output. If mangling is not used, nonportable names will fail to extract, and be skipped with a message and tally in program output. Despite auto-mangling support, running the fixer

script

is still recommended before unzipping on Windows, to avoid the potential perils of automatic name changes. When unzipping on Android ziptools does not mangle, even for shared storage, for the reasons outlined in the preceding section. Because of this, running the fixer

script

is recommended before unzipping on Android shared storage, to avoid skips. If the fixer script is not used, nonportable names will fail to extract in some versions' shared storage, and be skipped with a message and tally in program output. When unzipping on Unix ziptools does not mangle, because filenames are much more liberal. Content lucky enough to live only on Unix generally doesn't need to care about any of this interoperability drama. If, however, content on Linux or macOS must ever be copied to Windows, Android, or most external drives, running the fixer

script

first can avoid later headaches along the content-maintenance road. In short, ziptools reports filename changes, and performs them only on Windows. Thus, its -nomangle is relevant only on Windows, and is irrelevant even there if you run the fixer

script

, because there will be no names to mangle. Takeaway: except when unzipping on Unix or safe Android storage, using the script always maximizes content integrity, and should be the opening act of most cross-platform plays.

For More Mangling Details

There's more to the mangling story elsewhere:

For another demo of Android's filename rules, see

this

For a peek at ziptools' mangling in action on Windows, see

this

For implementation details, see trymangle() in ziptools/

ziptools.py

For ziptools' related handling of Windows backslashes, see

ahead

For related coverage, see the Android Deltas Sync note

here

For more related coverage, see the Mergeall note

here

And for demos of the sorts of filenames disallowed and mangled on Windows and Android, see also the results

here

and

here

. As these show, filename rules vary slightly between Windows and Android; Android's shared storage limits names but its app-specific storage does not; Windows limits folder names but Android shared storage does not; and exceptions raised on errors differ between the two platforms. Alas, filename interoperability seems a pipe dream, and perhaps not accidentally so.

Etcetera

Beyond the preceding sections' features, the ziptools package also provides command-line zip and unzip programs that work portably on

, and more; runs all its code on either Python 3.X or 2.X; and comes with complete and open-source Python code that you can use, audit, and adapt.

Again, see the main

zip-create.py

and

zip-extract.py

command-line scripts for more usage details omitted here, and the

ziptools.py

module they use for lower-level implementation details. For an example client of ziptools command lines, see the README and scripts of this

package

Usage

This section provides expanded usage examples to demo typical ziptools roles, and documents additional usage details along the way. For more examples external to this guide, the test-case folders:

selftest/

cmdtest/

moretests/

all capture usage and runs, and each script and module in this package includes in-depth documentation strings with usage details omitted here for space. In addition, the documentation folders:

docetc/1.1-upgrades/

docetc/1.2-upgrades/

demonstrate the new features in versions 1.1 and 1.2; the screenshots

here

and

here

capture basic usage; and Mergeall's test

folder

illustrates its related symlink support.

General Concepts

Before we get into command details, this section presents ziptools usage pointers that span modes:

Output

Both ziptools creates and extracts print one or two lines per item to standard output. The first line gives the item's source name: the file for creates, and the zipfile entry for extracts. The second line gives the destination following a =>, and appears only if the item's destination differs from its source. In creates, a second line is printed if the -zip@path switch described ahead is used to rename source folders in the zipfile. In extracts, a second line is displayed if the item's target path differs from its source in the zipfile, after mapping Windows \ to zip's /; this occurs when extracting to a folder other than .. In all cases, run output may be routed with shell | and >, and function-call trace arguments. ziptools 1.3 changed extracts to print one or two lines, as described above. Extracts formerly printed two lines for every item for clarity, but 1.3 drops the second line when the source and target paths are the same—a common case when extracting to the current directory. In this case, the former target line is redundant; omitting it reduces extract output enormously; and the resulting behavior mimics creates. In all other cases, extracts still print two lines as before, to make source and target explicit.

Portability All code in this package works under both Python 3.X and 2.X, and on both Unix and Windows, though some advanced tools (e.g., symlinks and permissions) are platform or Python-version dependent. ziptools' zipfiles are generally interoperable with other zip tools, and vice versa, though some features may be supported unevenly elsewhere. When in doubt, use ziptools for both zips and unzips, and see

Portability

ahead for complete interoperability details. Pathnames Items added to zip archives are recorded with their relative or absolute paths given, less any Windows drive and UNC parts, and most .. relative-path syntax. Items are later unzipped to these recorded paths relative to an unzip target folder. Leading ..s may be recorded by zips, but are stripped by unzips. Tip: to reduce the length of paths saved and restored, cd to source folders before zipping —or see the next item on this list for a 1.2 alternative.

Windows backslashes:

per the zip standard, ziptools creates (zips) always use Unix / for path separators in zipfiles on all platforms, including Windows. This is ensured by the underlying zipfile module, which also replaces / with \ in paths and illegal characters with _ in filenames when extracting on Windows (the latter by default; see ziptools 1.3's mods

ahead

). While some Windows zip tools may erroneously record path separators as \, there is no way for ziptools extracts (unzips) to recognize these as such on Unix, because \ is also a valid filename character on Unix.

Hence, in ziptools:

When extracting on Unix, any Windows \ in zipfile paths are taken as part of a filename, and will not generate folders—even if they were meant to do so on Windows

Conversely, when extracting on Windows, all \ are taken as path separators and will generate folders—even if they were not meant to do so on Unix

The Windows extract behavior is a result of code in both ziptools itself and the underlying Python zipfile module. The Python module's behavior is a known

issue

, but ziptools' own code must be consistent with it because it can occur even when ziptools' code is not run (in brief, when ziptools'

mangling

is not required).

For finer details, search for "More on Backslashes" in

ziptools.py

, or try the

web

. If backslashes crop up in your content's filenames on Unix, your best option is to sidestep issues in full by running the fixer

script

before propagating to Windows with ziptools; this script changes backslashes to underscores before they can trigger unintended folders. And if backslashes show up in paths zipped on Windows, consider using another Windows zip tool that isn't an affront to both standards and interoperability.

Alternate zip paths

As of 1.2, the create -zip@path command-line switch (and corresponding function argument) allows zipped items to be recorded, and hence unzipped, under an alternate path, or none if path is .. This is useful to shorten long pathnames in zip sources and thus avoid nesting on unzips, and can make a pre-zip cd unnecessary. It can also be used as a renaming tool. This feature is best described by example. Normally, a source item A/B/C/D is zipped at path A/B/C/D, and later unzipped at the same path, relative to (i.e., nested in) the unzip target folder. Given -zip@C at zip time, the item is instead zipped and unzipped at C/D, and using -zip@. makes D a top-level, unnested item in both zip and unzip, stripping its A/B/C path of origin.

Really, this switch merely replaces the top-level provided path with another (or none, for .) in the zipfile. A -zip@X, for example, records A/B/C/D as X/D in the zipfile and propagates it as such on unzips, thereby renaming the path of origin. When used, this feature also yields two output lines per item to show zip paths used (zip-list.py shows results too).

Two fine points: first, note that the alternate path is applied to every source item zipped, and so may be more useful when zipping either a single item or many items in the same folder (else same-named items from different folders will collide). Second, the path replacement applies only to the top-level paths given; items nested in zipped folders are still nested the same in the zip—though at a root path which has been shortened, expanded, or arbitrarily renamed by -zip@path.

For full fidelity, see zipatmunge() in module

ziptools.py

. For a demo of this feature at work, see the console log

here

. Per this demo, -zip@. can also be used to remove folder nesting for all items that match * wildcards—as described further in the next section.

Windows PowerShell users:

you must quote the new switch in command lines as either "-zip@path" or '-zip@path'.

PowerShell

uses a proprietary scripting

language

which treats @

specially

, and unfortunately clashes with ziptools' argument. You do not need to quote this argument on Windows in either Command Prompt; the standard bash shell in Windows 10's

Linux

subsystem (WSL); or the Unix shells available in the

Cygwin

package. Users familiar with Unix may prefer the last two of these options, captured

here

and

here

. Windows, being Windows, has long spun a convoluted and fragmented command-line tale; such is life in battling-towers development.

Source globbing

When using the create script (only), source items may use filename expansion (a.k.a. globbing) on both Unix and Windows: * matches any number of characters; ? matches any single character; [xxx] matches any character in xxx; [!xxx] matches any character not in xxx; and brackets escape literals (e.g., [?] matches ?). All items with matching names are zipped. This feature also works when arguments are prompted in interactive mode (see ahead). As examples, a source item *.py on any platform zips all Python source files in the current folder as top-level, unnested items. Similarly, a source item folderpath/*.py zips all Python files located in another folder; use the -zip@path feature of the preceding note to shorten, change, or remove folderpath for each matching file zipped. For instance, given source item folderpath/*, -zip@. zips all the matching items as top-level, unnested items, and -zip@newfolder zips all the matching items in a different folder name (this is a folder rename). For a demo of globbing and its interplay with -zip@, see the console logs

here

and

here

Source .

Using . as a zip source item zips all items in the current directory as unnested items, and is functionally equivalent to a lone * to match all items (except that * omits any .* items hidden per Unix convention). This may be more useful for command lines than for function calls in program-library mode, given that callers might be running in a program folder. Target . Using . as the unzip target folder unzips each item in the zipfile to the current directory, with no extra folder nesting apart from that recorded in the zipfile itself (unnested items in the zip are unnested in .). Caution: this silently overwrites any same-named items in .; name another folder instead of . if you wish to avoid this.

Existence

In all modes, an error is reported for zip source items that do not exist, but unzip target folders are automatically created if they do not exist. In command-line mode, nonexistent sources are detected before a zip starts, and abort the request with an error message. In program-library mode, nonexistent sources are simply skipped with a message, because callers are responsible for input correctness.

Errors

Both zips and unzips in ziptools ignore propagation failures that involve metadata, and continue to process the rest of the item archive with a message and minor metadata loss. This includes failures for permissions, modtimes, and symlinks; in the latter case, a stub file or forged link may replace a failed symlink. More generally, unzips (extracts) continue with a message when a single item fails, so the rest of the archive is made available (search output for "**SKIP"). By contrast, zips (creates) throw an uncaught exception and stop when an item cannot be zipped, because this is a core data loss. On zip aborts, you must address the cause of the failure (e.g., permissions).

Compression By default, items in archives created by ziptools itself are compressed using the Python zipfile module's ZIP_DEFLATED setting. This isn't configurable (until 1.3: see the following update), but it's the usual method for zips, and generally does a reasonable job. Archives unzipped by ziptools support any compression method supported by the installed zipfile module, including ZIP_DEFLATED; see this module's

docs

for more details.

New uncompressed option:

In ziptools 1.3, the create (zip) script adds a -nocompress command-line flag; if used, this disables the standard compression normally applied to zipped items, and stores them in the zipfile uncompressed (technically, by using the ZIP_STORED format of zipfile and the zip standard). The create function call also gains a parallel nocompress argument. In ziptools extracts, uncompressed zipfiles work automatically and require no extra flags or arguments.

While not generally required for common and smaller archives, the new create uncompressed options radically speed both zips and unzips of very large archives, at a modest cost in extra zipfile size, and no loss of data or metadata. On macOS, for example:

With compression, a 208.4G archive of 178.9K items took 1.5 hours to zip and 20 minutes to unzip, producing a 195.8G zipfile.

With compression disabled using the new switch or argument, this same archive zipped and unzipped in just 22 minutes and 5 minutes, respectively, and yielded a 208.5G zipfile.

This translates to roughly 4X faster zips and unzips for archives like that tested—and may move the needle from impractical to practical. Copying large content collections to smartphones, for example, may fail unless zipped for transit.

File size

ziptools always uses

ZIP64

extensions when needed to support files much larger than zip's former limits, via this option in Python's zipfile

module

. While file size is practically unlimited with ZIP64, some other tools may fail or refuse to extract zipfiles larger than 2G (e.g., Unix unzip), and others may balk at 4G. If you run into limits, your best option may be to find or install a Python 2.X or 3.X on the unzip host, and run ziptools' zip-extract.py.

Performance

Short story: ziptools' zip and unzip performance is on a par with the Unix command-line zip and unzip programs. Per a recent stress test on macOS, zipping a 208G content collection of 163K files and 16K folders took 1:24 (hours:minutes) with ziptools' zip-create.py, using Python 3.8. Doing the same with the Unix zip -r command took 1:18 and yielded a zipfile 3G larger at 198G. Unzipping this archive on the same platform was faster, at 0:24 for ziptools' zip-extract.py, and 0:20 for the Unix unzip. None of these times qualify as zippy (pun intended), and may or may not be reduced by disabling compression (see above). On the other hand, zipfiles avoid data-loss perils of filesystems, platforms, and explorers; both ziptools' and Unix's results for this test were proven correct (per the bytewise diffall.py and timestamp mergeall.py in

Mergeall

); and runtimes are naturally much quicker for content of smaller and more reasonable dimensions.

Update: see also the new -nocompress option described

above

; where speed matters more than space, this can make ziptools zips and unzips shockingly faster (4X as tested, to be precise) than the foregoing implies.

See the create and extract

scripts

for more on pathnames and other details in ziptools. The following sections move on to present examples by usage mode.

Program-Library Mode

ziptools can be used from both command lines shown ahead, and direct Python program calls like those demonstrated here.

In program mode only, callers are responsible for expanding any wildcard operators like * in source filenames (a.k.a. globbing), before calling ziptools' create function; use Python's glob.glob. Program mode can also leverage other Python tools to construct source lists, as is responsible for source existence per

above

Cruft skipping is enabled in program mode by passing a dictionary of skip and keep patterns to the create function's cruftpatts. This argument defaults to {} which disables skipping. To enable, pass either a custom definition or the default imported from

zipcruft.py

As usual in Python for simple source-code zips, ziptools' install (unzip) folder must be on your module search path. An export command on Unix or similar set on Windows in your shell start-up files generally suffices, though other options abound; your platform's documentation and other

resources

can provide more details.

See

ziptools.py

for more on program usage and the arguments demonstrated here, as well as additional arguments omitted here for space:

# Setup (Z is ziptools' install folder) $ export PYTHONPATH=$Z:$PYTHONPATH # Unix $ set PYTHONPATH=%Z%;%PYTHONPATH% # Windows # Basic usage import ziptools ziptools.createzipfile(zipto, sources) ziptools.extractzipfile(zipfrom, unzipto) # Test folders ziptools.createzipfile('test-1-2.zip', ['test1', 'test2']) ziptools.extractzipfile('test-1-2.zip', '.') # Websites, with cruft skips from ziptools.zipcruft import cruft_skip_keep ziptools.createzipfile('website.zip', ['website'], cruftpatts=cruft_skip_keep) ziptools.extractzipfile('website.zip', '~/public_html', permissions=True) # Development ziptools.createzipfile('devtree.zip', ['dev'], trace=(lambda *p, **k: None)) ziptools.extractzipfile('devtree.zip', '.', permissions=True) # Symlink options ziptools.createzipfile('filledintree.zip', ['skeleton'], atlinks=True) ziptools.extractzipfile('nonportable_devtree.zip', '.', nofixlinks=True) # Manual globs from glob import glob ziptools.createzipfile('allsourcecode.zip', glob('*.py') + glob('*.c')) # Use 1.2 alternate path to minimize or eliminate folder nesting in zips ziptools.createzipfile('folder.zip', ['long/path/to/folder'], zipat='to') ziptools.createzipfile('pycode.zip' glob('some/other/folder/*.py'), zipat='.') # Use 1.3 compression disable to speed zip and unzip of large archives ziptools.createzipfile('folder.zip', ['folder'], cruftpatts=cruft_skip_keep, nocompress=true)

Command-Line Mode

The following sorts of ziptools command lines may be run from your system's command shell (e.g., Terminal on Unix, Command Prompt on Windows, or Termux on Android); another program (e.g., using Python's os.popen); or IDEs that support system command lines (e.g.,

PyEdit

In all cases and on all platforms, ziptools' create script expands filename wildcards (e.g., * and ?) not already expanded by a command shell, and . in creates and extracts refers to the current directory. See the prior section for more on

expansion

and

dots

For brevity and convenience, most of these examples use a Unix shell-variable reference $Z and assume that this variable has been set by an export or similar prior to script runs (e.g., $Z/zip-create.py). In Windows Command Prompt, a %Z% after running a set command may be used instead (e.g., %Z%\zip-create.py)

See

zip-create.py

and

zip-extract.py

for more on the command lines and switches demonstrated here:

# Setup (optional) $ export Z=ziptoolspath # Unix $ set Z=ziptoolspath # Windows # Test folders c:\...\ziptools> zip-create.py cmdtest\ziptest.zip selftest\test1 selftest\test2 c:\...\ziptools> zip-list.py cmdtest\ziptest.zip c:\...\ziptools> zip-extract.py cmdtest\ziptest.zip cmdtest\unzipped # Websites ...local$ python3 $Z/zip-create.py ~/website.zip . -skipcruft ...remote$ python2 $Z/zip-extract.py ~/website.zip public_html -permissions # Distributions ...devdir$ python3 $Z/zip-create.py program.zip programdir -skipcruft ...usedir$ python3 $Z/zip-extract.py program.zip . # Development ...dir1$ python $Z/zip-create.py devtree.zip dev -skipcruft > /dev/null ...dir2$ python $Z/zip-extract.py devtree.zip . -permissions # Special cases: populating from links, retaining link separators ...dir1$ python $Z/zip-create.py devtree.zip dev -skipcruft -atlinks ...dir2$ python $Z/zip-extract.py devtree.zip . -nofixlinks # Individual items ...here$ python $Z/zip-create.py allcode.zip a.py b.py c.py d.py ...here$ python $Z/zip-create.py allcode.zip a.py b.py folder -skipcruft # Shell pattern expansion: supported on all platforms in [1.1] ...here$ python $Z/zip-create.py allcode.zip *.py ...here$ python $Z/zip-create.py allcode.zip *.py test[12].txt doc?.html # Use items in a folder as top-level items, not nested in their folder --cd source dir ...src$ python $Z/zip-create.py ../allcode.zip * -skipcruft --cd dest dir, copy allcode.zip to . ...dst$ python $Z/zip-extract.py allcode.zip . -permissions # Use 1.2 alternate path to minimize or eliminate folder nesting in zips ...here$ python $Z/zip-create.py folder.zip long/path/to/folder -zip@to -skipcruft ...here$ python $Z/zip-create.py pycode.zip some/other/folder/*.py -zip@. # Use 1.3 compression disable to speed zip and unzip of large archives ...here$ python $Z/zip-create.py folder.zip folder -skipcruft -nocompress

Interactive Mode

When ziptools is run from a command line with no arguments, it falls back on asking for inputs interactively, as in the examples that follow. Inputs normally come from a user at the console, but may also be taken from a piped-in replies file or shell 'here' document (though command-line arguments work too; for background, try

this

). In these listings, substitute \ for all / when working on Windows. In this mode,

expansion

and

dots

work the same as they do in command lines.

For both creates and extracts, interactive mode also asks you to verify the run before starting it, and a control+c keypress combination at any console prompt terminates the session without starting a zip or unzip. For extracts, interactive mode (only) also asks if you wish to clean (i.e., delete) the contents of the target folder first, if the folder already exists. If you opt to not clean, the unzip will add to the folder's current content.

To extract an existing zipfile to ., the current directory:

.../test-symlinks$ $Z/zip-extract.py Running ziptools 1.3 Zip file to extract? save-test1-test2.zip Folder to extract in (use . for here) ? . Do not localize symlinks (y=yes)? Retain access permissions (y=yes)? Do not mangle filenames (y=yes)? About to UNZIP save-test1-test2.zip, to ., localizing any links, not retaining permissions, mangling filenames Confirm with 'y'? y Clean target folder first (yes=y)? n Unzipping from save-test1-test2.zip to . Extracted test1/ ...etc... To create a new zipfile in a folder, from items in two other folders (run in ziptools' own folder for variety):

/Code/ziptools$ zip-create.py Running ziptools 1.3 Zip file to create? cmdtest/ziptest Items to zip (comma separated)? selftest/test1, selftest/test2 Skip cruft items (y=yes)? y Follow links to targets (y=yes)? n Alternate zip path (.=unnested, enter=none) ? Disable item compression? (y=yes)? About to ZIP ['selftest/test1', 'selftest/test2'], to cmdtest/ziptest.zip, skipping cruft, not following links, zip@ path (unused), compressing items Confirm with 'y'? y Zipping ['selftest/test1', 'selftest/test2'] to cmdtest/ziptest.zip Cruft patterns: {'skip': ['.*', '[dD]esktop.ini', 'Thumbs.db', '~*', '$*', '*.py[co]'], 'keep': ['.htaccess']} Adding folder selftest/test1 --Skipped cruft file selftest/test1/.DS_Store ...etc... To extract the zipfile just created, to another folder:

/Code/ziptools$ py3 zip-extract.py Running ziptools 1.3 Zip file to extract? cmdtest/ziptest Folder to extract in (use . for here) ? cmdtest/target Do not localize symlinks (y=yes)? Retain access permissions (y=yes)? y Do not mangle filenames (y=yes)? y About to UNZIP cmdtest/ziptest.zip, to cmdtest/target, localizing any links, retaining permissions not mangling filenames Confirm with 'y'? y Clean target folder first (yes=y)? y Removing cmdtest/target/selftest Unzipping from cmdtest/ziptest.zip to cmdtest/target Extracted selftest/test1/ => cmdtest/target/selftest/test1 ...etc... To list the created zipfile's contents:

/Code/ziptools> zip-list.py Running ziptools 1.3 Zipfile to list? cmdtest/ziptest.zip File Name Modified Size selftest/test1/ 2016-10-02 09:01:58 0 selftest/test1/d1/ 2016-09-30 16:41:12 0 selftest/test1/d1/fa1.txt 2014-02-07 16:38:58 0 selftest/test1/d3/ 2016-10-02 09:05:02 0 selftest/test1/d3/.htaccess 2015-03-31 16:55:44 271 ...etc... To extract using absolute paths, on Unix and Windows:

/...$ py3 /Code/ziptools/zip-extract.py Running ziptools 1.3 Zip file to extract? /Users/blue/Desktop/website.zip Folder to extract in (use . for here) ? /Users/blue/Desktop/temp/website Do not localize symlinks (y=yes)? n Retain access permissions (y=yes)? n Do not mangle filenames (y=yes)? n About to UNZIP /Users/blue/Desktop/website.zip, to /Users/blue/Desktop/temp/website, localizing any links, not retaining permissions mangling filenames Confirm with 'y'? y ...etc... c:\...> py -3 C:\Code\ziptools\zip-extract.py Running ziptools 1.3 Zip file to extract? C:\Users\me\Desktop\website.zip Folder to extract in (use . for here) ? C:\Users\me\Desktop\temp\website Do not localize symlinks (y=yes)? n Retain access permissions (y=yes)? n Do not mangle filenames (y=yes)? y About to UNZIP C:\Users\me\Desktop\website.zip, to C:\Users\me\Desktop\temp\website, localizing any links, not retaining permissions not mangling filenames Confirm with 'y'? y ...etc...

Portability

This section provides the full story on running ziptools in different contexts. In short:

ziptools runs well on both Python 3.X and 2.X, and on both the Windows and Unix platforms.

ziptools' zipfiles can be unzipped by most other tools, and it can unzip most other tools' zipfiles.

The net result is that ziptools can serve as your go-to tool for managing zip archives on all your devices. As examples, check out ziptools running on

, and

(if you haven't by now).

That said, interoperability is rarely perfect, and the following sections enumerate the footnotes that apply. Some of these can be mitigated by choosing an appropriate Python (as source code, ziptools is especially vulnerable to version skew). Most, however, reflect immutable constraints of other tools, platforms, or Pythons.

Other Tools

ziptools is broadly interoperable with other zip tools: its zipfiles can generally be unzipped by other tools, and it can usually unzip zipfiles created by other tools. As an example, zips created by ziptools running on Python 2.X are correctly unzipped by ziptools under 3.X—and vice versa. Thanks to the zip standard, ziptools' has also been verified to play well with Unix command-line zip and unzip, Finder zips on macOS, explorer zips on Windows, and an assortment of other tools.

The chief caveat here is that some tools may not support ziptools' features as fully as ziptools. In particular, modtimes, symlinks and permissions may not propagate as well; large files may not be supported by some unzips, per

above

; and ziptools' UTC-timestamp modtimes may be ignored, yielding DST

skew

. For best results, use ziptools on both the zip and unzip ends of your archive transfers.

Platforms

As suggested by some of the examples earlier in this guide, ziptools works equally well on Unix and Windows. It's been recently verified on Windows 7 and 10, macOS El Capitan and later, Ubuntu Linux, and Android 7 through 11 (including Nougat and Pie). Nevertheless, a handful of well-known platform idiosyncrasies can impact zip results:

Unix ziptools works well on

Unix

—and, by extension,

Linux

—with no notable caveats. Many of ziptools' extensions deal with Unix-oriented tools (e.g., symlinks and permissions), whose support is naturally best on their home base. In fact, because the fit is so good, this guide's examples are all run on macOS Unix, unless noted otherwise. Still, Unix inherits a legacy Windows constraint: by spec, zip archives mimic the "local time" modtime scheme of MS-DOS and Windows FAT, instead of using Unix UTC time. This skews times across timezone changes, and can yield different time results from different unzip tools across DST changes. ziptools 1.1 deferred to Python libraries for DST changes and did not address timezones, but 1.2 now accommodates both by saving UTC timestamps to zipfile extra fields. See the earlier

coverage

of the 1.2 change, and search for "DST" in

ziptools.py

for more on this topic.

A footnote on Linux: although it can largely be lumped in with Unix in all ziptools regards, it does differ in one arguably trivial way: as tested, Linux silently fails to set symlink permissions, leaving them a fixed 0o777. This also happens when changed from a shell chmod, and means ziptools can't propagate symlink permissions (only) on this platform. Linux also doesn't recognize symlinks created on USB drives by macOS, but the latter may be "cheating" with custom formats. For more on the Linux ziptools story, see its Ubuntu demo

session

On Windows ziptools works well on

Windows

—and even has Windows-specific support for long paths and command-line argument globbing. Some utility is limited, however, and some tools require extra steps. Unix-style permissions propagation, for example, translates only to Windows read-only

flags

and is otherwise irrelevant. An item with permissions 0o444 (read-only) on Unix will unzip on Windows as

read-only

, and return to unzip on Unix as 0o444, but all other Unix permissions will be lost in transit. See

this demo

for a Windows round trip's before and after. Moreover, symlinks on Windows work only on NTFS drives, and even then unzipping—and hence creating—symlinks at all has historically required special Windows permissions. To enable this permission, right-click Command Prompt and select "Run as administrator," or enable Developer Mode in recent versions of Windows 10 (and 11: see the

update

). For more details, see section "Symlinks—Copied, not Followed" in Mergeall's User Guide

here

. In ziptools, symlinks survive zips from and unzips to NTFS drives, but they are lost elsewhere, and symlink permission failures on unzips generate a message and stub file but do not end the run (see the

demo

Some Unix-focused tools also have uneven support on Windows. Symlinks to files and folders work on Windows in ziptools, for instance, but under Python 3.X only, and even then do not propagate modtimes or permissions. Python 2.X doesn't support symlinks on Windows

at all

, so ziptools does what it can: on unzips, symlinks are replaced with

stub files

so that the rest of the archive can be extracted; on zips, symlinks are followed because Python 2.X's library does not detect them, resulting in unavoidable data copies. The only remedy for these is Python 3.X.

Windows 11 update

: per Mergeall's

coverage

, symlinks can be created on Windows 11 by just setting Developer mode in Settings => Privacy & Security; no special permission or "Run as administrator" mode is required. With both Windows Developer mode and this system's path-separator adjustments covered

earlier

, ziptools is generally able to recreate Unix symlinks on Windows NTFS drives. Moreover, because a zip is a single self-contained file, the filesystem used to transport it to or from Windows is irrelevant; exFAT drives, for example, will do as go-betweens, even though they don't support raw symlinks on Windows.

On Android

Android is essentially Unix (really, Linux) with proprietary—and even extreme—access constraints: ziptools works on this platform

too

(e.g., in the Termux and Pydroid 3 apps' command lines), but you must zip and unzip to and from folders accessible to your Python app only. Unlike in Unix and Windows, file storage in Android is a fenced-in resource on unrooted devices. In particular, file writes required for ziptools may be available only in app-specific folders, but the rules have varied across storage mediums and Android releases, are scheduled to change again soon, and are too complex to cover here; visit

this page

for more on which folders are open to zips. In terms of specific features, symlink creation normally fails on Android today with an "Operation not permitted" permissions error, due to this platform's emulated filesystem stack. This happens both in ziptools and otherwise; here's the error on unrooted

and

. See the searches

and

and the related note

for background on this topic. As on Windows, ziptools on Android replaces such failing symlinks with dummy stub files, so unzips can proceed to extract an archive's non-link items.

For similar filesystem-emulation reasons, Android also silently refuses to support permissions propagation in ziptools or otherwise on unrooted

Pie

, and throws errors when propagating modtimes and permissions prior to

Oreo

. Per the same

note

, the pre-Oreo modtime constraint has been lifted in recent Androids by replacing a FUSE filesystem implementation, but undoubtedly still impacts many users of older devices. In ziptools, errors don't stop unzips on Android, but permissions won't propagate, and modtimes won't propagate on 2016's Nougat and earlier. Study the

Nougat

and

Pie

demo sessions to learn more.

Android 11 update 1

: a new demo was added to the docetc folder in version 1.3, to show the results of ziptools extracts (unzips) on Android 11. That platform now has three internal storage categories that differ in speed, accessibility, and longevity, and the demo tests all three on a Pixel 4a. In short, content and modtimes work well in all three storage types, but permissions and symlinks are supported only in app-private storage, which is not accessible to any but the sole owning app (apart the non-POSIX access Termux provides with the Storage Access Framework, as covered

here

by Android Deltas Sync). For the complete Android 11 ziptools story, see the

demo

, especially its overview and summary

file

Android 11 update 2: on further testing, it turns out that Android 11's filesystem behavior varies by both storage type and device vendor. Specifically, a Galaxy Z Fold3 has nearly the same rules as a Pixel 4a, but the former supports Unix permissions in app-specific storage, while the latter does not. In other words, there is no single Android 11: its behavior can vary arbitrarily from device to device. The Android 11

demo

has been updated for the new findings, but Android 11's fragmentation obviously makes it difficult to document, program, and use. Quality engineering will rarely thrive where it must build its house upon the sand.

Android 11 update 3: in late 2021, Mergeall

3.3

uncovered a likely bug in Android 11 shared storage which may preclude writing files with non-ASCII Unicode names in composed (e.g., NFC) format. Though rare, if unzips of such files fail on this platform, see the online work-around

script

, as well its Android Deltas Sync

coverage

Android 12 and 13 update

: in late 2022, ziptools has been used regularly and successfully on Androids 12 and 12L, though 12 introduced a new child-process

constraint

that may impact some users. Android 13 remains to be tested, but is expected to change little at the systems level where ziptools runs; it may further lock down app-specific storage, but the impact of this remains to

be seen

Other factors not called out in the foregoing list may also place hurdles in the way of cross-platform content. On unzips, for instance, illegal (non-portable) characters in filenames may optionally be mangled to _ on Windows to allow saves, even though this may break later syncs back to the source platform and overwrite files (see the new 1.3 policy and options for this

above

). Moreover, pathnames botched by some Windows zip programs may cause problems on Unix, as described

earlier

. In addition, unzipping to drives using some filesystems, including exFAT, may strip archives of their Unix

permissions

. See the earlier

notes

for more details on the latter, and always test ziptools in your use case before adopting it broadly.

Lest this all sound too dire, keep in mind that archives which stick to basic files and folders are immune to most of the interoperability concerns noted here. Where that's impossible, archiving is usually better when Unix-oriented tools like symlinks and permissions stay on Unix (and the nod to Unix compatibility on Windows and Android is recognized as half-hearted at best).

What about iOS?:

ziptools probably can also be used on iOS devices from a Python app (e.g., Pythonista, with either its Python 3.X or 2.X), but this hasn't been tested; is subject to iOS's extreme access constraints; and may require either program-call mode, or a shim wrapper to spawn a ziptools command line. iOS storage has also historically been less open on unrooted phones than Android; iOS 13 adds some filesystem functionality in its

Files app

(including macOS-like quickviews and support for zips), but the scope of content is still limited. That said, Android seems bent on imposing similar

restrictions

, so your mobile mileage may vary.

Pythons

Despite the prior section's critiques, platforms are not the only source of portability limits for ziptools. Because Python's support for some tools can vary by its version number even on the same platform, this section summarizes portability rules from the perspective of Python itself. Being coded in Python and shipped as source, ziptools naturally inherits these same constraints.

Broadly speaking, ziptools' most recent release has been verified to work on Pythons 2.7 through 3.9 and the zipfile modules these Pythons include; later Pythons are expected to run ziptools correctly too unless Python or its zipfile change incompatibly (see also this section's update ahead). ziptools also works almost identically on Python 3.X and 2.X, and the two Pythons perform equally well in almost all regards—including support for non-ASCII

filenames

in 2.X as of 1.1.

Still, Python 3.X holds a slight advantage for archives containing symlinks on Unix, and Python 3.X is fully required to retain symlinks on Windows. In complete detail, Python's current symlink support across platforms is both convoluted and mixed:

On Unix On Unix (and its kind), Python 3.X can read and write symlinks, and can propagate their permissions and modtimes. Python 2.X can read and write symlinks, and can propagate their permissions but not their modtimes. On Windows As noted earlier, Python 3.X can read and write symlinks on Windows, but is unable to propagate either their modtimes or their permissions. Python 2.X cannot

read

write

, or even

detect

symlinks on Windows, and hence does not support them there at all. Additionally, only Python 3.2 and later can detect folder-link cycles on Windows, though very few Python 3.0 or 3.1 installs likely see action today. On Android Per the prior section, platform prohibitions derail symlinks in Python 3.X; though untested, the same fate likely applies to 2.X. In sum, if you do not zip symlinks, Python 2.X is as good a choice as 3.X; if you do, 3.X is a better option everywhere. As noted in the prior section, it's also generally advised to maintain your symlinks on Unix, though Python 3.X makes them as portable as possible if they cross platform boundaries. By contrast, permissions may be lost altogether if round tripped between some platforms—in any Python.

It's also worth adding that Python 3.X may hold a speed advantage over 2.X. Although the same ziptools code runs unchanged on 3.X and 2.X, the underlying Python zipfile module differs between the two lines. In testing with 3.9 and 2.7, content results are the same (sans the functional differences noted above), but speed can differ: 2.X and 3.X perform roughly the same, except that 2.X extracts (unzips) can check in three times slower than 3.X. The cause of this remains unisolated, but Python 3.X may also be the better host if speed is a premium.

Despite all the platform and Python limitations and tradeoffs outlined in the last two sections, it's important to note zipfiles are still both practical and widely useful, and serve as the basis for countless content archives and media packages. In computing, the line between unusable and workable is always up to practice to define.

Python failsafe copy: as of 1.3, ziptools now ships a working copy of Python 3.X's zipfile module, as a safeguard against future changes. Find it in the docetc extras folder

here

, and copy it up to the root of the install package if zipfile ever changes in ways that break ziptools. ziptools does not need or use this file today, and using it would preclude adapting any future improvements in it, but it's available both for reference and as a last-resort measure. The copy is from 3.9 and won't work with Python 2.X, though 2.X's recent "

end of life

" ensures that it won't be changed and break in ways that are all too common in 3.X. Retirement has its upsides.