XXCOPY
[ Back to Table of Contents ] [ << ] [ >> ]

XXCOPY TECHNICAL BULLETIN #05


From:    Kan Yabumoto           tech@xxcopy.com
To:      XXCOPY user
Subject: The Exclusion specifier in XXCOPY
Date:    2010-12-07  (revised)
===============================================================================


Much of the mostly hidden power of XXCOPY lies in the exclusion
mechanism.  Starting with v.2.97.7 (2009-01-13), the exclusion
specifier is fully compatible with the "Wild-Wildcard" feature.
That is, you may use wildcard characters (* and ?) anywhere in
the exclusion specifier in any number of times.  (Earlier 
versions of XXCOPY had complicated rules on the wildcard usage).


XXCOPY Exclusion switch syntax

  /X<xspec>       excludes the file or directory item given by
                  <xspec> which is an exclusion specifier.
                  If the specifier contains an embedded space,
                  the specifier must be surrounded by a pair
                  of double-quotes (").

  /EX<xfile>      specifies a text file whose name is <xfile>
                  which contains a list of <xspec> separated by space.



  You may specify as many exclusion specifiers as you like.


Some examples of the /X switch

  The exclusion parameter must be specified immeidately after /X
  (without a space).

    /Xc:\mydir\myfile.txt // specifies just a single file
    /X*.tmp               // all files that end with ".tmp"
    /Xabc*                // all files that start with "abc"

  To improve readability, we suggest inserting a colon after X.
  Here are more examples:

    /X:mydir\             // the entire directory, "mydir" in the source
    /X:mydir\*\*          // same as /Xmydir\ which is a shortcut
    /X:mydir\*\*.tmp      // inside mydir, all files matching "*.tmp"
    /X:my*xyz\*\abc*.c    // in directories that match my*xyz, all "abc*.c"
    /X:*\cache\           // multiple-level subdirectories
    /X:*\cache\*\*        // same as above with a trailing backslash
    /X:*\cach?\*\*        // multiple-level subdir spec may have wildcards
    /X:"\Program Files\"  // use quotes (") with a pattern with a space
    /X:C:\windows\        // it's OK. Two colons with a full path name

  Here, you may see the glimpse of the powerful syntax in the exclusion
  specifier.  The first example seems the most straight forward.  The
  fourth example which ends with a backslash is a shorthand of for the
  common case of excluding a directory (it abbreviates "*\*" which follows).
  Therefore, all of the above examples except the first one contain
  or imply at least one wildcard specifier.  The last example includes
  one asterisks in each of the three parts.

  Don't worry about the complexity yet.  At least the first example shows
  a case which you can use it immediately without any further reading.
  Yes, if you have energy to list all of the files you want to exclude,
  you may painstakingly list all of such files by giving the full
  file specification of each file.  Since you will soon run out of the
  command line space, you will probably want to set up a text file using
  the /EX switch.


  E.g.,  /EXmyexcl.lst

   and myexcl.lst  contains the following specifiers:

     // this is a comment line
     c:\win386.swp               // comment may start like this
     c:\autoexec.bat  myfile.tmp // one line may have multiple items
     "c:\program files"          // use quotes (") for embedded space
     mydir\myfile.txt            // pathspec relative to the source dir
     yourdir\                    // entire yourdir\*\*


Characteristics of exclusion specifier (a short explanation):

    1.  The Wild-Wildcard feature fully supported for exclusion.
    2.  A pattern for direcotires always ends with a backslash.
    3.  A pattern for files has no trailing backslash.
    4.  A relative specifier is relative to the source base directory.


Syntactic rules of exclusion specifier:

    1.  Add the exclusion item immediately (without space) after /X.

         /Xmyfile.txt          // exclude "myfile.txt" anywhere

    2.  You may insert an optional colon (:) between /X and the rest.

         /X:xyz.doc            // the colon improves the readability

    3.  The trailing backslash distinguishes directory from file.

         /X:Windows\          // specifies the "Windows" directory
         /X:pagefile.sys      // specifies the "pagefile.sys" file

    4.  Use a pair of quotes (") for an exclusion item with embeded space.

         /X:"\Program Files\"  // the space would split the string

    5.  Absolute exclusion specifiers start with a backslash
        (with an optional drive letter or UNC pattern).

         /X:c:\mydir\myfile.txt     // an absolute pattern
         /X:\mydir\myrecord.doc     // you may omit the drive letter
         /X:\\john\c\windows\       // UNC always starts with \\

    6.  Relative exclusion specifiers (without leading backslash)
        are relative to the source base specifier.

         /X:dir_abc\myfile.txt      // relative to the source base dir

    7.  A bare exclusion item (without any backslash) is treated
        as a file name pattern that applies to all directories.

         /X:desktop.ini             // equivalent to /X:*\desktop.ini
         /X:*.mp3                   // equivalent to /X:*\*.mp3

        This feature allows for a filename pattern to apply for all
        directories.  On the other hand, you may want to specify a
        file pattern that explicitly apply only to the files in the
        source base directory.  In such a case, use a ".\" prefix.

         /X:.\*.tmp                 // for files in the top directory

    8.  A bare directory pattern (the only backslash is at the end)
        is treated a pattern that applies only to the source base
        directory, unlike the bare file name case exlained above.

         /X:dir_xyz\                // applies only the top directory
         /X:.\dir_xyz\              // the same as the one above


The Wild-Wildcard feature in exclusion specifier:

    Starting with v.2.97.7 (2009-01-01), you may use wildcard character
    anywhere in an exclusion specifier for any number of times.

       Astrisk (*) denotes a string of zero or more of any character.

       Question Mark (?) denotes exactly one character.


         /X:*.mp3      // any file that ends with ".mp3"
         /X:abc*\      // any directory that starts with "abc"
         /X:xyz???     // starts with xyx followed with 3 more chars
         /X:*a*        // a filename that has at least one letter "a"


    The Special "wild subdirectory"  (..\*\..) notation:

        The \*\ (an asterisk sandwiched by backslashes) sequence
        denotes a pathname pattern with zero or more of directories.

         /X:c:\mydir\*\abc\    excludes the following directories

            c:\mydir\abc\
            c:\mydir\xyz\abc\
            c:\mydir\klm\xyz\abc\

        Such a pattern does not always match the first one.

         /X:c:\mydir\*\a\b\    excludes the following directories

            c:\mydir\a\b\           // the first instance
            c:\mydir\a\xyz\a\b\     // 1st \a\ does not but 2nd one does


 Syntax rule for the Exclusion List File (/EX).

    An "Exclusion List File" specified in the /EX switch is a plain
    text file which contains a list of exclusion specifiers.
    You may list as many exclusion specifiers in one line.
    Exclusion specifiers are separated by one or more blank, tab,
    and/or newline character.  An exclusion specifier cannot be
    broken into two or more lines.  When a space character is
    embedded, the exclusion specifier must be surrounded by a
    pair of double-quotes (").  A line may contain a comment field
    which will be ignored by XXCOPY.  A comment field starts with
    two consecutive slashes (//) or two consecutive colons (::)
    and ends at the end of the line.  You may also use another
    comment style (starts with /* and ends with */ that can span
    over multiple lines).



A few examples of common exclusion specifiers.

    The directory template may have the following four variations
    in the ending.

      dir_pattern\         // full directory
      dir_pattern\*\*      // full directory (same as above)
      dir_pattern\*        // files in the directory (one-level)
      dir_pattern\?*\      // all subdirectories but not files

      Even though the second example is to specify files (as opposed
      to directories) without the mandatory trailing backslash,
      it behaves exactly like the first one because it covers
      all files in all subdirectories which is equivalent to the
      whole directory contents that is the directory itself.


    Examples:

        /X:dirtmpl\*\*     // excludes all matching directories in the
                           // base (src) directory and its contents
        /X:dirtmpl\        // same as above (the triling backslash
                           // denotes everything inside the directory)
        /X:.\dirtmpl\      // in the case of the directory template,
                           // it applies to one directory relative to
                           // the base (src) directory (1-Level)
        /x:*\dirtmp\       // you may make a directory template apply
                           // to many instances (Multi-Level)


        /x:c:\windows\*    // specifies all the files in the first
                           // level of the c:\Windows directory such
                           // as, EXPLORER.EXE, WIN.INI,

        /x:c:\windows\?*\  // this does not includes the first level
                           // files but all subdirectories in it such
                           // as \WINDOWS\SYSTEM\  \WINDOWS\DESKTOP\, etc.


     Note that the following two are distinct:

         /Xdir_spec\*     // one layer only (subdirectories not excluded)
         /Xdir_spec\*\*   // the entire dir_spec directory is excluded


     Beware of a few pitfalls:

         xxcopy mydir\  x:\mydst\  /S /X:mydir\a*\

            The exclusion item specifies a pattern (directory starting with
            letter a) as a relative path.  As you explained above, it is
            relative to the source base directory.  Therefore, this command
            excludes directories that matches  mydir\mydir\a*\  not
            the subdirectories immediately in the mydir\.  Rather, the
            exclusion specifier stipulate mydir inside the source base
            directory which happens to spell the same.

         xxcopy \src\  \dst\      /S /X:*.tmp

            The exclusion specifier is equivalent to /X:*\*.tmp that applies
            to all subdirectories.

         xxcopy "C:\Program Files\"  \dst\   /S /X:Microsoft*\

            The exclusion specifier deos not implicitly applies to all
            subdirectories as the filename template does.  In this case, only
            directories whose name starts with Microsoft immediately inside
            the C:\Program Files\ directory, but not at other levels.


The optimization of exclusion matching.

    In a very large scale backup operation, an XXCOPY job may encompass
    an entire volume as the source directory (such as c:\*).  To make
    the matters worse, the more files the source directory contains,
    the more the need for the exclusion specifiers grows.  Therefore,
    it is entirely possible that the entire C: drive may contain
    70,000 files and the total number of exclusion items the user specify
    in the exclusion list file with the /EX switch may contain literally
    hundreds of various exclusion specifiers.  If we were to test every
    file against this very large number of exclusion list, the combination
    will easily reach tens of millions which would slow down the entire
    backup process.  Therefore, XXCOPY performs preprocessing steps
    to analyze the set of exclusion specifiers.  First by classifying
    them into the five classes, some redundant exclusion specifiers can
    be removed.


Debug feature

    Because a liberal use of wildcard characters in exclusion feature
    could become fairly complex, sometimes the user may want to test
    if a given set of exclusion items really achieve the intended
    result.

    The /oX switch generates a list of exclusion specifiers after
    XXCOPY's initial exclusion parameter optimization step.  The list
    is organized by the internal classification of the wildcard usage.
    It is most effective in combination with the /DEBUG function.
    (This output is mostly for Pixelab's support activities.)

    The /ZLX function provides a special test by listing all the files
    and directories that would be excluded without actually making any
    copy or delete operation.  It suppresses the display of regular
    (selected) files and generates a list of files and directories that
    would be excluded by the /X and /EX switches.  It is analogous to
    the /L function that lists the files that would be selected by
    a given set of command arguments.  


Automatically excluded files.

    Since the few output files (e.g., the error log files) which are
    generated by the XXCOPY program itself could not be successfully
    included in the current copying job if any of them happens to be
    in the source directory (or its subdirectories), those files will
    be automatically excluded.


Note on earlier (pre v.2.97.7) versions:

    In earlier versions of XXCOPY (prior to v.2.97.7), the exclusion
    feature was the hardest one to fully master by XXCOPY's users.
    That was because the exclusion specifier in earlier versions
    did not support the "Wild-Wildcard" capablitity to the fullest
    extent.  Rather, XXCOPY provided a total of 11 classes of wildcard
    usages with a complex set of syntactic rules.  So, the many users
    had to stick with a few simplest ways to add wildcard characters
    for the exclusion items.  It was simply very difficult to implement
    an efficent algorithm to support such a feature.

    In the old (pre-v.2.97.7) versions, the exclusion feature defined
    a special syntactic rule as follows:

          dir_pattern\?\*

    When an exclusion item with the \?\* ending, it excludes the
    subdirectories of the directories that matches the pattern.  The
    above exclusion pattern can be written as

         dir_pattern\?*\

    which does not rely on the special syntactic rule (which is now
    being abolished).


Use of the /GX switch:

    In order to support the old syntax for the \?\* ending in the
    exclusion specifier, we have added the /GX switch with which
    a command line with the older exclusion specifier is treated
    in the old way.

    However, we urge all users to replace all usages of \?\* ending
    with \?*\ that excludes exactly the same group of subdirectories.



© Copyright 2016 Pixelab All rights reserved.

[ XXCOPY Home ] [ Table of Contents ] [ << ] [ >> ]

Join the XXCOPY group