Wiki

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.
====== BASH - Globs ======

**Glob** or **Globstar** is the common name for a set of Bash features that match or expand specific types of patterns.

Some synonyms for globbing (depending on the context in which it appears) are pattern matching, pattern expansion, filename expansion, and so on.

A glob may look like ***.txt** and, when used to match filenames, is sometimes called a "wildcard".

Traditional shell globs use a very simple syntax, which is less expressive than a Regular Expression.

Most characters in a glob are treated literally, but a * matches 0 or more characters, a ? matches precisely one character, and [...] matches any single character in a specified set (see Ranges below).

All globs are implicitly anchored at both start and end. 

----

===== Summary =====

|*|Matches any string, of any length|
|foo*|Matches any string beginning with foo|
|*x*|Matches any string containing an x (beginning, middle or end)|
|*.tar.gz|Matches any string ending with .tar.gz|
|*.[ch]|Matches any string ending with .c or .h|
|foo?|Matches foot or foo$ but not fools|

----

===== Expansion of Glob - Filenames =====

Bash expands globs which appear unquoted in commands, by matching filenames relative to the current directory.

The expansion of the glob results in 1 or more words (0 or more, if certain options are set), and those words (filenames) are used in the command.

<code bash>
tar xvf *.tar
# Expands to: tar xvf file1.tar file2.tar file42.tar ...
</code>

----

===== Expansion of Glob - Filename with Whitespace =====

Even if a file contains internal whitespace, the expansion of a glob that matches that file will still preserve each filename as a single word.

For example,

<code bash>
# This is safe even if a filename contains whitespace:
for f in *.tar; do
  tar tvf "$f"
done

# But this one is not:
for f in $(ls | grep '\.tar$'); do
  tar tvf "$f"
done
</code>

<WRAP info>
**NOTE:**  In the second example above, the output of **ls** is filtered, and then the result of the whole pipeline is divided into words, to serve as iterative values for the loop.

This word-splitting will occur at internal whitespace within each filename, which makes it useless in the general case.

The first example has no such problem, because the filenames produced by the glob do not undergo any further word-splitting.

For more such examples, see BashPitfalls.

</WRAP>

----

===== Pattern Matching =====

Globs are also used to match patterns in a few places in Bash.

The most traditional is in the case command:

<code bash>
case "$input" in
  [Yy]|'') confirm=1;;
  [Nn]*) confirm=0;;
  *) echo "I don't understand.  Please try again.";;
esac
</code>

<WRAP info>
**NOTE:**  Patterns (which are separated by | characters) are matched against the first word after the case itself.

The first pattern which matches, "wins", causing the corresponding commands to be executed.

</WRAP>

----

===== Comparison Globs =====

Bash also allows globs to appear on the right-hand side of a comparison inside a <nowiki>[[</nowiki> command:

<code bash>
if [[ $output = *[Ee]rror* ]]; then ...
</code>

----

===== Pattern Stripping =====

Globs are used during parameter expansion to indicate patterns which may be stripped out, or replaced, during a substitution.

<code bash>
filename=${path##*/}    # strip leading pattern that matches */ (be greedy)
dirname=${path%/*}      # strip trailing pattern matching /* (non-greedy)

printf '%s\n' "${arr[@]}"          # dump an array, one element per line
printf '%s\n' "${arr[@]/error*/}"  # dump array, removing error* if matched
</code>

----

===== Ranges =====

Globs can specify a range or class of characters, using square brackets.

This gives you the ability to match against a set of characters.

For example:

|[abcd]|Matches a or b or c or d|
|[a-d]|The same as above, if globasciiranges is set or your locale is C or POSIX. Otherwise, implementation-defined.|
|[!aeiouAEIOU]|Matches any character except a, e, i, o, u and their uppercase counterparts|
|<nowiki>[[:alnum:]]</nowiki>|Matches any alphanumeric character in the current locale (letter or number)|
|<nowiki>[[:space:]]</nowiki>|Matches any whitespace character|
|<nowiki>[![:space:]]</nowiki>|Matches any character that is not whitespace|
|<nowiki>[[:digit:]_.]</nowiki>|Matches any digit, or _ or .|

<WRAP info>
**NOTE:**  In most shell implementations, one may also use ^ as the range negation character, e.g. [^[:space:]].

However, POSIX specifies ! for this role, and therefore ! is the standard choice.

Recent Bash versions Interpret [a-d] as [abcd].  To match a literal **-**, include it as first or last character.

</WRAP>

----

===== Options which change globbing behavior =====

==== Extended Globs - extglob ====

In addition to the traditional globs (supported by all Bourne-family shells) that we've seen so far, Bash (and Korn Shell) offers extended globs, which have the expressive power of regular expressions.

Korn shell enables these by default; in Bash, you must run the command in your shell (or at the start of your script -- see note on parsing below) to use them.

<code bash>
shopt -s extglob
</code>

The pattern matching reference describes the syntax, which is reproduced here:

<code bash>
?(pattern-list)
    Matches zero or one occurrence of the given patterns. 
*(pattern-list)
    Matches zero or more occurrences of the given patterns. 
+(pattern-list)
    Matches one or more occurrences of the given patterns. 
@(pattern-list)
    Matches one of the given patterns. 
!(pattern-list)
    Matches anything except one of the given patterns. 
</code>

<WRAP info>
**NOTE:**  Patterns in a list are separated by | characters.
</WRAP>


----

==== Example of using Extended Globs ====

Extended globs allow you to solve a number of problems which otherwise require a rather surprising amount of ugly hacking; for example,

<code bash>
# To remove all the files except ones matching *.jpg:
rm !(*.jpg)

# All except *.jpg and *.gif and *.png:
rm !(*.jpg|*.gif|*.png)
</code>

or

<code bash>
# To copy all the MP3 songs except one to your device.
cp !(04*).mp3 /mnt
</code>

----

==== Extended Globs with Parameter Expansion ====

To use an extglob in a parameter expansion (this can also be done in one BASH statement with read):

<code bash>
# To trim leading and trailing whitespace from a variable.
x=${x##+([[:space:]])}; x=${x%%+([[:space:]])}
</code>

----

==== Nested Extended Glob Patterns ====

Extended glob patterns can be nested, too.

<code bash>
[[ $fruit = @(ba*(na)|a+(p)le) ]] && echo "Nice fruit"
</code>



<WRAP info>
**NOTE:**  extglob changes the way certain characters are parsed.

It is necessary to have a newline (not just a semicolon) between **shopt -s extglob** and any subsequent commands to use it.

You cannot enable extended globs inside a group command that uses them, because the entire block is parsed before the **shopt** is evaluated.

The typical function body is a group command.  An unpleasant workaround could be to use a subshell command list as the function body.
</WRAP>

Therefore, if you use this option in a script, it is best put right under the shebang line.

<code bash>
#!/usr/bin/env bash
shopt -s extglob   # and others, such as nullglob dotglob.
</code>

If your code must be sourced and needs extglob, ensure it preserves the original setting from your shell:

<code bash>
# Remember whether extglob was originally set, so we know whether to unset it.
shopt -q extglob; extglob_set=$?
# Set extglob if it wasn't originally set.
((extglob_set)) && shopt -s extglob
# Note, 0 (true) from shopt -q is "false" in a math context.

# The basic concept behind the following is to delay parsing of the globs until evaluation.
# This matters at group commands, such as functions in { } blocks.

declare -a s='( !(x) )'
echo "${s[@]}"

echo "${InvalidVar:-!(x)}"

eval 'echo !(x)'  # using eval if no other option.

# Unset extglob if it wasn't originally set.
((extglob_set)) && shopt -u extglob

This should also apply for other shell options.
</code>

----

===== Null Glob - nullglob =====

nullglob expands non-matching globs to zero arguments, rather than to themselves.

<code bash>
$ ls *.c
ls: cannot access *.c: No such file or directory

# With nullglob set.
shopt -s nullglob
ls *.c
# Runs "ls" with no arguments, and lists EVERYTHING
</code>

Typically, nullglob is used to count the number of files matching a pattern:

<code bash>
shopt -s nullglob
files=(*)
echo "There are ${#files[@]} files in this directory."
</code>

Without nullglob, the glob would expand to a literal * in an empty directory, resulting in an erroneous count of 1.

----

==== Null Blob BUG ====

<WRAP important>
**WARNING:**  Enabling nullglob on a wide scope can trigger bugs caused by bad programming practices.

It "breaks" the expectations of many utilities.

Removing array elements:

<code bash>
shopt -s nullglob
unset array[1]
#unsets nothing

unset -v "array[1]"
#correct
</code>

Array member assignments in compound form using subscripts:

<code bash>
shopt -s nullglob
array=([1]=*)
# Results in an empty array.
</code>

This was reported as a [[http://lists.gnu.org/archive/html/bug-bash/2012-08/msg00032.html|bug]] in 2012, yet is unchanged to this day.
</WRAP>

Apart from few builtins that use modified parsing under special conditions (e.g. declare) always use Quotes when arguments to simple commands could be interpreted as globs.

Enabling failglob, nullglob, or both during development and testing can help catch mistakes early.

To prevent pathname expansion occurring in unintended places, you can set **failglob**.  However, you must then guarantee all intended globs match at least one file.  Also note that the result of a glob expansion does not always differ from the glob itself.  failglob won't distinguish echo ? from echo '?' in a directory containing only a file named ?.  nullglob will.


----

==== Null Glob Portability ====

"null globbing" is not specified by POSIX.

In portable scripts, you must explicitly check that a glob match was successful by checking that the files actually exist.

<code bash>
# POSIX

for x in *; do
  [ -e "$x" ] || break
  ...
done

f() {
  [ -e "$1" ] || return 1

  for x do
    ...
  done
}

f * || echo "No files found"
</code>

Some modern POSIX-compatible shells allow null globbing as an extension.

<code bash>
# Bash
shopt -s nullglob
</code>

In ksh93, there is no toggle-able option.  Rather, that the "nullglob" behavior is to be enabled is specified inline using the "N" option to the ∼() sub-pattern syntax.

<code bash>
# ksh93

for x in ~(N)*; do
  ...
done
</code>

In zsh, an toggle-able option(NULL_GLOB) or a glob qualifier(N) can be used.

<code bash>
# zsh
for x in *(N); do ...; done # or setopt NULL_GLOB
</code>

mksh doesn't yet support nullglob (maintainer says he'll think about it).

----

===== Dot Glob - dotglob =====

By convention, a filename beginning with a dot is "hidden", and not shown by ls.

Globbing uses the same convention -- filenames beginning with a dot are not matched by a glob, unless the glob also begins with a dot.

Bash has a dotglob option that lets globs match "dot files":

<code bash>
shopt -s dotglob nullglob
files=(*)
echo "There are ${#files[@]} files here, including dot files and subdirs"
</code>

It should be noted that when dotglob is enabled, * will match files like .bashrc but not the . or .. directories.

This is orthogonal to the problem of matching "just the dot files" -- a glob of .* will match . and .., typically causing problems.

----

===== Glob Star - globstar =====

(since bash 4.0-alpha)

globstar recursively repeats a pattern containing **<nowiki>**</nowiki>**.

<code bash>
shopt -s globstar; tree
.
├── directory2
│   ├── directory3
│   ├── file1.c
│   └── file2
├── file1
└── file2.c
</code>

Suppose that for the following examples.

Matching files:

<code bash>
$ files=(**)
# equivalent to: files=(* */* */*/*)
# finds all files recursively

$ files=(**/*.c)
# equivalent to: files=(*.c */*.c */*/*.c)
# finds all *.c files recursively
# corresponds to: find -name "*.c"
# Caveat: **.c will not work, as it expands to *.c/*.c/…
</code>

<WRAP info>
**NOTE:**  To disable globstar use

<code bash>
shopt -u globstar
</code>

See:  **help shopt** for details.
</WRAP>


----

Assume you have a folder structure:

<code>
.
├── bar
│   ├── foo
│   │   └── baz
│   │       └── hurz
│   │           └── lolz
│   │               └── hello.txt
│   └── poit.txt
└── fnord.txt
</code>

Then **ls** with single star **<nowiki>*</nowiki>** would list:

<code bash>
ls *.txt
fnord.txt
</code>

The double star operator **<nowiki>**</nowiki>** will work on the subfolders.  The output will look like:

<code bash>
ls **/*.txt
bar/foo/baz/hurz/lolz/hello.txt  bar/poit.txt  fnord.txt
</code>

----

<WRAP info>
**NOTE:**  Just like **<nowiki>*</nowiki>**, **<nowiki>**</nowiki>** followed by a **<nowiki>/</nowiki>** will only match directories:
</WRAP>


<code bash>
files=(**/)
# Finds all subdirectories.

files=(. **/)
# Finds all subdirectories, including the current directory.
# Corresponds to: find -type d.
</code>

----

===== Fail Glob - failglob =====

If a pattern fails to match, bash reports an expansion error.

This can be useful at the commandline:

<code bash>
# Good at the command line!
$ > *.foo # creates file '*.foo' if glob fails to match
$ shopt -s failglob
$ > *.foo # doesn't get executed
-bash: no match: *.foo
</code>

----

===== GLOBIGNORE =====

The Bash variable (not shopt) GLOBIGNORE allows you to specify patterns a glob should not match.

This lets you work around the infamous "I want to match all of my dot files, but not . or .." problem:

<code bash>
$ echo .*
. .. .bash_history .bash_logout .bashrc .inputrc .vimrc
$ GLOBIGNORE=.:..
$ echo .*
.bash_history .bash_logout .bashrc .inputrc .vimrc
</code>

----

==== Unset GLOBIGNORE ====

<code bash>
$ GLOBIGNORE=
$ echo .*
. .. .bash_history .bash_logout .bashrc .inputrc .vimrc
</code>


----

===== No Case Match - nocasematch =====

Globs inside <nowiki>[[</nowiki> and case commands are matched case-insensitive:

<code bash>
foo() {
  local f r=0 nc=0
  shopt -q nocasematch && nc=1 || shopt -s nocasematch
  for f; do
    [[ $f = *.@(txt|jpg) ]] || continue
    cmd -on "$f" || r=1
  done
  ((nc)) || shopt -u nocasematch
  return $r
}
</code>

This is conventionally done this way:

<code bash>
case $f in
  *.[Tt][Xx][Tt]|*.[Jj][Pp][Gg]) : ;;
  *) continue
esac
</code>

and in earlier versions of bash we'd use a similar glob:

<code bash>
[[ $f = *.@([Tt][Xx][Tt]|[Jj][Pp][Gg]) ]] || continue
</code>

or with no extglob:

<code bash>
[[ $f = *.[Tt][Xx][Tt] ]] || [[ $f = *.[Jj][Pp][Gg] ]] || continue
</code>

Here, one might keep the tests separate for maintenance; they can be easily reused and dropped, without having to concern oneself with where they fit in relation to an internal ||. 

Note also:

<code bash>
[[ $f = *.@([Tt][Xx][Tt]|[Jj][Pp]?([Ee])[Gg]) ]]
</code>

Variants left as an exercise.

----

===== No Case Glob - nocaseglob =====

(since bash 2.02-alpha1)

This option makes pathname expansion case-insensitive.

In contrast, nocasematch operates on matches in <nowiki>[[</nowiki> and case commands.