<h2>Edit File: pygettext3.py</h2><form method="post"><textarea name="content" rows="20" cols="80">#! /usr/bin/python3.9
# -*- coding: iso-8859-1 -*-
# Originally written by Barry Warsaw &lt;barry@python.org&gt;
#
# Minimally patched to make it even more xgettext compatible
# by Peter Funk &lt;pf@artcom-gmbh.de&gt;
#
# 2002-11-22 J�rgen Hermann &lt;jh@web.de&gt;
# Added checks that _() only contains string literals, and
# command line args are resolved to module lists, i.e. you
# can now pass a filename, a module or package name, or a
# directory (including globbing chars, important for Win32).
# Made docstring fit in 80 chars wide displays using pydoc.
#

# for selftesting
try:
    import fintl
    _ = fintl.gettext
except ImportError:
    _ = lambda s: s

__doc__ = _(&quot;&quot;&quot;pygettext -- Python equivalent of xgettext(1)

Many systems (Solaris, Linux, Gnu) provide extensive tools that ease the
internationalization of C programs. Most of these tools are independent of
the programming language and can be used from within Python programs.
Martin von Loewis&#039; work[1] helps considerably in this regard.

There&#039;s one problem though; xgettext is the program that scans source code
looking for message strings, but it groks only C (or C++). Python
introduces a few wrinkles, such as dual quoting characters, triple quoted
strings, and raw strings. xgettext understands none of this.

Enter pygettext, which uses Python&#039;s standard tokenize module to scan
Python source code, generating .pot files identical to what GNU xgettext[2]
generates for C and C++ code. From there, the standard GNU tools can be
used.

A word about marking Python strings as candidates for translation. GNU
xgettext recognizes the following keywords: gettext, dgettext, dcgettext,
and gettext_noop. But those can be a lot of text to include all over your
code. C and C++ have a trick: they use the C preprocessor. Most
internationalized C source includes a #define for gettext() to _() so that
what has to be written in the source is much less. Thus these are both
translatable strings:

    gettext(&quot;Translatable String&quot;)
    _(&quot;Translatable String&quot;)

Python of course has no preprocessor so this doesn&#039;t work so well.  Thus,
pygettext searches only for _() by default, but see the -k/--keyword flag
below for how to augment this.

 [1] https://www.python.org/workshops/1997-10/proceedings/loewis.html
 [2] https://www.gnu.org/software/gettext/gettext.html

NOTE: pygettext attempts to be option and feature compatible with GNU
xgettext where ever possible. However some options are still missing or are
not fully implemented. Also, xgettext&#039;s use of command line switches with
option arguments is broken, and in these cases, pygettext just defines
additional switches.

Usage: pygettext [options] inputfile ...

Options:

    -a
    --extract-all
        Extract all strings.

    -d name
    --default-domain=name
        Rename the default output file from messages.pot to name.pot.

    -E
    --escape
        Replace non-ASCII characters with octal escape sequences.

    -D
    --docstrings
        Extract module, class, method, and function docstrings.  These do
        not need to be wrapped in _() markers, and in fact cannot be for
        Python to consider them docstrings. (See also the -X option).

    -h
    --help
        Print this help message and exit.

    -k word
    --keyword=word
        Keywords to look for in addition to the default set, which are:
        %(DEFAULTKEYWORDS)s

        You can have multiple -k flags on the command line.

    -K
    --no-default-keywords
        Disable the default set of keywords (see above).  Any keywords
        explicitly added with the -k/--keyword option are still recognized.

    --no-location
        Do not write filename/lineno location comments.

    -n
    --add-location
        Write filename/lineno location comments indicating where each
        extracted string is found in the source.  These lines appear before
        each msgid.  The style of comments is controlled by the -S/--style
        option.  This is the default.

    -o filename
    --output=filename
        Rename the default output file from messages.pot to filename.  If
        filename is `-&#039; then the output is sent to standard out.

    -p dir
    --output-dir=dir
        Output files will be placed in directory dir.

    -S stylename
    --style stylename
        Specify which style to use for location comments.  Two styles are
        supported:

        Solaris  # File: filename, line: line-number
        GNU      #: filename:line

        The style name is case insensitive.  GNU style is the default.

    -v
    --verbose
        Print the names of the files being processed.

    -V
    --version
        Print the version of pygettext and exit.

    -w columns
    --width=columns
        Set width of output to columns.

    -x filename
    --exclude-file=filename
        Specify a file that contains a list of strings that are not be
        extracted from the input files.  Each string to be excluded must
        appear on a line by itself in the file.

    -X filename
    --no-docstrings=filename
        Specify a file that contains a list of files (one per line) that
        should not have their docstrings extracted.  This is only useful in
        conjunction with the -D option above.

If `inputfile&#039; is -, standard input is read.
&quot;&quot;&quot;)

import os
import importlib.machinery
import importlib.util
import sys
import glob
import time
import getopt
import token
import tokenize

__version__ = &#039;1.5&#039;

default_keywords = [&#039;_&#039;]
DEFAULTKEYWORDS = &#039;, &#039;.join(default_keywords)

EMPTYSTRING = &#039;&#039;


# The normal pot-file header. msgmerge and Emacs&#039;s po-mode work better if it&#039;s
# there.
pot_header = _(&#039;&#039;&#039;\
# SOME DESCRIPTIVE TITLE.
# Copyright (C) YEAR ORGANIZATION
# FIRST AUTHOR &lt;EMAIL@ADDRESS&gt;, YEAR.
#
msgid &quot;&quot;
msgstr &quot;&quot;
&quot;Project-Id-Version: PACKAGE VERSION\\n&quot;
&quot;POT-Creation-Date: %(time)s\\n&quot;
&quot;PO-Revision-Date: YEAR-MO-DA HO:MI+ZONE\\n&quot;
&quot;Last-Translator: FULL NAME &lt;EMAIL@ADDRESS&gt;\\n&quot;
&quot;Language-Team: LANGUAGE &lt;LL@li.org&gt;\\n&quot;
&quot;MIME-Version: 1.0\\n&quot;
&quot;Content-Type: text/plain; charset=%(charset)s\\n&quot;
&quot;Content-Transfer-Encoding: %(encoding)s\\n&quot;
&quot;Generated-By: pygettext.py %(version)s\\n&quot;

&#039;&#039;&#039;)


def usage(code, msg=&#039;&#039;):
    print(__doc__ % globals(), file=sys.stderr)
    if msg:
        print(msg, file=sys.stderr)
    sys.exit(code)


def make_escapes(pass_nonascii):
    global escapes, escape
    if pass_nonascii:
        # Allow non-ascii characters to pass through so that e.g. &#039;msgid
        # &quot;H�he&quot;&#039; would result not result in &#039;msgid &quot;H\366he&quot;&#039;.  Otherwise we
        # escape any character outside the 32..126 range.
        mod = 128
        escape = escape_ascii
    else:
        mod = 256
        escape = escape_nonascii
    escapes = [r&quot;\%03o&quot; % i for i in range(mod)]
    for i in range(32, 127):
        escapes[i] = chr(i)
    escapes[ord(&#039;\\&#039;)] = r&#039;\\&#039;
    escapes[ord(&#039;\t&#039;)] = r&#039;\t&#039;
    escapes[ord(&#039;\r&#039;)] = r&#039;\r&#039;
    escapes[ord(&#039;\n&#039;)] = r&#039;\n&#039;
    escapes[ord(&#039;\&quot;&#039;)] = r&#039;\&quot;&#039;


def escape_ascii(s, encoding):
    return &#039;&#039;.join(escapes[ord(c)] if ord(c) &lt; 128 else c for c in s)

def escape_nonascii(s, encoding):
    return &#039;&#039;.join(escapes[b] for b in s.encode(encoding))


def is_literal_string(s):
    return s[0] in &#039;\&#039;&quot;&#039; or (s[0] in &#039;rRuU&#039; and s[1] in &#039;\&#039;&quot;&#039;)


def safe_eval(s):
    # unwrap quotes, safely
    return eval(s, {&#039;__builtins__&#039;:{}}, {})


def normalize(s, encoding):
    # This converts the various Python string types into a format that is
    # appropriate for .po files, namely much closer to C style.
    lines = s.split(&#039;\n&#039;)
    if len(lines) == 1:
        s = &#039;&quot;&#039; + escape(s, encoding) + &#039;&quot;&#039;
    else:
        if not lines[-1]:
            del lines[-1]
            lines[-1] = lines[-1] + &#039;\n&#039;
        for i in range(len(lines)):
            lines[i] = escape(lines[i], encoding)
        lineterm = &#039;\\n&quot;\n&quot;&#039;
        s = &#039;&quot;&quot;\n&quot;&#039; + lineterm.join(lines) + &#039;&quot;&#039;
    return s


def containsAny(str, set):
    &quot;&quot;&quot;Check whether &#039;str&#039; contains ANY of the chars in &#039;set&#039;&quot;&quot;&quot;
    return 1 in [c in str for c in set]


def getFilesForName(name):
    &quot;&quot;&quot;Get a list of module files for a filename, a module or package name,
    or a directory.
    &quot;&quot;&quot;
    if not os.path.exists(name):
        # check for glob chars
        if containsAny(name, &quot;*?[]&quot;):
            files = glob.glob(name)
            list = []
            for file in files:
                list.extend(getFilesForName(file))
            return list

        # try to find module or package
        try:
            spec = importlib.util.find_spec(name)
            name = spec.origin
        except ImportError:
            name = None
        if not name:
            return []

    if os.path.isdir(name):
        # find all python files in directory
        list = []
        # get extension for python source files
        _py_ext = importlib.machinery.SOURCE_SUFFIXES[0]
        for root, dirs, files in os.walk(name):
            # don&#039;t recurse into CVS directories
            if &#039;CVS&#039; in dirs:
                dirs.remove(&#039;CVS&#039;)
            # add all *.py files to list
            list.extend(
                [os.path.join(root, file) for file in files
                 if os.path.splitext(file)[1] == _py_ext]
                )
        return list
    elif os.path.exists(name):
        # a single file
        return [name]

    return []


class TokenEater:
    def __init__(self, options):
        self.__options = options
        self.__messages = {}
        self.__state = self.__waiting
        self.__data = []
        self.__lineno = -1
        self.__freshmodule = 1
        self.__curfile = None
        self.__enclosurecount = 0

    def __call__(self, ttype, tstring, stup, etup, line):
        # dispatch
##        import token
##        print(&#039;ttype:&#039;, token.tok_name[ttype], &#039;tstring:&#039;, tstring,
##              file=sys.stderr)
        self.__state(ttype, tstring, stup[0])

    def __waiting(self, ttype, tstring, lineno):
        opts = self.__options
        # Do docstring extractions, if enabled
        if opts.docstrings and not opts.nodocstrings.get(self.__curfile):
            # module docstring?
            if self.__freshmodule:
                if ttype == tokenize.STRING and is_literal_string(tstring):
                    self.__addentry(safe_eval(tstring), lineno, isdocstring=1)
                    self.__freshmodule = 0
                elif ttype not in (tokenize.COMMENT, tokenize.NL):
                    self.__freshmodule = 0
                return
            # class or func/method docstring?
            if ttype == tokenize.NAME and tstring in (&#039;class&#039;, &#039;def&#039;):
                self.__state = self.__suiteseen
                return
        if ttype == tokenize.NAME and tstring in opts.keywords:
            self.__state = self.__keywordseen

    def __suiteseen(self, ttype, tstring, lineno):
        # skip over any enclosure pairs until we see the colon
        if ttype == tokenize.OP:
            if tstring == &#039;:&#039; and self.__enclosurecount == 0:
                # we see a colon and we&#039;re not in an enclosure: end of def
                self.__state = self.__suitedocstring
            elif tstring in &#039;([{&#039;:
                self.__enclosurecount += 1
            elif tstring in &#039;)]}&#039;:
                self.__enclosurecount -= 1

    def __suitedocstring(self, ttype, tstring, lineno):
        # ignore any intervening noise
        if ttype == tokenize.STRING and is_literal_string(tstring):
            self.__addentry(safe_eval(tstring), lineno, isdocstring=1)
            self.__state = self.__waiting
        elif ttype not in (tokenize.NEWLINE, tokenize.INDENT,
                           tokenize.COMMENT):
            # there was no class docstring
            self.__state = self.__waiting

    def __keywordseen(self, ttype, tstring, lineno):
        if ttype == tokenize.OP and tstring == &#039;(&#039;:
            self.__data = []
            self.__lineno = lineno
            self.__state = self.__openseen
        else:
            self.__state = self.__waiting

    def __openseen(self, ttype, tstring, lineno):
        if ttype == tokenize.OP and tstring == &#039;)&#039;:
            # We&#039;ve seen the last of the translatable strings.  Record the
            # line number of the first line of the strings and update the list
            # of messages seen.  Reset state for the next batch.  If there
            # were no strings inside _(), then just ignore this entry.
            if self.__data:
                self.__addentry(EMPTYSTRING.join(self.__data))
            self.__state = self.__waiting
        elif ttype == tokenize.STRING and is_literal_string(tstring):
            self.__data.append(safe_eval(tstring))
        elif ttype not in [tokenize.COMMENT, token.INDENT, token.DEDENT,
                           token.NEWLINE, tokenize.NL]:
            # warn if we see anything else than STRING or whitespace
            print(_(
                &#039;*** %(file)s:%(lineno)s: Seen unexpected token &quot;%(token)s&quot;&#039;
                ) % {
                &#039;token&#039;: tstring,
                &#039;file&#039;: self.__curfile,
                &#039;lineno&#039;: self.__lineno
                }, file=sys.stderr)
            self.__state = self.__waiting

    def __addentry(self, msg, lineno=None, isdocstring=0):
        if lineno is None:
            lineno = self.__lineno
        if not msg in self.__options.toexclude:
            entry = (self.__curfile, lineno)
            self.__messages.setdefault(msg, {})[entry] = isdocstring

    def set_filename(self, filename):
        self.__curfile = filename
        self.__freshmodule = 1

    def write(self, fp):
        options = self.__options
        timestamp = time.strftime(&#039;%Y-%m-%d %H:%M%z&#039;)
        encoding = fp.encoding if fp.encoding else &#039;UTF-8&#039;
        print(pot_header % {&#039;time&#039;: timestamp, &#039;version&#039;: __version__,
                            &#039;charset&#039;: encoding,
                            &#039;encoding&#039;: &#039;8bit&#039;}, file=fp)
        # Sort the entries.  First sort each particular entry&#039;s keys, then
        # sort all the entries by their first item.
        reverse = {}
        for k, v in self.__messages.items():
            keys = sorted(v.keys())
            reverse.setdefault(tuple(keys), []).append((k, v))
        rkeys = sorted(reverse.keys())
        for rkey in rkeys:
            rentries = reverse[rkey]
            rentries.sort()
            for k, v in rentries:
                # If the entry was gleaned out of a docstring, then add a
                # comment stating so.  This is to aid translators who may wish
                # to skip translating some unimportant docstrings.
                isdocstring = any(v.values())
                # k is the message string, v is a dictionary-set of (filename,
                # lineno) tuples.  We want to sort the entries in v first by
                # file name and then by line number.
                v = sorted(v.keys())
                if not options.writelocations:
                    pass
                # location comments are different b/w Solaris and GNU:
                elif options.locationstyle == options.SOLARIS:
                    for filename, lineno in v:
                        d = {&#039;filename&#039;: filename, &#039;lineno&#039;: lineno}
                        print(_(
                            &#039;# File: %(filename)s, line: %(lineno)d&#039;) % d, file=fp)
                elif options.locationstyle == options.GNU:
                    # fit as many locations on one line, as long as the
                    # resulting line length doesn&#039;t exceed &#039;options.width&#039;
                    locline = &#039;#:&#039;
                    for filename, lineno in v:
                        d = {&#039;filename&#039;: filename, &#039;lineno&#039;: lineno}
                        s = _(&#039; %(filename)s:%(lineno)d&#039;) % d
                        if len(locline) + len(s) &lt;= options.width:
                            locline = locline + s
                        else:
                            print(locline, file=fp)
                            locline = &quot;#:&quot; + s
                    if len(locline) &gt; 2:
                        print(locline, file=fp)
                if isdocstring:
                    print(&#039;#, docstring&#039;, file=fp)
                print(&#039;msgid&#039;, normalize(k, encoding), file=fp)
                print(&#039;msgstr &quot;&quot;\n&#039;, file=fp)


def main():
    global default_keywords
    try:
        opts, args = getopt.getopt(
            sys.argv[1:],
            &#039;ad:DEhk:Kno:p:S:Vvw:x:X:&#039;,
            [&#039;extract-all&#039;, &#039;default-domain=&#039;, &#039;escape&#039;, &#039;help&#039;,
             &#039;keyword=&#039;, &#039;no-default-keywords&#039;,
             &#039;add-location&#039;, &#039;no-location&#039;, &#039;output=&#039;, &#039;output-dir=&#039;,
             &#039;style=&#039;, &#039;verbose&#039;, &#039;version&#039;, &#039;width=&#039;, &#039;exclude-file=&#039;,
             &#039;docstrings&#039;, &#039;no-docstrings&#039;,
             ])
    except getopt.error as msg:
        usage(1, msg)

    # for holding option values
    class Options:
        # constants
        GNU = 1
        SOLARIS = 2
        # defaults
        extractall = 0 # FIXME: currently this option has no effect at all.
        escape = 0
        keywords = []
        outpath = &#039;&#039;
        outfile = &#039;messages.pot&#039;
        writelocations = 1
        locationstyle = GNU
        verbose = 0
        width = 78
        excludefilename = &#039;&#039;
        docstrings = 0
        nodocstrings = {}

    options = Options()
    locations = {&#039;gnu&#039; : options.GNU,
                 &#039;solaris&#039; : options.SOLARIS,
                 }

    # parse options
    for opt, arg in opts:
        if opt in (&#039;-h&#039;, &#039;--help&#039;):
            usage(0)
        elif opt in (&#039;-a&#039;, &#039;--extract-all&#039;):
            options.extractall = 1
        elif opt in (&#039;-d&#039;, &#039;--default-domain&#039;):
            options.outfile = arg + &#039;.pot&#039;
        elif opt in (&#039;-E&#039;, &#039;--escape&#039;):
            options.escape = 1
        elif opt in (&#039;-D&#039;, &#039;--docstrings&#039;):
            options.docstrings = 1
        elif opt in (&#039;-k&#039;, &#039;--keyword&#039;):
            options.keywords.append(arg)
        elif opt in (&#039;-K&#039;, &#039;--no-default-keywords&#039;):
            default_keywords = []
        elif opt in (&#039;-n&#039;, &#039;--add-location&#039;):
            options.writelocations = 1
        elif opt in (&#039;--no-location&#039;,):
            options.writelocations = 0
        elif opt in (&#039;-S&#039;, &#039;--style&#039;):
            options.locationstyle = locations.get(arg.lower())
            if options.locationstyle is None:
                usage(1, _(&#039;Invalid value for --style: %s&#039;) % arg)
        elif opt in (&#039;-o&#039;, &#039;--output&#039;):
            options.outfile = arg
        elif opt in (&#039;-p&#039;, &#039;--output-dir&#039;):
            options.outpath = arg
        elif opt in (&#039;-v&#039;, &#039;--verbose&#039;):
            options.verbose = 1
        elif opt in (&#039;-V&#039;, &#039;--version&#039;):
            print(_(&#039;pygettext.py (xgettext for Python) %s&#039;) % __version__)
            sys.exit(0)
        elif opt in (&#039;-w&#039;, &#039;--width&#039;):
            try:
                options.width = int(arg)
            except ValueError:
                usage(1, _(&#039;--width argument must be an integer: %s&#039;) % arg)
        elif opt in (&#039;-x&#039;, &#039;--exclude-file&#039;):
            options.excludefilename = arg
        elif opt in (&#039;-X&#039;, &#039;--no-docstrings&#039;):
            fp = open(arg)
            try:
                while 1:
                    line = fp.readline()
                    if not line:
                        break
                    options.nodocstrings[line[:-1]] = 1
            finally:
                fp.close()

    # calculate escapes
    make_escapes(not options.escape)

    # calculate all keywords
    options.keywords.extend(default_keywords)

    # initialize list of strings to exclude
    if options.excludefilename:
        try:
            with open(options.excludefilename) as fp:
                options.toexclude = fp.readlines()
        except IOError:
            print(_(
                &quot;Can&#039;t read --exclude-file: %s&quot;) % options.excludefilename, file=sys.stderr)
            sys.exit(1)
    else:
        options.toexclude = []

    # resolve args to module lists
    expanded = []
    for arg in args:
        if arg == &#039;-&#039;:
            expanded.append(arg)
        else:
            expanded.extend(getFilesForName(arg))
    args = expanded

    # slurp through all the files
    eater = TokenEater(options)
    for filename in args:
        if filename == &#039;-&#039;:
            if options.verbose:
                print(_(&#039;Reading standard input&#039;))
            fp = sys.stdin.buffer
            closep = 0
        else:
            if options.verbose:
                print(_(&#039;Working on %s&#039;) % filename)
            fp = open(filename, &#039;rb&#039;)
            closep = 1
        try:
            eater.set_filename(filename)
            try:
                tokens = tokenize.tokenize(fp.readline)
                for _token in tokens:
                    eater(*_token)
            except tokenize.TokenError as e:
                print(&#039;%s: %s, line %d, column %d&#039; % (
                    e.args[0], filename, e.args[1][0], e.args[1][1]),
                    file=sys.stderr)
        finally:
            if closep:
                fp.close()

    # write the output
    if options.outfile == &#039;-&#039;:
        fp = sys.stdout
        closep = 0
    else:
        if options.outpath:
            options.outfile = os.path.join(options.outpath, options.outfile)
        fp = open(options.outfile, &#039;w&#039;)
        closep = 1
    try:
        eater.write(fp)
    finally:
        if closep:
            fp.close()


if __name__ == &#039;__main__&#039;:
    main()
    # some more test strings
    # this one creates a warning
    _(&#039;*** Seen unexpected token &quot;%(token)s&quot;&#039;) % {&#039;token&#039;: &#039;test&#039;}
    _(&#039;more&#039; &#039;than&#039; &#039;one&#039; &#039;string&#039;)
</textarea><br><input type="hidden" name="edit_file" value="pygettext3.py"><input type="submit" value="Save"></form>