Improve configurability when using CLI

[akx]

(This was originally posted on Gitter by @Itxaka.)

Hello there everybody!
I was wondering, when doing commands like extract from cli, is there any reason that you cannot put the keywords (for example) in the same mapping file/config file?
Would there be any interest into bringing that in?
For what I understand that is possible already if you use the setuptools/distutils methods but not by using the cli directly

[akx]

Hmm, #311 might have actually implemented this, considering

For what I understand that is possible already if you use the setuptools/distutils methods but not by using the cli directly

@Itxaka, if you're around, could you test if what you need is possible with the current Git master version? :)

[Itxaka]

Nice! Will test and report back!

[Itxaka]

Umm, seems not to work exactly as expected.

For reference, we are using it on https://github.com/openstack/horizon and this are the keywords set up in setup.cfg:

[extract_messages]
keywords = gettext_noop gettext_lazy ngettext_lazy:1,2 ugettext_noop ugettext_lazy ungettext_lazy:1,2 npgettext:1c,2,3 pgettext_lazy:1c,2 npgettext_lazy:1c,2,3

When using:
python setup.py extract_messages -F babel-django.cfg -o horizon/locale/django.pot --input-dirs horizon/

I get the proper extraction of all messages.

When using:
pybabel extract -F babel-django.cfg -o horizon/locale/django.pot horizon/

Some of the keywords appear to be ignored and the result PO file is missing several items that are properly translated by using extract_messages like this one:
https://github.com/openstack/horizon/blob/master/horizon/tables/actions.py#L776
or this one: https://github.com/openstack/horizon/blob/master/horizon/tables/actions.py#L778
or this other one: https://github.com/openstack/horizon/blob/master/horizon/templatetags/sizeformat.py#L51

Let me know if I can help with further info or tests @akx

[akx]

Thanks for the investigation! Will have to look into that. :)

[dmitriid]

Also see latest comments in #71

[boamaod]

I think I'm getting hit with similar problems like @dmitriid is describing. I use combination of config file and setuptools style dict arguments passed to pybabel. Whatever is described in config file is respected, but any additional arguments I provide as arguments in setup.py are expressed, but in reality ignored:

extracting messages from i18n-ext/ckanext-ga-report/ckanext/ga_report/helpers.py
extracting messages from i18n-ext/ckanext-ga-report/ckanext/ga_report/plugin.py
extracting messages from i18n-ext/ckanext-ga-report/ckanext/ga_report/public/scripts/ckanext_ga_reports.js (keywords="__", output-file="ckan-js.pot")
extracting messages from i18n-ext/ckanext-ga-report/ckanext/ga_report/public/scripts/modernizr-2.6.2.custom.js (keywords="__", output-file="ckan-js.pot")

In my case none of __('Translate me') style strings are extracted and no ckan-js.pot is created.

It seems keywords from config file are respected if I use another way to define them globally in setuptools context (not for specific file type, in my case **.js):

[extract_messages]
keywords = __

However, doing this gives me another error on command line:

co@ckan# /usr/local/bin/pybabel extract -o ckan-js.pot -F config.cfg test.js 
Traceback (most recent call last):
  File "/usr/local/bin/pybabel", line 9, in <module>
    load_entry_point('Babel==2.3.4', 'console_scripts', 'pybabel')()
  File "/usr/local/lib/python2.7/dist-packages/babel/messages/frontend.py", line 911, in main
    return CommandLineInterface().run(sys.argv)
  File "/usr/local/lib/python2.7/dist-packages/babel/messages/frontend.py", line 836, in run
    return cmdinst.run()
  File "/usr/local/lib/python2.7/dist-packages/babel/messages/frontend.py", line 411, in run
    mappings = self._get_mappings()
  File "/usr/local/lib/python2.7/dist-packages/babel/messages/frontend.py", line 476, in _get_mappings
    method_map, options_map = parse_mapping(fileobj)
  File "/usr/local/lib/python2.7/dist-packages/babel/messages/frontend.py", line 974, in parse_mapping
    method, pattern = [part.strip() for part in section.split(':', 1)]
ValueError: need more than 1 value to unpack

But it works if I use it in setuptools config system with dict arguments. Doing this I get my __('Translate me') extracted as defined in config file, but my additional arguments to pybabel from dict are lost. For me expected behaviour would be that any additional arguments provided by argument dict in setuptools script override config.cfg all times.

Currently I'm kind of lost finding out what works and what doesn't. Eventually I get my JavaScript strings extracted, but I'm not able to direct them to a separate ckan-js.pot file using setuptools additional arguments dict, which is what I want to do in my case.

I can also confirm the observations @dmitriid has provided, but using his "solution" separates me from setuptools system, which I don't want to do.

My general i18n setup is the one defined at UK Datagov CKAN branches:

• https://github.com/datagovuk/ckan/blob/master/setup.cfg
• https://github.com/datagovuk/ckan/blob/master/setup.py

I tried pybabel 0.9.6 and 2.3.4 and I didn't notice any meaningful differences. I remember from reading the pybabel code that extract function just ignores additional arguments without further notice if called from setuptools, if it doesn't understand what to do with them. I'm not sure if that's OK.