Plans for python support on lyx-2.2

[José Matos]

Hi all,
I suggest to update the python code used by lyx to support both python 2 and python 3 with the same code.

For those who (did not) read the summit notes from Guido:
http://thread.gmane.org/gmane.comp.python.devel/147048

There is one point particularly that I noticed:
"- The recommended and least painful way to develop for Python 2 and 3 is definitely to use a single source that runs under both without translation; we no longer recommend auto-generating Python 3 compatible source code using 2to3, for a variety of reasons. Several people attested that single-source has worked well for them; Mercurial is using the 2to3 approach but they're not too happy with it."

We have already a ticket opened for this:
http://www.lyx.org/trac/ticket/9006

Also of notice for some of the points raised in the ticket regarding python 2.7 support:
"- We should make an effort to publicize that we're NOT sunsetting Python 2.7 just yet; support will continue (hopefully with ample support from distro vendors), and someone should update PEP 373. (Unclear what the new EOL is but we should definitely rescind the currently published schedule.)"

As you can see here:
http://legacy.python.org/dev/peps/pep-0373/
"The End Of Life date (EOL, sunset date) for Python 2.7 has been moved five years into the future, to 2020."

So IMHO I suggest code that is able to support both python 2 and python 3.

Before proceeding here there two related issues that we should discuss:

What is the minimum python 2 version that we will support?

What is the minimum python 3 version that we will support?

My suggestions are python 2.7 and python 3.3 as the minimal versions. Since I suggest to support both versions that means that you only need to have one of them available.

Before proceeding with the changes we should add tests to guarantee that the code works as intended on both versions.

I do not intend to work on this until the end of the semester (mid June FWIW) but I would like to start the discussion here.

Regards,

[Pavel Sanda]

Post by José Matos
Before proceeding with the changes we should add tests to guarantee that the code works as intended on both versions.
I do not intend to work on this until the end of the semester (mid June FWIW) but I would like to start the discussion here.

The more versions supported the bigger maintenance burden. I would think twice whether we should simply switch just to python 3.
(I hope x in 3.x doesn't matter much, right?)

Pavel

[Benjamin Piwowarski]

Hi,

I started to convert to python 3 some central scripts (configure.py a among those). Most of the problems are due to print calls, and the remaining problems were straightforward to solve. I will push this to master soon.

Benjamin

Post by Pavel Sanda

Post by José Matos
Before proceeding with the changes we should add tests to guarantee that the code works as intended on both versions.
I do not intend to work on this until the end of the semester (mid June FWIW) but I would like to start the discussion here.

The more versions supported the bigger maintenance burden. I would think twice whether we should simply switch just to python 3.
(I hope x in 3.x doesn't matter much, right?)
Pavel

[Pavel Sanda]

Post by Benjamin Piwowarski
I started to convert to python 3 some central scripts (configure.py a among those). Most of the problems are due to print calls, and the remaining problems were straightforward to solve. I will push this to master soon.

Benjamin,
such patches should be always sent to mailing list for discussion before pushing anything.
Pavel

[Stephan Witt]

Post by Pavel Sanda

Post by José Matos
Before proceeding with the changes we should add tests to guarantee that the code works as intended on both versions.
I do not intend to work on this until the end of the semester (mid June FWIW) but I would like to start the discussion here.

The more versions supported the bigger maintenance burden. I would think twice whether we should simply switch just to python 3.
(I hope x in 3.x doesn't matter much, right?)
Pavel

I would think twice whether we should drop support for 2.7.
IMHO, this should be done only if all platforms support 3.x
without additional user action. I cannot say anything about
the amount of maintenance burden. But the python people do
recommend it, AFAIU.

Stephan

[Guenter Milde]

Post by Pavel Sanda

Post by José Matos
Before proceeding with the changes we should add tests to guarantee
that the code works as intended on both versions.
I do not intend to work on this until the end of the semester (mid
June FWIW) but I would like to start the discussion here.

The more versions supported the bigger maintenance burden. I would
think twice whether we should simply switch just to python 3. (I hope x
in 3.x doesn't matter much, right?)

I also vote for keeping compatibility with both 2.7 and 3.x (x >=3) for the
next couple of LyX versions:

* There are still a lot of installations, where
2.7 (or older) is the default Python version.

* As we can have one common source for both major versions, the added burden
is managable.

I am also willing to help, if required.

Günter

[Josh Hieronymus]

Post by Guenter Milde

Post by Pavel Sanda

Post by José Matos
Before proceeding with the changes we should add tests to guarantee
that the code works as intended on both versions.
I do not intend to work on this until the end of the semester (mid
June FWIW) but I would like to start the discussion here.

The more versions supported the bigger maintenance burden. I would
think twice whether we should simply switch just to python 3. (I hope x
in 3.x doesn't matter much, right?)

I also vote for keeping compatibility with both 2.7 and 3.x (x >=3) for the
* There are still a lot of installations, where
2.7 (or older) is the default Python version.
* As we can have one common source for both major versions, the added burden
is managable.
I am also willing to help, if required.
GÃŒnter

For most code, using a common source shouldn't cause problems between the
2.7 and 3.X versions of Python. In certain cases where problems could
arise, such as with print statements, it is possible to write code that is
in the style of 3.X but is still compatible with 2.7 by importing the
proper items from the __future__ module. For other cases, which I believe
to be relatively few, we should be fine for the most part detecting the
version of Python and running version-specific code in those cases. I
imagine that the biggest issues that would arise would probably be caused
by using third-party modules that haven't been updated to work with Python
3.X yet.

Josh

[Benjamin Piwowarski]

There is not much to do for python 3 compatibility, but still
 I attached the patch this time.

Benjamin
 

Post by Guenter Milde

Post by Pavel Sanda

Post by José Matos
Before proceeding with the changes we should add tests to guarantee
that the code works as intended on both versions.
I do not intend to work on this until the end of the semester (mid
June FWIW) but I would like to start the discussion here.

The more versions supported the bigger maintenance burden. I would
think twice whether we should simply switch just to python 3. (I hope x
in 3.x doesn't matter much, right?)

I also vote for keeping compatibility with both 2.7 and 3.x (x >=3) for the
* There are still a lot of installations, where
  2.7 (or older) is the default Python version.
* As we can have one common source for both major versions, the added burden
  is managable.
I am also willing to help, if required.
GÃŒnter

For most code, using a common source shouldn't cause problems between the 2.7 and 3.X versions of Python. In certain cases where problems could arise, such as with print statements, it is possible to write code that is in the style of 3.X but is still compatible with 2.7 by importing the proper items from the __future__ module. For other cases, which I believe to be relatively few, we should be fine for the most part detecting the version of Python and running version-specific code in those cases. I imagine that the biggest issues that would arise would probably be caused by using third-party modules that haven't been updated to work with Python 3.X yet.

Josh

[Josh Hieronymus]

Post by Benjamin Piwowarski
There is not much to do for python 3 compatibility, but still
 I attached

the patch this time.

Post by Benjamin Piwowarski
Benjamin

I have one comment and one question regarding this patch.

The comment: if we're only concerned with supporting Python 2.7 and up, we
could include the statement "from __future__ import print_function" in any
file in which we use the print function, as this allows us to use all of
the features of the Python 3.X print function and avoids some potentially
unwanted behavior when printing multiple values. For example, in Python
2.X, "print 'Alice', 'Bob'" prints the line "Alice Bob", but
"print('Alice', 'Bob')" prints the line "('Alice', 'Bob')", since the
parentheses are not interpreted as enclosing the arguments for a function,
but as demarcating a tuple.

The question: why are we switching from opening files in binary mode to
opening files in text "t" mode? Was there a reason that we were using
binary mode in the first place?

Josh

[Jean-Marc Lasgouttes]

Post by Josh Hieronymus
The comment: if we're only concerned with supporting Python 2.7 and up,
we could include the statement "from __future__ import print_function"
in any file in which we use the print function, as this allows us to use
all of the features of the Python 3.X print function and avoids some
potentially unwanted behavior when printing multiple values.

Stephan, what version of python can be expected on Mac OS X 10.5 and
10.6? 10.7 has python 2.7.1.

JMarc

[Cyrille Artho]

Mac OS 10.9 (Mavericks) has 2.7.5 installed. So we should keep 2.7.X
compatibility. I think most people still use 2.7, as 3.X is relatively new
(in terms of software maintenance/update cycles) and it takes considerable
effort to switch a project over to 3.X.

Post by Josh Hieronymus
The comment: if we're only concerned with supporting Python 2.7 and up,
we could include the statement "from __future__ import print_function"
in any file in which we use the print function, as this allows us to use
all of the features of the Python 3.X print function and avoids some
potentially unwanted behavior when printing multiple values.

Stephan, what version of python can be expected on Mac OS X 10.5 and 10.6?
10.7 has python 2.7.1.
JMarc

[Stephan Witt]

Post by Josh Hieronymus
The comment: if we're only concerned with supporting Python 2.7 and up,
we could include the statement "from __future__ import print_function"
in any file in which we use the print function, as this allows us to use
all of the features of the Python 3.X print function and avoids some
potentially unwanted behavior when printing multiple values.

Stephan, what version of python can be expected on Mac OS X 10.5 and 10.6? 10.7 has python 2.7.1.

On 10.6.8 it's 2.6.1
On 10.5.8 it's 2.5.1

Stephan

[José Matos]

The question:... Was there a reason that we were using
binary mode in the first place?
Josh

IIRC it was to deal with the end line char in windows, but this was to support python 2.2 and my memory is fuzzy about further details. This was before the universal end line support in python.

[Benjamin Piwowarski]

The question:... Was there a reason that we were using
binary mode in the first place?
Josh

IIRC it was to deal with the end line char in windows, but this was to support python 2.2 and my memory is fuzzy about further details. This was before the universal end line support in python.

OK, so going to text mode should not be a problem anymore.

Benjamin 

[Josh Hieronymus]

if we're only concerned with supporting Python 2.7 and up, we could

include the statement "from __future__ import print_function" in any file
in which we use the print function

I misremembered this before, but Python 2.6 supports the import of the
print function from the __future__ module as well. Earlier versions of
Python, however do not.

Josh

[Benjamin Piwowarski]

Here is a first patch for python 2 & 3 compatibility of configure.py and prefs2prefs.py which are among the most central python scripts.

I did not push the changes because of the “open” function (in prefs2prefs) which had to be changed from binary mode to text mode (otherwise, regular expressions cannot be applied). I tested them with both python 2 and 3, and did not find any problem with this, but would like some feedback before applying this patch.
 
Benjamin

Post by Pavel Sanda

Post by José Matos
Before proceeding with the changes we should add tests to guarantee
that the code works as intended on both versions.
I do not intend to work on this until the end of the semester (mid
June FWIW) but I would like to start the discussion here.

The more versions supported the bigger maintenance burden. I would
think twice whether we should simply switch just to python 3. (I hope x
in 3.x doesn't matter much, right?)

I also vote for keeping compatibility with both 2.7 and 3.x (x >=3) for the
next couple of LyX versions:

* There are still a lot of installations, where
2.7 (or older) is the default Python version.

* As we can have one common source for both major versions, the added burden
is managable.

I am also willing to help, if required.

GÃŒnter

[Richard Heck]

Post by Benjamin Piwowarski
Here is a first patch for python 2 & 3 compatibility of configure.py
and prefs2prefs.py which are among the most central python scripts.
I did not push the changes because of the "open" function (in
prefs2prefs) which had to be changed from binary mode to text mode
(otherwise, regular expressions cannot be applied). I tested them with
both python 2 and 3, and did not find any problem with this, but would
like some feedback before applying this patch.

Hmm. Very short patch. (You forgot to attach.)

Richard

[Georg Baum]

Post by Guenter Milde

Post by Pavel Sanda

Post by José Matos
Before proceeding with the changes we should add tests to guarantee
that the code works as intended on both versions.
I do not intend to work on this until the end of the semester (mid
June FWIW) but I would like to start the discussion here.

The more versions supported the bigger maintenance burden. I would
think twice whether we should simply switch just to python 3. (I hope x
in 3.x doesn't matter much, right?)

This is the key question IMHO: If 3.2.1 is usable, I'd simply go for python
3 only, and require at least 3.2.1 (it is old enough). This is no problem on
any current and not so current linux distribution, not on windows (we ship
our own python), adn neither on OS X (AFAIK, correct me if I am wrong).

Post by Guenter Milde
I also vote for keeping compatibility with both 2.7 and 3.x (x >=3) for
* There are still a lot of installations, where
2.7 (or older) is the default Python version.

Which version is default does not matter IMHO: The same code that currently
selects python 2 if both are avbaiable can be used to select python 3.

Post by Guenter Milde
* As we can have one common source for both major versions, the added burden
is managable.

It is still a burden, and if you don't test regularily with both version
then bugs _will_ occur.


Georg

[Jean-Marc Lasgouttes]

Post by Georg Baum
This is the key question IMHO: If 3.2.1 is usable, I'd simply go for python
3 only, and require at least 3.2.1 (it is old enough). This is no problem on
any current and not so current linux distribution, not on windows (we ship
our own python), adn neither on OS X (AFAIK, correct me if I am wrong).

Mac OS X 10.7 here, I only have python 2.[567]. A few days ago, there
was only 10.6 on this machine, and I doubt it was better then.

JMarc

[Stephan Witt]

Post by Georg Baum

Post by Guenter Milde

Post by Pavel Sanda

Post by José Matos
Before proceeding with the changes we should add tests to guarantee
that the code works as intended on both versions.
I do not intend to work on this until the end of the semester (mid
June FWIW) but I would like to start the discussion here.

The more versions supported the bigger maintenance burden. I would
think twice whether we should simply switch just to python 3. (I hope x
in 3.x doesn't matter much, right?)

This is the key question IMHO: If 3.2.1 is usable, I'd simply go for python
3 only, and require at least 3.2.1 (it is old enough). This is no problem on
any current and not so current linux distribution, not on windows (we ship
our own python), adn neither on OS X (AFAIK, correct me if I am wrong).

Sorry, you're wrong. My Mac comes with 2.7.2 and I have to install some
additional software package to use python 3. It's possible, but I don't
recommend to force users to do so. It's a hassle and I've heard of users
getting problems with software suddenly broken because of the installation
of the "alien" python 3 package - this was LyX. But it may be other software
as well.

Stephan

Post by Georg Baum

Post by Guenter Milde
I also vote for keeping compatibility with both 2.7 and 3.x (x >=3) for
* There are still a lot of installations, where
2.7 (or older) is the default Python version.

Which version is default does not matter IMHO: The same code that currently
selects python 2 if both are avbaiable can be used to select python 3.

Post by Guenter Milde
* As we can have one common source for both major versions, the added burden
is managable.

It is still a burden, and if you don't test regularily with both version
then bugs _will_ occur.
Georg

[Georg Baum]

Post by Stephan Witt
Sorry, you're wrong. My Mac comes with 2.7.2 and I have to install some
additional software package to use python 3. It's possible, but I don't
recommend to force users to do so. It's a hassle and I've heard of users
getting problems with software suddenly broken because of the installation
of the "alien" python 3 package - this was LyX. But it may be other
software as well.

OK, this is of course a good reason to continue to support 2.x. I am a bit
surprised, since neither on windows nor on linux parallel installation of
2.x and 3.x is a problem.


Georg

[Neal Becker]

I think the python-future package is helpful

[José Matos]

Post by Neal Becker
I think the python-future package is helpful

Do you mean

from __future__ import print_function, division

if so I agree. :-)

[Benjamin Piwowarski]

Post by Neal Becker
I think the python-future package is helpful

Do you mean

from __future__ import print_function, division

if so I agree. :-)

Yes, but it would be good not to use it, in order to support OS X systems more ancient than the current one - they are running python 2.5-2.6. 

benjamin

[Richard Heck]

Post by Benjamin Piwowarski

Post by José Matos

Post by Neal Becker
I think the python-future package is helpful

Do you mean
from __future__ import print_function, division
if so I agree. :-)

Yes, but it would be good not to use it, in order to support OS X
systems more ancient than the current one - they are running python
2.5-2.6.

We should decide on the minimum version of Python we want to support for
2.2.

Richard

[Richard Heck]

Post by Benjamin Piwowarski
Yes, but it would be good not to use it, in order to support OS X systems more ancient than the current one - they are running python 2.5-2.6.
benjamin

python 2.7 was released on July of 2010. The next version of lyx will be available probably on 2016, late 2015 at best (taking into account the previous releases).
If we go with python 2.7 we will be able to support python 2 and python 3 with the same code base, using a python version version released more than 5 years ago. Doing this while retaining support for previous versions will be an harder work that will be thrown out in the next version.
My proposal is then to support both python 2.7 and 3.3+. For those that have python 3.1 and 3.2 (no one in his right mind should be using python 3.0) they will probably also have python 2.7 so our bases are covered there.

If there are, at that time, cases where 2.7 is not going to be
available, then we can package Python ourselves with the binary, as we
already do on Windows.

What's the reason to require 3.3+? (I'm not familiar with developments
here.)

Richard

[José Matos]

Post by Richard Heck
If there are, at that time, cases where 2.7 is not going to be
available, then we can package Python ourselves with the binary, as we
already do on Windows.
What's the reason to require 3.3+? (I'm not familiar with developments
here.)
Richard

Some sort of compromise. :-)

Think about it as the surprise test paradox:
http://en.wikipedia.org/wiki/Unexpected_hanging_paradox

The question regarding which is the minimum version of python 3 to support is more or less like this:

* python 3.0 was the first implementation and it has some "features" that were later restored to behave as python 2.x

* python 3.1 is a iterative improvement over 3.0 and definitively a better choice than 3.0

* the same can be said about python 3.2

* and python 3.3, where for example the unicode strings u"" were re-enacted after being missing from the previous three releases.

* python 3.4 is even a better release in terms of porting code from python 2.7

* python 3.5 will be an even better to help to the porting effect.

So the ideal, IMHO, would be to based as the minimal version for python 3 the current 3.4, as a second best to use 3.3 as the starting point.

In terms of the support going back to 3.2 at least in terms of linux, every distribution that supports python 3.2 also supports python 2.7 so there is no gain supporting 3.2.

The main argument here is heuristic. I hope that this makes some sense. :-)

[Pavel Sanda]

Post by José Matos
* python 3.0 was the first implementation and it has some "features" that were later restored to behave as python 2.x

:)) Seems the python devs really know what they are doing.

Pavel

[José Matos]

Post by Pavel Sanda
:)) Seems the python devs really know what they are doing.
Pavel

$ python -m this
The Zen of Python, by Tim Peters

...
Special cases aren't special enough to break the rules.
Although practicality beats purity.
...
Now is better than never.
Although never is often better than *right* now.
...

I like the fact that they admit their mistakes. :-)

[José Matos]

Post by Benjamin Piwowarski
Yes, but it would be good not to use it, in order to support OS X systems more ancient than the current one - they are running python 2.5-2.6.
benjamin

python 2.7 was released on July of 2010. The next version of lyx will be available probably on 2016, late 2015 at best (taking into account the previous releases).

If we go with python 2.7 we will be able to support python 2 and python 3 with the same code base, using a python version version released more than 5 years ago. Doing this while retaining support for previous versions will be an harder work that will be thrown out in the next version.

My proposal is then to support both python 2.7 and 3.3+. For those that have python 3.1 and 3.2 (no one in his right mind should be using python 3.0) they will probably also have python 2.7 so our bases are covered there.

[Georg Baum]

Post by Benjamin Piwowarski
Yes, but it would be good not to use it, in order to support OS X systems
more ancient than the current one - they are running python 2.5-2.6.
benjamin

python 2.7 was released on July of 2010. The next version of lyx will be
available probably on 2016, late 2015 at best (taking into account the
previous releases).
If we go with python 2.7 we will be able to support python 2 and python 3
with the same code base, using a python version version released more than
5 years ago. Doing this while retaining support for previous versions will
be an harder work that will be thrown out in the next version.
My proposal is then to support both python 2.7 and 3.3+. For those that
have python 3.1 and 3.2 (no one in his right mind should be using python
3.0) they will probably also have python 2.7 so our bases are covered
there.

This plan looks sensible. As far as OS X 10.7 is concerned:
https://www.python.org/download/mac claims that you can install their binary
packages in parallel to the system python, so if anybody really wants to use
a current version of LyX on a 5 year old OS, the he could as well do the
extra work and install an additional python package.


Georg

[José Matos]

Post by Georg Baum
https://www.python.org/download/mac claims that you can install their binary
packages in parallel to the system python, so if anybody really wants to use
a current version of LyX on a 5 year old OS, the he could as well do the
extra work and install an additional python package.
Georg

Would anyone oppose to change the requirements for devel as stated above?

python 2.7 or 3.3+

[Richard Heck]

Post by José Matos

Post by Georg Baum
https://www.python.org/download/mac claims that you can install their binary
packages in parallel to the system python, so if anybody really wants to use
a current version of LyX on a 5 year old OS, the he could as well do the
extra work and install an additional python package.
Georg

Would anyone oppose to change the requirements for devel as stated above?
python 2.7 or 3.3+

I don't, but I'd suggest you reissue this question with a new subject
line, so everyone will see it.

Richard

[José Matos]

Post by Richard Heck
I don't, but I'd suggest you reissue this question with a new subject
line, so everyone will see it.
Richard

Fair enough. :-)

[Cyrille Artho]

Post by José Matos
Would anyone oppose to change the requirements for devel as stated above?
python 2.7 or 3.3+

I think it's a very good idea as Python 2.7 is still default on Mac OS
and perhaps also other platforms.

[Neal Becker]

Post by José Matos

Post by Neal Becker
I think the python-future package is helpful

Do you mean
from __future__ import print_function, division
if so I agree. :-)

No, I mean:
https://pypi.python.org/pypi/future

[Benjamin Piwowarski]

Hi,

Again - on OS X 10.7, this will not work. And I don’t think the scripts used in LyX are that complex that we cannot handle them with using the future statement; for the two first scripts I converted, the work was trivial and did not require it. 

We can maybe start by not using it and discussing when/if needed. 

Benjamin

Post by José Matos

Post by Neal Becker
I think the python-future package is helpful

Do you mean
from __future__ import print_function, division
if so I agree. :-)

No, I mean:
https://pypi.python.org/pypi/future

[Kornel Benko]

Post by Benjamin Piwowarski
Hi,
Again - on OS X 10.7, this will not work. And I don’t think the scripts used in LyX are that complex that we cannot handle them with using the future statement; for the two first scripts I converted, the work was trivial and did not require it.

At least prints could be rewritten with
import sys
...
sys.out.write(...)
or
output = open(...)
output.write(....)

No dependencies on python2 or python3.
Just my 2 cents.

Post by Benjamin Piwowarski
We can maybe start by not using it and discussing when/if needed.
Benjamin

Post by José Matos

Post by Neal Becker
I think the python-future package is helpful

Do you mean
from __future__ import print_function, division
if so I agree. :-)

https://pypi.python.org/pypi/future

Kornel

[José Matos]

Post by Kornel Benko
At least prints could be rewritten with
import sys
...
sys.out.write(...)
or
output = open(...)
output.write(....)
No dependencies on python2 or python3.
Just my 2 cents.

But then it would not look like neither python 2 or python 3. It is not idiomatic in any version.

The print function support both cases in a more natural way:

In [1]: help (print)
...
print(...)
print(value, ..., sep=' ', end='\n', file=sys.stdout, flush=False)
...

[Kornel Benko]

Post by José Matos

Post by Kornel Benko
At least prints could be rewritten with
import sys
...
sys.out.write(...)
or
output = open(...)
output.write(....)
No dependencies on python2 or python3.
Just my 2 cents.

But then it would not look like neither python 2 or python 3. It is not idiomatic in any version.

I don't get it. It is not what we want?

Post by José Matos
In [1]: help (print)
...
print(...)
print(value, ..., sep=' ', end='\n', file=sys.stdout, flushúlse)
...

How works this code in python2?
Sorry for my apparent ignorance.

Kornel

[José Matos]

Post by Kornel Benko
How works this code in python2?
Sorry for my apparent ignorance.
Kornel

Adding to that file
from __future__ import print_function

this works since python 2.6