Program Listing environment to include non-ASCII characters

[Stratos kovalkov Papadoudis]

Modern source authoring supports non-ASCII text, appearing in source
comments and/or output text.

EXAMPLES OF USAGE

- box initial program description with box drawing characters isntead of a
spam of ASCII symbols and/or characters:

┌─┬─┐╒═╤═╕╓─╥─╖╔═╦═╗ +-+-+
│ │ ││ │ │║ ║ ║║ ║ ║ | | |
├─┼─┤╞═╪═╡╟─╫─╢╠═╬═╣ instead of +-+-+
│ │ ││ │ │║ ║ ║║ ║ ║ | | |
└─┴─┘╘═╧═╛╙─╨─╜╚═╩═╝ +-+-+

- using the same box drawing to format program output nicely.

- use non-latin characters to output symbol closer to the mathematical
equivalent: scalar field strength φ=0.100Ε+001, with 'φ' instead of 'phi' or
other

The last reason appears to be most common, but the other ones are not
insignificant. Imagine for example how much clearer git's commit tree with
box drawing instead of ASCII will be on stdout.

Now what does git code in a lyx file? Probably nothing, but whenever a
programmer wishes to list their code in their dissertation for example, they
cannot do so without LyX deleting all the non-ASCII characters. (like it
happened to me)

Is there a really serious limitation that prevents lyx from outputting
program listings in UTF-8? If there is real none, is there room to implement
this enhancement in the near future?

I did not post a ticket for this yet due to it's ambiguous nature.

[Guenter Milde]

Post by Stratos kovalkov Papadoudis
Modern source authoring supports non-ASCII text, appearing in source
comments and/or output text.

...

Post by Stratos kovalkov Papadoudis
Is there a really serious limitation that prevents lyx from outputting
program listings in UTF-8? If there is real none, is there room to
implement this enhancement in the near future?

The limitation is in the "listings" package http://www.ctan.org/pkg/listings
which is not 8-bit-clean.

This is outside the scope of LyX development.
As this is a really complex package, there is no easy fix -- if you (or some
other) would volunteer to "upgrade" the listings package to full Unicode
support, this would be welcomed not only by LyX users but the whole
LaTeX community.

As a workaround: If you are prepared to sacrifice syntax highlighting,
you could use the LyXCode style or Insert>File>sub-document>unformatted
for progam insets. With Xe/LuaTeX, this should give you the chance to use
non-ASCII characters (of course, the font(s) must support them, with
Xe/LuaTeX, you don't get errors for missing characters, check the output).

Günter

[Stratos kovalkov Papadoudis]

Thank you for your reply. As a matter of fact I did discover that it was a
Tex issue in the past and forgot all about it, you are right. I am willing to
work on this in the future, when my schedule lightens a bit, as even still
using non-ASCII characters is still very odd community-wise; my colaborators
where kinda wondering, and sometimes I even wonder myself. But those points
are valid still, especially about non-latin characters in program output.

Thanks again, once I have something ready to commit, I'll let the group know.

Trent

[Guenter Milde]

Post by Stratos kovalkov Papadoudis
Thank you for your reply. As a matter of fact I did discover that it was a
Tex issue in the past and forgot all about it, you are right. I am willing to
work on this in the future, when my schedule lightens a bit, as even still
using non-ASCII characters is still very odd community-wise; my colaborators
where kinda wondering, and sometimes I even wonder myself. But those points
are valid still, especially about non-latin characters in program output.

Interesting alternatives could be
http://www.ctan.org/tex-archive/macros/latex/contrib/texments/
and
http://www.ctan.org/tex-archive/macros/latex/contrib/pythontex
as both use the Pygments_ syntax highlighter.
While I don't know, whether these packages support the full Unicode range, I
know that Pygments does, so it may be easier to either just use of fix one
of them (if you have Python instlled, but this is always the case if you use
LyX).

A plan could be:

* test both packages with LaTeX, decide which better fits the needs

* test with LyX (using ERT, user preamble, ...)

* maybe provide a "LyX module" supporting syntax highlight via Pygments
Document it and upload to the wiki.

* in addition or alternatively, add a ticket for LyX support for one of the
packages (the better one).

Günter

[Guenter Milde]

Post by Guenter Milde

Post by Stratos kovalkov Papadoudis
Thank you for your reply. As a matter of fact I did discover that it was a
Tex issue in the past and forgot all about it, you are right. I am willing to
work on this in the future, when my schedule lightens a bit, as even still
using non-ASCII characters is still very odd community-wise; my colaborators
where kinda wondering, and sometimes I even wonder myself. But those points
are valid still, especially about non-latin characters in program output.

Interesting alternatives could be
http://www.ctan.org/tex-archive/macros/latex/contrib/texments/
and
http://www.ctan.org/tex-archive/macros/latex/contrib/pythontex
as both use the Pygments_ syntax highlighter.

There is also the new package PygmenTeX:

Location on CTAN: tex-archive/macros/latex/contrib/pygmentex/

Post by Guenter Milde
While I don't know, whether these packages support the full Unicode range, I
know that Pygments does, so it may be easier to either just use of fix one
of them (if you have Python instlled, but this is always the case if you use
LyX).
* test both packages with LaTeX, decide which better fits the needs
* test with LyX (using ERT, user preamble, ...)
* maybe provide a "LyX module" supporting syntax highlight via Pygments
Document it and upload to the wiki.
* in addition or alternatively, add a ticket for LyX support for one of the
packages (the better one).
Günter