Bug where undefined control sequence errors are ignored if they appear after certain math formulas

[Yoni]

Hi LyX developers,

I've encountered this bug in LyX 2.0.6 (on Windows 8.1), and it still
exists as of LyX 2.1.2.

After playing around with macros a little bit, I noticed that sometimes I
would rename a macro (i.e. from \mymacro to \myothermacro) and rebuild the
pdf, I would not get errors about \mymacro being unrecognized. (The control
is simply ignored.) I would get the error in a new document but not in the
document I was working on.

After trying to place invalid control sequences in several places in the
file I noticed that if I put it in the beginning of the document, these
errors are raised, but in the ending, they don't get raised. By doing a
"binary search" I pinpointed an exact point after which these errors are
suppressed.

I don't understand the conditions that ensure this, but I managed to create
a small example that reproduces it:

LyX 2.1.2 on Windows 8.1 default install, default document settings, create
a new document and write two lines:

The first line has "blah blah blah blah blah blah blah blah blah blah"
(must be exactly 10 blahs) and then the
formula $B^{d+1}\left(0,\frac{r}{\sqrt{d}}\right)\subset\left[-\frac{r}{\sqrt{d}},\frac{r}{\sqrt{d}}\right]^{d+1}$

In the second line, just put an invalid control sequence like "\blah".

At least on my computer, compiling this won't raise an error. If you remove
one of the blahs or change the formula too much, you'll get the error again
though. I think part of the contributing factors for the bug is that the
formula appears on the far right side of the line.

I am attaching a file that has the above contents.

Yoni

[Richard Heck]

Post by Yoni
Hi LyX developers,
I've encountered this bug in LyX 2.0.6 (on Windows 8.1), and it still
exists as of LyX 2.1.2.
After playing around with macros a little bit, I noticed that
sometimes I would rename a macro (i.e. from \mymacro to \myothermacro)
and rebuild the pdf, I would not get errors about \mymacro being
unrecognized. (The control is simply ignored.) I would get the error
in a new document but not in the document I was working on.
After trying to place invalid control sequences in several places in
the file I noticed that if I put it in the beginning of the document,
these errors are raised, but in the ending, they don't get raised. By
doing a "binary search" I pinpointed an exact point after which these
errors are suppressed.
I don't understand the conditions that ensure this, but I managed to
LyX 2.1.2 on Windows 8.1 default install, default document settings,
The first line has "blah blah blah blah blah blah blah blah blah blah"
(must be exactly 10 blahs) and then the
formula $B^{d+1}\left(0,\frac{r}{\sqrt{d}}\right)\subset\left[-\frac{r}{\sqrt{d}},\frac{r}{\sqrt{d}}\right]^{d+1}$
In the second line, just put an invalid control sequence like "\blah".
At least on my computer, compiling this won't raise an error. If you
remove one of the blahs or change the formula too much, you'll get the
error again though. I think part of the contributing factors for the
bug is that the formula appears on the far right side of the line.
I am attaching a file that has the above contents.

Hmm. That is very weird. I do get an error here (Fedora 20).

Richard

[Yoni]

If you remove the bad control and view with pdflatex, do you have the
formula appear at the very end of the line without breaking? That's how it
appears for me. I think this is needed for the bug to occur and that's why
I need exactly 10 blahs.

Maybe on your Fedora the fonts are slightly different and they pull or push
the formula? Try adding or removing one or two blahs. If it doesn't
reproduce, maybe it's Windows only somehow, or only my computer.

If you want me to try something, let me know.

Post by Richard Heck

Post by Yoni
Hi LyX developers,
I've encountered this bug in LyX 2.0.6 (on Windows 8.1), and it still
exists as of LyX 2.1.2.
After playing around with macros a little bit, I noticed that sometimes I
would rename a macro (i.e. from \mymacro to \myothermacro) and rebuild the
pdf, I would not get errors about \mymacro being unrecognized. (The control
is simply ignored.) I would get the error in a new document but not in the
document I was working on.
After trying to place invalid control sequences in several places in the
file I noticed that if I put it in the beginning of the document, these
errors are raised, but in the ending, they don't get raised. By doing a
"binary search" I pinpointed an exact point after which these errors are
suppressed.
I don't understand the conditions that ensure this, but I managed to
LyX 2.1.2 on Windows 8.1 default install, default document settings,
The first line has "blah blah blah blah blah blah blah blah blah blah"
(must be exactly 10 blahs) and then the formula $B^{d+1}\left(0,\frac{r}{\
sqrt{d}}\right)\subset\left[-\frac{r}{\sqrt{d}},\frac{r}{\
sqrt{d}}\right]^{d+1}$
In the second line, just put an invalid control sequence like "\blah".
At least on my computer, compiling this won't raise an error. If you
remove one of the blahs or change the formula too much, you'll get the
error again though. I think part of the contributing factors for the bug is
that the formula appears on the far right side of the line.
I am attaching a file that has the above contents.

Hmm. That is very weird. I do get an error here (Fedora 20).
Richard

[Richard Heck]

Post by Yoni
If you remove the bad control and view with pdflatex, do you have the
formula appear at the very end of the line without breaking? That's
how it appears for me. I think this is needed for the bug to occur and
that's why I need exactly 10 blahs.

No effect.

Post by Yoni
Maybe on your Fedora the fonts are slightly different and they pull or
push the formula? Try adding or removing one or two blahs. If it
doesn't reproduce, maybe it's Windows only somehow, or only my computer.

I'm guessing it has to do with our having different TeX installations.
MikTeX for you, I believe, but TeXLive here.

If you look at Document> LaTeX Log after compilation, are there errors
reported there? You'll need to scroll through it all, probably.

Richard

PS Try to post inline, as it makes the discussion easier for other
people to follow.

Post by Yoni
If you want me to try something, let me know.
Hi LyX developers,
I've encountered this bug in LyX 2.0.6 (on Windows 8.1), and
it still exists as of LyX 2.1.2.
After playing around with macros a little bit, I noticed that
sometimes I would rename a macro (i.e. from \mymacro to
\myothermacro) and rebuild the pdf, I would not get errors
about \mymacro being unrecognized. (The control is simply
ignored.) I would get the error in a new document but not in
the document I was working on.
After trying to place invalid control sequences in several
places in the file I noticed that if I put it in the beginning
of the document, these errors are raised, but in the ending,
they don't get raised. By doing a "binary search" I pinpointed
an exact point after which these errors are suppressed.
I don't understand the conditions that ensure this, but I
LyX 2.1.2 on Windows 8.1 default install, default document
The first line has "blah blah blah blah blah blah blah blah
blah blah" (must be exactly 10 blahs) and then the formula
$B^{d+1}\left(0,\frac{r}{\sqrt{d}}\right)\subset\left[-\frac{r}{\sqrt{d}},\frac{r}{\sqrt{d}}\right]^{d+1}$
In the second line, just put an invalid control sequence like "\blah".
At least on my computer, compiling this won't raise an error.
If you remove one of the blahs or change the formula too much,
you'll get the error again though. I think part of the
contributing factors for the bug is that the formula appears
on the far right side of the line.
I am attaching a file that has the above contents.
Hmm. That is very weird. I do get an error here (Fedora 20).
Richard

[Yoni]

On 2 October 2014 00:50, Richard Heck <***@lyx.org> wrote:

I'm guessing it has to do with our having different TeX installations.

Post by Richard Heck
MikTeX for you, I believe, but TeXLive here.

Yep, MiKTeX it is.

Post by Richard Heck
If you look at Document> LaTeX Log after compilation, are there errors
reported there? You'll need to scroll through it all, probably.

The only thing out of the ordinary in the log is at the very end - there is
this message, which isn't treated as an error (nor as a warning) by LyX:

Overfull \hbox (16.17464pt too wide) in paragraph at lines 10--11
[]\T1/cmr/m/n/10 blah blah blah blah blah blah blah blah blah blah
$\OML/cmm/m/
it/10 B[] [] \OMS/cmsy/m/n/10

This appears regardless of whether I add an invalid control sequence
afterwards. But if I remove the formula or remove a "blah", then this
overfull message disappears, and invalid control sequence errors appear
again. So the problem appears to be related to this overfull \hbox
condition.

Post by Richard Heck
PS Try to post inline, as it makes the discussion easier for other people
to follow.

Indeed. I don't use mailing lists much :)

[Yoni]

rather than the usual summary of TeX memory usage, so something is still

not right, but a pdf is produced nonetheless.

Not only the usual summary is not displayed, but since errors are ignored
and a pdf is made, there is really no way for the user to notice if
something is wrong with the compilation (maybe a macro was incorrectly used
etc.) except by noticing it by chance.

[aparsloe]

Post by Richard Heck

Post by Yoni
Hi LyX developers,
I've encountered this bug in LyX 2.0.6 (on Windows 8.1), and it still
exists as of LyX 2.1.2.
After playing around with macros a little bit, I noticed that
sometimes I would rename a macro (i.e. from \mymacro to
\myothermacro) and rebuild the pdf, I would not get errors about
\mymacro being unrecognized. (The control is simply ignored.) I would
get the error in a new document but not in the document I was working
on.
After trying to place invalid control sequences in several places in
the file I noticed that if I put it in the beginning of the document,
these errors are raised, but in the ending, they don't get raised. By
doing a "binary search" I pinpointed an exact point after which these
errors are suppressed.
I don't understand the conditions that ensure this, but I managed to
LyX 2.1.2 on Windows 8.1 default install, default document settings,
The first line has "blah blah blah blah blah blah blah blah blah
blah" (must be exactly 10 blahs) and then the formula
$B^{d+1}\left(0,\frac{r}{\sqrt{d}}\right)\subset\left[-\frac{r}{\sqrt{d}},\frac{r}{\sqrt{d}}\right]^{d+1}$
In the second line, just put an invalid control sequence like "\blah".
At least on my computer, compiling this won't raise an error. If you
remove one of the blahs or change the formula too much, you'll get
the error again though. I think part of the contributing factors for
the bug is that the formula appears on the far right side of the line.
I am attaching a file that has the above contents.

Hmm. That is very weird. I do get an error here (Fedora 20).
Richard

On 64-bit Windows 7 & MiKTeX 2.9, I get the error, but I have to add an
11-th blah to the first line to trigger it. The latex log has

LaTeX Font Info: External font `cmex10' loaded for size
(Font) <5> on input line 19.
! Undefined control sequence.
l.21 \blah

If I remove the 11-th blah, the document compiles, but the latex log is
incomplete. It ends with

LaTeX Font Info: External font `cmex10' loaded for size
(Font) <5> on input line 12.

Overfull \hbox (16.17464pt too wide) in paragraph at lines 12--13
[]\T1/cmr/m/n/10 blah blah blah blah blah blah blah blah blah blah
$\OML/cmm/m/
it/10 B[] [] \OMS/cmsy/m/n/10

rather than the usual summary of TeX memory usage, so something is
still not right, but a pdf is produced nonetheless.

Andrew

[Yoni]

Hi everyone,

I played around some more and I found what the bug is, more or less.

After compiling, I took a look at the temp directory where LyX was working.
In my case this was C:\Temp\lyx_tmpdir.aORPZMkU6924\lyx_tmpbuf0

There is a log file in there (generated presumably by pdflatex). At the end

Post by Yoni
Overfull \hbox (16.17464pt too wide) in paragraph at lines 10--11
[]\T1/cmr/m/n/10 blah blah blah blah blah blah blah blah blah blah

$\OML/cmm/m/

Post by Yoni
it/10 B[] [] \OMS/cmsy/m/n/10 [][]$
[]
! Undefined control sequence.
l.12 blah \blah
blahblah
The control sequence at the end of the top line
of your error message was never \def'ed. If you have
misspelled it (e.g., `\hobx'), type `I' and the correct
spelling (e.g., `I\hbox'). Otherwise just continue,
and I'll forget about whatever was undefined.

And then the usual memory information dump.

Post by Yoni
Overfull \hbox (16.17464pt too wide) in paragraph at lines 10--11
[]\T1/cmr/m/n/10 blah blah blah blah blah blah blah blah blah blah

$\OML/cmm/m/

Post by Yoni
it/10 B[] [] \OMS/cmsy/m/n/10

And that's it.


Notice how the last line LyX sees is cut in the middle. In fact for some
reason there is an EOF character between the "/m/n/10" and the "[][]$"! It
probably didn't make it into this email, but it's there in the log. Sure
enough, viewing this log with a hex editor reveals a 0x1A byte, like this:


".../cmsy/m/n/10 \x1a [][]$ \r\n []....."


If I run pdflatex directly on a .tex file that LyX exports, again - it
generates this EOF character in the log!


I have no idea why pdftex inserted an EOF character in there (maybe it's a
bug in pdflatex or maybe it's on purpose), but clearly LyX stops reading at
the EOF, and in my opinion, this is a bug in LyX.


I suggest to patch LyX so it opens the log file in binary reading mode
instead of text reading mode, so it doesn't stop at EOF characters.


Yoni