Is the user-dir/scripts folder added to PATH for converter script execution?

[Scott Kostyshak]

I've always used the full paths to scripts in converters, but I
interpret the Customization manual to suggest that I only need to use
the base name:

"Save it in your local LyX directory—say,
/home/you/.lyx/scripts/pdfcopier.sh—and make it executable... enter
pdfcopier.sh $$i $$o into the Copier field"

When I do that, LyX does not find the script (whether or not I reconfigure):

env: pdfcopier.sh: No such file or directory
support/Systemcall.cpp (275): Systemcall: 'pdfcopier.sh
"newfile1.tex"' finished with exit code 127
Error: Cannot view file
----------------------------------------
File does not exist: /tmp/lyx_tmpdir.oSJyNpB31268/lyx_tmpbuf0/newfile1.pdf

Further, the wiki page here:
http://wiki.lyx.org/LyX/UserDir
suggests that the "scripts" folder is "Only useful if you hack on [the
library] scripts."

Am I misinterpreting the Customization manual or following the
instructions wrong?

Scott

[Jean-Marc Lasgouttes]

Le 23/01/2014 06:27, Scott Kostyshak a écrit :
> I've always used the full paths to scripts in converters, but I
> interpret the Customization manual to suggest that I only need to use
> the base name:
>
> "Save it in your local LyX directory—say,
> /home/you/.lyx/scripts/pdfcopier.sh—and make it executable... enter
> pdfcopier.sh $$i $$o into the Copier field"

I thought that one had to use

$$s/pdfcopier.sh $$i $$o into the Copier field"

JMarc

[Scott Kostyshak]

On Thu, Jan 23, 2014 at 5:06 AM, Jean-Marc Lasgouttes
<***@lyx.org> wrote:
> Le 23/01/2014 06:27, Scott Kostyshak a écrit :
>
>> I've always used the full paths to scripts in converters, but I
>> interpret the Customization manual to suggest that I only need to use
>> the base name:
>>
>> "Save it in your local LyX directory—say,
>> /home/you/.lyx/scripts/pdfcopier.sh—and make it executable... enter
>> pdfcopier.sh $$i $$o into the Copier field"
>
>
> I thought that one had to use
>
> $$s/pdfcopier.sh $$i $$o into the Copier field"

$$s is for the system directory. So to reach those scripts one could
do $$s/scripts/. I don't believe there is a variable for the user
scripts directory and it is this directory that I understand the
Customization manual to suggest.

Scott

[Jean-Marc Lasgouttes]

Le 23/01/2014 17:22, Scott Kostyshak a écrit :
> $$s is for the system directory. So to reach those scripts one could
> do $$s/scripts/. I don't believe there is a variable for the user
> scripts directory and it is this directory that I understand the
> Customization manual to suggest.

Actually, the filettols.h file says:

/** Takes a command such as "python $$s/scripts/convertDefault.py
file.in file.out"
* and replaces "$$s/" with the path to the LyX support directory
containing
* this script. If the script is not found, "$$s/" is removed.
Executing the
* command will still fail, but the error message will make some sort of
* sense ;-)
*/
std::string const libScriptSearch(std::string const & command,
quote_style style = quote_shell);

$$s is not really a variable like the others, but a kind of macro. The
real syntax is actually $$s/foo/bar, and this searches for bar in
user_dir/foo and then system_dir/foo.

Therefore $$s/scripts/pdfcopier.sh is correct and finds the script
wherever it is.

JMarc

[Scott Kostyshak]

On Thu, Jan 23, 2014 at 11:32 AM, Jean-Marc Lasgouttes
<***@lyx.org> wrote:
> Le 23/01/2014 17:22, Scott Kostyshak a écrit :
>
>> $$s is for the system directory. So to reach those scripts one could
>> do $$s/scripts/. I don't believe there is a variable for the user
>> scripts directory and it is this directory that I understand the
>> Customization manual to suggest.
>
>
> Actually, the filettols.h file says:
>
> /** Takes a command such as "python $$s/scripts/convertDefault.py file.in
> file.out"
> * and replaces "$$s/" with the path to the LyX support directory
> containing
> * this script. If the script is not found, "$$s/" is removed. Executing
> the
> * command will still fail, but the error message will make some sort of
> * sense ;-)
> */
> std::string const libScriptSearch(std::string const & command,
> quote_style style = quote_shell);
>
> $$s is not really a variable like the others, but a kind of macro. The real
> syntax is actually $$s/foo/bar, and this searches for bar in user_dir/foo
> and then system_dir/foo.
>
> Therefore $$s/scripts/pdfcopier.sh is correct and finds the script wherever
> it is.

Ah now I'm starting to understand. Do you happen to know under which
cases $$s is interpreted? Putting the same copier command in different
converters leads to the $$s sometimes being substituted and sometimes
not. And contrary to the message you cited, the $$s/ is still included
for me when the path is not found (I put in a bogus name). So this
makes me think there's something that I'm still not understanding.

Scott

[Scott Kostyshak]

On Thu, Jan 23, 2014 at 12:14 PM, Scott Kostyshak <***@lyx.org> wrote:
> On Thu, Jan 23, 2014 at 11:32 AM, Jean-Marc Lasgouttes
> <***@lyx.org> wrote:
>> Le 23/01/2014 17:22, Scott Kostyshak a écrit :
>>
>>> $$s is for the system directory. So to reach those scripts one could
>>> do $$s/scripts/. I don't believe there is a variable for the user
>>> scripts directory and it is this directory that I understand the
>>> Customization manual to suggest.
>>
>>
>> Actually, the filettols.h file says:
>>
>> /** Takes a command such as "python $$s/scripts/convertDefault.py file.in
>> file.out"
>> * and replaces "$$s/" with the path to the LyX support directory
>> containing
>> * this script. If the script is not found, "$$s/" is removed. Executing
>> the
>> * command will still fail, but the error message will make some sort of
>> * sense ;-)
>> */
>> std::string const libScriptSearch(std::string const & command,
>> quote_style style = quote_shell);
>>
>> $$s is not really a variable like the others, but a kind of macro. The real
>> syntax is actually $$s/foo/bar, and this searches for bar in user_dir/foo
>> and then system_dir/foo.
>>
>> Therefore $$s/scripts/pdfcopier.sh is correct and finds the script wherever
>> it is.
>
> Ah now I'm starting to understand. Do you happen to know under which
> cases $$s is interpreted? Putting the same copier command in different
> converters leads to the $$s sometimes being substituted and sometimes
> not. And contrary to the message you cited, the $$s/ is still included
> for me when the path is not found (I put in a bogus name). So this
> makes me think there's something that I'm still not understanding.
>
> Scott

If latex=pdflatex is specified, the $$s is not substituted. The only
substitution is for $$i. This is not documented. I have a script that
I want to use to run pdflatex (and do other things like copy the log
file to a different directory). Because my script runs pdflatex,
specifying latex=pdflatex seems like the right thing to do. I'm
actually not sure what latex=pdflatex does. Does it tell LyX to make
sure that the converter is run as many times as necessary? I would
like to put this information in the customization manual, which
currently only says "This converter runs some form of LaTeX. This will
make LyX's LaTeX error logs available." I'm guessing it does more than
that.

The attached patch fixes the "problem" for me. I have no idea if it is
correct. Is there a reason not to do the substitution, i.e. do we want
to disallow user scripts when latex=pdflatex?

Also, it seems that now latex is not a flag and should be listed under
"The following three flags are not really flags at all because they
take an argument in the key = value format:"
Is that correct?

Scott

[Scott Kostyshak]

On Sat, Jan 25, 2014 at 1:18 PM, Scott Kostyshak <***@lyx.org> wrote:
> On Thu, Jan 23, 2014 at 12:14 PM, Scott Kostyshak <***@lyx.org> wrote:
>> On Thu, Jan 23, 2014 at 11:32 AM, Jean-Marc Lasgouttes
>> <***@lyx.org> wrote:
>>> Le 23/01/2014 17:22, Scott Kostyshak a écrit :
>>>
>>>> $$s is for the system directory. So to reach those scripts one could
>>>> do $$s/scripts/. I don't believe there is a variable for the user
>>>> scripts directory and it is this directory that I understand the
>>>> Customization manual to suggest.
>>>
>>>
>>> Actually, the filettols.h file says:
>>>
>>> /** Takes a command such as "python $$s/scripts/convertDefault.py file.in
>>> file.out"
>>> * and replaces "$$s/" with the path to the LyX support directory
>>> containing
>>> * this script. If the script is not found, "$$s/" is removed. Executing
>>> the
>>> * command will still fail, but the error message will make some sort of
>>> * sense ;-)
>>> */
>>> std::string const libScriptSearch(std::string const & command,
>>> quote_style style = quote_shell);
>>>
>>> $$s is not really a variable like the others, but a kind of macro. The real
>>> syntax is actually $$s/foo/bar, and this searches for bar in user_dir/foo
>>> and then system_dir/foo.
>>>
>>> Therefore $$s/scripts/pdfcopier.sh is correct and finds the script wherever
>>> it is.
>>
>> Ah now I'm starting to understand. Do you happen to know under which
>> cases $$s is interpreted? Putting the same copier command in different
>> converters leads to the $$s sometimes being substituted and sometimes
>> not. And contrary to the message you cited, the $$s/ is still included
>> for me when the path is not found (I put in a bogus name). So this
>> makes me think there's something that I'm still not understanding.
>>
>> Scott
>
> If latex=pdflatex is specified, the $$s is not substituted. The only
> substitution is for $$i. This is not documented. I have a script that
> I want to use to run pdflatex (and do other things like copy the log
> file to a different directory). Because my script runs pdflatex,
> specifying latex=pdflatex seems like the right thing to do. I'm
> actually not sure what latex=pdflatex does. Does it tell LyX to make
> sure that the converter is run as many times as necessary? I would
> like to put this information in the customization manual, which
> currently only says "This converter runs some form of LaTeX. This will
> make LyX's LaTeX error logs available." I'm guessing it does more than
> that.
>
> The attached patch fixes the "problem" for me. I have no idea if it is
> correct. Is there a reason not to do the substitution, i.e. do we want
> to disallow user scripts when latex=pdflatex?
>
> Also, it seems that now latex is not a flag and should be listed under
> "The following three flags are not really flags at all because they
> take an argument in the key = value format:"
> Is that correct?
>
> Scott

Is there any reason not to make any of the other substitutions also? I
would like to either fix this for 2.1 or fix the Customization manual.

Scott

[Jean-Marc Lasgouttes]

Le 27/01/14 19:07, Scott Kostyshak a écrit :
> Is there any reason not to make any of the other substitutions also? I
> would like to either fix this for 2.1 or fix the Customization manual.

I think there is no reason for not doing it, but in this case it should
be done with a centralized method if possible. This seems better than
replicating code in many places.

JMarc

[Scott Kostyshak]

On Tue, Jan 28, 2014 at 6:01 PM, Jean-Marc Lasgouttes
<***@lyx.org> wrote:
> Le 27/01/14 19:07, Scott Kostyshak a écrit :
>
>> Is there any reason not to make any of the other substitutions also? I
>> would like to either fix this for 2.1 or fix the Customization manual.
>
>
> I think there is no reason for not doing it, but in this case it should be
> done with a centralized method if possible. This seems better than
> replicating code in many places.
>
> JMarc

Sounds good. I will work on this. Scott

[Jean-Marc Lasgouttes]

Le 25/01/14 19:18, Scott Kostyshak a écrit :
> If latex=pdflatex is specified, the $$s is not substituted. The only
> substitution is for $$i. This is not documented. I have a script that
> I want to use to run pdflatex (and do other things like copy the log
> file to a different directory). Because my script runs pdflatex,
> specifying latex=pdflatex seems like the right thing to do. I'm
> actually not sure what latex=pdflatex does. Does it tell LyX to make
> sure that the converter is run as many times as necessary? I would
> like to put this information in the customization manual, which
> currently only says "This converter runs some form of LaTeX. This will
> make LyX's LaTeX error logs available." I'm guessing it does more than
> that.
>
> The attached patch fixes the "problem" for me. I have no idea if it is
> correct. Is there a reason not to do the substitution, i.e. do we want
> to disallow user scripts when latex=pdflatex?

The patch looks good. An alternative and more general solution could be
to o that in SystemCall::startscript(); so that all commands honor this
$$s/foo/bar notation. For example, it would make sense for the parselog
flag too.

> Also, it seems that now latex is not a flag and should be listed under
> "The following three flags are not really flags at all because they
> take an argument in the key = value format:"
> Is that correct?

Yes.

JMarc

[Scott Kostyshak]

On Tue, Jan 28, 2014 at 5:56 PM, Jean-Marc Lasgouttes
<***@lyx.org> wrote:
> Le 25/01/14 19:18, Scott Kostyshak a écrit :
>
>> If latex=pdflatex is specified, the $$s is not substituted. The only
>> substitution is for $$i. This is not documented. I have a script that
>> I want to use to run pdflatex (and do other things like copy the log
>> file to a different directory). Because my script runs pdflatex,
>> specifying latex=pdflatex seems like the right thing to do. I'm
>> actually not sure what latex=pdflatex does. Does it tell LyX to make
>> sure that the converter is run as many times as necessary? I would
>> like to put this information in the customization manual, which
>> currently only says "This converter runs some form of LaTeX. This will
>> make LyX's LaTeX error logs available." I'm guessing it does more than
>> that.
>>
>> The attached patch fixes the "problem" for me. I have no idea if it is
>> correct. Is there a reason not to do the substitution, i.e. do we want
>> to disallow user scripts when latex=pdflatex?
>
>
> The patch looks good. An alternative and more general solution could be to o
> that in SystemCall::startscript(); so that all commands honor this
> $$s/foo/bar notation. For example, it would make sense for the parselog flag
> too.

Makes sense. I was a little worried about performance from
libScriptSearch but I guess it is not an issue compared to the
overhead of running an external command anyway.

What about the substitutions of the other converter variables? Should
they also all go to SystemCall::startscript() ?

Scott

[Jean-Marc Lasgouttes]

Le 29/01/14 06:01, Scott Kostyshak a écrit :
> Makes sense. I was a little worried about performance from
> libScriptSearch but I guess it is not an issue compared to the
> overhead of running an external command anyway.
>
> What about the substitutions of the other converter variables? Should
> they also all go to SystemCall::startscript() ?

I do not think that you have the relevant information there. Actually,
doing all the variables at one place may not make sense: if one has to
pass lots of variables to one function, it may be as easy to just do the
conversions by hand.

JMarc

[Scott Kostyshak]

On Wed, Jan 29, 2014 at 3:07 AM, Jean-Marc Lasgouttes
<***@lyx.org> wrote:
> Le 29/01/14 06:01, Scott Kostyshak a écrit :
>
>> Makes sense. I was a little worried about performance from
>> libScriptSearch but I guess it is not an issue compared to the
>> overhead of running an external command anyway.
>>
>> What about the substitutions of the other converter variables? Should
>> they also all go to SystemCall::startscript() ?
>
>
> I do not think that you have the relevant information there. Actually, doing
> all the variables at one place may not make sense: if one has to pass lots
> of variables to one function, it may be as easy to just do the conversions
> by hand.

Ah, right. OK then, I will make a patch for using libScriptSearch in
SystemCall::startscript().

Scott

[Scott Kostyshak]

On Wed, Jan 29, 2014 at 3:37 AM, Scott Kostyshak <***@lyx.org> wrote:
> On Wed, Jan 29, 2014 at 3:07 AM, Jean-Marc Lasgouttes
> <***@lyx.org> wrote:
>> Le 29/01/14 06:01, Scott Kostyshak a écrit :
>>
>>> Makes sense. I was a little worried about performance from
>>> libScriptSearch but I guess it is not an issue compared to the
>>> overhead of running an external command anyway.
>>>
>>> What about the substitutions of the other converter variables? Should
>>> they also all go to SystemCall::startscript() ?
>>
>>
>> I do not think that you have the relevant information there. Actually, doing
>> all the variables at one place may not make sense: if one has to pass lots
>> of variables to one function, it may be as easy to just do the conversions
>> by hand.
>
> Ah, right. OK then, I will make a patch for using libScriptSearch in
> SystemCall::startscript().
>
> Scott

Attached is a patch. Note that I also made the change in
ForkedCall::startScript. I'm not confident in this patch and have only
tested it briefly.

Comments?

Scott

[Jean-Marc Lasgouttes]

31/01/2014 07:02, Scott Kostyshak:
> On Wed, Jan 29, 2014 at 3:37 AM, Scott Kostyshak <***@lyx.org> wrote:
>> Ah, right. OK then, I will make a patch for using libScriptSearch in
>> SystemCall::startscript().
>
> Attached is a patch. Note that I also made the change in
> ForkedCall::startScript. I'm not confident in this patch and have only
> tested it briefly.

Hello Scott,

The patch looks reasonable and I have only a couple of remarks

* the description of $$s in start[Ss]cript shall be made more precise.
The best is probably to mention that libScriptSearch is applied to the
command.

* the special code to handle python in Systemcall::startscript is
probably also of interest to ForkedCall::startScript. It may be good to
factor it out somewhere, or even to include it in libScriptSearch (which
should then be renamed).

In any case, this is probably not usable to 2.1 (maybe backported
later). I understand this was the precise reason why you wanted to do
that. In this case, a solution maybe to, in parallel, reserrect your old
simple patch to add libScriptSearch here and there.

JMarc

[Scott Kostyshak]

On Mon, Feb 3, 2014 at 6:42 AM, Jean-Marc Lasgouttes <***@lyx.org> wrote:
> 31/01/2014 07:02, Scott Kostyshak:
>>
>> On Wed, Jan 29, 2014 at 3:37 AM, Scott Kostyshak <***@lyx.org> wrote:
>>>
>>> Ah, right. OK then, I will make a patch for using libScriptSearch in
>>> SystemCall::startscript().
>>
>>
>> Attached is a patch. Note that I also made the change in
>> ForkedCall::startScript. I'm not confident in this patch and have only
>> tested it briefly.
>
>
> Hello Scott,
>
> The patch looks reasonable and I have only a couple of remarks
>
> * the description of $$s in start[Ss]cript shall be made more precise. The
> best is probably to mention that libScriptSearch is applied to the command.

Done in the first patch attached. I did not explain what
libScriptSearch does though (let me know if I should).

> * the special code to handle python in Systemcall::startscript is probably
> also of interest to ForkedCall::startScript. It may be good to factor it out
> somewhere, or even to include it in libScriptSearch (which should then be
> renamed).

Good idea. I put the quote-related python code in libScriptSearch in
the first patch.

I put the other python code (substitutions) in libScriptSearch in the
second patch. Do you have an idea for the new name of libScriptSearch
once this is done? I guess the unifying theme is substituting the
command. 'commandSub'? Or perhaps 'commandPrep' (preparing the command
to be run) ? I used 'commandPrep' in the patch.

> In any case, this is probably not usable to 2.1 (maybe backported later). I
> understand this was the precise reason why you wanted to do that. In this
> case, a solution maybe to, in parallel, reserrect your old simple patch to
> add libScriptSearch here and there.

Agreed that it's not for 2.1. Since no one has reported this before
and I guess it's been around for a while, I'm not worried. For 2.1 I'm
fine with my old simple patch or with modifying Customization.lyx to
mention this temporary bug (otherwise the example there does not
work). Which do you prefer?

Scott

[Scott Kostyshak]

On Mon, Feb 3, 2014 at 6:42 AM, Jean-Marc Lasgouttes <***@lyx.org> wrote:
> 31/01/2014 07:02, Scott Kostyshak:

> In any case, this is probably not usable to 2.1 (maybe backported later). I
> understand this was the precise reason why you wanted to do that. In this
> case, a solution maybe to, in parallel, reserrect your old simple patch to
> add libScriptSearch here and there.

Committed at 731b8610.

For the two general patches, I will plan to commit them after 2.1 is
released. Let me know if you have comments on them.

Scott

[Scott Kostyshak]

On Fri, Feb 21, 2014 at 12:57 PM, Scott Kostyshak <***@lyx.org> wrote:
> On Mon, Feb 3, 2014 at 6:42 AM, Jean-Marc Lasgouttes <***@lyx.org> wrote:
>> 31/01/2014 07:02, Scott Kostyshak:
>
>> In any case, this is probably not usable to 2.1 (maybe backported later). I
>> understand this was the precise reason why you wanted to do that. In this
>> case, a solution maybe to, in parallel, reserrect your old simple patch to
>> add libScriptSearch here and there.
>
> Committed at 731b8610.
>
> For the two general patches, I will plan to commit them after 2.1 is
> released. Let me know if you have comments on them.

Any other comments? If not I will commit the two patches to master
that are attached (they are the same as before except that I revert
the temporary fix at 731b8610 in one of them now).

Scott

[Jean-Marc Lasgouttes]

28/04/2014 01:57, Scott Kostyshak:
> Any other comments? If not I will commit the two patches to master
> that are attached (they are the same as before except that I revert
> the temporary fix at 731b8610 in one of them now).

I am not sure that I understand the second patch: what are the changes
in commandPrep wrt libScritpSearch exactly? How come that you did not
need to change the documentation of the function?

From what you write, it seems that the function is specialized for
python now. But we have other types of scripts, especially R scripts. Do
these still work?

JMarc

[Scott Kostyshak]

On Mon, Apr 28, 2014 at 5:44 AM, Jean-Marc Lasgouttes
<***@lyx.org> wrote:
> 28/04/2014 01:57, Scott Kostyshak:
>
> I am not sure that I understand the second patch: what are the changes in
> commandPrep wrt libScritpSearch exactly? How come that you did not need to
> change the documentation of the function?

The second patch was what I interpreted (and agreed with) what you wrote:

On Mon, Feb 3, 2014 at 6:42 AM, Jean-Marc Lasgouttes <***@lyx.org> wrote:
> * the special code to handle python in Systemcall::startscript is probably also of interest to ForkedCall::startScript. It may be good to factor it out somewhere, or even to include it in libScriptSearch (which should then be renamed).

The idea is that the command string is also parsed to look for python.
This parsing can be done all together (with the macro substitutions)
and that's why the function is renamed.

> From what you write, it seems that the function is specialized for python
> now. But we have other types of scripts, especially R scripts. Do these
> still work?

We do not parse looking for R commands in Systemcall::startscript as
we were doing for Python. The second patch should not change any
functionality, just organization (I will state this explicitly in the
commit message). Just in case, I tested exporting the knitr manual and
it exported successfully.

Scott

[Jean-Marc Lasgouttes]

29/04/2014 19:29, Scott Kostyshak:
> We do not parse looking for R commands in Systemcall::startscript as
> we were doing for Python. The second patch should not change any
> functionality, just organization (I will state this explicitly in the
> commit message). Just in case, I tested exporting the knitr manual and
> it exported successfully.

OK, then.

JMarc

[Scott Kostyshak]

On Mon, May 12, 2014 at 4:45 AM, Jean-Marc Lasgouttes
<***@lyx.org> wrote:
> 29/04/2014 19:29, Scott Kostyshak:
>
>> We do not parse looking for R commands in Systemcall::startscript as
>> we were doing for Python. The second patch should not change any
>> functionality, just organization (I will state this explicitly in the
>> commit message). Just in case, I tested exporting the knitr manual and
>> it exported successfully.
>
>
> OK, then.

Committed at 1821c6d8 and 8b66f9ce.

Scott