8000 Julia console support · Issue #98 · gpoore/pythontex · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

Julia console support #98

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
mykelk opened this issue Mar 23, 2017 · 28 comments
Closed

Julia console support #98

mykelk opened this issue Mar 23, 2017 · 28 comments

Comments

@mykelk
Copy link
mykelk commented Mar 23, 2017

What is involved in adding Julia console support? Is this part of the roadmap, or is it really tricky in the near term?

@gpoore
Copy link
Owner
gpoore commented Mar 24, 2017

General support for non-Python consoles will involve some reworking of the PythonTeX internals. Since PythonTeX is written in Python, and there's the code module, Python was a special (and easy) case. The main work, though, may relate to the individual languages. For Julia, I think there are at least three main options.

  • It might be possible to communicate with a Julia process via pipes, using Python's subprocess and some code to allow non-blocking communication. I had only marginal success with this when I tried last summer. Things may have improved. I was able to make some progress with the pipes approach with Ruby's irb, but my sense is that this approach will always be fragile, especially if anything beyond ASCII is involved.
  • The ideal case would be Julia code, probably based on the Julia REPL, that reads a file as if it were console input, and then outputs the complete result.
  • It should be possible to use Jupyter kernels to run code in many other languages. I'm interested in looking into that option eventually. However, that also may not do what you want, depending on how close you want your results to be to the real Julia REPL.

If someone can solve the problem of taking Julia code, processing it as console input, and then creating console-style output, then I think that adding PythonTeX support would be relatively quick and straightforward.

@mykelk
Copy link
Author
mykelk commented Mar 25, 2017

It looks like we have a solution to your second option here.

@gpoore
Copy link
Owner
gpoore commented Mar 25, 2017

That looks like a good option.

Getting this to work would involve creating a new set of templates in pythontex_engines.py for Julia console/terminal content, and then writing a wrapper around Weave.jl to get the output into the form PythonTeX needs. The wrapper could probably be Python or Julia. Feel free to make an attempt at that if you want.

I have several deadlines that will keep me busy for the next couple weeks, but I can probably make an attempt myself at some point after that. The work I'm finishing up over the next couple of weeks will actually lay the groundwork for redoing language support in PythonTeX. The goal is to make things much more extensible, and split all of the templates out of pythontex_engines.py to make things simpler. I want adding new language support (at least in non-console cases) to be as simple as dropping a few config files in a template directory.

@mpastell
Copy link

Whats the required output format for Pythontex? I can add it to Weave.jl if needed.

@mykelk
Copy link
Author
mykelk commented Mar 25, 2017

I can be patient for your refactor, @gpoore. Thanks for all your work on pythontex. It is a really great tool.

@gpoore
Copy link
Owner
gpoore commented Apr 14, 2017

I'm back to looking at this...the config format for redoing PythonTeX language support and eventually allowing some features to be used with Markdown is largely complete (https://bespon.org/).

@mpastell The script format with #+ term=true looks like it will be fine for an input format for Weave.jl. I think the tex output will work fine as well. PythonTeX stores some code output in cache files, and other output in LaTeX macros in detokenized form. It looks simple to parse the Weave.jl output into the appropriate form, using the \begin{juliaterm} and \end{juliaterm} delimiters.

Unless I'm overlooking something, there does appear to be one limitation in Weave.jl, compared to Python's code module for emulating an interactive console or terminal. With code, something like a syntax error doesn't stop code execution; you just get the error message that you would actually get in interactive mode. It looks like Weave.jl stops whenever it encounters a syntax error (and presumably other sorts of errors). In many cases, that probably would be the desired behavior, and it will be simple to parse the weave output (which is nice and informative) to provide a traceback into the tex source. However, there are cases when it might be convenient to write about errors, and be able to show an example of what they look like in interactive mode.

I'm starting into the PythonTeX refactor that should get this working, and will try to post some updates if it looks like that will be slow.

@mykelk
Copy link
Author
mykelk commented Apr 14, 2017

@gpoore Awesome. We're looking forward to this and will start using it in our book as soon as it is available.

@mykelk
Copy link
Author
mykelk commented May 24, 2017

How is the refactor going? Is help needed?

gpoore added a commit that referenced this issue May 25, 2017
@gpoore
Copy link
Owner
gpoore commented May 25, 2017

I've added a basic juliaconsole environment in the last commit.

This is built on top of the existing code-style environment, because I haven't had a chance yet to redo the code execution part of things. There isn't syntax highlighting yet, but I can add that easily as long as this is giving you what you need. There is a warning because of Weave.jl output messages, but I should be able to remove that. Otherwise, the main limitation I'm aware of is that errors aren't supported by Weave.jl, and if Weave.jl fails, PythonTeX isn't currently able to provide a traceback into the .tex source.

Sample document:

\documentclass{article}
\usepackage[usefamily=juliacon]{pythontex}
\begin{document}
text

\begin{juliaconsole}
x = 1:10
y = Dict(:language => "julia")
\end{juliaconsole}

\end{document}

@mpastell
Copy link

This sounds great! I will add support for errors in Weave.jl when I find the time,

@mykelk
Copy link
Author
mykelk commented Jun 6, 2017

Yep, this does what we need. Syntax highlighting would be great! Thanks!

@gpoore
Copy link
Owner
gpoore commented Jun 29, 2017

Syntax highlighting now works, and there are no longer incorrect error/warning messages.

You can set the highlighting style with something like this in the preamble:

\setpythontexpygopt[juliacon]{style=default}

The one current drawback is that changing highlighting style won't necessarily take effect unless code is modified.

@mykelk mykelk closed this as completed Jul 23, 2017
@mykelk
Copy link
Author
mykelk commented Jul 23, 2017

Just FYI for new users in the future, if you don't have Weave installed, you might get an error message like this:

Traceback (most recent call last): File "c:\texlive\2016\texmf-dist\scripts\pythontex\pythontex.py", line 62, in <module> pythontex.main() File "c:\texlive\2016\texmf-dist\scripts\pythontex\pythontex2.py", line 2716, in main do_multiprocessing(data, temp_data, old_data, engine_dict) File "c:\texlive\2016\texmf-dist\scripts\pythontex\pythontex2.py", line 1371, in do_multiprocessing result = task.get() File "C:\app\Miniconda\lib\multiprocessing\pool.py", line 567, in get raise self._value IOError: [Errno 2] No such file or directory: u'pythontex-files-test\\juliacon_default_default.tex'

To fix, just go into Julia and run Pkg.add("Weave").

@mykelk
Copy link
Author
mykelk commented Jul 24, 2017

When I run your example document:

\documentclass{article}
\usepackage[usefamily=juliacon]{pythontex}
\begin{document}
text

\begin{juliaconsole}
x = 1:10
y = Dict(:language => "julia")
\end{juliaconsole}

\end{document}

It seems to evaluate every line and return its output, with a blank line between each output. However, this seems to be different from how the Python console works (sec. 4.2.3 of pythontex manual). Is this intentional? I think it prevents us from using multiline commands. Am I missing something?

@gpoore
Copy link
Owner
gpoore commented Jul 24, 2017

@mykelk I'll add a note to the docs about installing Weave.jl.

In terms of the output, I believe the empty lines after all output are mimicking the Julia console exactly. When I run julia on the command line, and then try something like the example document, this is the output I get:

julia>  x = 1:10
1:10

julia>  y = Dict(:language => "julia")
Dict{Symbol,String} with 1 entry:
  :language => "julia"

Meanwhile, when I run python on the command line,

>>> x = range(1,10)
>>> x
range(1, 10)
>>> y = {"language": "python"}

This looks like there are just differences in how Julia and Python format their console output. There may be a way to modify that, but that would involve a Weave.jl setting.

Multiline Julia seems to work fine. For example, reformatting the sample document:

\documentclass{article}
\usepackage[usefamily=juliacon]{pythontex}
\begin{document}
text

\begin{juliaconsole}
x = 1:10
y = Dict(
         :language => "julia"
    )
\end{juliaconsole}

\end{document}

@mykelk
Copy link
Author
mykelk commented Jul 25, 2017

Ah, okay. Yes, it looks like multiline works. But, does the highlighting look like this for you?
image
Note that just the first line of the multiline is syntax highlighted.

@mlhetland
Copy link

It also seems that comments mess things up:

\begin{juliaconsole}
2*2
2^5
# This is a comment
2^5
\end{juliaconsole}

This yields:

julia> 2*2
4

julia> 2^5
32

julia> # This is a comment
2^5
32

@gpoore
Copy link
Owner
gpoore commented Jul 25, 2017

@mykelk This is what I have, with Pygments 2.2.0. The continuation lines seem to work fine. Maybe you have an older version of Pygments?

untitled

@mlhetland That looks like a limitation in Weave.jl.

@mlhetland
Copy link

Ah, OK. Thanks.
Re your highlighting example, here: It seems like your second parenthesis is gray?

@gpoore
Copy link
Owner
gpoore commented Jul 25, 2017

@mlhetland Yes, the second (closing) parenthesis is gray. I'm not sure if that's due to a limitation in Pygments when working with multiline elements, or (probably more likely) an indication that the Pygments Julia console lexer needs improvement.

@gpoore
Copy link
Owner
gpoore commented Jul 26, 2017

@mykelk All empty console lines can be removed by adding this to the preamble of the .tex document.

\makeatletter
\AtBeginEnvironment{juliaconsole}{%
  \def\FV@@PreProcessLine{%
    \expandafter\ifstrempty\expandafter{\FV@Line}{}{%
      \FV@StepLineNo
      \FV@Gobble
      \expandafter\FV@ProcessLine\expandafter{\FV@Line}}}%
}
\makeatother

This discards all empty lines in Julia console environments (doesn't check for julia> on the next line), so it shouldn't be used when console code or output contains literal empty lines. There would probably be a way to have it do the lookahead for julia> on the next line (let me know if you need that at some point), but this should cover the most typical cases. Eventually, I'll add something like this (remove empty lines) as an option for the fvextra package, so there will be built-in ways to do such things. Removing empty lines is a feature that has been requested in other contexts.

@mykelk
Copy link
Author
mykelk commented Jul 26, 2017

Oh, wow, that is exactly what I want!!! Thanks so much.

@nathancarter
Copy link
Contributor

Is this feature supposed to exist in pythontex 0.16? It is not working for me as follows.

Created this file, test.tex, as others have done, above:

\documentclass{article}

\usepackage[usefamily=juliacon]{pythontex}

\begin{document}

\begin{juliaconsole}
x = 1:10
y = Dict(:language => "julia")
\end{juliaconsole}

\end{document}

Ran pythontex on it, getting this output:

This is PythonTeX 0.16
Traceback (most recent call last):
  File "/Library/TeX/texbin/pythontex", line 62, in <module>
    pythontex.main()
  File "/usr/local/texlive/2018/texmf-dist/scripts/pythontex/pythontex2.py", line 2716, in main
    do_multiprocessing(data, temp_data, old_data, engine_dict)
  File "/usr/local/texlive/2018/texmf-dist/scripts/pythontex/pythontex2.py", line 1371, in do_multiprocessing
    result = task.get()
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/pool.py", line 558, in get
    raise self._value
IOError: [Errno 2] No such file or directory: 'pythontex-files-test/juliacon_default_default.tex'

If I do usefamily=julia instead, then the juliaconsole environment isn't recognized.

@mykelk
Copy link
Author
mykelk commented Sep 10, 2018

Which version of Julia are you using? I don't think Julia 1.0 is supported yet.

@nathancarter
Copy link
Contributor
$ julia --version
julia version 0.7.0

@mykelk
Copy link
Author
mykelk commented Sep 10, 2018

What does the juliacon_default_default.err file say in the pythontex-files-test folder that is generated?

@mykelk
Copy link
Author
mykelk commented Sep 11, 2018

@nathancarter Do you have Weave installed? I was able to get it working after it was installed.

@nathancarter
Copy link
Contributor

That solved it. My apologies if that's documented somewhere and I missed it. I came to this feature through this Stack Exchange question and answer and thus didn't read too many docs. Thanks for the help!!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

5 participants
0