conjure.remote.stdio2: Rebuilding the foundation of stdio support

[Olical]

This is sort of in response to an email from @russtoku, so thank you very much for prompting the thought on this! Hopefully writing down my thoughts in a public way like this will prove beneficial to a few people.

The current conjure.remote.stdio

Clients that rely on interacting with processes over stdio all share a dependency on conjure.remote.stdio, it's a generic wrapper of the libuv process management functions that allows clients to start, manage and talk to a sub process. It uses some pattern matching (Lua lacks regular expressions but it does have "patterns", worth getting familiar with it) to work out when the REPL has sent a "prompt" to stdout, signifying that it's completed something and / or is ready to accept input.

So if we want to wait until the REPL prints $> , we'd configure the prompt pattern for the stdio remote to look for that, then we can send code to it and the system will extract the text between the $> prompts as the output of the command. Let's look at the following interaction:

$> (+ 10 20)
30
$> 

Conjure sent the (+ 10 20) and now it spots the $> . It's been storing all output from when it last sent an input, so when it sees the next prompt it cuts off the result string, prints it and resets it. This is extremely simplistic and doesn't account for a bunch of things in different REPLs.

It assumes your REPL only ever prints stdout / eval results and a prompt of some kind that remains fairly stable. That's a big assumption! And they say to assume is to make an ass out of u and me.

The desired future of conjure.remote.stdio2

So, this is too simplistic and tightly coupled to stdio. It's tightly coupled to the idea of a REPL with a prompt and some sort of user -> system -> user -> system only flow. Anything that deviates from that just won't work or will be very wonky. So, let's split these ideas up a little:

• IO adaptors, we want something that abstracts away stdio, sockets, potentially HTTP, who knows! Just something that turns something into an interface that supports subscription to their outputs and publishing to their inputs. This may well just be the libuv socket interface it that works for network sockets as well as sub-process stdio in the exact same way.
  • stdio
  • socket file
  • network socket
• Stateful helpers. We need a wrapper around this IO layer or maybe a library of functions you combine with it that allows you to store values going in and out of the IO side with functions that fire when certain strings fly past with hooks that allow you to say "hey! That's a result!". So some helpers that allow you to manage the state of an IO stream and it's various meanings from it's outputs.
  • We'd probably only have one of these, but we could implement different versions if required. It just takes some IO layer as an input and provides a more stateful higher level interface with extension points. On it's own, it does nothing, but it contains very common things used in the next layer:
• Language / REPL specific configuration! This is where the client wraps up it's own knowledge of it's REPL. So it knows what markers to look for when some code was evaluated and the result was displayed. It can tell if the REPL has hung (i.e., some input was sent but nothing else came out for a while... maybe emit a warning?).

So we'd have the IO separate from the stateful side and helpers separate from the domain specific ideas of each individual REPL and language. Specialisations here would include things like how some Lisps drop into a debug interface on error, so if you invoke something it doesn't like it will display a DIFFERENT prompt that indicates that you can now inspect the state of the REPL to debug what happened.

They essentially default to a stepping debugger when something goes wrong, we can't support that right now but this IO/state/domain specific separation would allow us to express that pretty well. We can't just view REPLs as "all you need is the last line", we need to store some scroll back and inspect it when new lines arrive to build context.

Conclusion

So I want to separate the IO from the parsing logic and I want to make the REPLs more stateful (in a contained way) relying on shared tooling keep code to a minimum. Some REPLs that are simpler and really just need that original REPL prompt pattern style from before could share a module with similar ideas. Common Lisp could build on this idea and add support for the debugger side.

I haven't nailed down that stateful layer or shared functions just yet. That's something that will have to remain simple, open and flexible. It'll be built out for one implementation first while leaving room to expand for languages that require more tools and ideas.

I want to see what others think before I really commit to this sort of architecture and would appreciate help in implementing this sort of things for the various clients we have. Feedback is much appreciated. Thanks!

[Olical]

This is actually wider than stdio, it's about laying out common interface layers to get everything to plug together like lego instead of something bespoke each time. Here's the rough structure I'm thinking of.

[russtoku]

An Observation of a stdio-based Client

While working on a Conjure client for SQL REPLs like PostgreSQL's psql, I had to modify the next-in-queue function in fnl/conjure/remote/stdio.fnlto get the client to work. There was never any prompt returned from psql; only the response to a query or meta-command. I think this is because psql is being run "headless" in the sub-process spawned by libuv (:help luv in Neovim). See #511.

The really interesting thing about psql is that it has a \watch meta-command that executes a query after an interval of time passes (default is 2 seconds). This switches the interaction to a more asynchronous mode in the sense that responses are returned without a request being made. To stop \watch , you send an interrupt signal (typically Ctl-C in a terminal) to psql. This is like the Linux watch command.

This reinforces the problem pointed out in The desired future of conjure.remote.stdio2 section above:

So, this is too simplistic and tightly coupled to stdio. It's tightly coupled to the idea of a REPL with a prompt and some sort of user -> system -> user -> system only flow. Anything that deviates from that just won't work or will be very wonky.

The nrepl docs has a nice comparison table of alternatives. The description of the Clojure socket REPL sounds like the stdio-based client (REPL in sub-process); synchronous request and response.

What Problem Are We Trying to Solve?

As the nrepl doc on alternatives points out, one of the main problems that nrepl tries to solve is distinguishing output (like from a print statement) from return value (like from evaluating a form or calling a function). This is also a problem that Conjure clients need to solve. The challenge is how to make that easier for stdio-based REPLs.

How Do Some REPLs Behave?

When using a programming language's REPL (also known as an interpreter), it's not clear what the difference is. For a non-Lisp language like Python, the REPL doesn't make a distinction. The standard Python interpreter responds the same with eveything (printed output, return value, and error message). In contrast, the iPython interpreter tags printed output and return value with Out[n] but errors are not tagged and there is no Out[n] response.

Here are some examples:

• Standard Python interpreter

>>> import pandas

>>> mydata = {
...   'cars': ["BMW", "Volvo", "Ford"],
...   'passings': [3, 7, 2]
... }

>>> type(mydata)
<class 'dict'>

>>> mydata
{'cars': ['BMW', 'Volvo', 'Ford'], 'passings': [3, 7, 2]}

>>> str(mydata)
"{'cars': ['BMW', 'Volvo', 'Ford'], 'passings': [3, 7, 2]}"

>>> repr(mydata)
"{'cars': ['BMW', 'Volvo', 'Ford'], 'passings': [3, 7, 2]}"

>>> df = pandas.DataFrame(mydata)

>>> type(df)
<class 'pandas.core.frame.DataFrame'>

>>> df
    cars  passings
0    BMW         3
1  Volvo         7
2   Ford         2

>>> str(mydata)
"{'cars': ['BMW', 'Volvo', 'Ford'], 'passings': [3, 7, 2]}"

>>> repr(mydata)
"{'cars': ['BMW', 'Volvo', 'Ford'], 'passings': [3, 7, 2]}"

>>> undef
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>

>>> def undef:
  File "<stdin>", line 1
    def undef:
             ^
SyntaxError: expected '('

• iPython interpreter

In [1]: import pandas

In [2]: mydata = {
   ...:   'cars': ["BMW", "Volvo", "Ford"],
   ...:   'passings': [3, 7, 2]
   ...:   }

In [3]: type(mydata)
Out[3]: dict

In [4]: mydata
Out[4]: {'cars': ['BMW', 'Volvo', 'Ford'], 'passings': [3, 7, 2]}

In [5]: str(mydata)
Out[5]: "{'cars': ['BMW', 'Volvo', 'Ford'], 'passings': [3, 7, 2]}"

In [6]: repr(mydata)
Out[6]: "{'cars': ['BMW', 'Volvo', 'Ford'], 'passings': [3, 7, 2]}"

In [7]: df = pandas.DataFrame(mydata)

In [8]: type(df)
Out[8]: pandas.core.frame.DataFrame

In [9]: df
Out[9]: 
    cars  passings
0    BMW         3
1  Volvo         7
2   Ford         2

In [10]: str(df)
Out[10]: '    cars  passings\n0    BMW         3\n1  Volvo         7\n2   Ford         2'

In [11]: repr(df)
Out[11]: '    cars  passings\n0    BMW         3\n1  Volvo         7\n2   Ford         2'

In [12]: undef
---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
Cell In[12], line 1
----> 1 undef

NameError: name 'undef' is not defined

In [13]: def undef:
   ...: 
   ...: 
  Cell In[13], line 1
    def undef:
             ^
SyntaxError: expected '('

So, an automated client must use context (what was sent to the REPL) to make that distinction.

BUT maybe there doesn't need to be any distinction. I chose to not make any distinction between printed output and return value for a Conjure SQL client. All output from psql is treated like a return value so you'll see it in the buffer that you're editing.

Here's an example of editing an SQL file. The inserted output is truncated by the width of the window so you get some idea of what's going on. To see the full output, you can open the log buffer (a very nice Conjure feature).

SELECT * FROM customers; => id | name | address | city | state | zip_code | coun

bad statement; => ERROR: syntax error at or near "bad" LINE 1: bad statement; ^

\dt => List of relations Schema | Name | Type | Owner --------+-----------+-----

[pyrmont]

Hi 👋 I realise it's been a while but @sogaiu directed me to this discussion as I collaborate with him on an improved Conjure client for Janet. For background, the client I'm considering would use the proto-repl protocol over stdio.

I think Conjure should offer:

1. IO libraries for various IO interfaces
2. a single messaging library that abstracts the IO libraries
3. protocol libraries for various protocols

The protocol libraries would be built on the messaging library which would be built on the interface libraries.

This would allow a client author to choose the level appropriate for their client. If you need to control the IO directly, you can build on top of the IO library. If you're trying to implement an unsupported protocol, you'd use the messaging library but handle the protocol encoding and decoding in your client. If you're making a client that uses a supported protocol, you've got it the easiest: just use the respective library and away you go.

So, for example, what I'm looking for is a very simple library that, based on a configuration table, (1) starts my REPL process and hooks up the IO, (2) sets a general response callback and (3) returns a 'REPL object' I can use for calling a send function. Since the protocol I want to implement is stateless, I don't need (or want) the library to try to match messages with callbacks. Just call my response callback with every message you get and I'll handle interpreting them.