General new "high level" API discussions

[bdpedigo]

Just wanted to start a general thread about the new high-level/developer friendly API. I'm sure we can have other more specific threads.

[bdpedigo]

One question I have - is there anything in this new API that could not be thought of as "pipeline"? E.g. eats networkx graph and spits out some sort of info about each node, edge, graph, etc.

I'm wondering if one option would be to have a pipeline module (possibly with more submodules)

[daxpryce]

Honestly pipeline is sounding about the best I can think of at the moment. When I think about what I'm trying to achieve, it's:

• opinionated (I want to avoid having a graph be any number of datatypes, as a starting point - I'd feel differently about it if we had function/method overloading in Python, but we do not)
• strongly typed
• easiest graph datatype to prune/cut/ask questions of/iterate over
• fixed return types (zero Union[T, U, V] return types)

[daxpryce]

One thing I didn't write is that I want a very limited set of graph types and return embedding types that can be fed into other parts of the pipeline package without any other gentle massaging in the meantime. Quickly chain-able? I'm not sure the verbiage I really want to use, but something in that general vicinity of intent

[bdpedigo]

sure but the internals will end up being a conversation I think - since these will be the return types of the other, lower level ASE-type function, no?

if so, I'd fully support a "low-level" (for lack of a better name??) API that eats and spits out arrays (sparse or numpy) and a high-level API that eats networkx and spits... something as appropriate for the problem

[daxpryce]

I was going to suggest we make some private functions that bypass some of the overloady-checks since we'll have some guarantees that happen due to it being in networkx land. In fact, we may want to consider just not supporting networkx in the graspologic.embed module and point people toward pipeline if they want an networkxified flow (which incidentally should make it easier/better to get ase/lse/etc graduated to other libraries??)

which is basically what you're saying, right?

Re: appropriateness, one of the things we had as a type was an EmbeddingContainer before (we can call it whatever we want) - but the point is having a numpy 2d array return needs to have the labels returned too (or, arguably, zipped up into an ordered dict... or maybe ... hrm maybe it should behave as a dictionary but maintain the core arrays so it can be faster if we do any clustering over them. Anyway I'm thinking out loud here so I'll stop :D)

[bdpedigo]

In fact, we may want to consider just not supporting networkx in the graspologic.embed module and point people toward pipeline if they want an networkxified flow (which incidentally should make it easier/better to get ase/lse/etc graduated to other libraries??)

yes this is what I'm saying. If we could be more opinionated at the high-level and low-level api that could streamline a lot of code I'm guessing

one of the things we had as a type was an EmbeddingContainer before (we can call it whatever we want) - but the point is having a numpy 2d array return needs to have the labels returned too (or, arguably, zipped up into an ordered dict... or maybe ... hrm maybe it should behave as a dictionary but maintain the core arrays so it can be faster if we do any clustering over them.

While we're being opinionated my opinionated answer is (at least in the case of embedding-type things) the internal workflow is:

• immediately pull a nodelist
• immediately do graph -> some kind of matrix representation
• do the rest of the stuff...
• likely will end up with some kind of array
• right before return, bundle up the array/nodelist in some way that makes sense for the application. If it were up to me, would be a pandas dataframe where the index is the nodelist. I know some folks don't like pandas but it is literally a wrapper around a numpy array with support for arbitrary-keyed rows which is exactly what we have. We could fight about the exact output type. but regardless, just bundle up the data at the end.

This way, everything low-level just has to worry about arrays, and us as the developers will do the work internally of keeping track of which node is which

[daxpryce]

One of the things @nyecarr and I were talking about is how we think any API we make should be built around or after we run through a couple notebooks on our most common patterns. Things like the trimming the hairball approach (i.e. making cuts to optimize for modularity modulated by trying to keep as many edges and vertices as possible), vertex recommendations processes and whatnot.

pipeline as a name also could let us slide layouts.auto into it, since it's effectively a pipeline as well.

I think our current plan is going to be to pull a lot of the topologic api over (just the function names and types as a starting point, and only really for ase, lse, omni, (n2v???), maybe the io package (or something similar), etc).

[daxpryce]

it would probably be something more like:
graspologic.pipeline.embed.adjacency_spectral or something - subpackage and all that. Even if it's the same name, we're asking for it in a different package, and that should be ok - we're doing the same thing, just with a different entry point and set of parameters, yanno?

[bdpedigo]

maybe, im not sure how I feel, i think the name overlap could get confusing.

[bdpedigo]

I do like the "run through a couple notebooks idea". We talked about how we don't have enough examples on real data. One possible option is we start by adding these notebooks to an "Examples" section. That would give everyone the chance to look at and comment and also add to our docs. Then we could basically take that notebook, make it into a function, and drop it in at the end/beginning of the notebook and say "see look how easy this is in one line if you dont want to worry about the details"

[bdpedigo]

might mesh well with the book stuff @loftusa

[daxpryce]

Oh, it's going to be somewhat confusing no matter what, I think - we're providing two pointedly different calls to the same behaviors. As long as we don't ever advocate someone do from graspologic.pipeline.embed import adjacency_spectral_embed it should be clean enough - at lest for my comfort level (you can certainly disagree, and I'm open to other ideas!) I will say though, that when we do this, we should put something in the documentation spelling out how the two different APIs are meant to work, intended audiences, and expected usage patterns we're assigning to those audiences.

[daxpryce]

I also think we should consider breaking the utils module into a package and splitting it into multiple files, but just for code organization purposes (meaning it should still be graspologic.utils.do_a_thing() even if it's technically in graspologic/utils/connected_components.py. This isn't really a high level api discussion thing specifically, but I was just thinking about how we had a bunch of utilities we liked having too and that file is already stupidly large. Thoughts?

[bdpedigo]

there's no good reason they are all in one file, that's just where they are, but I'd be happy to see them split up.