Sparse to Dense

[brett1479]

In choice_calcs.py line 930, it appears the library is calling rows_to_obs.toarray() which converts a sparse array to a potentially huge dense array (for my use case, the resulting sparse array is small, and the dense version is about 200 GiB).
Here is the existing code:

weights_per_obs =\
        np.max(rows_to_obs.toarray() * weights[:, None], axis=0)

Is the intended behavior given below?

M = rows_to_obs.multiply(weights.reshape(-1,1))
weights_per_obs = np.max(M, axis=0).toarray().reshape(-1)

If so, I think this is a simple fix.

[mathijsvdv]

I have run into the exact same issue in my usage of Pylogit and had to work around it with a similar rewrite. I like your simple fix: it's very concise and will preserve the sparse matrix structure.

Perhaps you can create a Pull request where you implement this exact fix and have Timothy review it. Here, make sure to include the same fix in nested_choice_calcs.py on lines 569 and 746.

[friedertheo]

Hi @brett1479 and @mathijsvdv,

Does a similar trick also work for the mixed logit calculation? Because when estimating it, the dense array becomes super large. Using xlogit (or Stata's cmxtmixlogit) needs 1/20 of the memory pylogit requires. However, I need pylogit's feature for constrained optimization.

Any help is very much appreciated!