Returning Pauli or SparsePauliOp analysis results implicitly calls their to_matrix method when adding to dataframe

[samanthavbarron]

This line ends up trying to convert Operators to arrays, if included in the analysis results. If the operator is on a large number of qubits, this produces a large array.

https://github.com/Qiskit-Extensions/qiskit-experiments/blob/0bbd426dc257be6d35daf04d94bbb31e0d059648/qiskit_experiments/framework/analysis_result_table.py#L153

[aeddins-ibm]

Just to emphasize the severity: for an Operator on tens of qubits, "large" here can mean petabytes, so the program immediately crashes.

[chriseclectic]

This issue is likely related to an issue with Numpy itself which I opened awhile ago and I believe is still unresolved:

• BUG: Setting an object array via slicing with nested lists invokes the objects __array__ methods unnecessarily numpy/numpy#24336
• Quantum Info classes __array__ method is invoked when setting object array values via slicing Qiskit/qiskit#10561
  and hence Pandas internal use of numpy, and Qiskit Experiments internal use of Pandas could be rigger this.

A minimal example for how this breaks in qiskit experiments is below:

from qiskit_experiments.framework import BaseAnalysis, AnalysisResultData, ExperimentData
from qiskit.quantum_info import Pauli

class BrokenAnalysis(BaseAnalysis):
    """This will fail because Panda's trys to convert a Pauli to a dense array"""
        
    def _run_analysis(self, experiment_data):
        return [AnalysisResultData("pauli", Pauli(100 * "X"))], []
    
analysis = BrokenAnalysis()
analysis.run(ExperimentData())

Analysis callback failed [Experiment ID: 8cf08621-dfb8-4cf8-97ad-b794207b1981][Analysis Callback ID: 64ef09198f934e1dac00450ed401faaf]:
Traceback (most recent call last):
  File "/Users/cjwood/mambaforge/envs/qiskit1/lib/python3.11/site-packages/qiskit_experiments/framework/experiment_data.py", line 960, in _run_analysis_callback
    callback(self, **kwargs)
  File "/Users/cjwood/mambaforge/envs/qiskit1/lib/python3.11/site-packages/qiskit_experiments/framework/base_analysis.py", line 202, in run_analysis
    expdata.add_analysis_results(**table_format)
  File "/Users/cjwood/mambaforge/envs/qiskit1/lib/python3.11/site-packages/qiskit/utils/deprecation.py", line 183, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/Users/cjwood/mambaforge/envs/qiskit1/lib/python3.11/site-packages/qiskit_experiments/framework/experiment_data.py", line 86, in _wrapped
    return_val = func(self, *args, **kwargs)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/cjwood/mambaforge/envs/qiskit1/lib/python3.11/site-packages/qiskit_experiments/framework/experiment_data.py", line 1377, in add_analysis_results
    uid = self._analysis_results.add_data(
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/cjwood/mambaforge/envs/qiskit1/lib/python3.11/site-packages/qiskit_experiments/framework/analysis_result_table.py", line 153, in add_data
    self._data.loc[index, :] = pd.array(list(template.values()), dtype=object)
                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/cjwood/mambaforge/envs/qiskit1/lib/python3.11/site-packages/pandas/core/construction.py", line 388, in array
    return NumpyExtensionArray._from_sequence(data, dtype=dtype, copy=copy)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/cjwood/mambaforge/envs/qiskit1/lib/python3.11/site-packages/pandas/core/arrays/numpy_.py", line 130, in _from_sequence
    result = np.asarray(scalars, dtype=dtype)  # type: ignore[arg-type]
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/cjwood/mambaforge/envs/qiskit1/lib/python3.11/site-packages/qiskit/quantum_info/operators/symplectic/pauli.py", line 226, in __array__
    return np.asarray(self.to_matrix(), dtype=dtype)
                      ^^^^^^^^^^^^^^^^
  File "/Users/cjwood/mambaforge/envs/qiskit1/lib/python3.11/site-packages/qiskit/quantum_info/operators/symplectic/pauli.py", line 424, in to_matrix
    return self._to_matrix(self.z, self.x, self._phase[0], sparse=sparse)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/cjwood/mambaforge/envs/qiskit1/lib/python3.11/site-packages/qiskit/quantum_info/operators/symplectic/base_pauli.py", line 445, in _to_matrix
    indptr = np.arange(dim + 1, dtype=np.uint)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ValueError: Maximum allowed size exceeded