Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Change RB test parameters to make the test faster and more robust #1295

Merged
merged 3 commits into from
Oct 31, 2023

Conversation

wshanks
Copy link
Collaborator

@wshanks wshanks commented Oct 27, 2023

test_add_more_circuit_yields_lower_variance started failing with qiskit-aer 0.13.0. The test was only comparing the results of 3 RB sampels to 5 RB samples and looking for the case with more samples to have lower variance. These numbers of samples were likely not enough to ensure that the case with more samples always had lower variance across a wide range of random seeds. Here the high number of samples was increased to 30.

To make sure that the fit results were reasonable and that the test run time did not get too long, these additional changes were:

  • Convert the test from two qubit RB to single qubit RB. This makes the simulation faster.
  • Increase the single qubit gate error. This makes the qubit depolarize faster so that the RB curve can be traced out with less gates.

`test_add_more_circuit_yields_lower_variance` started failing with
qiskit-aer 0.13.0. The test was only comparing the results of 3 RB
sampels to 5 RB samples and looking for the case with more samples to
have lower variance. These numbers of samples were likely not enough to
ensure that the case with more samples always had lower variance across
a wide range of random seeds. Here the high number of samples was
increased to 30.

To make sure that the fit results were reasonable and
that the test run time did not get too long, these additional changes
were:

* Convert the test from two qubit RB to single qubit RB. This makes the
  simulation faster.
* Increase the single qubit gate error. This makes the qubit depolarize
  faster so that the RB curve can be traced out with less gates.
@wshanks wshanks added backport stable potential The issue or PR might be minimal and/or import enough to backport to stable Changelog: None Do not include in changelog labels Oct 27, 2023
@wshanks wshanks requested a review from itoko October 27, 2023 19:59
@wshanks
Copy link
Collaborator Author

wshanks commented Oct 27, 2023

Here is what the two cases look like now (starting to push the limit a bit of noisy data vs fast test time):

image

@wshanks
Copy link
Collaborator Author

wshanks commented Oct 27, 2023

And here the test code pulled out in case you want to play with the numbers in a notebook or script which is what I did:

from qiskit_aer import AerSimulator
from qiskit_aer.noise import NoiseModel, depolarizing_error
from qiskit_experiments.library import StandardRB


# depolarizing error
p1q = 0.15
p2q = 0.04
pvz = 0.0
pcz = 0.06

basis_gates = ["rz", "sx", "cx"]

# setup noise model
sx_error = depolarizing_error(p1q, 1)
rz_error = depolarizing_error(pvz, 1)

noise_model = NoiseModel()
noise_model.add_all_qubit_quantum_error(sx_error, "sx")
noise_model.add_all_qubit_quantum_error(rz_error, "rz")

noise_model = noise_model

# Need level1 for consecutive gate cancellation for reference EPC value calculation
transpiler_options = {
    "basis_gates": basis_gates,
}

backend = AerSimulator(noise_model=noise_model, seed_simulator=123)

exp1 = StandardRB(
    physical_qubits=(0,),
    lengths=list(range(1, 30, 3)),
    seed=444,
    backend=backend,
    num_samples=3,
)
exp1.analysis.set_options(gate_error_ratio=None)
exp1.set_transpile_options(**transpiler_options)
expdata1 = exp1.run().block_for_results()

exp2 = StandardRB(
    physical_qubits=(0,),
    lengths=list(range(1, 30, 3)),
    seed=123,
    backend=backend,
    num_samples=30,
)
exp2.analysis.set_options(gate_error_ratio=None)
exp2.set_transpile_options(**transpiler_options)
expdata2 = exp2.run().block_for_results()

Copy link
Collaborator

@coruscating coruscating left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks

@wshanks wshanks enabled auto-merge October 31, 2023 14:27
@wshanks wshanks added this pull request to the merge queue Oct 31, 2023
Merged via the queue into qiskit-community:main with commit b1a42e3 Oct 31, 2023
10 checks passed
mergify bot pushed a commit that referenced this pull request Oct 31, 2023
)

`test_add_more_circuit_yields_lower_variance` started failing with
qiskit-aer 0.13.0. The test was only comparing the results of 3 RB
sampels to 5 RB samples and looking for the case with more samples to
have lower variance. These numbers of samples were likely not enough to
ensure that the case with more samples always had lower variance across
a wide range of random seeds. Here the high number of samples was
increased to 30.

To make sure that the fit results were reasonable and that the test run
time did not get too long, these additional changes were:

* Convert the test from two qubit RB to single qubit RB. This makes the
simulation faster.
* Increase the single qubit gate error. This makes the qubit depolarize
faster so that the RB curve can be traced out with less gates.

(cherry picked from commit b1a42e3)
coruscating pushed a commit that referenced this pull request Oct 31, 2023
…ckport #1295) (#1304)

This is an automatic backport of pull request #1295 done by
[Mergify](https://mergify.com).


---


<details>
<summary>Mergify commands and options</summary>

<br />

More conditions and actions can be found in the
[documentation](https://docs.mergify.com/).

You can also trigger Mergify actions by commenting on this pull request:

- `@Mergifyio refresh` will re-evaluate the rules
- `@Mergifyio rebase` will rebase this PR on its base branch
- `@Mergifyio update` will merge the base branch into this PR
- `@Mergifyio backport <destination>` will backport this PR on
`<destination>` branch

Additionally, on Mergify [dashboard](https://dashboard.mergify.com) you
can:

- look at your merge queues
- generate the Mergify configuration with the config editor.

Finally, you can contact us on https://mergify.com
</details>

Co-authored-by: Will Shanks <willshanks@us.ibm.com>
nkanazawa1989 pushed a commit to nkanazawa1989/qiskit-experiments that referenced this pull request Jan 17, 2024
…skit-community#1295)

`test_add_more_circuit_yields_lower_variance` started failing with
qiskit-aer 0.13.0. The test was only comparing the results of 3 RB
sampels to 5 RB samples and looking for the case with more samples to
have lower variance. These numbers of samples were likely not enough to
ensure that the case with more samples always had lower variance across
a wide range of random seeds. Here the high number of samples was
increased to 30.

To make sure that the fit results were reasonable and that the test run
time did not get too long, these additional changes were:

* Convert the test from two qubit RB to single qubit RB. This makes the
simulation faster.
* Increase the single qubit gate error. This makes the qubit depolarize
faster so that the RB curve can be traced out with less gates.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport stable potential The issue or PR might be minimal and/or import enough to backport to stable Changelog: None Do not include in changelog
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants