Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add a baseline CFIS dataset #40

Merged
merged 2 commits into from
Nov 18, 2021
Merged

Add a baseline CFIS dataset #40

merged 2 commits into from
Nov 18, 2021

Conversation

EiffL
Copy link
Member

@EiffL EiffL commented Nov 7, 2021

This PR adds a first pass of a tf dataset of CFIS-like images, this aims at solving #39.

This is a first pass because we will probably want to add additional variants on that dataset in the future, but it is a starting point

@EiffL
Copy link
Member Author

EiffL commented Nov 7, 2021

Super easy to use:

import autometacal
import tensorflow_dataset as tfds
import tensorflow as tf

# We create a function to add noise on the fly for data augmentation
@tf.function
def add_noise(example):
    im_noise = example['obs'] + example['noise_std'] * tf.random.normal([51,51])
    return im_noise, example

dset = tfds.load('CFIS/parametric_shear_1k', split='train')
dset = dset.map(add_noise)

for im, example in dset:
  # im contains the noisy observation
  # example contains psf image, observation without noise, and galaxy magnitude
  ....

@aguinot
Copy link

aguinot commented Nov 8, 2021

I checked the notebook, I am a bit confused by the results:

  • Do you know why autometacal is "slower"? (from the tqdm print)
  • The response are different from ngmix and autocal (and they are not negligible).
  • Autocal is 1 magnitude better than ngmix, do you think this is because of autodiff? If so, this is very cool!
  • I think I would make a test at very high SNR (like 50000-100000) just to make sure everything is working well. Maybe you have already tried?

@EiffL
Copy link
Member Author

EiffL commented Nov 9, 2021

Dont look too closely at the notebook. The response computed by autometacal is wrong, cf #26.
The tqdm looks slower but that's because I do like 10000 instead of 1000 galaxies.

The main point of this PR is to add the dataset, the notebook is to illustrate how to load and use the dataset, but it's optional, if you think it's not useful I can drop it for now.

Copy link
Member Author

@EiffL EiffL left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have a few comments

'psf': tfds.features.Tensor(shape=[self.builder_config.stamp_size,
self.builder_config.stamp_size],
dtype=tf.float32),
# 'gal_kimage': tfds.features.Tensor(shape=[2, self.builder_config.kstamp_size,
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what do you have these commented lines? If they are not needed we should remove to avoid cluttering the code

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

that;s a good point, I'll rmove this stuff for now

Copy link
Contributor

@andrevitorelli andrevitorelli left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

gtg

@andrevitorelli andrevitorelli merged commit 974b271 into main Nov 18, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Build a realistic galaxy sample, with some typical SNR, shape, size
3 participants