Numpy memory use at step increments #206
Unanswered
MartinKlefas
asked this question in
Q&A
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
I realise I've probably set something up naively here, but I used the included
StyleGAN2-SPD-ADA.yaml
config as a template to train on my own dataset of images. The dataset itself isn't very large, being only 280,000 images across a couple of classes, and after a while I got it running on a medium sized VM with access to 128GB of RAM and 2x 48GB GPUs.However, at step 2,000 it came out of the steps loop to do metrics and tried to make itself a numpy array that would have been (N_Images, N_images) in shape - or 594 GiB in main memory...
My question is really: What did I do to ask for that numpy array? it made some performance scores without it:
[INFO] 2024-03-25 15:31:41 > Step: 2000 Progress: 1.0% Elapsed: 1:11:25 Gen_loss: 5.902 Dis_loss: 0.2644 Cls_loss: N/A Topk: N/A aa_p: 0.07962
[INFO] 2024-03-25 15:31:41 > Visualize (num_rows x 8) fake image canvans.
[INFO] 2024-03-25 15:31:56 > Save image canvas to /workspace/weights_out/figures/Restained Crypts v3 Labels-Crypt-StyleGAN2-SPD-ADA-train-2024_03_25_14_19_56/generated_canvas_2000.png
[INFO] 2024-03-25 15:32:01 > Start Evaluation (2000 Step): Restained Crypts v3 Labels-Crypt-StyleGAN2-SPD-ADA-train-2024_03_25_14_19_56
[INFO] 2024-03-25 15:32:01 > generate images and stack features (282304 images).
[INFO] 2024-03-25 16:20:12 > 282304 real images is used for evaluation.
[INFO] 2024-03-25 16:20:12 > Inception score (Step: 2000, 282304 generated images): 1.2603631019592285
[INFO] 2024-03-25 16:20:29 > FID score (Step: 2000, Using train moments): 419.71832077560316
[INFO] 2024-03-25 16:20:29 > Best FID score (Step: 2000, Using train moments): 419.71832077560316
The command I used to run it was:
CUDA_VISIBLE_DEVICES=0,1 python3 main.py -t -metrics is fid prdc -cfg configs/AFHQv2/StyleGAN3-r-paper.yaml -data dataset512/ -save /workspace/weights_out/
Was it the PRDC Metrics? Can I skip them without too much pain??
Beta Was this translation helpful? Give feedback.
All reactions