bbox loss激增问题

[TimLiujn]

我在训练r_net的时候bbox loss突然变得很大，请问有其他人遇到类似问题吗
train Rnet argument:
Namespace(annotation_file='/app/ljn_work/DFace/anno_store/imglist_anno_24.txt', batch_size=512, end_epoch=22, frequent=200, lr=0.01, model_store_path='/app/ljn_work/DFace/model_store', use_cuda=True, **{'': None})
append flipped images to imdb 824034
../core/models.py:8: UserWarning: nn.init.xavier_uniform is now deprecated in favor of nn.init.xavier_uniform_.
nn.init.xavier_uniform(m.weight.data)
../core/models.py:9: UserWarning: nn.init.constant is now deprecated in favor of nn.init.constant_.
nn.init.constant(m.bias, 0.1)
2019-01-23 14:21:15.507047 : Epoch: 1, Step: 0, accuracy: 0.9690722227096558, det loss: 0.5977954864501953, bbox loss: 0.1574765294790268, all_loss: 0.6765337586402893, lr:0.01
2019-01-23 14:22:00.736671 : Epoch: 1, Step: 200, accuracy: 0.9730290770530701, det loss: 0.11306899785995483, bbox loss: 0.023932723328471184, all_loss: 0.125035360455513, lr:0.01
2019-01-23 14:22:44.877566 : Epoch: 1, Step: 400, accuracy: 0.9812500476837158, det loss: 0.06650176644325256, bbox loss: 0.03081391006708145, all_loss: 0.08190871775150299, lr:0.01
2019-01-23 14:23:23.430212 : Epoch: 1, Step: 600, accuracy: 0.9613820910453796, det loss: 0.10355271399021149, bbox loss: 0.028969649225473404, all_loss: 0.11803753674030304, lr:0.01
2019-01-23 14:24:00.741899 : Epoch: 1, Step: 800, accuracy: 0.9894291758537292, det loss: 0.05135280638933182, bbox loss: 0.027542466297745705, all_loss: 0.0651240423321724, lr:0.01
2019-01-23 14:24:37.632884 : Epoch: 1, Step: 1000, accuracy: 0.9835051894187927, det loss: 0.05876293033361435, bbox loss: 0.019213180989027023, all_loss: 0.06836952269077301, lr:0.01
2019-01-23 14:25:11.963026 : Epoch: 1, Step: 1200, accuracy: 0.9896907806396484, det loss: 0.06266145408153534, bbox loss: 0.0241145770996809, all_loss: 0.07471874356269836, lr:0.01
2019-01-23 14:25:45.191520 : Epoch: 1, Step: 1400, accuracy: 0.9833679795265198, det loss: 0.06828348338603973, bbox loss: 0.027324318885803223, all_loss: 0.08194564282894135, lr:0.01
2019-01-23 14:26:12.594181 : Epoch: 1, Step: 1600, accuracy: 0.9831578731536865, det loss: 0.08070078492164612, bbox loss: 0.026685547083616257, all_loss: 0.0940435603260994, lr:0.01
2019-01-23 14:26:41.969361 : Epoch: 1, Step: 1800, accuracy: 0.9835051894187927, det loss: 0.06561711430549622, bbox loss: 0.02304217964410782, all_loss: 0.07713820040225983, lr:0.01
2019-01-23 14:27:06.414403 : Epoch: 1, Step: 2000, accuracy: 0.9767932295799255, det loss: 0.09401784837245941, bbox loss: 0.023107564076781273, all_loss: 0.10557162761688232, lr:0.01
2019-01-23 14:27:29.553256 : Epoch: 1, Step: 2200, accuracy: 0.9832635521888733, det loss: 0.054206281900405884, bbox loss: 0.0239733774214983, all_loss: 0.06619296967983246, lr:0.01
2019-01-23 14:27:53.957669 : Epoch: 1, Step: 2400, accuracy: 0.9854772090911865, det loss: 0.047386255115270615, bbox loss: 0.023768266662955284, all_loss: 0.05927038937807083, lr:0.01
2019-01-23 14:28:15.802341 : Epoch: 1, Step: 2600, accuracy: 0.9852631092071533, det loss: 0.0627639964222908, bbox loss: 0.023052219301462173, all_loss: 0.07429010421037674, lr:0.01
2019-01-23 14:28:37.131607 : Epoch: 1, Step: 2800, accuracy: 0.9853556156158447, det loss: 0.04311512038111687, bbox loss: 0.0205087848007679, all_loss: 0.053369514644145966, lr:0.01
2019-01-23 14:28:56.894010 : Epoch: 1, Step: 3000, accuracy: 0.9937106370925903, det loss: 0.033722564578056335, bbox loss: 0.025765910744667053, all_loss: 0.04660551995038986, lr:0.01
2019-01-23 14:29:15.391435 : Epoch: 1, Step: 3200, accuracy: 0.9895616173744202, det loss: 0.044307757169008255, bbox loss: 0.020762424916028976, all_loss: 0.054688967764377594, lr:0.01
Epoch: 1, accuracy: 0.9821656346321106, cls loss: 0.09693042933940887, bbox loss: 0.03235609456896782
2019-01-23 14:29:17.076650 : Epoch: 2, Step: 0, accuracy: 0.9917526245117188, det loss: 0.017674211412668228, bbox loss: 0.027816535905003548, all_loss: 0.03158247843384743, lr:0.01
2019-01-23 14:29:34.908802 : Epoch: 2, Step: 200, accuracy: 0.9853862524032593, det loss: 0.04590099677443504, bbox loss: 0.026683641597628593, all_loss: 0.059242818504571915, lr:0.01
2019-01-23 14:29:54.435913 : Epoch: 2, Step: 400, accuracy: 0.9690722227096558, det loss: 0.11844029277563095, bbox loss: 0.017504854127764702, all_loss: 0.12719272077083588, lr:0.01
2019-01-23 14:30:12.778035 : Epoch: 2, Step: 600, accuracy: 0.9917355179786682, det loss: 0.03938473388552666, bbox loss: 0.017851099371910095, all_loss: 0.048310283571481705, lr:0.01
2019-01-23 14:30:30.442959 : Epoch: 2, Step: 800, accuracy: 0.9873417019844055, det loss: 0.0521685853600502, bbox loss: 0.02776484563946724, all_loss: 0.06605100631713867, lr:0.01
2019-01-23 14:30:48.199636 : Epoch: 2, Step: 1000, accuracy: 0.9792531728744507, det loss: 0.0700664296746254, bbox loss: 0.03486182913184166, all_loss: 0.08749734610319138, lr:0.01
2019-01-23 14:31:06.811424 : Epoch: 2, Step: 1200, accuracy: 0.9812108874320984, det loss: 0.051916588097810745, bbox loss: 0.03027299977838993, all_loss: 0.06705308705568314, lr:0.01
2019-01-23 14:31:24.524263 : Epoch: 2, Step: 1400, accuracy: 0.9917184710502625, det loss: 0.042696353048086166, bbox loss: 0.02568719908595085, all_loss: 0.05553995072841644, lr:0.01
2019-01-23 14:31:42.106860 : Epoch: 2, Step: 1600, accuracy: 0.9894958734512329, det loss: 0.05638515576720238, bbox loss: 0.029507212340831757, all_loss: 0.07113876193761826, lr:0.01
2019-01-23 14:32:01.420803 : Epoch: 2, Step: 1800, accuracy: 0.9874476790428162, det loss: 0.04238443076610565, bbox loss: 0.030263887718319893, all_loss: 0.057516373693943024, lr:0.01
2019-01-23 14:32:20.253376 : Epoch: 2, Step: 2000, accuracy: 0.9812889695167542, det loss: 0.060417406260967255, bbox loss: 0.026619980111718178, all_loss: 0.07372739911079407, lr:0.01
2019-01-23 14:32:38.544129 : Epoch: 2, Step: 2200, accuracy: 0.9733605980873108, det loss: 0.7360723614692688, bbox loss: 51090.0234375, all_loss: 25545.748046875, lr:0.01
2019-01-23 14:32:56.030824 : Epoch: 2, Step: 2400, accuracy: 0.9766454696655273, det loss: 0.6453104615211487, bbox loss: 30.824626922607422, all_loss: 16.05762481689453, lr:0.01
2019-01-23 14:33:13.531097 : Epoch: 2, Step: 2600, accuracy: 0.9664570093154907, det loss: 0.9268267154693604, bbox loss: 19.710317611694336, all_loss: 10.78198528289795, lr:0.01
2019-01-23 14:33:31.707023 : Epoch: 2, Step: 2800, accuracy: 0.9897958636283875, det loss: 0.2819491922855377, bbox loss: 12.830120086669922, all_loss: 6.697009086608887, lr:0.01
2019-01-23 14:33:49.352857 : Epoch: 2, Step: 3000, accuracy: 0.958071231842041, det loss: 1.1585333347320557, bbox loss: 9.914958000183105, all_loss: 6.1160125732421875, lr:0.01
2019-01-23 14:34:07.073843 : Epoch: 2, Step: 3200, accuracy: 0.9707112908363342, det loss: 0.8092767596244812, bbox loss: 8.420356750488281, all_loss: 5.0194549560546875, lr:0.01
Epoch: 2, accuracy: 0.9806320071220398, cls loss: 0.30325907468795776, bbox loss: 3010.118896484375
2019-01-23 14:34:08.732263 : Epoch: 3, Step: 0, accuracy: 0.95208340883255, det loss: 1.3239864110946655, bbox loss: 8.500407218933105, all_loss: 5.574190139770508, lr:0.01
2019-01-23 14:34:27.355744 : Epoch: 3, Step: 200, accuracy: 0.9832285046577454, det loss: 0.4634133577346802, bbox loss: 5.173794746398926, all_loss: 3.0503106117248535, lr:0.01
2019-01-23 14:34:44.825701 : Epoch: 3, Step: 400, accuracy: 0.9689441323280334, det loss: 0.8581061959266663, bbox loss: 4.081416130065918, all_loss: 2.8988142013549805, lr:0.01
2019-01-23 14:35:02.375759 : Epoch: 3, Step: 600, accuracy: 0.970954418182373, det loss: 0.8025608658790588, bbox loss: 6.820148944854736, all_loss: 4.212635517120361, lr:0.01
2019-01-23 14:35:20.082355 : Epoch: 3, Step: 800, accuracy: 0.9731959104537964, det loss: 0.7406253814697266, bbox loss: 2.788095474243164, all_loss: 2.1346731185913086, lr:0.01
2019-01-23 14:35:37.779874 : Epoch: 3, Step: 1000, accuracy: 0.9753085970878601, det loss: 0.6822474598884583, bbox loss: 4.362936019897461, all_loss: 2.863715410232544, lr:0.01
2019-01-23 14:35:55.549916 : Epoch: 3, Step: 1200, accuracy: 0.9730849266052246, det loss: 0.7436921000480652, bbox loss: 2.563908100128174, all_loss: 2.025646209716797, lr:0.01
2019-01-23 14:36:13.278228 : Epoch: 3, Step: 1400, accuracy: 0.9568789005279541, det loss: 1.1914814710617065, bbox loss: 2.419104814529419, all_loss: 2.401033878326416, lr:0.01
2019-01-23 14:36:30.979793 : Epoch: 3, Step: 1600, accuracy: 0.9684209823608398, det loss: 0.8725585341453552, bbox loss: 4.835862159729004, all_loss: 3.290489673614502, lr:0.01
2019-01-23 14:36:48.621762 : Epoch: 3, Step: 1800, accuracy: 0.9569671750068665, det loss: 1.1890400648117065, bbox loss: 8.744510650634766, all_loss: 5.561295509338379, lr:0.01
2019-01-23 14:37:06.089376 : Epoch: 3, Step: 2000, accuracy: 0.9728601574897766, det loss: 0.7499024868011475, bbox loss: 2.765618324279785, all_loss: 2.13271164894104, lr:0.01
2019-01-23 14:37:23.618019 : Epoch: 3, Step: 2200, accuracy: 0.9752065539360046, det loss: 0.6850666403770447, bbox loss: 5.427116394042969, all_loss: 3.398624897003174, lr:0.01
2019-01-23 14:37:41.202316 : Epoch: 3, Step: 2400, accuracy: 0.9769391417503357, det loss: 0.6371933221817017, bbox loss: 5.064423084259033, all_loss: 3.169404983520508, lr:0.01
2019-01-23 14:37:59.325220 : Epoch: 3, Step: 2600, accuracy: 0.9734693169593811, det loss: 0.7330679297447205, bbox loss: 2.8945021629333496, all_loss: 2.18031907081604, lr:0.01
2019-01-23 14:38:17.459984 : Epoch: 3, Step: 2800, accuracy: 0.9732509851455688, det loss: 0.7391014099121094, bbox loss: 11.652641296386719, all_loss: 6.565422058105469, lr:0.01
2019-01-23 14:38:35.265657 : Epoch: 3, Step: 3000, accuracy: 0.9793815016746521, det loss: 0.5697118043899536, bbox loss: 6.923870086669922, all_loss: 4.031646728515625, lr:0.01
2019-01-23 14:38:52.865394 : Epoch: 3, Step: 3200, accuracy: 0.9728601574897766, det loss: 0.7499023675918579, bbox loss: 1.9425570964813232, all_loss: 1.7211809158325195, lr:0.01
Epoch: 3, accuracy: 0.9707667231559753, cls loss: 0.807744562625885, bbox loss: 5.115347862243652
2019-01-23 14:38:54.500485 : Epoch: 4, Step: 0, accuracy: 0.9523809552192688, det loss: 1.3157627582550049, bbox loss: 1.4270354509353638, all_loss: 2.029280424118042, lr:0.01
2019-01-23 14:39:12.029839 : Epoch: 4, Step: 200, accuracy: 0.9629629254341125, det loss: 1.0233712196350098, bbox loss: 0.9527443051338196, all_loss: 1.4997433423995972, lr:0.01
2019-01-23 14:39:29.662662 : Epoch: 4, Step: 400, accuracy: 0.970954418182373, det loss: 0.8025607466697693, bbox loss: 3.181978464126587, all_loss: 2.393549919128418, lr:0.01
2019-01-23 14:39:47.202199 : Epoch: 4, Step: 600, accuracy: 0.9764453768730164, det loss: 0.6508377194404602, bbox loss: 10.882843017578125, all_loss: 6.092259407043457, lr:0.01
2019-01-23 14:40:05.007325 : Epoch: 4, Step: 800, accuracy: 0.9690722227096558, det loss: 0.8545677065849304, bbox loss: 0.8961852192878723, all_loss: 1.302660346031189, lr:0.01
2019-01-23 14:40:22.690702 : Epoch: 4, Step: 1000, accuracy: 0.9680171012878418, det loss: 0.8837214112281799, bbox loss: 1.7729414701461792, all_loss: 1.7701921463012695, lr:0.01
2019-01-23 14:40:40.455843 : Epoch: 4, Step: 1200, accuracy: 0.9813664555549622, det loss: 0.5148637890815735, bbox loss: 11.610335350036621, all_loss: 6.320031642913818, lr:0.01
2019-01-23 14:40:58.200712 : Epoch: 4, Step: 1400, accuracy: 0.9712526202201843, det loss: 0.7943210005760193, bbox loss: 2.4853084087371826, all_loss: 2.036975145339966, lr:0.01
2019-01-23 14:41:15.930705 : Epoch: 4, Step: 1600, accuracy: 0.9689441323280334, det loss: 0.858106255531311, bbox loss: 0.6418619155883789, all_loss: 1.1790372133255005, lr:0.01
2019-01-23 14:41:33.608430 : Epoch: 4, Step: 1800, accuracy: 0.9769874215126038, det loss: 0.6358603239059448, bbox loss: 0.512661337852478, all_loss: 0.8921909928321838, lr:0.01
2019-01-23 14:41:51.154037 : Epoch: 4, Step: 2000, accuracy: 0.9790794849395752, det loss: 0.5780549049377441, bbox loss: 40.10780334472656, all_loss: 20.631956100463867, lr:0.01
2019-01-23 14:42:08.722124 : Epoch: 4, Step: 2200, accuracy: 0.975051999092102, det loss: 0.6893393397331238, bbox loss: 1.4536627531051636, all_loss: 1.4161707162857056, lr:0.01
2019-01-23 14:42:26.425914 : Epoch: 4, Step: 2400, accuracy: 0.9730290770530701, det loss: 0.745235025882721, bbox loss: 0.9394894242286682, all_loss: 1.2149797677993774, lr:0.01
2019-01-23 14:42:43.953294 : Epoch: 4, Step: 2600, accuracy: 0.9667359590530396, det loss: 0.9191191792488098, bbox loss: 1.7752183675765991, all_loss: 1.8067283630371094, lr:0.01
2019-01-23 14:43:01.647522 : Epoch: 4, Step: 2800, accuracy: 0.9771784543991089, det loss: 0.6305834650993347, bbox loss: 0.3447815477848053, all_loss: 0.8029742240905762, lr:0.01
2019-01-23 14:43:19.346791 : Epoch: 4, Step: 3000, accuracy: 0.9688149690628052, det loss: 0.8616742491722107, bbox loss: 0.2147742360830307, all_loss: 0.9690613746643066, lr:0.01
2019-01-23 14:43:37.291739 : Epoch: 4, Step: 3200, accuracy: 0.966386616230011, det loss: 0.928773820400238, bbox loss: 10.592591285705566, all_loss: 6.225069522857666, lr:0.01
Epoch: 4, accuracy: 0.9708623290061951, cls loss: 0.8051031231880188, bbox loss: 5.281894683837891
2019-01-23 14:43:39.106912 : Epoch: 5, Step: 0, accuracy: 0.9645833969116211, det loss: 0.9785986542701721, bbox loss: 1.4899259805679321, all_loss: 1.7235616445541382, lr:0.01
2019-01-23 14:43:58.427369 : Epoch: 5, Step: 200, accuracy: 0.9628098607063293, det loss: 1.0276000499725342, bbox loss: 77.7536392211914, all_loss: 39.9044189453125, lr:0.01
2019-01-23 14:44:16.214350 : Epoch: 5, Step: 400, accuracy: 0.9744136929512024, det loss: 0.706977128982544, bbox loss: 54.73612976074219, all_loss: 28.075042724609375, lr:0.01
2019-01-23 14:44:34.805013 : Epoch: 5, Step: 600, accuracy: 0.9648759961128235, det loss: 0.9705110788345337, bbox loss: 52.41902542114258, all_loss: 27.180023193359375, lr:0.01
2019-01-23 14:44:53.576852 : Epoch: 5, Step: 800, accuracy: 0.9708333611488342, det loss: 0.8059048056602478, bbox loss: 30.6917667388916, all_loss: 16.15178871154785, lr:0.01
2019-01-23 14:45:11.230668 : Epoch: 5, Step: 1000, accuracy: 0.9670103192329407, det loss: 0.9115387797355652, bbox loss: 43.69115447998047, all_loss: 22.757116317749023, lr:0.01
2019-01-23 14:45:28.921397 : Epoch: 5, Step: 1200, accuracy: 0.9648033380508423, det loss: 0.9725204110145569, bbox loss: 23.02033805847168, all_loss: 12.48268985748291, lr:0.01
2019-01-23 14:45:47.105647 : Epoch: 5, Step: 1400, accuracy: 0.9775509834289551, det loss: 0.6202881932258606, bbox loss: 29.870832443237305, all_loss: 15.555704116821289, lr:0.01
2019-01-23 14:46:05.092484 : Epoch: 5, Step: 1600, accuracy: 0.9722814559936523, det loss: 0.7658918499946594, bbox loss: 17.052854537963867, all_loss: 9.292319297790527, lr:0.01
2019-01-23 14:46:23.110959 : Epoch: 5, Step: 1800, accuracy: 0.9728601574897766, det loss: 0.7499023675918579, bbox loss: 12.887855529785156, all_loss: 7.1938300132751465, lr:0.01
2019-01-23 14:46:40.897368 : Epoch: 5, Step: 2000, accuracy: 0.970954418182373, det loss: 0.8025608062744141, bbox loss: 10.125185012817383, all_loss: 5.8651533126831055, lr:0.01
2019-01-23 14:46:58.737332 : Epoch: 5, Step: 2200, accuracy: 0.9689441323280334, det loss: 0.858106255531311, bbox loss: 10.612981796264648, all_loss: 6.164597034454346, lr:0.01
2019-01-23 14:47:16.662478 : Epoch: 5, Step: 2400, accuracy: 0.9579832553863525, det loss: 1.160967230796814, bbox loss: 7.97253942489624, all_loss: 5.1472368240356445, lr:0.01
2019-01-23 14:47:34.737180 : Epoch: 5, Step: 2600, accuracy: 0.9834368824958801, det loss: 0.45765671133995056, bbox loss: 5.740257263183594, all_loss: 3.3277852535247803, lr:0.01
2019-01-23 14:47:52.946507 : Epoch: 5, Step: 2800, accuracy: 0.9614561200141907, det loss: 1.065007209777832, bbox loss: 8.753331184387207, all_loss: 5.4416728019714355, lr:0.01
2019-01-23 14:48:12.018501 : Epoch: 5, Step: 3000, accuracy: 0.9708939790725708, det loss: 0.8042293190956116, bbox loss: 10.31871509552002, all_loss: 5.963586807250977, lr:0.01
2019-01-23 14:48:30.444587 : Epoch: 5, Step: 3200, accuracy: 0.9701492786407471, det loss: 0.8248065710067749, bbox loss: 9.488470077514648, all_loss: 5.569041728973389, lr:0.01
Epoch: 5, accuracy: 0.969167172908783, cls loss: 0.851945161819458, bbox loss: 23.91911506652832
2019-01-23 14:48:32.246242 : Epoch: 6, Step: 0, accuracy: 0.9649484753608704, det loss: 0.968510091304779, bbox loss: 8.402690887451172, all_loss: 5.16985559463501, lr:0.01
2019-01-23 14:48:51.166888 : Epoch: 6, Step: 200, accuracy: 0.9625779986381531, det loss: 1.0340090990066528, bbox loss: 4.893960952758789, all_loss: 3.480989456176758, lr:0.01
2019-01-23 14:49:09.799202 : Epoch: 6, Step: 400, accuracy: 0.9445585608482361, det loss: 1.5319045782089233, bbox loss: 6.67972993850708, all_loss: 4.871769428253174, lr:0.01
2019-01-23 14:49:27.814443 : Epoch: 6, Step: 600, accuracy: 0.9708939790725708, det loss: 0.8042293787002563, bbox loss: 5.477999687194824, all_loss: 3.543229103088379, lr:0.01
2019-01-23 14:49:45.420536 : Epoch: 6, Step: 800, accuracy: 0.9814814329147339, det loss: 0.5116856098175049, bbox loss: 4.063371658325195, all_loss: 2.5433714389801025, lr:0.01
2019-01-23 14:50:02.968040 : Epoch: 6, Step: 1000, accuracy: 0.9707724452018738, det loss: 0.8075873255729675, bbox loss: 11.518770217895508, all_loss: 6.566972255706787, lr:0.01
2019-01-23 14:50:20.716327 : Epoch: 6, Step: 1200, accuracy: 0.9708333611488342, det loss: 0.805904746055603, bbox loss: 2.895217180252075, all_loss: 2.2535133361816406, lr:0.01
2019-01-23 14:50:38.586131 : Epoch: 6, Step: 1400, accuracy: 0.9749478101730347, det loss: 0.6922176480293274, bbox loss: 38.158931732177734, all_loss: 19.771682739257812, lr:0.01
2019-01-23 14:50:56.286343 : Epoch: 6, Step: 1600, accuracy: 0.9744136929512024, det loss: 0.706977128982544, bbox loss: 23.079925537109375, all_loss: 12.246939659118652, lr:0.01
2019-01-23 14:51:13.872885 : Epoch: 6, Step: 1800, accuracy: 0.9686192274093628, det loss: 0.8670822381973267, bbox loss: 4.911552429199219, all_loss: 3.3228583335876465, lr:0.01
2019-01-23 14:51:31.560673 : Epoch: 6, Step: 2000, accuracy: 0.9686847925186157, det loss: 0.8652721047401428, bbox loss: 2.223311185836792, all_loss: 1.9769277572631836, lr:0.01

[WalkingWoo]

缩小学习率

[zhouzhubin]

@TimLiujn 你调小解决了这个问题了吗?我也出现了一样的,有时候还会出现nan