-
-
Notifications
You must be signed in to change notification settings - Fork 5
/
Copy pathmmirs_pipeline_taskfile.py
1937 lines (1567 loc) · 68.5 KB
/
mmirs_pipeline_taskfile.py
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
"""
mmirs_pipeline_taskfile
====
Python code to create task files for MMIRS IDL pipeline:
http://tdc-www.harvard.edu/software/mmirs_pipeline.html
Operates in a given path containing raw files, find common files, and create
task files to execute with MMIRS IDL data reduction pipeline, mmirs-pipeline
This code will create several files:
(1) 'obs_summary.tbl'
- A summary of observations in a specified 'rawdir' with FITS header
information
(2) MMIRS input taskfiles
- FITS header compliant ASCII files that are provided to the
MMIRS IDL pipeline
(3) 'IDL_input_[name].lis'
- A list of files to run the MMIRS IDL pre-processing (i.e., non-linearity
correction)
Here, [name] is a naming convention that contains the name of the
target, LS or MOS mode, and grism and filter combinations. It is defined
from mmirs_pipeline_taskfile.
TO EXECUTE:
0. First, you will need python. This code has been tested to work with v2.7.15 and
3.6.8. I recommend installing through anaconda:
https://www.anaconda.com/download/
Next you will need the Astropy package. This code has been tested to work
with v1.3, v2.0.2, and v3.1.2 of Astropy. Install via the conda command:
conda install astropy
In addition, you will need the astroquery package (tested with 0.3.7).
Install via pip command:
pip install astroquery
1. After python, astropy, and astroquery are successfully installed, you
need to clone Chun Ly's MMTtools package to your local machine:
git clone https://github.com/astrochun/MMTtools.git
2. Within ipython (or python) import this code:
from MMTtools import mmirs_pipeline_taskfile
Note: If this does not work, MMTtools is not within your PYTHONPATH
environment
3. Remove FITS files from your raw path that you do not want this code to detect.
This code does a file search for '*.????.fits*'. If there are bad FITS
files or those that are saturated, relocate them to another folder or
delete them.
4. Call code for specified path in ipython (or python):
rawdir = '/path/to/raw/files/' <- Should end with forward slash
mmirs_pipeline_taskfile.create(rawdir, w_dir='', dither='ABApBp',
bright=True)
Notes:
1. w_dir can be changed. Default is to create a 'reduced' folder in [rawdir]
2. dither: If NOT specified, code will determine dither pattern based on
FITS header
3. Set bright to True of False if there is a bright object in slit or MOS
4. Set inter to True if multiple calibration datasets (including telluric star)
are available. The code will prompt user to select.
5. Note that mmirs-pipeline will look for a 'calib_MMIRS' folder. This
is needed in the pre-processing (e.g., applying non-linearity correction).
This Python code will prompt you to provide a path such that a symbolic
link is created.
For example, if mmirs-pipeline is installed in /codes/idl, then you would
type '/codes/idl', and it would create a symbolic link as follow:
ln -s /codes/idl/mmirs-pipeline/pipeline/calib_MMIRS [rawdir]/calib_MMIRS
If your [rawdir] contains multiple targets, mmirs_pipeline_taskfile _should_
separate out the targets in a respective manner.
5. Next install IDL and set-up it up to have an appropriate license.
Then have Igor Chilingarian's mmirs-pipeline on your computer and included
in your IDL_PATH environment:
git clone https://bitbucket.org/chil_sai/mmirs-pipeline
Also make sure that you have the IDL Astrolib installed and it is in your
IDL_PATH environment:
git clone https://github.com/wlandsman/IDLAstro
Note: legend.pro has been deprecated (part of IDL as al_legend.pro),
which will cause mmirs-pipeline to crash. Either change mmirs-pipeline to
use al_legend or include this legend.pro somewhere in your IDL_PATH:
https://idlastro.gsfc.nasa.gov/ftp/obsolete/legend.pro
6. Run the IDL script run_mmirs_pipeline_nonlin_script.idl that is
automatically generated from step 3 in the [rawdir] path:
idl run_mmirs_pipeline_nonlin_script.idl
Note: All the pre-processed files will be placed in the 'preproc' folder
within [rawdir].
7. After creating pre-processed files, you can now run the MMIRS pipeline via
the IDL scripts (run_mmirs_pipeline_[name].idl) that are automatically
generated from step 3 in the [rawdir] path
If your [rawdir] contains multiple targets, mmirs_pipeline_taskfile _should_
separate out the targets in a respective manner. Thus, there should be
multiple run_mmirs_pipeline.idl scripts
TIPS:
This code has a log file that is created in rawdir called
'mmirs_pipeline_taskfile.log'. It logs everything that is written to
stdout
If a bug is encountered please submit an issue ticket here:
https://github.com/astrochun/MMTtools/issues
Also, please email the creator, Chun Ly, at astro.chun [at] gmail.com
your mmirs_pipeline_taskfile.log, any error messages on the ipython
or python screen, and your 'obs_summary.tbl' file
"""
__version__ = '0.2' # Set on 15/09/2019. PyPI-ed and a few minor fixes
#__version__ = '0.2beta' # Set on 26/04/2019. - Py2 and 3 compatibility
#__version__ = '0.1' # Set on 16/02/2018
import sys, os
from os.path import exists
#Py3 compatibility
py_vers = sys.version_info.major
if py_vers == 3:
import subprocess as commands
raw_input = input
if py_vers == 2:
import commands
from astropy.io import ascii as asc
from astropy.io import fits
import numpy as np
import glob
from astropy.table import Table
from astropy import log
from astropy.time import Time # + on 31/01/2018
from socket import gethostname
from itertools import groupby, count
# Mod on 01/05/2018
use_simbad = 0
if use_simbad:
from astroquery.simbad import Simbad
if not any('sptype' in sfield for sfield in Simbad.get_votable_fields()):
Simbad.add_votable_fields('sptype')
import collections
import logging
formatter = logging.Formatter('%(asctime)s - %(module)12s.%(funcName)20s - %(levelname)s: %(message)s')
sh = logging.StreamHandler(sys.stdout)
sh.setLevel(logging.INFO)
sh.setFormatter(formatter)
# + on 30/11/2017
co_filename = __file__
co_path = os.path.dirname(co_filename) + '/'
# Fix if code is placed locally | Mod on 02/03/2018
if co_path == '/': co_path = ''
class mlog:
'''
Main class to log information to stdout and ASCII file
To execute:
mylog = mlog(rawdir)._get_logger()
Parameters
----------
rawdir : str
Full path for where raw files are
Returns
-------
Notes
-----
Created by Chun Ly, 19 February 2018
'''
def __init__(self,rawdir):
self.LOG_FILENAME = rawdir + 'mmirs_pipeline_taskfile.log'
self._log = self._get_logger()
def _get_logger(self):
loglevel = logging.INFO
log = logging.getLogger(self.LOG_FILENAME) # + Mod on 14/12/2017
if not getattr(log, 'handler_set', None):
log.setLevel(logging.INFO)
sh = logging.StreamHandler()
sh.setFormatter(formatter)
log.addHandler(sh)
fh = logging.FileHandler(self.LOG_FILENAME)
fh.setLevel(logging.INFO)
fh.setFormatter(formatter)
log.addHandler(fh)
log.setLevel(loglevel)
log.handler_set = True
return log
#enddef
def get_header_info(files0):
'''
Get information from FITS header
Parameters
----------
files0 : list
List containing full path to MMIRS files
Returns
-------
tab0: astropy.table.table
Astropy Table containing FITS header
Notes
-----
Created by Chun Ly, 28 November 2017
- Added documentation
- Return Astropy Table
- List for variables with strings
- Get proper FITS extension
- Add [seqno] for sorting purposes
Modified by Chun Ly, 30 November 2017
- Add airmass info
- Add date-obs info
- Re-order table columns
Modified by Chun Ly, 8 December 2017
- Add PI and PropID info
Modified by Chun Ly, 11 December 2017
- Add instrument elevation offset (dithers along slit)
Modified by Chun Ly, 17 February 2018
- Bug fix: Handle FITS files that are not multi-extension FITS
(i.e., only one read)
Modified by Chun Ly, 2 May 2019
- Check that FITS header contains all keys - From Sean Moran (CfA)
'''
n_files0 = len(files0)
filename = [] #np.array(['']*n_files0)
seqno = []
exptime = [] #np.zeros(n_files0)
airmass = [] #np.zeros(n_files0)
dateobs = [] #np.array(['']*n_files0)
object0 = [] #np.array(['']*n_files0)
imagetyp = [] #np.array(['']*n_files0)
aptype = [] #np.array(['']*n_files0)
aperture = [] #np.array(['']*n_files0)
filter0 = [] #np.array(['']*n_files0)
disperse = [] #np.array(['']*n_files0)
pi = [] # + on 08/12/2017
propid = [] # + on 08/12/2017
instel = [] #np.zeros(n_files0) # + on 11/12/2017
for ii in range(n_files0):
zhdr = fits.getheader(files0[ii], ext=0)
extend = 'EXTEND' in zhdr.keys()
if not extend:
hdr = zhdr.copy()
else:
hdr = fits.getheader(files0[ii], ext=1)
haskeys = 'OBJECT' in hdr.keys() and 'EXPTIME' in hdr.keys() and \
'AIRMASS' in hdr.keys() and 'FILENAME' in hdr.keys() and \
'DATE-OBS' in hdr.keys() and 'IMAGETYP' in hdr.keys() and \
'APTYPE' in hdr.keys() and 'APERTURE' in hdr.keys() and \
'FILTER' in hdr.keys() and 'DISPERSE' in hdr.keys() and \
'PI' in hdr.keys() and 'PROPID' in hdr.keys() and 'INSTEL' in hdr.keys()
if haskeys:
exptime.append(hdr['EXPTIME'])
airmass.append(hdr['AIRMASS'])
t_filename = hdr['FILENAME'].split('/')[-1]
seqno.append(np.int(t_filename.split('.')[-1]))
filename.append(t_filename)
dateobs.append(hdr['DATE-OBS'])
object0.append(hdr['OBJECT'])
imagetyp.append(hdr['IMAGETYP'])
aptype.append(hdr['APTYPE'])
aperture.append(hdr['APERTURE'])
filter0.append(hdr['FILTER'])
disperse.append(hdr['DISPERSE'])
pi.append(hdr['PI']) # + on 08/12/2017
propid.append(hdr['PROPID']) # + on 08/12/2017
instel.append(hdr['INSTEL']) # + on 11/12/2017
#endfor
arr0 = [filename, seqno, dateobs, object0, pi, propid, imagetyp,
aptype, exptime, airmass, aperture, filter0, disperse, instel]
names0 = ('filename','seqno','dateobs','object','PI','PropID','imagetype',
'aptype','exptime','airmass','aperture','filter','disperse','instel')
tab0 = Table(arr0, names=names0)
tab0.sort('seqno')
return tab0
#enddef
def get_header_comments(f0):
'''
Extract FITS header comments. This function is to resolve issue with FITS
block size, which prevents fits.Header.tofile() from working
NOTE: This code is now obsolete
Parameters
----------
f0 : list
List containing full header as strings
Returns
-------
comm0 : list
List containing FITS header comments
Notes
-----
Created by Chun Ly, 24 January 2018
Modified by Chun Ly, 25 January 2018
- Moved up to work with read_template()
Modified by Chun Ly, 22 February 2018
- Updated documentation
'''
split0 = [str0.split(' / ')[-1].replace('\n','') for str0 in f0]
comm0 = [split0[xx] if split0[xx] != f0[xx].replace('\n','') else ''
for xx in range(len(split0))]
return comm0
#enddef
def remove_padding(outfile, mylog=None):
'''
Removes space padding for FITS keyword value string. The padding is done
automatically by astropy.io.fits. This breaks mmirs-pipeline as the
IDL code does not remove space paddings
Parameters
----------
c_hdr0 : astropy.io.fits.header.Header
FITS header
outfile : str
Full path of filename to write updated FITS header file
Returns
-------
None
Notes
-----
Created by Chun Ly, 21 February 2018
- Add mylog keyword input; Implement stdout and ASCII logging with mlog()
- Bug fix: indentation mistake
- Bug fix: CONTINUE cards are lost. Major modification: Need to modify
ASCII file directly instead of FITS header object
'''
if type(mylog) == type(None): mylog = log
f1 = open(outfile, 'r')
str_hdr = f1.readlines()
f1.close()
i_fix = [xx for xx in range(len(str_hdr)) if '=' in str_hdr[xx]]
for tt in range(len(i_fix)):
split0 = str_hdr[i_fix[tt]].split('= ')
if len(split0) > 1:
right0 = split0[1]
if (right0[:3] != "' '") and ('OGIP' not in right0) and \
('SIMPLE' not in split0[0]):
lloc = right0.find("'")
rloc = right0.rfind("'")
if lloc != -1 and rloc != -1:
t_val = right0[lloc:rloc+1]
r_t_val = t_val.replace(' ','')
right0 = r_t_val.ljust(len(t_val), ' ')
str_hdr[i_fix[tt]] = split0[0].ljust(8)+'= '+right0+'\n'
else:
t_split = right0.split('/')
t_val = t_split[0]
r_t_val = t_val.replace(' ','')
if '/' in right0:
right0 = r_t_val+' /'+t_split[1].replace("\n",'')
else:
right0 = r_t_val
str_hdr[i_fix[tt]] = split0[0].ljust(8)+'= '+right0+'\n'
#endfor
mylog.info('Updating : '+os.path.basename(outfile))
f1 = open(outfile, 'w')
f1.writelines(str_hdr)
f1.close()
#enddef
def read_template(longslit=False, mos=False, mylog=None):
'''
Read in MMIRS templates to populate with information
Parameters
----------
longslit : bool
Indicate whether to use longslit template. Default: False
Either longslit=True or mos=True
mos : bool
Indicate whether to use MOS template. Default: False
Either mos=True or longslit=True
Returns
-------
temp_hdr : astropy.io.fits.header.Header
Astropy FITS-formatted header class
Notes
-----
Created by Chun Ly, 29 November 2017
Modified by Chun Ly, 30 November 2017
- Output ordered dictionary
Modified by Chun Ly, 11 December 2017
- Bug fix for splitting with '='
- Bug fix for COMMENT entries
Modified by Chun Ly, 18 December 2017
- Switch from dict to FITS header for simplification
- Update documentation
Modified by Chun Ly, 24 January 2018
- Return string list version of template
Modified by Chun Ly, 25 January 2018
- Call get_header_comments() to get FITS keywords' comments
- Return hdr0_comm, FITS keywords' comments
Modified by Chun Ly, 17 February 2018
- Remove return of f0 and hdr0_comm (obsolete)
Modified by Chun Ly, 19 February 2018
- Add mylog keyword input; Implement stdout and ASCII logging with mlog()
Modified by Chun Ly, 22 February 2018
- Remove extraneous commented out code
'''
if type(mylog) == type(None): mylog = log
if longslit == False and mos == False:
mylog.warn('Must specify longslit or mos keyword!!!')
mylog.warn('Exiting!')
return
if longslit == True:
temp_file = co_path + 'mmirs_longslit_template.txt'
if mos == True:
temp_file = co_path + 'mmirs_mos_template.txt'
#log.info('## Reading : '+temp_file)
f = open(temp_file)
f0 = f.readlines()
# + on 18/12/2017
temp_hdr = fits.Header.fromstring("".join(f0), sep='\n')
# hdr0_comm = get_header_comments(f0) # + on 25/01/2018
return temp_hdr
#enddef
def group_range(iterable):
l = list(iterable)
if len(l) > 1:
return range(l[0], l[-1]+1)
else:
return [l[0]]
def get_calib_files(name, tab0, mylog=None, inter=False):
'''
Get appropriate calibration files for each dataset
Parameters
----------
name : str
object + aperture + filter + disperse name from organize_targets()
tab0: astropy.table.table
Astropy Table containing FITS header info
Returns
-------
calib_dict0 : dict
Ordered dictionary containing information of calibration files
Notes
-----
Created by Chun Ly, 1 December 2017
Modified by Chun Ly, 5 December 2017
- Determine and return dark frames for science exposures
Modified by Chun Ly, 8 December 2017
- Return ordered dict instead of individual variables
Modified by Chun Ly, 11 December 2017
- Use PI and PropID to further filter out comps and flats. Need a time constraint
- Include darks for comps, flats
Modified by Chun Ly, 26 January 2018
- Minor code documentation
Modified by Chun Ly, 20 February 2018
- Add mylog keyword input; Implement stdout and ASCII logging with mlog()
Modified by Chun Ly, 1 May 2018
- Handle mos case when object is not the same as filename
(i.e., mask vs mask_filter)
- Handle case when comps are not available
- Handle case when flats are not available
- Handle mylog calls and calib_dict0 when comps/flats are not available
Modified by Chun Ly, 25 May 2018
- Change from underscore to colon separator
Modified by Chun Ly, 3 July 2018
- Handle use of dimflat
'''
if type(mylog) == type(None): mylog = log # + on 20/02/2018
len0 = len(tab0)
itype0 = tab0['imagetype']
aper0 = tab0['aperture']
filt0 = tab0['filter']
disp0 = tab0['disperse']
# + on 11/12/2017
pi = tab0['PI']
propid = tab0['PropID']
t_str = name.split(':')
t_obj = t_str[0]
t_ap = t_str[1]
t_filt = t_str[2]
t_disp = t_str[3]
i_obj = [ii for ii in range(len0) if t_obj in tab0['object'][ii]]
# + on 11/12/2017
t_pi = pi[i_obj[0]]
t_propid = propid[i_obj[0]]
# + on 05/12/2017
exptime = tab0['exptime']
dark_etime = list(set(np.array(exptime)[i_obj]))
dark_str0 = []
for etime in dark_etime:
i_dark = [ii for ii in range(len0) if
(itype0[ii] == 'dark' and exptime[ii] == etime)]
t_txt = ",".join(tab0['filename'][i_dark])
dark_str0.append(t_txt)
mylog.info("List of science dark files for %.3fs : %s" % (etime, t_txt))
## COMPS
# Mod on 11/12/2017
i_comp = [ii for ii in range(len0) if
(itype0[ii] == 'comp' and aper0[ii] == t_ap and
filt0[ii] == t_filt and disp0[ii] == t_disp and \
pi[ii] == t_pi and propid[ii] == t_propid)]
if len(i_comp) != 0: # Mod 01/05/2018
i_comp_grp = [group_range(g) for _, g in
groupby(i_comp, key=lambda n, c=count(): n-next(c))]
if len(i_comp_grp) > 1:
mylog.warn('Too many comp sets!!!') # Mod on 20/02/2018
if inter:
for cc in range(len(i_comp_grp)):
tmpt = tab0[i_comp_grp[cc][0]]
tmp_num = tab0['seqno'][i_comp_grp[cc]]
comp_str = '(%i) %i-%i ' % (cc, min(tmp_num), max(tmp_num))
comp_str += '%s %s+%s %s' % (tmpt['aperture'], tmpt['filter'],
tmpt['disperse'], tmpt['PropID'])
mylog.info(comp_str)
raw_in = raw_input("Select from above comp sets : ")
mylog.info("User selected : (%s) " % np.int(raw_in))
i_comp = i_comp_grp[np.int(raw_in)]
comp_str0 = ",".join(tab0['filename'][i_comp])
comp_itime = tab0['exptime'][i_comp[0]]
# Darks for comps
id_comp = [ii for ii in range(len0) if
(itype0[ii] == 'dark' and tab0['exptime'][ii] == comp_itime)]
comp_dark = ",".join(tab0['filename'][id_comp])
else:
mylog.warn('No comps found!!')
## FLATS
# Mod on 11/12/2017
i_flat = [ii for ii in range(len0) if
('flat' in itype0[ii] and aper0[ii] == t_ap and \
filt0[ii] == t_filt and disp0[ii] == t_disp and \
pi[ii] == t_pi and propid[ii] == t_propid)]
if len(i_flat) != 0: # Mod 01/05/2018
i_flat_grp = [group_range(g) for _, g in
groupby(i_flat, key=lambda n, c=count(): n-next(c))]
if len(i_flat_grp) > 1:
mylog.warn('Too many flat sets!!!') # Mod on 20/02/2018
if inter:
for cc in range(len(i_flat_grp)):
tmpt = tab0[i_flat_grp[cc][0]]
tmp_num = tab0['seqno'][i_flat_grp[cc]]
flat_str = '(%i) %i-%i ' % (cc, min(tmp_num), max(tmp_num))
flat_str += '%s %s+%s %s' % (tmpt['aperture'], tmpt['filter'],
tmpt['disperse'], tmpt['PropID'])
mylog.info(flat_str)
raw_in = raw_input("Select from above flat sets : ")
mylog.info("User selected : (%s) " % np.int(raw_in))
i_flat = i_flat_grp[np.int(raw_in)]
flat_str0 = ",".join(tab0['filename'][i_flat])
flat_itime = tab0['exptime'][i_flat[0]]
# Darks for flats
id_flat = [ii for ii in range(len0) if
(itype0[ii] == 'dark' and tab0['exptime'][ii] == flat_itime)]
flat_dark = ",".join(tab0['filename'][id_flat])
else:
mylog.warn('No flats found!!')
# Mod on 20/02/2018
if len(i_comp) != 0:
mylog.info("List of comp files : "+comp_str0)
mylog.info("List of comp dark files : "+comp_dark)
# Mod on 20/02/2018
if len(i_flat) != 0:
mylog.info("List of flat files : "+flat_str0)
mylog.info("List of flat dark files : "+flat_dark)
calib_dict0 = collections.OrderedDict()
# Mod on 01/05/2018
if len(i_comp) != 0:
calib_dict0['comp_str'] = comp_str0
calib_dict0['comp_dark'] = comp_dark # + on 11/12/2017
else:
calib_dict0['comp_str'] = ''
calib_dict0['comp_dark'] = ''
if len(i_flat) != 0:
calib_dict0['flat_str'] = flat_str0
calib_dict0['flat_dark'] = flat_dark # + on 11/12/2017
else:
calib_dict0['flat_str'] = ''
calib_dict0['flat_dark'] = ''
calib_dict0['dark_etime'] = dark_etime
calib_dict0['dark_str'] = dark_str0
return calib_dict0
#enddef
def handle_tellurics(tab0, object0, PropID, i_tell, obj_etime, tell_comb0,
idx, target_setup, mmirs_setup0, inter=False, mylog=None):
'''
Handle when multiple telluric datasets are available
Parameters
----------
tab0: astropy.table.table
Astropy Table containing FITS header
object0 : list
List of strings of the source name (handle MOS case)
PropID: str
Proposal ID full name (for science exposures)
i_tell : list or np.array
Index of entries that are telluric stars
obj_etime : list
Telluric name and exposure time
tell_comb0 : list
List of strings that combines telluric name and exposure time
idx : list or np.array
Index of entries for a given target
target_setup : str
String that combines aperture, filter, and disperse for specific
target
mmirs_setup0 : list
List of strings that combines aperture, filter, and disperse.
This gets proper setup for telluric star identification
inter : boolean
For interactive telluric star selection. Default: False
Returns
-------
rev_tell_comb0 : list
Improved telluric name and exposure time
Notes
-----
Created by Chun Ly, 5 March 2018
Modified by Chun Ly, 6 March 2018
- Simplification improvements
- Add idx input
- Add mylog keyword input; Implement stdout and ASCII logging with mlog()
- Simplification improvements (cont'd)
- Add pass_score to tracking
- Check if sci data is bracketed with telluric data
- Return improved list of telluric data
Modified by Chun Ly, 6 March 2018
- Bug fix: log -> mylog changes
Modified by Chun Ly, 1 May 2018
- Fix to work with python3
- Handle no telluric case for rev_tell_comb0
- Fix typo for rev_tell_comb0
Modified by Chun Ly, 4 June 2018
- List all tellurics if more than one
Modified by Chun Ly, 8 June 2018
- Include inter keyword option
- Add user prompts to identify telluric star
- Bug fix: indexing issue
Modified by Chun Ly, 9 June 2018
- Add mmirs_setup0 and target_setup inputs
- Require target_setup in telluric selection
Modified by Chun Ly, 10 June 2018
- Require target_setup in telluric selection (cont'd)
- mylog.info spec setup info for telluric star list based on PropID
- Clean up code
- Include PropID info for interactive case
- Bug fix: Correct index from user inputs
Modified by Chun Ly, 3 July 2018
- Allow negative input for interactive telluric selection (no telluric)
- Bug fix for handling list input -> str
Modified by Chun Ly, 12 July 2018
- Bug fix: Fix TypeError for tmp_num: a number is required, not
numpy.string_
'''
if type(mylog) == type(None): mylog = log # + on 06/03/2018
obj_etime = np.array(obj_etime) # Mod on 06/03/2018
mmirs_setup0 = np.array(mmirs_setup0) # + on 09/06/2018
pass_score = 0 # + on 06/03/2018
# First distinguish by PropID
i_prop = [xx for xx in range(len(tab0)) if tab0['PropID'][xx] == PropID]
i_pid = np.array(list(set(i_prop) & set(i_tell)))
if len(i_pid) > 0:
obj_etime_pid = list(set(obj_etime[i_pid]))
setup_pid = list(set(mmirs_setup0[i_pid])) # + on 10/06/2018
if len(obj_etime_pid) == 1:
mylog.info('Only one telluric dataset found using PropID !!!')
mylog.info('## '+obj_etime_pid[0]+' '+setup_pid[0]) # + on 10/06/2018
pass_score = 1 # + on 06/03/2018
rev_tell_comb0 = list(obj_etime_pid) # + on 06/03/2018
else:
mylog.warn('More than one telluric dataset found using PropID : '+\
str(len(obj_etime_pid)))
for tell_name_test in obj_etime_pid: # + on 04/06/2018
tell_idx = [xx for xx in range(len(i_pid)) if
obj_etime[i_pid][xx] == tell_name_test]
# + on 10/06/2018
mylog.info('## '+tell_name_test+' '+mmirs_setup0[i_pid[tell_idx[0]]])
else:
mylog.warn('No tellurics found using PropID !!!')
rev_tell_comb0 = list(tell_comb0) # + on 06/03/2018
# Determine if telluric data bracket sci data | + on 06/03/2018
if not pass_score:
i_obj = [ii for ii in range(len(tab0)) if
(tab0['imagetype'][ii] == 'object' and tab0['aptype'][ii] != 'open')]
tab0_nocalib = tab0[i_obj]
obj_etime_nocalib = obj_etime[i_obj]
mmirs_setup_nocalib = mmirs_setup0[i_obj] # + on 09/06/2018
tell_idx_min = np.zeros(len(tell_comb0))
tell_idx_max = np.zeros(len(tell_comb0))
for tt in range(len(tell_comb0)):
# Mod on 09/06/2018
t_idx = [xx for xx in range(len(tab0_nocalib)) if
(obj_etime_nocalib[xx] == tell_comb0[tt] and
mmirs_setup_nocalib[xx] == target_setup)]
#t_idx = np.array(i_obj)[t_idx]
tell_idx_min[tt], tell_idx_max[tt] = min(t_idx), max(t_idx)
idx_nocalib = [ii for ii in range(len(tab0_nocalib)) if
obj_etime_nocalib[ii] == obj_etime[idx[0]]]
#idx_nocalib = np.array(i_obj)[idx_nocalib]
sci_idx_min, sci_idx_max = min(idx_nocalib), max(idx_nocalib)
tmp_tell_comb0 = [] # + on 06/03/2018
# Mod on 07/06/2018
if inter == False:
# Check before
bef0 = np.where(tell_idx_max - sci_idx_min == -1)[0]
if len(bef0) == 1:
mylog.info('Telluric data found before science data : '+\
tell_comb0[bef0[0]])
tmp_tell_comb0.append(tell_comb0[bef0[0]])
pass_score += 1
else:
mylog.info('NO telluric data before science data')
# Check after
aft0 = np.where(tell_idx_min - sci_idx_max == 1)[0]
if len(aft0) == 1:
mylog.info('Telluric data found after science data : '+tell_comb0[aft0[0]])
tmp_tell_comb0.append(tell_comb0[aft0[0]]) # + on 06/03/2018
pass_score += 1
else:
mylog.info('NO telluric data after science data')
# + on 06/03/2018
if len(bef0) == 1 or len(aft0) == 1:
rev_tell_comb0 = tmp_tell_comb0
if len(bef0) == 0 or len(aft0) == 0:
rev_tell_comb0 = []
else:
print(tab0[idx])
# Check before
bef0 = np.where(tell_idx_max - sci_idx_min < 0)[0]
if len(bef0) > 0:
mylog.info('Select telluric star BEFORE science observations : ')
for bb in range(len(bef0)):
bb_bef = bef0[bb]
t_idx = [xx for xx in range(len(tab0_nocalib)) if
(obj_etime_nocalib[xx] == tell_comb0[bb_bef] and
mmirs_setup_nocalib[xx] == target_setup)]
tmpt = tab0_nocalib[t_idx][0]
tmp_num = np.int_(tab0_nocalib['seqno'][t_idx])
tell_str = '(%i) %i-%i ' % (bb, min(tmp_num), max(tmp_num))
tell_str += '%s %s %s+%s %s' % (tell_comb0[bb_bef], tmpt['aperture'],
tmpt['filter'], tmpt['disperse'],
tmpt['PropID'])
mylog.info(tell_str)
raw_bef = raw_input("Select from above telluric star to use : ")
if np.int(raw_bef) < 0:
mylog.info('Negative value provided -> No before telluric star')
else:
tmp_tell_comb0.append(tell_comb0[bef0[np.int(raw_bef)]])
mylog.info("User selected : (%s) " % raw_bef)
# Check after
aft0 = np.where(tell_idx_min - sci_idx_max > 0)[0]
if len(aft0) > 0:
mylog.info('Select telluric star AFTER science observations : ')
for bb in range(len(aft0)):
bb_aft = aft0[bb]
t_idx = [xx for xx in range(len(tab0_nocalib)) if
(obj_etime_nocalib[xx] == tell_comb0[bb_aft] and
mmirs_setup_nocalib[xx] == target_setup)]
tmpt = tab0_nocalib[t_idx][0]
tmp_num = np.int_(tab0_nocalib['seqno'][t_idx])
tell_str = '(%i) %i-%i ' % (bb, min(tmp_num), max(tmp_num))
tell_str += '%s %s %s+%s %s' % (tell_comb0[bb_aft], tmpt['aperture'],
tmpt['filter'], tmpt['disperse'],
tmpt['PropID'])
mylog.info(tell_str)
raw_aft = raw_input("Select from above telluric star to use : ")
if np.int(raw_aft) < 0:
mylog.info('Negative value provided -> No after telluric star')
else:
tmp_tell_comb0.append(tell_comb0[aft0[np.int(raw_aft)]])
mylog.info("User selected : (%s) " % raw_aft)
rev_tell_comb0 = tmp_tell_comb0
#endif
return rev_tell_comb0
#enddef
def get_tellurics(tab0, idx, comb0, object0, mmirs_setup0, inter=False, mylog=None):
'''
Determining tellurics to use
Parameters
----------
tab0: astropy.table.table
Astropy Table containing FITS header
idx : list or np.array
Index of entries for a given target
comb0 : list
List of strings that combines name, aperture, filter, and disperse
object0 : list
List of strings that contains name of source. This is from
organize_targets() and handles mask observations
mmirs_setup0 : list
List of strings that combines aperture, filter, and disperse.
This handle MOS cases to get proper setup for telluric star identification
inter : boolean
For interactive telluric star selection. Default: False
Returns
-------
tell_dict0 : dict
Dictionary containing telluric filenames, exptime, and associated darks
Notes
-----
Created by Chun Ly, 25 January 2018
- Require that str_tell be a list
Modified by Chun Ly, 26 January 2018
- Minor code documentation
- Bug fix: incorrect indexing, tmp -> i_tell
Modified by Chun Ly, 28 January 2018
- Include exptime in separating telluric datasets
- Get darks for each telluric datasets, str_dark
- Return full telluric information through dictionary, tell_dict0
Modified by Chun Ly, 29 January 2018
- Bug fixes: len(n_tell) -> n_tell, str(etime), specify size when calling range
- Import astroquery.simbad to get spectral type for telluric star; export
in tell_dict0
Modified by Chun Ly, 18 February 2018
- Handle mos case for tellurics with object0 input
- Bug fix : obj -> object0
Modified by Chun Ly, 20 February 2018
- Add mylog keyword input; Implement stdout and ASCII logging with mlog()
Modified by Chun Ly, 5 March 2018
- Call handle_tellurics() function
Modified by Chun Ly, 6 March 2018
- Pass idx to handle_tellurics()
- Get improved telluric list from handle_tellurics()
Modified by Chun Ly, 7 March 2018
- Pass mylog to handle_tellurics()
- str_tell, tell_etime, str_dark and tell_stype definitions after
handle_tellurics() call
Modified by Chun Ly, 12 March 2018
- Add BD telluric star in telluric list comprehension
- Handle BD telluric star names for call to Simbad
Modified by Chun Ly, 17 March 2018
- mmirs-pipeline only accept a0v or g0v stars, changing this to work with
templates
Modified by Chun Ly, 1 May 2018
- Turn off Simbad query if use_simbad=0
- Add mmirs_setup0 input
- Bug fix: mmirs_setup -> mmirs_setup0
Modified by Chun Ly, 8 June 2018