FSSC Science Tools

Data Retrieval, Preparation, and Exploration

This tutorial is designed to not only make you comfortable running a LAT analysis but also make you familiar with the data and where to go if you want to find more information. At the end of this tutorial and the pulsar analysis tutorial you should be able to know what all of the various file formats are and be able to perform a simple analysis.

In [1]:
from IPython.display import Image,HTML

Multiple levels of documentation

Available from FSSC WebsiteDataDocumentation

In [2]:
HTML("<iframe src='http://fermi.gsfc.nasa.gov/ssc/data/analysis/documentation/' width='1000' height='500'></iframe>")
Out[2]:
  • Installation
    • Details the installation procedure
  • LAT Analysis Start Page
    • Summary of the important resources that are needed for standard LAT analysis
  • Analysis Threads
    • Examples of actual science analysis where you can follow along step-by-step
  • Cicerone
    • General information on the satellite and instruments
    • Describes instrumentation and data acquisition
    • Explains analysis methods
  • Individual Tool Descriptions
    • Explains individual tools in detail
    • (Identical to the information available via fhelp)

Downloading and installing the software

  • Important things about software installation
    • Pay attention to the supported platforms, if yours isn't supported, it probably isn't supported.
    • Figure out your distribution with uname -a
In [3]:
%system uname -a
Out[3]:
['Darwin gs66-kaylee 13.4.0 Darwin Kernel Version 13.4.0: Wed Mar 18 16:20:14 PDT 2015; root:xnu-2422.115.14~1/RELEASE_X86_64 x86_64']

Science Analysis Threads

Go to the FSSC WebsiteDataData AnalysisAnalysis Threads

  • A good place to start is the data analysis threads. These are step-by-step guides that reproduce actual science results.
  • The best way to learn how to do something is to do it.
  • Lots of good stuff:
In [4]:
HTML("<iframe src='http://fermi.gsfc.nasa.gov/ssc/data/analysis/scitools/' width='1000' height='500'></iframe>")
Out[4]:

What if I can't figure it out?

Contact us! fermihelp@milkyway.gsfc.nasa.gov . Most of us are nice...most of the time.

Getting the data

Go to the FSSC WebsiteDataData Access

In [5]:
HTML("<iframe src='http://fermi.gsfc.nasa.gov/ssc/data/access/' width='1000' height='500'></iframe>")
Out[5]:

Lits of data products are available. What you use depends on what you're doing. Here are some highlights:

  • I want to perform a detailed source anaylsis:

    • LAT Data Server (Pass 7 Reprocessed data)
  • I want to perform an analysis of a short timescale flare like a GRB or a Solar Flare

    • LAT Low-Energy (LLE) Data
    • Fermi Solar Flare Observations
    • GBM Data
  • I want to monitor all sources in the full sky

    • Pass 7 Reprocessed Weekly files
  • I want to be alerted to flares in known sources

    • LAT Monitored Source List Light Curves
    • Aperture Photometry Light Curves for the LAT 2-year Point Source Catalog
  • I want to find out about known sources, GRBs, or solar flares.

    • LAT Burst Catalog
    • LAT 4-year Point Source Catalog
    • LAT List of Flaring Sources
    • LAT Pulsar Catalog
    • LAT Hard Sources Catalog
    • GBM Burst Catalog
    • GBM Earth Occultation Sources
    • Fermi Solar Flare Data
  • I want to know what the spacecraft was doing

    • Spacecraft Data
    • Spacecraft Pointing Files

Don't forget to pay attention to the caveats for the data products you are using. You never know, it might say don't do what you're trying to do or tell you to do something else!

Pick a pulsar

Download and unzip/untar the second pulsar catalog archive. You should find the timing files under 2PC_auxiliary_files_v04/par_files . You should look at the .par file for your source of interest to determine the validity interval for the timing solution.

In [6]:
HTML("<iframe src='http://fermi.gsfc.nasa.gov/ssc/data/access/lat/2nd_PSR_catalog/' width='850' height='500'></iframe>")
Out[6]:
In [44]:
%system fv /Users/eferrara/FSSC/VM/shared/SecondPulsarCatalogFIles/2PC_auxiliary_files_v04/PSR_data/FITS/PSRJ0248+6021_2PC_data.fits
Out[44]:
[]
In [26]:
Image(filename='images/2PC_J0248_psr_data.png')
Out[26]:
In [43]:
%system cat /Users/eferrara/FSSC/VM/shared/SecondPulsarCatalogFIles/2PC_auxiliary_files_v04/par_files/PSRJ0007+7303_2PC_2.par
Out[43]:
['PSRJ           J0007+7303',
 'RAJ            00:07:01.56',
 'DECJ           73:03:08.3',
 'F0             3.1656273496983659374  0.00000000362589170414',
 'F1             -3.5815680310574102946e-12  2.6873539591288110219e-16',
 'F2             8.8657070674040351073e-22  2.2520701819818733927e-23',
 'PEPOCH         55600',
 'POSEPOCH       55600',
 'DMEPOCH        55600',
 'GLEP_2         55465.649999999999999',
 'GLPH_2         0',
 'GLF0_2         3.9512901101121276016e-06',
 'GLF1_2         -3.5090255812328969218e-14',
 'START          55010.798475736919499',
 'FINISH         55961.097272243612679',
 'TZRMJD         55481.148395659399906744',
 'TZRFRQ         0',
 'TZRSITE        coe',
 'TRES           2539.761',
 'EPHVER         5',
 'CLK            TT(TAI)',
 'MODE           1',
 'UNITS          TDB',
 'EPHEM          DE405',
 'NITS           1',
 'NTOA           100',
 'CHI2R          3.0727  79',
 'WAVEEPOCH      55600',
 'WAVE_OM        0.0033517327682003  0',
 'WAVE1          1969.7662484155  -5004.6447565024',
 'WAVE2          -3028.7565222605  3252.5253098962',
 'WAVE3          2902.3584103003  -1389.5124815123',
 'WAVE4          -2019.0833293061  146.75472363379',
 'WAVE5          1043.4985284817  323.23722654411',
 'WAVE6          -387.71605561554  -308.38955660858',
 'WAVE7          92.345169199857  157.81039021016',
 'WAVE8          -8.4642199625054  -51.299017043953',
 'WAVE9          -1.9995719403781  10.057763762042',
 'WAVE10         0.54767649563687  -0.91893036659144']

This pulsar, in the SNR CTA1, was the first to be discovered by the Fermi-LAT. Even though the 2PC catalog used three years of LAT data, the timing solution is only valid for 2.6 years (MJD 55010 - 55961). This is not uncommon for young pulsars, as frequent glitches require these sources to be recharacterized on a regular basis. You can see from the number of wave terms at the end of the file that this is a difficult pulsar to time.

So, instead, let's look at a pulsar that is being regularly timed by radio telescopes.

In [42]:
%system cat /Users/eferrara/FSSC/VM/shared/SecondPulsarCatalogFIles/2PC_auxiliary_files_v04/par_files/PSRJ0248+6021_2PC.par
Out[42]:
['PSRJ           J0248+6021',
 'RAJ            02:48:18.6271820',
 'DECJ           +60:21:34.92726',
 'F0             4.6060228132407530083  1  0.00000000000296652185',
 'F1             -1.1676372961612546569e-12  1  1.1798851346771129241e-19',
 'F2             3.1898613815637571987e-24  1  5.617460107829222999e-27',
 'F3             -2.874207791559120525e-32  1  5.0270445136628880472e-34',
 'F4             1.7679367024475266472e-39  1  1.1627013748622322271e-41',
 'PEPOCH         54000.000168324422816',
 'POSEPOCH       54000.000168324422816',
 'DMEPOCH        54000.000168324422816',
 'DM             369.47966018314969219  0.01408210007430153732',
 'DM1            0.28364834104484394758  0.00983348499993829013',
 'PMRA           -24.836168842180075172  1.72832902680280420249',
 'PMDEC          59.210606343041537677  2.23010128724490330754',
 'PX             0.1',
 'GLEP_1         54897.41',
 'GLEP_2         53900',
 'GLEP_3         55657  1',
 'GLPH_3         -1.1269157593186818712  1  0.00450649843720785526',
 'GLF0_1         3.4384114658567793291e-06  0.00000000000315676397',
 'GLF0_2         1.3685030498465448546e-11  0.00000000000360783942',
 'GLF0_3         2.3231669426707142958e-06  1  0.00000000125882805502',
 'GLF1_1         -3.6240716054499395222e-15  1.1206123234362808014e-19',
 'GLF1_3         -3.4883804242361686036e-15  1  1.7471109817374488138e-16',
 'START          53306.990912117056723',
 'FINISH         55789.209591344144428',
 'TZRMJD         54546.63838513551049167',
 'TZRFRQ         2048',
 'TZRSITE        f',
 'TRES           171.461',
 'EPHVER         5',
 'CLK            TT(TAI)',
 'MODE           1',
 'EPHEM          DE200',
 'NITS           1',
 'NTOA           911',
 'CHI2R          15.1055  901']

Much better! No wave terms means the timing model describes the data very well!

The 2PC Catalog has a huge amount of useful information, including:

  • An overview FITS file that summarizes the content of the paper tables for each pulsar
  • Timing parameter files for all the pulsars in the catalog
  • Plots showing the radio and gamma-ray light curves, the best-fit gamma-ray light curve, and the gamma-ray light curves in different energy ranges
  • Plots showing the results of the spectral analysis, with the best-fit model overplotted on the individual spectral points
  • Individual FITS (and ascii) files for each pulsar that include the data points and errors used for:
    • all the gamma-ray light curves
    • the best-fit gamma-ray light curve
    • the radio light curve
    • the individual spectral points
    • the best-fit spectral model

This is a very useful archive for preparing for publications. I encourage you to take a look at the contents.

In [41]:
%system ls /Users/eferrara/FSSC/VM/shared/SecondPulsarCatalogFIles/2PC_auxiliary_files_v04/
Out[41]:
['2PC_catalog_v04.asc',
 '2PC_catalog_v04.fits',
 'PSR_data',
 'images',
 'online_supplement_overview.pdf',
 'par_files']

Get the LAT data for your pulsar

Now we need the data for the pulsar. Since this pulsar is in the plane of the Galaxy (b=0.697), I have downloaded data for a 30-degree ROI. The data have been selected to match the validity period of the timing file. I'm using a directory called 'psr_data' in my shared folder.

In [13]:
HTML("<iframe src='http://fermi.gsfc.nasa.gov/cgi-bin/ssc/LAT/LATDataQuery.cgi' width='1000' height='500'></iframe>")
Out[13]:

How do I fill in these parameters?

(You can click on the field name for more details.)

  • Object name or coordinates: you can use the NED/Simbad/GRB Name or the coordinates (like RA,Dec)
    • If you use coordinate lookup, be careful with the coordinates it uses. A name may correspond to several different positions.
  • Search Radius: This depends on your source region. I usually do an anlysis on 10 - 15 degrees but I always download 30 degrees of data so I have freedom later to change my mind. If you want to grab more than this, you probably should think about using the weekly files.
  • Observation dates: Lots of options (see the help). xtime is very useful.
  • Energy Range: separated by a comma.
  • LAT data type: Photon or Extended, more on this later.
  • Spacecraft data: yes. You want this.

I've already downloaded these data. You can get them in the data directory in the USB drive we've given you. Here's the resulting page (for the record).

In [14]:
Image(filename='images/FSSC_P8_DataServer1.png')
Out[14]:

If you click on the link to get to the query result you get this:

In [15]:
Image(filename='images/FSSC_P8_DataServer2.png')
Out[15]:

You could click the Available links or you could just use the wget commands (I added the '-q' for quiet).

In [17]:
%system wget -q http://fermi.gsfc.nasa.gov/FTP/fermi/data/lat/queries/L151119044834C9B75A5E69_SC00.fits
%system wget -q http://fermi.gsfc.nasa.gov/FTP/fermi/data/lat/queries/L151119044834C9B75A5E69_PH01.fits
%system wget -q http://fermi.gsfc.nasa.gov/FTP/fermi/data/lat/queries/L151119044834C9B75A5E69_PH03.fits
%system wget -q http://fermi.gsfc.nasa.gov/FTP/fermi/data/lat/queries/L151119044834C9B75A5E69_PH07.fits
%system wget -q http://fermi.gsfc.nasa.gov/FTP/fermi/data/lat/queries/L151119044834C9B75A5E69_PH04.fits
%system wget -q http://fermi.gsfc.nasa.gov/FTP/fermi/data/lat/queries/L151119044834C9B75A5E69_PH06.fits
%system wget -q http://fermi.gsfc.nasa.gov/FTP/fermi/data/lat/queries/L151119044834C9B75A5E69_PH02.fits
%system wget -q http://fermi.gsfc.nasa.gov/FTP/fermi/data/lat/queries/L151119044834C9B75A5E69_PH05.fits
%system wget -q http://fermi.gsfc.nasa.gov/FTP/fermi/data/lat/queries/L151119044834C9B75A5E69_PH08.fits
%system wget -q http://fermi.gsfc.nasa.gov/FTP/fermi/data/lat/queries/L151119044834C9B75A5E69_PH00.fits
In [40]:
%system ls /Users/eferrara/FSSC/VM/shared/psr_data/L151119044834C9B75A5E69_*.fits
Out[40]:
['/Users/eferrara/FSSC/VM/shared/psr_data/L151119044834C9B75A5E69_PH00.fits',
 '/Users/eferrara/FSSC/VM/shared/psr_data/L151119044834C9B75A5E69_PH01.fits',
 '/Users/eferrara/FSSC/VM/shared/psr_data/L151119044834C9B75A5E69_PH02.fits',
 '/Users/eferrara/FSSC/VM/shared/psr_data/L151119044834C9B75A5E69_PH03.fits',
 '/Users/eferrara/FSSC/VM/shared/psr_data/L151119044834C9B75A5E69_PH04.fits',
 '/Users/eferrara/FSSC/VM/shared/psr_data/L151119044834C9B75A5E69_PH05.fits',
 '/Users/eferrara/FSSC/VM/shared/psr_data/L151119044834C9B75A5E69_PH06.fits',
 '/Users/eferrara/FSSC/VM/shared/psr_data/L151119044834C9B75A5E69_PH07.fits',
 '/Users/eferrara/FSSC/VM/shared/psr_data/L151119044834C9B75A5E69_PH08.fits',
 '/Users/eferrara/FSSC/VM/shared/psr_data/L151119044834C9B75A5E69_SC00.fits']

I always record the information from my query with my data. The same information is also stored in the header of the file and can be viewed with gtvcut.

In [64]:
%system cat /Users/eferrara/FSSC/VM/shared/psr_data/query.txt
Out[64]:
['Equatorial coordinates (degrees)\t(42.0777,60.3595)',
 'Time range (MET)\t(239557417,335145602)',
 'Time range (Gregorian)\t(2008-08-04 15:43:36,2011-08-16 00:00:00)',
 'Energy range (MeV)\t(100,300000)',
 'Search radius (degrees)\t30']

So, what do we have now?

We've got nine photon files (this is a result of how the data server processes queries) and one spacecraft file (always one). Let's take a look at these with the ftool fv and see what we have.

In [39]:
%system fv /Users/eferrara/FSSC/VM/shared/psr_data/L151119044834C9B75A5E69_PH00.fits
Out[39]:
[]

Here's what you should see. Let's poke at the GUI for a bit and talk about what we have.

In [27]:
Image(filename='images/J0248_event_file.png')
Out[27]:

This is a FITS file which is an open standard defining a digital file format useful for storage, transmission and processing of scientific and other images. It has three Header Data Units (HDUs); the first is (and always must be) an image even if it's empty (as it is in this case). The second and third are binary types and store data. All of our data are in the second hdu (index = 1) and you can see that there are 22 columns and 149459 rows. This means we have 149,459 events in this single photon file! The last HDU is a binary table called GTI which details the good time intervals for the data in this file. If you poke the 'Header' button for the first HDU (the empty image) you get this.

In [29]:
Image(filename='images/FSSC_fv2.png')
Out[29]:

All sorts of useful information. Note that there are some useful ftools to do this.

In [38]:
%system ftlist /Users/eferrara/FSSC/VM/shared/psr_data/L151119044834C9B75A5E69_PH00.fits H
Out[38]:
['',
 '        Name               Type       Dimensions',
 '        ----               ----       ----------',
 'HDU 1   Primary Array      Null Array                               ',
 'HDU 2   EVENTS             BinTable    23 cols x 950176 rows        ',
 'HDU 3   GTI                BinTable     2 cols x 2382 rows          ']

Let's close the header and click on the 'All' button for the sedcond HDU (the one with all of our data). It should look like this:

In [32]:
Image(filename='images/FSSC_fv3.png')
Out[32]:

There's a row for every event and each column details all of the information about that event. You could click the header to find more details or check out the information about the columns on the FSSC site. We can have a look at them. And you can see what the 'extended' data are all about.

In [33]:
HTML("<iframe src='http://fermi.gsfc.nasa.gov/ssc/data/analysis/documentation/Cicerone/Cicerone_Data/LAT_Data_Columns.html' width='1000' height='500'></iframe>")
Out[33]:
In [36]:
%system pwd
Out[36]:
['/Users/eferrara/FSSC/VM/shared/Notebooks']

The extended file just has some more information about each event. This page also details what's in the spacecraft file. You can use 'fv' to plot some of these parameters or we can actually plot them here in python.

In [34]:
import pyfits
In [68]:
sc_hdulist = pyfits.open('/Users/eferrara/FSSC/VM/shared/psr_data/L151119044834C9B75A5E69_SC00.fits')
In [69]:
sc_hdulist.info()
Filename: /Users/eferrara/FSSC/VM/shared/psr_data/L151119044834C9B75A5E69_SC00.fits
No.    Name         Type      Cards   Dimensions   Format
0    PRIMARY     PrimaryHDU      28   ()           uint8   
1    SC_DATA     BinTableHDU    265   5088941R x 30C   [D, D, 3E, E, E, D, E, E, E, E, E, E, L, E, E, E, E, E, E, E, J, B, I, D, D, D, D, D, E, E]   

For simplicity, we will rename the spacecraft file to "spacecraft.fits".

In [ ]:
%system mv /Users/eferrara/FSSC/VM/shared/psr_data/L151119044834C9B75A5E69_SC00.fits /Users/eferrara/FSSC/VM/shared/psr_data/spacecraft.fits

Filtering and Selecting Data

You can filter and select on any of the parameters of the events but there are some recommendations and good practices listed at the FSSC site.

Event Selection Recommendations (P8R2)

Analysis Type Minimum Energy
(emin)
Maximum Energy
(emax)
Max Zenith Angle
(zmax)
Event Class
(evclass)
IRF Name
Galactic Point Source Analysis 100 (MeV) 500000 (MeV) 90 (degrees) 128 P8R2_SOURCE_V6
Off-plane Point Source Analysis 100 (MeV) 500000 (MeV) 90 (degrees) 128 P8R2_SOURCE_V6
Burst and Transient Analysis (<200s) 100 (MeV) 500000 (MeV) 100 (degrees) 16 P8R2_TRANSIENT020_V6
Galactic Diffuse Analysis 100 (MeV) 500000 (MeV) 90 (degrees) 128 P8R2_SOURCE_V6
Extra-Galactic Diffuse Analysis 100 (MeV) 500000 (MeV) 90 (degrees) 1024 P8R2_ULTRACLEANVETO_V6
Impulsive Solar Flare Analysis 100 (MeV) 500000 (MeV) 100 (degrees) 65536 P8R2_TRANSIENT015S_V6

Let's talk about these for a bit:

  • The minimum and maximum energy cuts are pretty obvious. Don't really want to go down below 100 MeV and most of the time could actually raise this to 200 MeV.
  • Zenith Angle: This cut is there to remove events originating from the Earth's limb.
  • Event Class: The LAT team makes cuts on the raw data that classifies events based on the probability that they are photons, and on desired level of EGB contanimation. Each class has it's own Instrument Response Functions (the description of how the instrument responds to events).

What makes up an IRF?

You can find details of the Instrument Response Functions at the LAT Performance Page

If you install the Fermi Science Tools, the IRFs are also available in the following directory:

$FERMI_DIR/refdata/fermi/caldb/CALDB/data/glast/lat/bcf

In [70]:
HTML("<iframe src='http://www.slac.stanford.edu/exp/glast/groups/canda/lat_Performance.htm' width='1000' height='500'></iframe>")
Out[70]:
In [47]:
from os import environ
In [48]:
environ['CALDB']
Out[48]:
'/usr/local/ScienceTools/refdata/fermi/caldb/CALDB'
In [51]:
ls /usr/local/ScienceTools/refdata/fermi/caldb/CALDB/data/glast/lat/bcf
ea/                           irf_index.fits                valid_evclass_selections.txt
edisp/                        psf/                          valid_evtype_selections.txt
In [52]:
ls '/usr/local/ScienceTools/refdata/fermi/caldb/CALDB/data/glast/lat/bcf/ea'
aeff_P6_v11_diff_back.fits              aeff_P7REP_TRANSIENT_V15_back.fits      aeff_P8R2_TRANSIENT015S_V6_EDISP.fits
aeff_P6_v11_diff_front.fits             aeff_P7REP_TRANSIENT_V15_front.fits     aeff_P8R2_TRANSIENT015S_V6_FB.fits
aeff_P6_v1_diff_back.fits               aeff_P7REP_ULTRACLEAN_V10_back.fits     aeff_P8R2_TRANSIENT015S_V6_PSF.fits
aeff_P6_v1_diff_front.fits              aeff_P7REP_ULTRACLEAN_V10_front.fits    aeff_P8R2_TRANSIENT020E_V6_EDISP.fits
aeff_P6_v1_source_back.fits             aeff_P7REP_ULTRACLEAN_V15_back.fits     aeff_P8R2_TRANSIENT020E_V6_FB.fits
aeff_P6_v1_source_front.fits            aeff_P7REP_ULTRACLEAN_V15_front.fits    aeff_P8R2_TRANSIENT020E_V6_PSF.fits
aeff_P6_v1_trans_back.fits              aeff_P7SOURCE_V6MC_back.fits            aeff_P8R2_TRANSIENT020_V6_EDISP.fits
aeff_P6_v1_trans_front.fits             aeff_P7SOURCE_V6MC_front.fits           aeff_P8R2_TRANSIENT020_V6_FB.fits
aeff_P6_v3_diff_back.fits               aeff_P7SOURCE_V6_back.fits              aeff_P8R2_TRANSIENT020_V6_PSF.fits
aeff_P6_v3_diff_front.fits              aeff_P7SOURCE_V6_front.fits             aeff_P8R2_TRANSIENT100E_V6_EDISP.fits
aeff_P6_v3_source_back.fits             aeff_P7TRANSIENT_V6_back.fits           aeff_P8R2_TRANSIENT100E_V6_FB.fits
aeff_P6_v3_source_front.fits            aeff_P7TRANSIENT_V6_front.fits          aeff_P8R2_TRANSIENT100E_V6_PSF.fits
aeff_P6_v3_trans_back.fits              aeff_P7ULTRACLEAN_V6_back.fits          aeff_P8R2_TRANSIENT100S_V6_EDISP.fits
aeff_P6_v3_trans_front.fits             aeff_P7ULTRACLEAN_V6_front.fits         aeff_P8R2_TRANSIENT100S_V6_FB.fits
aeff_P7CLEAN_V6_back.fits               aeff_P8R2_CLEAN_V6_EDISP.fits           aeff_P8R2_TRANSIENT100S_V6_PSF.fits
aeff_P7CLEAN_V6_front.fits              aeff_P8R2_CLEAN_V6_FB.fits              aeff_P8R2_TRANSIENT100_V6_EDISP.fits
aeff_P7REP_CLEAN_V10_back.fits          aeff_P8R2_CLEAN_V6_PSF.fits             aeff_P8R2_TRANSIENT100_V6_FB.fits
aeff_P7REP_CLEAN_V10_front.fits         aeff_P8R2_SOURCE_V6_EDISP.fits          aeff_P8R2_TRANSIENT100_V6_PSF.fits
aeff_P7REP_CLEAN_V15_back.fits          aeff_P8R2_SOURCE_V6_FB.fits             aeff_P8R2_ULTRACLEANVETO_V6_EDISP.fits
aeff_P7REP_CLEAN_V15_front.fits         aeff_P8R2_SOURCE_V6_PSF.fits            aeff_P8R2_ULTRACLEANVETO_V6_FB.fits
aeff_P7REP_SOURCE_V10_back.fits         aeff_P8R2_TRANSIENT010E_V6_EDISP.fits   aeff_P8R2_ULTRACLEANVETO_V6_PSF.fits
aeff_P7REP_SOURCE_V10_front.fits        aeff_P8R2_TRANSIENT010E_V6_FB.fits      aeff_P8R2_ULTRACLEAN_V6_EDISP.fits
aeff_P7REP_SOURCE_V15_back.fits         aeff_P8R2_TRANSIENT010E_V6_PSF.fits     aeff_P8R2_ULTRACLEAN_V6_FB.fits
aeff_P7REP_SOURCE_V15_front.fits        aeff_P8R2_TRANSIENT010_V6_EDISP.fits    aeff_P8R2_ULTRACLEAN_V6_PSF.fits
aeff_P7REP_TRANSIENT_V10_back.fits      aeff_P8R2_TRANSIENT010_V6_FB.fits       aeff_p6_v3_dataclean_back.fits
aeff_P7REP_TRANSIENT_V10_front.fits     aeff_P8R2_TRANSIENT010_V6_PSF.fits      aeff_p6_v3_dataclean_front.fits
In [54]:
aeff_P8R2_CLEAN_V6_FB_HDUs = pyfits.open('/usr/local/ScienceTools/refdata/fermi/caldb/CALDB/data/glast/lat/bcf/ea/aeff_P7CLEAN_V6_back.fits')
In [55]:
print aeff_P8R2_CLEAN_V6_FB_HDUs.info()
Filename: /usr/local/ScienceTools/refdata/fermi/caldb/CALDB/data/glast/lat/bcf/ea/aeff_P7CLEAN_V6_back.fits
No.    Name         Type      Cards   Dimensions   Format
0    PRIMARY     PrimaryHDU      10   ()           uint8   
1    EFFECTIVE AREA  BinTableHDU     58   1R x 5C      [60E, 60E, 32E, 32E, 1920E]   
2    PHI_DEPENDENCE  BinTableHDU     61   1R x 6C      [18E, 18E, 8E, 8E, 144E, 144E]   
3    EFFICIENCY_PARAMS  BinTableHDU     38   2R x 1C      [6E]   
None

Filtering the Data

So, for the point source analysis we're doing, we'll need to use source class events. We use gtselect, a 'balistic' tool to do the selection. There are ways to do this from within python (I'll show you these later) but we'll run this from a terminal right now. Actually, it's a good time to talk about how to find information about these tools.

All of the fermi ballistic tools are here

In [56]:
environ['FERMI_DIR']
Out[56]:
'/usr/local/ScienceTools/'
In [57]:
ls /usr/local/ScienceTools/bin/gt*
/usr/local/ScienceTools/bin/gtbary*      /usr/local/ScienceTools/bin/gtfindsrc*   /usr/local/ScienceTools/bin/gtpsf*
/usr/local/ScienceTools/bin/gtbin*       /usr/local/ScienceTools/bin/gtirfs*      /usr/local/ScienceTools/bin/gtpspec*
/usr/local/ScienceTools/bin/gtbindef*    /usr/local/ScienceTools/bin/gtlike*      /usr/local/ScienceTools/bin/gtptest*
/usr/local/ScienceTools/bin/gtbkg*       /usr/local/ScienceTools/bin/gtltcube*    /usr/local/ScienceTools/bin/gtpulsardb*
/usr/local/ScienceTools/bin/gtburst*     /usr/local/ScienceTools/bin/gtltcubesun* /usr/local/ScienceTools/bin/gtrspgen*
/usr/local/ScienceTools/bin/gtburstfit*  /usr/local/ScienceTools/bin/gtltsum*     /usr/local/ScienceTools/bin/gtselect*
/usr/local/ScienceTools/bin/gtdiffrsp*   /usr/local/ScienceTools/bin/gtltsumsun*  /usr/local/ScienceTools/bin/gtsrcid*
/usr/local/ScienceTools/bin/gtdispcube*  /usr/local/ScienceTools/bin/gtmktime*    /usr/local/ScienceTools/bin/gtsrcmaps*
/usr/local/ScienceTools/bin/gtephem*     /usr/local/ScienceTools/bin/gtmodel*     /usr/local/ScienceTools/bin/gtsrcprob*
/usr/local/ScienceTools/bin/gtexpcube*   /usr/local/ScienceTools/bin/gtobspsf*    /usr/local/ScienceTools/bin/gtsuntemp*
/usr/local/ScienceTools/bin/gtexpcube2*  /usr/local/ScienceTools/bin/gtophase*    /usr/local/ScienceTools/bin/gttsmap*
/usr/local/ScienceTools/bin/gtexphpsun*  /usr/local/ScienceTools/bin/gtorbsim*    /usr/local/ScienceTools/bin/gtvcut*
/usr/local/ScienceTools/bin/gtexpmap*    /usr/local/ScienceTools/bin/gtpphase*    /usr/local/ScienceTools/bin/gtversion
/usr/local/ScienceTools/bin/gtexposure*  /usr/local/ScienceTools/bin/gtpsearch*

You can find more information out about a tool by using fhelp.

In [58]:
%system fhelp gtselect
Out[58]:
['NAME',
 '',
 '    gtselect - Performs selection cuts on event data files.',
 '',
 '',
 'USAGE',
 '    ',
 '    gtselect infile outfile ra dec rad tmin tmax emin emax zmin zmax',
 '             evclass evtype ',
 '',
 '',
 'DESCRIPTION',
 '',
 '    gtselect creates a filtered FITS file by selecting rows from an input',
 '    event data file based on user-specified cuts that are applied to',
 '    each row of the input file. This application enables detailed',
 '    selections to be made on Fermi photon and event data obtained from the ',
 '    FSSC data server or simulated data generated using gtobssim',
 '    (see gtobssim help). The most common selections are these',
 '    involving time range (minimum and maximum time) and energy range',
 '    (minimum and maximum energy), and event class.  For each cut that is applied Data',
 '    Subspace (DSS) keywords are written to the EVENTS header of the',
 '    output FITS file that describe the selection.  This information is',
 '    used by the likelihood tools and gtrspgen for computing',
 '    exposure-related information. ',
 '    ',
 '    ',
 'NOTES',
 '',
 '    - The selection cone center (right ascension and declination)',
 '      must match exactly that which was used in the original selection',
 '      from the data server.',
 '',
 '    - both evclass and evtype are "hidden" parameters, i.e. they are',
 '      not propmted for but must be supplied on the command line',
 '   ',
 '    - evclass, recommended selection for standard analysis = 128 (source)',
 '',
 '      The event classification system in Pass 8 is similar to that employed ',
 '      in Pass 7 (i.e. a bitmask classification system for various event classes), ',
 '      Additionally, through the use of the new data filter EVENT_TYPE (see below), ',
 '      further subselection of events are allowed in data preparation. Each bit ',
 '      in the EVENT_CLASS variable corresponds to a ',
 '      particular set of the Instrument Response Functions (IRFs). The more ',
 '      used event classes (Transient, Source, Clean and UltraCleanVeto ) are ',
 '      still hierarchical, i.e., Transient > Source > Clean > UltraCleanVeto.',
 '      Most analyses will use either the Source or Clean event samples.',
 '      **NOTE** once you perform selection on event class or event type, you ',
 '      cannot revert to a more general selection at a later stage of your analysis.',
 '',
 '      Table of standard (non-transient) event classes (for a complete',
 '      list see the FSSC Website or Cicerone)',
 '',
 '      -----------------------------------------',
 '      |Class (Pass 8)|  evclass\t\t\t   ',
 '      |----------------------------------------',
 '      |Source        |  128\t |\t  ',
 '      |Clean\t     |  256\t |\t  ',
 '      |UltraCleanVeto|  1024\t |\t  ',
 '      -----------------------------------------',
 '  ',
 '    - evtype, recommended selection for standard analysis = 3 (front +',
 '      back events)',
 '        ',
 '\tThe Pass 8 processing implements a new scheme for partitioning',
 '        the data within an event class with "event type" selections.',
 '        The event type selections partition the data within a class',
 '        into independent subsets. Currently, evtype has the ability to',
 '        select on 2 different types of event conversions (front/back),',
 '        4 different PSF classes, and 4 different energy reconstruction',
 '        classes. You may select a sub-sample within a given event type',
 '        (e.g. PSF2), but those events can only be used with other',
 '        events of the same type. If you select one of the "all events"',
 '        values (3 = front+back, 60 = PSF0+PSF1+PSF2+PSF3, 960 =',
 '        EDISP0+EDISP1+EDISP2+EDISP3). be aware that different IRFs',
 '        will be applied. If you do not select an "all events" value,',
 '        then the tool may apply results only to a subset of the',
 '        events.',
 '',
 ' ',
 'PARAMETERS',
 '',
 '    infile [filename]',
 '         Input event FITS file. This is the file containing the event data. ',
 '       ',
 '    outfile [filename]',
 '         Output event FITS file with events that satisfy the selections',
 '         performed with gtselect.',
 '       ',
 '    ra [double]',
 '         Right ascension of acceptance cone (J2000) in decimal degrees.',
 '         If the parameter is set to INDEF the right ascension will be read from the ',
 '         header of the input file. ',
 '       ',
 '    dec [double]',
 '         Declination of acceptance cone (J2000) in decimal degrees.',
 '         If the parameter is set to INDEF the declination will be read from the ',
 '         header of the input file. ',
 '   ',
 '    rad [double]',
 '         Radius of acceptance cone (decimal degrees). A value of 180 (default)',
 '         indicates that no acceptance cone cut will be applied.  If the',
 '         parameter is set to INDEF the radius will be read from the header',
 '         of the input file.',
 '              ',
 '    tmin [double]',
 '         Event arrival time lower limit in mission elapsed time (MET)',
 '         seconds. All the events in the output FITS file will have "event',
 '         arrival time" larger than this value. The reference time used for MET',
 '         is midnight (0h:0m:0s) on January 1, 2001, in Coordinated Universal',
 '         Time (UTC). The FERMI convention is that MJDREF=51910',
 '         (UTC)=51910.0007428703703703703 (TT); the fractional part of MJDREF in',
 '         the TT system compensates for the use of midnight in the UTC system as',
 '         the reference time.  MJDREF is divided into two keywords:',
 '         MJDREFI=51910, the integer part; and MJDREFF=7.428703703703703D-4, the',
 '         fractional part. If the parameter is set to INDEF (default) tmin will be',
 '         read from the header of the input file. For conversions',
 '         between MET and other commonly used time conventions we refer',
 '         the user to the HEASARC online tool',
 '         http://heasarc.gsfc.nasa.gov/cgi-bin/Tools/xTime/xTime.pl.',
 '',
 '    tmax [double]',
 '         Event arrival time upper limit in mission elapsed time (MET)',
 '         seconds. All the events in the output FITS file will have',
 '         "event arrival time" smaller than this value. A value of zero',
 '         in this context means that no upper limit will be applied. If',
 '         the parameter is set to INDEF (default) the tmax will be read',
 '         from the header of the input file.',
 '       ',
 '    emin [double]',
 '         Minimum event energy in MeV. All the events in the output FITS file',
 '         will have "event energy" larger than this value. Default is "100" MeV.',
 '       ',
 '    emax [double]',
 '         Maximum event energy in MeV. All the events in the output',
 '         FITS file will have "event energy" smaller than this',
 '         value. Default is "300000" MeV.',
 '      ',
 '    zmax [double]',
 '         Maximum apparent zenith angle (degrees). It ranges from 0 to 180 (default).',
 '    ',
 '    (evclsmin) [integer] ',
 '         This parameter is ignored for data reprocessed after Aug. 1, 2011 ',
 '         (Pass 7 series).',
 '         Mininum event class ID to include. Each set of IRFs will define a set ',
 '         of event classes numbered 0 to n, where n+1 is the number of event ',
 '         classes. These numbers will be ordered such that the more inclusive ',
 '         classes are numbered lowest. Default is "INDEF".',
 '   ',
 '     (evclsmax) [integer]',
 '         This parameter is ignored for data reprocessed after Aug. 1, 2011 ',
 '         (Pass 7 series).',
 '         Maximum event class ID to include. Default is "INDEF".',
 '',
 '     (evclass) [integer] ',
 '         Event class selection for pass 8. The default is 128 (source',
 '         class).',
 '  ',
 '     (evtype) [integer]',
 '         The event type selection for Pass 8 data. The event type is a',
 '         subselection of the event class. Users can select on range of',
 '         events based on conversion type (front/back), angular',
 '         reconstruction quality (PSF values), and energy',
 '         reconstruction quality (EDISP values). The default value is',
 '         INDEF, which will not apply a subselection.',
 '',
 '    (convtype) [integer]',
 '         Conversion type. This parameter screens events based on which',
 '         portion of the instrument the incident gamma-ray pair conversion',
 '         event occurred. 0=Front, 1=Back, -1=both (defalt value). Refer ',
 '         to the Cicerone manual for details',
 '         (http://fermi.gsfc.nasa.gov/ssc/data/analysis/documentation/Cicerone). ',
 '   ',
 '    (phasemin) [double]',
 '         Minimum pulsar phase to include. This cut is only applied if',
 '         the PULSE_PHASE column has been added to the FT1 file by the',
 '         gtpphase tool.  Default is "0".',
 '   ',
 '    (phasemax) [double]',
 '         Maximum pulsar phase to include. Default is "1".',
 '',
 '    (evtable) [string]',
 '         Event data extension. Default is "EVENTS".',
 '',
 '    (chatter) [integer]  ',
 '         This parameter fixes the output verbosity: no screen output (0),',
 '         nominal screen output (2), maximum verbosity (4). Default is "2".',
 '',
 '    (clobber) [boolean]',
 '         If true, an existing file of the same name will be overwritten. ',
 '         Default is "yes".',
 '    ',
 '    (debug) [boolean]',
 '         Activate debugging mode. Default is "no". When debug is "no", all ',
 '         exceptions that are not caught and',
 '         handled by individual tool-specific code are caught by a top-level',
 '         exception handler that displays information about the exception and',
 '         then exits. When debug is "yes", such exceptions are not caught by the',
 '         top level code. Instead the tool produces a segmentation violation,',
 '         which is more useful for debugging. When debugging mode is enabled,',
 '         the tool produces more verbose output describing any errors or',
 '         exceptions that are encountered.',
 '    ',
 '    (gui) [boolean]',
 '         Graphical User Interface (GUI) mode is activated if set to "yes".',
 '         Default is "no".',
 '    ',
 '    (mode) [string]',
 '         Mode of automatic parameters: "h" for batch, "ql" for interactive. ',
 '         Default is "ql".',
 '',
 '        ',
 'EXAMPLES',
 '',
 '    The way that the parameters are passed follows the FTOOLs model. They',
 '    could be passed by answering from a prompt, as a list in a command',
 '    line, or by modifying the parameter file. The command line option',
 '    facilitates calling gtselect from a script.',
 ' ',
 '    To prompt for gtselect type in the command line: ',
 '',
 '> gtselect ',
 '',
 '    This will prompt for parameter values. Not all parameter are prompted: ',
 '    some of them are "hidden". To change one of the "hidden" parameter, ',
 '    the user should specify the values in the command line or modify its ',
 '    mode by editing the parameter file. ',
 '    For example, to prevent overwrite the existing output file, the following ',
 '    command line as to be typed:',
 ' ',
 '> gtselect clobber=no  ',
 ' ',
 '    This is an example of how to run the tool: ',
 '    In this case the quasar 3C279 (centered on Ra=193.98, Dec=-5.82) was',
 '    simulated using gtobssim (type "fhelp gtobssim" for further explanation), ',
 '    although the indentical procedure could be followed for actual data. ',
 '    Note that a radius of 40 degrees was specified in the input simulation.',
 '    The energy for the simulated events ranges between 20 MeV and 200000 MeV. ',
 '    Using gtselect it possible to select the events with energy larger than ',
 '    100 MeV and within a radius of 20 degrees centered on Ra=193.98, ',
 '    Dec=-5.82 and a time range between 220838400 and 225590400 MET seconds:',
 '',
 '> gtselect evclass=2',
 'Input FT1 file[] 3C279_events_0000.fits',
 'Output FT1 file[] 3c279_filtered.fits',
 'RA for new search center (degrees) (0:360) [INDEF] 193.98',
 'Dec for new search center (degrees) (-90:90) [INDEF] -5.82',
 'radius of new search region (degrees) (0:180) [INDEF] 20',
 'start time (MET in s) (0:) [INDEF] 220838400',
 'end time (MET in s) (0:) [INDEF] 225590400',
 'lower energy limit (MeV) (0:) [100] 100',
 'upper energy limit (MeV) (0:) [300000] 100000',
 'maximum zenith angle value (degrees) (0:180) [180] 105',
 'Done.',
 '',
 '    The last example could be also run from the command line as:',
 '',
 '> gtselect infile=3C279_events_0000.fits outfile=3c279_filtered.fits \\',
 '  ra=193.98 dec=-5.82 rad=20 evclass=128 evtype=3 \\',
 '  tmin=220838400 tmax=225590400  emin=100 emax=100000 zmax=100',
 '',
 '    You can also use the defaults in the input file header by using',
 '    the INDEF syntax',
 '',
 '> gtselect infile=3C279_events_0000.fits outfile=3c279_filtered.fits \\',
 '  ra=INDEF dec=INDEF rad=INDEF evclass=128 evtype=INDEF tmin=INDEF \\',
 '  tmax=INDEF emin=100 emax=100000 zmax=100',
 '',
 '',
 'KNOWN BUGS',
 '',
 '    Use the INDEF values for right ascension, declination, and radius',
 '    are handled peculiarly. If any one of the parameters are set to',
 '    INDEF it is treated if all are set to INDEF. ',
 '    ',
 '',
 'SEE ALSO',
 '',
 '     fselect',
 '     ',
 '     gtobssim',
 '     ',
 '     gtpphase',
 '     ',
 '     gtrspgen',
 '     ',
 '',
 '   ']

We need to get ready to run gtselect now. Since we were given multiple separate photon files from the data server we need to make a file list so that when we run gtselect it knows to look in both files for events that match our selections.

Here's a quick way to do this:

In [65]:
%system ls /Users/eferrara/FSSC/VM/shared/psr_data/L151119044834C9B75A5E69_PH* > ~/FSSC/VM/shared/psr_data/events.txt
Out[65]:
[]
In [66]:
%system cat /Users/eferrara/FSSC/VM/shared/psr_data/events.txt
Out[66]:
['/Users/eferrara/FSSC/VM/shared/psr_data/L151119044834C9B75A5E69_PH00.fits',
 '/Users/eferrara/FSSC/VM/shared/psr_data/L151119044834C9B75A5E69_PH01.fits',
 '/Users/eferrara/FSSC/VM/shared/psr_data/L151119044834C9B75A5E69_PH02.fits',
 '/Users/eferrara/FSSC/VM/shared/psr_data/L151119044834C9B75A5E69_PH03.fits',
 '/Users/eferrara/FSSC/VM/shared/psr_data/L151119044834C9B75A5E69_PH04.fits',
 '/Users/eferrara/FSSC/VM/shared/psr_data/L151119044834C9B75A5E69_PH05.fits',
 '/Users/eferrara/FSSC/VM/shared/psr_data/L151119044834C9B75A5E69_PH06.fits',
 '/Users/eferrara/FSSC/VM/shared/psr_data/L151119044834C9B75A5E69_PH07.fits',
 '/Users/eferrara/FSSC/VM/shared/psr_data/L151119044834C9B75A5E69_PH08.fits']

Now we're ready to go. Note in the usage for gtselect that the event class and event types aren't listed. They are 'hidden parameters', so you'll need to run the tool like this:

gtselect evclass=128 evclass=3

There are a number of event classes you can use. Unless you're an advanced user, it's best to follow the analysis recommendations above.

Standard Hierarchy for LAT Event Classes
Event Class evclass Photon File Extended File Description
P8R2_TRANSIENT020 16 X Transient event class with background rate equal to two times the A10 IGRB reference spectrum.
P8R2_TRANSIENT010 64 X Transient event class with background rate equal to one times the A10 IGRB reference spectrum.
P8R2_SOURCE 128 X X This event class has a residual background rate that is comparable to P7REP_SOURCE. This is the recommended class for most analyses and provides good sensitivity for analysis of point sources and moderately extended sources.
P8R2_CLEAN 256 X X This class is identical to SOURCE below 3 GeV. Above 3 GeV it has a 2-4 times lower background rate than SOURCE and is slightly more sensitive to hard spectrum sources at high galactic latitudes.
P8R2_ULTRACLEAN 512 X X This class has a background rate between CLEAN and ULTRACLEANVETO.
P8R2_ULTRACLEANVETO 1024 X X This is the cleanest Pass 8 event class. Between 100 MeV and 10 GeV the background rate is between 2 and 4 times lower than the background rate of SOURCE class. This class is recommended to check for CR-induced systematics as well as for studies of diffuse emission that require low levels of CR contamination.
Extended Hierarchy
Event Class evclass Photon File Extended File Description
P8R2_TRANSIENT020E 8 X Extended version of the P8R2_TRANSIENT020 event class with a less restrictive fiducial cut on projected track length through the Calorimeter.
P8R2_TRANSIENT010E 64 X Extended version of the P8R2_TRANSIENT010 event class with a less restrictive fiducial cut on projected track length through the Calorimeter.
NON-ACD Hierarchy
Event Class evclass Photon File Extended File Description
P8R2_TRANSIENT015S 65536 X Transient event class designed for analysis of prompt solar flares in which pileup activity may be present. This class has a background rate equal to 1.5 times the A10 reference spectrum.

You don't really need to specify evtype unless you plan to use one of the event types. A value of 3 is used for FRONT+BACK data. Here is a list of the available values:

Conversion Type Partition
Event Type evtype Description
FRONT 1 Events converting in the Front-section of the Tracker. Equivalent to convtype=0.
BACK 2 Events converting in the Back-section of the Tracker. Equivalent to convtype=1.
PSF Type Partition
Event Type evtype Description
PSF0 4 First (worst) quartile in the quality of the reconstructed direction.
PSF1 8 Second quartile in the quality of the reconstructed direction.
PSF2 16 Third quartile in the quality of the reconstructed direction.
PSF3 32 Fourth (best) quartile in the quality of the reconstructed direction.
EDISP Type Partition
Event Type evtype Description
EDISP0 64 First (worst) quartile in the quality of the reconstructed energy.
EDISP1 128 Second quartile in the quality of the reconstructed energy.
EDISP2 256 Third quartile in the quality of the reconstructed energy.
EDISP3 512 Fourth (best) quartile in the quality of the reconstructed energy.

There's lots of info we need to give gtselect so it makes sense to run gtvcut on one of our photon files to figure out what we were doing.

In [67]:
%system gtvcut /Users/eferrara/FSSC/VM/shared/psr_data/L151119044834C9B75A5E69_PH00.fits EVENTS
Out[67]:
['DSTYP1: TIME',
 'DSUNI1: s',
 'DSVAL1: TABLE',
 'DSREF1: :GTI',
 '',
 'GTIs: (suppressed)',
 '',
 'DSTYP2: BIT_MASK(EVENT_CLASS,128,P8R2)',
 'DSUNI2: DIMENSIONLESS',
 'DSVAL2: 1:1',
 '',
 'DSTYP3: POS(RA,DEC)',
 'DSUNI3: deg',
 'DSVAL3: CIRCLE(42.0777,60.3595,30)',
 '',
 'DSTYP4: TIME',
 'DSUNI4: s',
 'DSVAL4: 239557417:335145602',
 '',
 'DSTYP5: ENERGY',
 'DSUNI5: MeV',
 'DSVAL5: 100:300000',
 '']

We can combine all the files without making any additional cuts, just to see how many total events were found for this source.

In [75]:
Image(filename='images/P8_gtselect.png')
Out[75]:

How many total events do we have?

In [74]:
allevt_hdulist = pyfits.open('/Users/eferrara/FSSC/VM/shared/psr_data/Pulsar_data.fits')
allevt_hdulist.info()
Filename: /Users/eferrara/FSSC/VM/shared/psr_data/Pulsar_data.fits
No.    Name         Type      Cards   Dimensions   Format
0    PRIMARY     PrimaryHDU      31   ()           uint8   
1    EVENTS      BinTableHDU    218   24303323R x 23C   [E, E, E, E, E, E, E, E, E, D, J, J, I, 3I, 32X, 32X, I, D, E, E, E, E, E]   
2    GTI         BinTableHDU     46   17212R x 2C   [D, D]   

You can run all the analysis tools on the command line, as above. However, it's easier to run using a python version of the science tools. I'm going to run this using the python toolset called gtapps.

In python, let's use gtselect to reduce the nomber of events e're working with:

In [101]:
import gt_apps
dir(gt_apps)
Out[101]:
['GtApp',
 'TsMap',
 '__builtins__',
 '__doc__',
 '__file__',
 '__name__',
 '__package__',
 'addCubes',
 'counts_map',
 'diffResps',
 'evtbin',
 'expCube',
 'expMap',
 'filter',
 'gtexpcube2',
 'like',
 'maketime',
 'model_map',
 'obsSim',
 'rspgen',
 'srcMaps']
In [71]:
from gt_apps import filter
filter.pars()
Out[71]:
' infile= outfile= ra="INDEF" dec="INDEF" rad="INDEF" tmin="INDEF" tmax="INDEF" emin=30.0 emax=300000.0 zmin=0.0 zmax=180.0 evclass="INDEF" evclsmin=0 evclsmax=10 evtype="INDEF" convtype=-1 phasemin=0.0 phasemax=1.0 evtable="EVENTS" chatter=2 clobber=yes debug=no gui=no mode="ql"'
In [72]:
filter['infile'] = '@/Users/eferrara/FSSC/VM/shared/psr_data/events.txt'
filter['outfile'] = '/Users/eferrara/FSSC/VM/shared/psr_data/PSRJ0248_filtered.fits'
filter['ra'] = 42.0
filter['dec'] = 60.4
filter['rad'] = 15
filter['tmin'] = 239557417
filter['tmax'] = 335145602
filter['emin'] = 300
filter['emax'] = 50000
filter['zmax'] = 90
filter['evclass'] = 128
filter['evtype'] = 3

You can check what the command will look like before you run it. This is also useful for logging, and when building scripts.

In [73]:
filter.command()
Out[73]:
'time -p /usr/local/ScienceTools/bin/gtselect infile=@/Users/eferrara/FSSC/VM/shared/psr_data/events.txt outfile=/Users/eferrara/FSSC/VM/shared/psr_data/PSRJ0248_filtered.fits ra=42.0777 dec=60.3595 rad=15.0 tmin=239557417.0 tmax=335145602.0 emin=300.0 emax=50000.0 zmin=0.0 zmax=90.0 evclass=128 evclsmin=0 evclsmax=10 evtype=3 convtype=-1 phasemin=0.0 phasemax=1.0 evtable="EVENTS" chatter=2 clobber=yes debug=no gui=no mode="ql"'
In [76]:
filter.run()
time -p /usr/local/ScienceTools/bin/gtselect infile=@/Users/eferrara/FSSC/VM/shared/psr_data/events.txt outfile=/Users/eferrara/FSSC/VM/shared/psr_data/PSRJ0248_filtered.fits ra=42.0777 dec=60.3595 rad=15.0 tmin=239557417.0 tmax=335145602.0 emin=300.0 emax=50000.0 zmin=0.0 zmax=90.0 evclass=128 evclsmin=0 evclsmax=10 evtype=3 convtype=-1 phasemin=0.0 phasemax=1.0 evtable="EVENTS" chatter=2 clobber=yes debug=no gui=no mode="ql"
Done.
real 60.96
user 37.27
sys 2.10

Some things to note here:

  • We put an '@' sign in front of the input filename which tells gtselect that this isn't actually a FITS file but a list of FITS files.
  • We used the recommended zenith angle cut of 90.
  • We could have used INDEF for the start and end times, rather than explicitly stating them.
  • We could not have used INDEF for the RA and Dec values because we wanted to change the selected radius. The science tools use a position keyword that includes all three values. If you declare one, you have to declare them all.
  • We used a position in the cut that differs from that used in the data server (42.0777,60.3595). If I had used the full 30-degree radius, this would have caused a problem later on. Note that the position and radius parameters are a tuple.

The Filtered Data

Let's take a look at what we have now.

In [77]:
filtered_data = pyfits.open('/Users/eferrara/FSSC/VM/shared/psr_data/PSRJ0248_filtered.fits')
In [78]:
filtered_data.info()
Filename: /Users/eferrara/FSSC/VM/shared/psr_data/PSRJ0248_filtered.fits
No.    Name         Type      Cards   Dimensions   Format
0    PRIMARY     PrimaryHDU      31   ()           uint8   
1    EVENTS      BinTableHDU    223   1044419R x 23C   [E, E, E, E, E, E, E, E, E, D, J, J, I, 3I, 32X, 32X, I, D, E, E, E, E, E]   
2    GTI         BinTableHDU     46   17212R x 2C   [D, D]   

Notice in the dimensions that we now have 1.04 million events in the combined event file. This is quite a change from having 24 million events returned by the original query. Lots of events have been filtered out.

The next thing to do is to make sure we update the good time interval (the GTI) column using the recommended GTI filters. These can also be found on the recommendations and good practices. Here's the relavant table:

Time Selection Recommendations

Analysis Type ROI-Based Zenith Angle Cut
(roicut)
Relational Filter Expression
(filter)
Galactic Point Source Analysis no (DATA_QUAL>0)&&(LAT_CONFIG==1)
Off-plane Point Source Analysis no (DATA_QUAL>0)&&(LAT_CONFIG==1)
Burst and Transient Analysis yes (DATA_QUAL>0)&&(LAT_CONFIG==1)
Galactic Diffuse Analysis no (DATA_QUAL>0)&&(LAT_CONFIG==1)
Extra-Galactic Diffuse Analysis no (DATA_QUAL>0)&&(LAT_CONFIG==1)
Burst and Transient Analysis yes (DATA_QUAL>0||DATA_QUAL==-1)&&(LAT_CONFIG==1)

IMPORTANT: For analyses where an ROI-based zenith cut is NOT performed, an exposure correction must be made using the "zmax" option in the gtltcube tool.

These filter expression is simply a boolean expression using columns in the spacecraft file. We do this via the gtmktime command, known as maketime in gt_apps.

In [79]:
from gt_apps import maketime
maketime['evfile'] = '/Users/eferrara/FSSC/VM/shared/psr_data/PSRJ0248_filtered.fits'
maketime['outfile'] = '/Users/eferrara/FSSC/VM/shared/psr_data/PSRJ0248_maketime.fits'
maketime['scfile'] = '/Users/eferrara/FSSC/VM/shared/psr_data/spacecraft.fits'
maketime['filter'] = 'DATA_QUAL>0 && LAT_CONFIG==1'
maketime['apply_filter'] = 'yes'
maketime['roicut'] = 'no'
maketime.run()
time -p /usr/local/ScienceTools/bin/gtmktime scfile=/Users/eferrara/FSSC/VM/shared/psr_data/spacecraft.fits sctable="SC_DATA" filter="DATA_QUAL>0 && LAT_CONFIG==1" roicut=no evfile=/Users/eferrara/FSSC/VM/shared/psr_data/PSRJ0248_filtered.fits evtable="EVENTS" outfile="/Users/eferrara/FSSC/VM/shared/psr_data/PSRJ0248_maketime.fits" apply_filter=yes overwrite=no header_obstimes=yes tstart=0.0 tstop=0.0 gtifile="default" chatter=2 clobber=yes debug=no gui=no mode="ql"
real 88.46
user 68.06
sys 2.28

Some things to note here:

  • We gave it the spacecraft file as input (this makes sense).
  • We said 'no' to the roi-based zenith angle cut, as pr the recommendations.
  • If we had said 'yes' to the roi-based zenith cut, it would have basically made this cut: "angsep(RA_ZENITH,DEC_ZENITH,RA_ROI,DEC_ROI) < ZMAX - ROI_RADIUS" which just makes sure that your ROI is close to zenith and away from the Earth's limb.

Rather than applying the exposure correction to the data file, we will apply it to the livetime calculation instead.

Now what do we have...

In [81]:
%system ftlist /Users/eferrara/FSSC/VM/shared/psr_data/PSRJ0248_filtered.fits H
Out[81]:
['',
 '        Name               Type       Dimensions',
 '        ----               ----       ----------',
 'HDU 1   Primary Array      Null Array                               ',
 'HDU 2   EVENTS             BinTable    23 cols x 1044419 rows       ',
 'HDU 3   GTI                BinTable     2 cols x 17212 rows         ']
In [82]:
%system ftlist /Users/eferrara/FSSC/VM/shared/psr_data/PSRJ0248_maketime.fits H
Out[82]:
['',
 '        Name               Type       Dimensions',
 '        ----               ----       ----------',
 'HDU 1   Primary Array      Null Array                               ',
 'HDU 2   EVENTS             BinTable    23 cols x 1044302 rows       ',
 'HDU 3   GTI                BinTable     2 cols x 17226 rows         ']

You can see that the maketime command only changed the GTI extension.

Things to think about when making data selections

  • What scientific question am I trying to answer?
    • Am I looking at a point source?
    • What spectral shape do I expect (hard spectral shapes could benefit from a higher low-energy cut).
    • Is this a short timescale event (~200 s, like a GRB)?
    • Am I interested in a small region or the whole sky?
    • Is the region close to the galactic plane (here lies analysis dragons, might want to raise the energy threshold, make your ROI smaller)

Always double check the recommended cuts, especially after a data/software release. Don't hesitate to ask the FSSC for advice. We are working scientists and have proably run into the types of problems you are seeing. And if we can't, we can always contact the software developers.

Quick Looks at the Data

These are quick ways you can have a look at the data you have. These are at a higher level than the previous methods but they are not statistically rigorous (rigor needs the Likelihood method). The data you are about to see have not been corrected for things like livetime, exposure, background, random acts of statistics etc. but go for it…

Using gtbin to make a counts map

gtbin is a pretty versitle tool: it can make counts maps, counts cubes (counts maps with an energy dimension), simple light curves and more.

We're going to make a counts map and plot it up with python. You can also use ds9 or fv to plot this.

In [88]:
from gt_apps import counts_map
counts_map['evfile'] = '/Users/eferrara/FSSC/VM/shared/psr_data/PSRJ0248_maketime.fits'
counts_map['scfile'] = 'NONE'
counts_map['outfile'] = '/Users/eferrara/FSSC/VM/shared/psr_data/PSRJ0248_cmap.fits'
counts_map['algorithm'] = 'CMAP'
counts_map['xref'] = 42.0
counts_map['yref'] = 60.4
counts_map['coordsys'] = 'CEL'
counts_map['nxpix'] = 400
counts_map['nypix'] = 400
counts_map['binsz'] = 0.05
counts_map['axisrot'] = 0.0
counts_map['proj'] = 'AIT'
counts_map.run()
time -p /usr/local/ScienceTools/bin/gtbin evfile=/Users/eferrara/FSSC/VM/shared/psr_data/PSRJ0248_maketime.fits scfile=NONE outfile=/Users/eferrara/FSSC/VM/shared/psr_data/PSRJ0248_cmap.fits algorithm="CMAP" ebinalg="LOG" emin=30.0 emax=200000.0 ebinfile=NONE tbinalg="LIN" tbinfile=NONE nxpix=400 nypix=400 binsz=0.05 coordsys="CEL" xref=42.0 yref=60.4 axisrot=0.0 rafield="RA" decfield="DEC" proj="AIT" hpx_ordering_scheme="RING" hpx_order=3 hpx_ebin=yes evtable="EVENTS" sctable="SC_DATA" efield="ENERGY" tfield="TIME" chatter=2 clobber=yes debug=no gui=no mode="ql"
This is gtbin version ScienceTools-v10r0p5-fssc-20150819
gtbin: WARNING: No spacecraft file: EXPOSURE keyword will be set equal to ontime.
real 1.42
user 1.36
sys 0.06
In [89]:
cmap = pyfits.open('/Users/eferrara/FSSC/VM/shared/psr_data/PSRJ0248_cmap.fits')
In [90]:
cmap.info()
Filename: /Users/eferrara/FSSC/VM/shared/psr_data/PSRJ0248_cmap.fits
No.    Name         Type      Cards   Dimensions   Format
0    PRIMARY     PrimaryHDU     131   (400, 400)   int32   
1    GTI         BinTableHDU     49   17226R x 2C   [D, D]   
In [93]:
%system ds9 /Users/eferrara/FSSC/VM/shared/psr_data/PSRJ0248_cmap.fits
Out[93]:
[]
In [94]:
Image(filename='images/P8_J0248_cmap.png')
Out[94]:

If we use the ROI-based zenith cut, how does the data file change?

In [99]:
%system ftlist /Users/eferrara/FSSC/VM/shared/psr_data/PSRJ0248_maketime.fits H
Out[99]:
['',
 '        Name               Type       Dimensions',
 '        ----               ----       ----------',
 'HDU 1   Primary Array      Null Array                               ',
 'HDU 2   EVENTS             BinTable    23 cols x 1044302 rows       ',
 'HDU 3   GTI                BinTable     2 cols x 17226 rows         ']
In [100]:
%system ftlist /Users/eferrara/FSSC/VM/shared/psr_data/PSRJ0248_maketime_roicut.fits H
Out[100]:
['',
 '        Name               Type       Dimensions',
 '        ----               ----       ----------',
 'HDU 1   Primary Array      Null Array                               ',
 'HDU 2   EVENTS             BinTable    23 cols x 695091 rows        ',
 'HDU 3   GTI                BinTable     2 cols x 15564 rows         ']
In [98]:
Image(filename='images/P8_J0248_cmap_roicut.png')
Out[98]:
In [ ]: