This tutorial is designed to not only make you comfortable running a LAT analysis but also make you familiar with the data and where to go if you want to find more information. At the end of this tutorial and the pulsar analysis tutorial you should be able to know what all of the various file formats are and be able to perform a simple analysis.
from IPython.display import Image,HTML
Available from FSSC Website→Data→Documentation
HTML("<iframe src='http://fermi.gsfc.nasa.gov/ssc/data/analysis/documentation/' width='1000' height='500'></iframe>")
Go to the FSSC Website→Data→Data Analysis→Software Download
%system uname -a
Go to the FSSC Website→Data→Data Analysis→Analysis Threads
HTML("<iframe src='http://fermi.gsfc.nasa.gov/ssc/data/analysis/scitools/' width='1000' height='500'></iframe>")
Contact us! fermihelp@milkyway.gsfc.nasa.gov . Most of us are nice...most of the time.
Go to the FSSC Website→Data→Data Access
HTML("<iframe src='http://fermi.gsfc.nasa.gov/ssc/data/access/' width='1000' height='500'></iframe>")
Lits of data products are available. What you use depends on what you're doing. Here are some highlights:
I want to perform a detailed source anaylsis:
I want to perform an analysis of a short timescale flare like a GRB or a Solar Flare
I want to monitor all sources in the full sky
I want to be alerted to flares in known sources
I want to find out about known sources, GRBs, or solar flares.
I want to know what the spacecraft was doing
Don't forget to pay attention to the caveats for the data products you are using. You never know, it might say don't do what you're trying to do or tell you to do something else!
Download and unzip/untar the second pulsar catalog archive. You should find the timing files under 2PC_auxiliary_files_v04/par_files . You should look at the .par file for your source of interest to determine the validity interval for the timing solution.
HTML("<iframe src='http://fermi.gsfc.nasa.gov/ssc/data/access/lat/2nd_PSR_catalog/' width='850' height='500'></iframe>")
%system fv /Users/eferrara/FSSC/VM/shared/SecondPulsarCatalogFIles/2PC_auxiliary_files_v04/PSR_data/FITS/PSRJ0248+6021_2PC_data.fits
Image(filename='images/2PC_J0248_psr_data.png')
%system cat /Users/eferrara/FSSC/VM/shared/SecondPulsarCatalogFIles/2PC_auxiliary_files_v04/par_files/PSRJ0007+7303_2PC_2.par
This pulsar, in the SNR CTA1, was the first to be discovered by the Fermi-LAT. Even though the 2PC catalog used three years of LAT data, the timing solution is only valid for 2.6 years (MJD 55010 - 55961). This is not uncommon for young pulsars, as frequent glitches require these sources to be recharacterized on a regular basis. You can see from the number of wave terms at the end of the file that this is a difficult pulsar to time.
So, instead, let's look at a pulsar that is being regularly timed by radio telescopes.
%system cat /Users/eferrara/FSSC/VM/shared/SecondPulsarCatalogFIles/2PC_auxiliary_files_v04/par_files/PSRJ0248+6021_2PC.par
Much better! No wave terms means the timing model describes the data very well!
The 2PC Catalog has a huge amount of useful information, including:
This is a very useful archive for preparing for publications. I encourage you to take a look at the contents.
%system ls /Users/eferrara/FSSC/VM/shared/SecondPulsarCatalogFIles/2PC_auxiliary_files_v04/
Now we need the data for the pulsar. Since this pulsar is in the plane of the Galaxy (b=0.697), I have downloaded data for a 30-degree ROI. The data have been selected to match the validity period of the timing file. I'm using a directory called 'psr_data' in my shared folder.
Go to the FSSC Website→Data→LAT Data Server
HTML("<iframe src='http://fermi.gsfc.nasa.gov/cgi-bin/ssc/LAT/LATDataQuery.cgi' width='1000' height='500'></iframe>")
How do I fill in these parameters?
(You can click on the field name for more details.)
I've already downloaded these data. You can get them in the data directory in the USB drive we've given you. Here's the resulting page (for the record).
Image(filename='images/FSSC_P8_DataServer1.png')
If you click on the link to get to the query result you get this:
Image(filename='images/FSSC_P8_DataServer2.png')
You could click the Available links or you could just use the wget commands (I added the '-q' for quiet).
%system wget -q http://fermi.gsfc.nasa.gov/FTP/fermi/data/lat/queries/L151119044834C9B75A5E69_SC00.fits
%system wget -q http://fermi.gsfc.nasa.gov/FTP/fermi/data/lat/queries/L151119044834C9B75A5E69_PH01.fits
%system wget -q http://fermi.gsfc.nasa.gov/FTP/fermi/data/lat/queries/L151119044834C9B75A5E69_PH03.fits
%system wget -q http://fermi.gsfc.nasa.gov/FTP/fermi/data/lat/queries/L151119044834C9B75A5E69_PH07.fits
%system wget -q http://fermi.gsfc.nasa.gov/FTP/fermi/data/lat/queries/L151119044834C9B75A5E69_PH04.fits
%system wget -q http://fermi.gsfc.nasa.gov/FTP/fermi/data/lat/queries/L151119044834C9B75A5E69_PH06.fits
%system wget -q http://fermi.gsfc.nasa.gov/FTP/fermi/data/lat/queries/L151119044834C9B75A5E69_PH02.fits
%system wget -q http://fermi.gsfc.nasa.gov/FTP/fermi/data/lat/queries/L151119044834C9B75A5E69_PH05.fits
%system wget -q http://fermi.gsfc.nasa.gov/FTP/fermi/data/lat/queries/L151119044834C9B75A5E69_PH08.fits
%system wget -q http://fermi.gsfc.nasa.gov/FTP/fermi/data/lat/queries/L151119044834C9B75A5E69_PH00.fits
%system ls /Users/eferrara/FSSC/VM/shared/psr_data/L151119044834C9B75A5E69_*.fits
I always record the information from my query with my data. The same information is also stored in the header of the file and can be viewed with gtvcut.
%system cat /Users/eferrara/FSSC/VM/shared/psr_data/query.txt
So, what do we have now?
We've got nine photon files (this is a result of how the data server processes queries) and one spacecraft file (always one). Let's take a look at these with the ftool fv and see what we have.
%system fv /Users/eferrara/FSSC/VM/shared/psr_data/L151119044834C9B75A5E69_PH00.fits
Here's what you should see. Let's poke at the GUI for a bit and talk about what we have.
Image(filename='images/J0248_event_file.png')
This is a FITS file which is an open standard defining a digital file format useful for storage, transmission and processing of scientific and other images. It has three Header Data Units (HDUs); the first is (and always must be) an image even if it's empty (as it is in this case). The second and third are binary types and store data. All of our data are in the second hdu (index = 1) and you can see that there are 22 columns and 149459 rows. This means we have 149,459 events in this single photon file! The last HDU is a binary table called GTI which details the good time intervals for the data in this file. If you poke the 'Header' button for the first HDU (the empty image) you get this.
Image(filename='images/FSSC_fv2.png')
All sorts of useful information. Note that there are some useful ftools to do this.
%system ftlist /Users/eferrara/FSSC/VM/shared/psr_data/L151119044834C9B75A5E69_PH00.fits H
Let's close the header and click on the 'All' button for the sedcond HDU (the one with all of our data). It should look like this:
Image(filename='images/FSSC_fv3.png')
There's a row for every event and each column details all of the information about that event. You could click the header to find more details or check out the information about the columns on the FSSC site. We can have a look at them. And you can see what the 'extended' data are all about.
HTML("<iframe src='http://fermi.gsfc.nasa.gov/ssc/data/analysis/documentation/Cicerone/Cicerone_Data/LAT_Data_Columns.html' width='1000' height='500'></iframe>")
%system pwd
The extended file just has some more information about each event. This page also details what's in the spacecraft file. You can use 'fv' to plot some of these parameters or we can actually plot them here in python.
import pyfits
sc_hdulist = pyfits.open('/Users/eferrara/FSSC/VM/shared/psr_data/L151119044834C9B75A5E69_SC00.fits')
sc_hdulist.info()
For simplicity, we will rename the spacecraft file to "spacecraft.fits".
%system mv /Users/eferrara/FSSC/VM/shared/psr_data/L151119044834C9B75A5E69_SC00.fits /Users/eferrara/FSSC/VM/shared/psr_data/spacecraft.fits
You can filter and select on any of the parameters of the events but there are some recommendations and good practices listed at the FSSC site.
Analysis Type | Minimum Energy (emin) |
Maximum Energy (emax) |
Max Zenith Angle (zmax) |
Event Class (evclass) |
IRF Name |
---|---|---|---|---|---|
Galactic Point Source Analysis | 100 (MeV) | 500000 (MeV) | 90 (degrees) | 128 | P8R2_SOURCE_V6 |
Off-plane Point Source Analysis | 100 (MeV) | 500000 (MeV) | 90 (degrees) | 128 | P8R2_SOURCE_V6 |
Burst and Transient Analysis (<200s) | 100 (MeV) | 500000 (MeV) | 100 (degrees) | 16 | P8R2_TRANSIENT020_V6 |
Galactic Diffuse Analysis | 100 (MeV) | 500000 (MeV) | 90 (degrees) | 128 | P8R2_SOURCE_V6 |
Extra-Galactic Diffuse Analysis | 100 (MeV) | 500000 (MeV) | 90 (degrees) | 1024 | P8R2_ULTRACLEANVETO_V6 |
Impulsive Solar Flare Analysis | 100 (MeV) | 500000 (MeV) | 100 (degrees) | 65536 | P8R2_TRANSIENT015S_V6 |
Let's talk about these for a bit:
You can find details of the Instrument Response Functions at the LAT Performance Page
If you install the Fermi Science Tools, the IRFs are also available in the following directory:
$FERMI_DIR/refdata/fermi/caldb/CALDB/data/glast/lat/bcf
HTML("<iframe src='http://www.slac.stanford.edu/exp/glast/groups/canda/lat_Performance.htm' width='1000' height='500'></iframe>")
from os import environ
environ['CALDB']
ls /usr/local/ScienceTools/refdata/fermi/caldb/CALDB/data/glast/lat/bcf
ls '/usr/local/ScienceTools/refdata/fermi/caldb/CALDB/data/glast/lat/bcf/ea'
aeff_P8R2_CLEAN_V6_FB_HDUs = pyfits.open('/usr/local/ScienceTools/refdata/fermi/caldb/CALDB/data/glast/lat/bcf/ea/aeff_P7CLEAN_V6_back.fits')
print aeff_P8R2_CLEAN_V6_FB_HDUs.info()
So, for the point source analysis we're doing, we'll need to use source class events. We use gtselect, a 'balistic' tool to do the selection. There are ways to do this from within python (I'll show you these later) but we'll run this from a terminal right now. Actually, it's a good time to talk about how to find information about these tools.
All of the fermi ballistic tools are here
environ['FERMI_DIR']
ls /usr/local/ScienceTools/bin/gt*
You can find more information out about a tool by using fhelp.
%system fhelp gtselect
We need to get ready to run gtselect now. Since we were given multiple separate photon files from the data server we need to make a file list so that when we run gtselect it knows to look in both files for events that match our selections.
Here's a quick way to do this:
%system ls /Users/eferrara/FSSC/VM/shared/psr_data/L151119044834C9B75A5E69_PH* > ~/FSSC/VM/shared/psr_data/events.txt
%system cat /Users/eferrara/FSSC/VM/shared/psr_data/events.txt
Now we're ready to go. Note in the usage for gtselect that the event class and event types aren't listed. They are 'hidden parameters', so you'll need to run the tool like this:
gtselect evclass=128 evclass=3
There are a number of event classes you can use. Unless you're an advanced user, it's best to follow the analysis recommendations above.
Standard Hierarchy for LAT Event Classes | ||||
---|---|---|---|---|
Event Class | evclass | Photon File | Extended File | Description |
P8R2_TRANSIENT020 | 16 | X | Transient event class with background rate equal to two times the A10 IGRB reference spectrum. | |
P8R2_TRANSIENT010 | 64 | X | Transient event class with background rate equal to one times the A10 IGRB reference spectrum. | |
P8R2_SOURCE | 128 | X | X | This event class has a residual background rate that is comparable to P7REP_SOURCE. This is the recommended class for most analyses and provides good sensitivity for analysis of point sources and moderately extended sources. |
P8R2_CLEAN | 256 | X | X | This class is identical to SOURCE below 3 GeV. Above 3 GeV it has a 2-4 times lower background rate than SOURCE and is slightly more sensitive to hard spectrum sources at high galactic latitudes. |
P8R2_ULTRACLEAN | 512 | X | X | This class has a background rate between CLEAN and ULTRACLEANVETO. |
P8R2_ULTRACLEANVETO | 1024 | X | X | This is the cleanest Pass 8 event class. Between 100 MeV and 10 GeV the background rate is between 2 and 4 times lower than the background rate of SOURCE class. This class is recommended to check for CR-induced systematics as well as for studies of diffuse emission that require low levels of CR contamination. |
Extended Hierarchy | ||||
Event Class | evclass | Photon File | Extended File | Description |
P8R2_TRANSIENT020E | 8 | X | Extended version of the P8R2_TRANSIENT020 event class with a less restrictive fiducial cut on projected track length through the Calorimeter. | |
P8R2_TRANSIENT010E | 64 | X | Extended version of the P8R2_TRANSIENT010 event class with a less restrictive fiducial cut on projected track length through the Calorimeter. | |
NON-ACD Hierarchy | ||||
Event Class | evclass | Photon File | Extended File | Description |
P8R2_TRANSIENT015S | 65536 | X | Transient event class designed for analysis of prompt solar flares in which pileup activity may be present. This class has a background rate equal to 1.5 times the A10 reference spectrum. |
You don't really need to specify evtype unless you plan to use one of the event types. A value of 3 is used for FRONT+BACK data. Here is a list of the available values:
Conversion Type Partition | ||
---|---|---|
Event Type | evtype | Description |
FRONT | 1 | Events converting in the Front-section of the Tracker. Equivalent to convtype=0. |
BACK | 2 | Events converting in the Back-section of the Tracker. Equivalent to convtype=1. |
PSF Type Partition | ||
Event Type | evtype | Description |
PSF0 | 4 | First (worst) quartile in the quality of the reconstructed direction. |
PSF1 | 8 | Second quartile in the quality of the reconstructed direction. |
PSF2 | 16 | Third quartile in the quality of the reconstructed direction. |
PSF3 | 32 | Fourth (best) quartile in the quality of the reconstructed direction. |
EDISP Type Partition | ||
Event Type | evtype | Description |
EDISP0 | 64 | First (worst) quartile in the quality of the reconstructed energy. |
EDISP1 | 128 | Second quartile in the quality of the reconstructed energy. |
EDISP2 | 256 | Third quartile in the quality of the reconstructed energy. |
EDISP3 | 512 | Fourth (best) quartile in the quality of the reconstructed energy. |
There's lots of info we need to give gtselect so it makes sense to run gtvcut on one of our photon files to figure out what we were doing.
%system gtvcut /Users/eferrara/FSSC/VM/shared/psr_data/L151119044834C9B75A5E69_PH00.fits EVENTS
We can combine all the files without making any additional cuts, just to see how many total events were found for this source.
Image(filename='images/P8_gtselect.png')
How many total events do we have?
allevt_hdulist = pyfits.open('/Users/eferrara/FSSC/VM/shared/psr_data/Pulsar_data.fits')
allevt_hdulist.info()
You can run all the analysis tools on the command line, as above. However, it's easier to run using a python version of the science tools. I'm going to run this using the python toolset called gtapps.
In python, let's use gtselect to reduce the nomber of events e're working with:
import gt_apps
dir(gt_apps)
from gt_apps import filter
filter.pars()
filter['infile'] = '@/Users/eferrara/FSSC/VM/shared/psr_data/events.txt'
filter['outfile'] = '/Users/eferrara/FSSC/VM/shared/psr_data/PSRJ0248_filtered.fits'
filter['ra'] = 42.0
filter['dec'] = 60.4
filter['rad'] = 15
filter['tmin'] = 239557417
filter['tmax'] = 335145602
filter['emin'] = 300
filter['emax'] = 50000
filter['zmax'] = 90
filter['evclass'] = 128
filter['evtype'] = 3
You can check what the command will look like before you run it. This is also useful for logging, and when building scripts.
filter.command()
filter.run()
Some things to note here:
Let's take a look at what we have now.
filtered_data = pyfits.open('/Users/eferrara/FSSC/VM/shared/psr_data/PSRJ0248_filtered.fits')
filtered_data.info()
Notice in the dimensions that we now have 1.04 million events in the combined event file. This is quite a change from having 24 million events returned by the original query. Lots of events have been filtered out.
The next thing to do is to make sure we update the good time interval (the GTI) column using the recommended GTI filters. These can also be found on the recommendations and good practices. Here's the relavant table:
Analysis Type | ROI-Based Zenith Angle Cut (roicut) |
Relational Filter Expression (filter) |
---|---|---|
Galactic Point Source Analysis | no | (DATA_QUAL>0)&&(LAT_CONFIG==1) |
Off-plane Point Source Analysis | no | (DATA_QUAL>0)&&(LAT_CONFIG==1) |
Burst and Transient Analysis | yes | (DATA_QUAL>0)&&(LAT_CONFIG==1) |
Galactic Diffuse Analysis | no | (DATA_QUAL>0)&&(LAT_CONFIG==1) |
Extra-Galactic Diffuse Analysis | no | (DATA_QUAL>0)&&(LAT_CONFIG==1) |
Burst and Transient Analysis | yes | (DATA_QUAL>0||DATA_QUAL==-1)&&(LAT_CONFIG==1) |
IMPORTANT: For analyses where an ROI-based zenith cut is NOT performed, an exposure correction must be made using the "zmax" option in the gtltcube tool.
These filter expression is simply a boolean expression using columns in the spacecraft file. We do this via the gtmktime command, known as maketime in gt_apps.
from gt_apps import maketime
maketime['evfile'] = '/Users/eferrara/FSSC/VM/shared/psr_data/PSRJ0248_filtered.fits'
maketime['outfile'] = '/Users/eferrara/FSSC/VM/shared/psr_data/PSRJ0248_maketime.fits'
maketime['scfile'] = '/Users/eferrara/FSSC/VM/shared/psr_data/spacecraft.fits'
maketime['filter'] = 'DATA_QUAL>0 && LAT_CONFIG==1'
maketime['apply_filter'] = 'yes'
maketime['roicut'] = 'no'
maketime.run()
Some things to note here:
Rather than applying the exposure correction to the data file, we will apply it to the livetime calculation instead.
Now what do we have...
%system ftlist /Users/eferrara/FSSC/VM/shared/psr_data/PSRJ0248_filtered.fits H
%system ftlist /Users/eferrara/FSSC/VM/shared/psr_data/PSRJ0248_maketime.fits H
You can see that the maketime command only changed the GTI extension.
Always double check the recommended cuts, especially after a data/software release. Don't hesitate to ask the FSSC for advice. We are working scientists and have proably run into the types of problems you are seeing. And if we can't, we can always contact the software developers.
These are quick ways you can have a look at the data you have. These are at a higher level than the previous methods but they are not statistically rigorous (rigor needs the Likelihood method). The data you are about to see have not been corrected for things like livetime, exposure, background, random acts of statistics etc. but go for it…
gtbin is a pretty versitle tool: it can make counts maps, counts cubes (counts maps with an energy dimension), simple light curves and more.
We're going to make a counts map and plot it up with python. You can also use ds9 or fv to plot this.
from gt_apps import counts_map
counts_map['evfile'] = '/Users/eferrara/FSSC/VM/shared/psr_data/PSRJ0248_maketime.fits'
counts_map['scfile'] = 'NONE'
counts_map['outfile'] = '/Users/eferrara/FSSC/VM/shared/psr_data/PSRJ0248_cmap.fits'
counts_map['algorithm'] = 'CMAP'
counts_map['xref'] = 42.0
counts_map['yref'] = 60.4
counts_map['coordsys'] = 'CEL'
counts_map['nxpix'] = 400
counts_map['nypix'] = 400
counts_map['binsz'] = 0.05
counts_map['axisrot'] = 0.0
counts_map['proj'] = 'AIT'
counts_map.run()
cmap = pyfits.open('/Users/eferrara/FSSC/VM/shared/psr_data/PSRJ0248_cmap.fits')
cmap.info()
%system ds9 /Users/eferrara/FSSC/VM/shared/psr_data/PSRJ0248_cmap.fits
Image(filename='images/P8_J0248_cmap.png')
If we use the ROI-based zenith cut, how does the data file change?
%system ftlist /Users/eferrara/FSSC/VM/shared/psr_data/PSRJ0248_maketime.fits H
%system ftlist /Users/eferrara/FSSC/VM/shared/psr_data/PSRJ0248_maketime_roicut.fits H
Image(filename='images/P8_J0248_cmap_roicut.png')