As I am writing this, it has been nearly a day and a half of doing nothing on the project. I just got through the brunt of the training to lead NSO next week, and wanted to update the blog on the progress I made before training but didn’t have time to share here. When I last updated the blog, I vaguely explained that my evening had been crazy. After meeting with Steve about the posters, I spent the evening beginning a code to find the 814 sources, average magnitudes between them and match them to their corresponding stars in 606. As was the case with the first few catmat codes, I felt that I was pushing the boundary of my coding comfort. With the 814 code, I felt this even more, and I came up with very inefficient ways to do the things I needed to do.

Yesterday I was able to get some rudimentary catalog matching running, but was feeling very flummoxed on how to proceed, I knew what had to be done, but didn’t think I had the programming chops to do it. It took me all morning and a lot of frustration, but I believe I finally produced a catalog of stars and a way of matching the same sources. The programs I wrote, catmat1,2, and 3 look at the fields, find the stars that are separated by less than half an arc second, delete them from the larger list, average their magnitudes and add them back to the larger list as one object with the average of their WCS coordinates and the average of their magnitudes. Catmat1 found the catalog between ibo01 and ibo02 and catmat2 found the catalog between ibo03 and ibo04. catmat3 looks at the catalog products of catmat1 and 2 and makes a master catalog with both of them. I am planning on using this master catalog to find matches for the 814 stars. There are 9 814 fields, and I haven’t decided exactly how I am going to tackle them. I am kind of fried from a pretty intense morning trying to make catmat work. I am thrilled it is working now, and proud of the on-the-fly learning I was able to do.

Yesterday I began to develop some catalog matching code. It worked this morning, but then I was faced with the more tedious and difficult task of turning all of this data into a catalog to match all of the stars that I screened for. I spent the morning thinking about this, and met with Dr. G in the afternoon to run some questions by her. I also was confused about how to best accomplish source discretion on all of the files. Should I keep awk parameters the same for every file or change it up for each one depending on how it looked?

This weekend, I honed my star detection a bit and overlaid the detected stars on a reference image to see how well the stars matched the actual stars on the image. I didn’t have a ton of success matching stars well, and wasn’t sure how to proceed. I felt like I needed somethin like the “threshold” parameter from our earlier aperture photometry source detection codes.

This week has been full of great progress on Dolphot, bounding closer to the goal of making Color-Magnitude Diagrams of NGC6819. The only problem is it is one of our last weeks! I wish we had a few more weeks to work together on this project before school started, but alas, the summer is cruelly short every time.

Yesterday saw me reduce the object list from dolphot output from one of the image sets to 887 probable stars. Today I set out to visualize these probable stars on the reference image they were extracted from. I wrote a python program using knowledge from back when we wrote our own aperture photometry codes, and I successfully displayed circular apertures at each probable star’s coordinates overlaid on the reference image. Success! Until I realized that the source detection was horribly wrong.

Yesterday Thom and I finished running dolphot on all 13 sets of images. Big thanks to our computers for their hours and hours of toil looking at many millions of “stars”. Now we have to step back and figure out how to find the really important stars from these long lists of possible objects. Yesterday I figured out how to use the text utility awk to produce a new file with only some of the rows from the original huge file. I had no luck using object type to whittle the list into something of reasonable length, and thought perhaps chi would be a good value to use to screen the objects since it has to do with how much they look like the PSF of an actual star. This morning, reading through the dolphot manual however, I noticed that it specifically warns against using chi as an absolute quality indicator. Saying, “Because they depend very heavily on how well you have tuned the NoiseMult parameter, the χ values tend to be unreliable for measuring goodness of fit.” I am not going to use chi as a singular cutoff. So we still need to find something that we will be able to cutoff based on.

It felt really good to be reunited with the team this morning. The tedium of running dolphot and getting huge outputs was weird to encounter at home, but I feel more confident taking on the big files dolphot spits out now that Thom and I are working together as a team. I am running dolphot on another folder of files right now, and reading a tutorial on awk, a text processing utility that dolphot recommends to interpret the output.

These last few days have been busy as I came down from Vermont and headed back to New Hampshire for one last night. I have a bit of time at the airport, before my flight boards. I wanted to just update the group that I will be back to work tomorrow morning. The rest of today I will be busy flying and driving home. I plan on working full-time between now and the 16th. Thom and I discussed the difficulties of working with Dolphot, although we plan on continuing to use it. We will need to divide to get through all of the data since the program takes so long to run, and I think it will be good when we are together at the end of this week going forward.

Yesterday the photometry part of dolphot finally ran for me. I hadn’t expected it to take as long as it did, so I hadn’t let it run before. Now I know that it takes a very long time to run. Although photometry ran and created several non-empty files, I still have no idea how to decipher what they say. I am trying a fresh run of pydolphot and dolphot to remove any issues that may have been caused from the fact I was running dolphot several times yesterday.

Yesterday I ended my workday with what I thought was an output for dolphot. There were new folders in my working directory that looked like dolphot results. They were empty when I opened them in atom, but I assumed that was an issue with atom and not them. Today, I checked them though, and they are truly empty. I ran dolphot again a few times and I added the PSFs (pretty important step I neglected yesterday). Now what I am thinking is that dolphot takes a really long time to run. Dolphot has been using over 90% of my cpu for the last two hours just running the photometry. looking in phot.log, I can see that it has gone through some of the preliminary steps and identified 124 stars for alignment. It is now on the “finding stars” step and keeps writing a new dot every half hour or so. I now have “Finding Stars: …” I am going to see if I can find some parameter I could tweak that may make it go faster. This is the photometry for one out of the 13 groups that we have of NGC6819, so I am a little baffled at how long it is taking. I generally distrust programs that take a really long time because they encounter little errors after running for two hours and then you don’t get anything out of them.

The benefit of borrowing my dad’s electric car to head down to New Hampshire for a few days is that I have four hour chunks of time while it charges to find a coffee shop and get some work done!

I was able to spend a short period of time this morning getting prepared to run more photometry. I now have all the files I need to have reference images and the images for photometry. I think I may have previously missed some of the flc images, as I now have 61 of them. I have to run to catch the boat to the island where I’m headed, but I will be back on Thursday to run more photometry.

I spoke with Thom this morning about what path he is taking with the reference files. I believe he is using each mast pipeline .drz image to do the photometry on the .flc images that are in the frame of that reference image. I am interested to see how this works. I am personally still not sure what the best way to proceed on reference images is. I am even confused about what the reference image does. My current understanding is that it provides the coordinate system over which to overlay dolphot’s photometry. I am confused about the necessity of it though, since the flt images have accurate wcs data. The enhanced resolution of drz images would enhance the source positional accuracy, so perhaps that is why we use dr(z/c) images as references.

Continuing Progress

Reference Files

Rory’s Astrophysics Blog is going interstate! I am in Chicago now, waiting for my flight to Boston (home). I am going to fill my readers in on a few things that have been giving me a lot of trouble and some ideas I have for next steps.

I have spent my morning thus far troubleshooting the pydolphot execution. I got the acs and wfc3 modules for dolphot I added the acs and wfc3 folders to the dolphot2.0 folder, edited the dolphot 2.0 makefile to uncomment the wfc3 line and acs line. I also made sure to change the directory for the makefile to match the directory in which I want dolphot. I used ‘make clean’ to wipe the previous make and reran the make in the new directory for dolphot (and its modules). The bin now contains dolphot modules and the acs and wfc3 modules needed to run it. Pydolphot still wouldn’t run, which is now obvious as I now understand what the PATH is.

Today I am continuing to try to get some results out of dolphot, doing some error-hopping (where I resolve one error then find the next error) This process continues until the program runs out of errors at which point it works successfully. For a while, I was editing the wrong make_dolparam.py file so when I ran what I thought was the new file it wasn’t actually a new one at all, so I kept getting the same exact error. Now the program is no longer giving me the ‘images not found’ error, it is rather giving me an error that takes about ten minutes to happen (since the program copies all of the files into the working directory). I am modifying the program to only load one file to identify the errors more quickly. Once I deal with the errors that occur when loading one file, hopefully loading all of the files will work.

New Programs, New Problems

This morning I am getting back to working with DOLPHOT. On Thursday I began to work my way through the pydolphot tutorial and completed splitting each image up into its component chips. Each image now has 15 sub-images divided into chips. There is something up though as I don’t have the 615 files I would expect out of 41 images. I only count 416 in my folder. I may have to run the program again and check for any errors. For now, I am going to keep going through the program’s steps to familiarize myself with DOLPHOT.

FWHM

DOLPHOT

I am going to work through the Dolphot tutorial here. I am just going to do it step by step. I will post the text of the tutorial and annotate in italics what I am doing as I go.

Calculating FWHM

Polishing the programs

Playing with source detection parameters

Source Detecting

Troubleshooting saving the outputs

Slowing Down

Troubleshooting Source Detection Code

Getting into it

Reading about photometry (continued!)

Reading about photometry

Reflection Posts

Slow and Steady

Github Magic

Finally

10 am Today I am starting to wrap my head around some of the stuff we are going to do this summer. I am beginning to go through the photutils tutorial and I think I might try to find a cup of coffee soon. I am excited for a cup of coffee and to learn how to do photometry and image analysis in python! I love the idea of using the powerful open-source stuff rather than the somewhat dated iraf environments I have used briefly in the past.

This is my first post. I finally figured out how to make a post!