Data Organization for Astrophotographers

D ATA O R G A N I Z AT I O N
F O R A S T R O P H O T O G R A P H E R S
A S T R O PA L O O Z A 2 0 2 2
J O S H K O VA C H , H E / H I M
@ S H E K I B O B O
@ S H E K I B O B O
@ S H E K I B O B O @ M A S T O D O N . S O C I A L
@ A S T R O S C H WA @ A S T R O D O N . S O C I A L

A b o u t M e
• Codeweaver, Astronomer, Enthusiast

A b o u t M e
• Grand Rapids, MI (Bortle 7ish)

A b o u t M e
• Software developer (10yrs)

A b o u t M e
• Ruby & web development

A b o u t M e
• Android app developer

A b o u t M e
• Amateur Astronomer

A b o u t M e
• Astronomical League Member at Large

A b o u t M e
• Grand Rapids Amateur Astronomical Association

A b o u t M e
• 41 Images on Astrobin (@Shekibobo)

A b o u t M e
• 41 Images on Astrobin (@Shekibobo)
• More not yet published

H i s t o r y
• Amateur Astrophotographer

H i s t o r y
• Comet C/2020 F3 NEOWISE

H i s t o r y
• Pixel 3 Astrophotography Mode

H i s t o r y
• Meade DS-90 EC

H i s t o r y
• Meade DS-90 EC
• Restored after 15 yrs in storage

H i s t o r y
• Meade DS-90 EC
• Tried so many phone adapters

H i s t o r y
• Meade DS-90 EC
• Tried so many phone adapters
• Good for moon, not for stars

M y E q u i p m e n t
• William Optics Redcat 51

• Great for wide-field nebulae

• Primary setup on EQM35-Pro

• Zhumell Z130 Tabletop Dobsonian

• Mostly for Visual

• Mostly for Visual
• Modified for Galaxy Season

C a p t u re E q u i p m e n t

• Intervalometer + DSLR

• Pixel Pro TW-283RX

• Program quantity, exposure time, delay

• Remote controllable

• Lots of time math

• Capture data is limited to DSLR file system

• ZWO ASIAir Plus

• ZWO ASIAir Plus
• Program entire plan, GOTO

• ZWO ASIAir Plus
• Control from phone or computer

• ZWO ASIAir Plus
• Control from phone or computer
• Rich capture data stored in FITS files

ASIAir - Dark File Name Examples

ASIAir - Flat File Name Examples

ASIAir - Light File Name Examples

ASIAir - Bias File Name Examples

P i x I n s i g h t : W B P P
• WBPP can process all files together,
but can also preprocess calibration
frames to save for later.

• Grouping Keywords work great, but
not when matching all raw
subframes.

• Grouping Keywords work great, but
not when matching all raw
subframes.
• Bias, Flats, Darks, and Lights all have
different matching rules.

B I A S F R A M E S
Master bias frame showing read noise on the imaging sensor.
Canon EOS Rebel T7
ISO 200, STF Stretch applied to enhance read noise

B I A S F R A M E S
• Creates read-out gradient for sensor
Canon EOS Rebel T7

B I A S F R A M E S
• Exposure: shorted possible (~250µs)
Canon EOS Rebel T7

B I A S F R A M E S
• Regenerate Bias frames monthly/quarterly
Canon EOS Rebel T7

B I A S F R A M E S
• Matching:
• Darks: ISO, Month/Season

B I A S F R A M E S
• Matching:
• Darks: ISO, Month/Season
• Flat Darks (optional): ISO, Month/Season

Series of master dark frames showing increasing thermal noise over range of ambient temperatures
Canon EOS Rebel T7
ISO 800, EXP 150.0s
CCD-TEMP range 11°C - 41°C
D A R K F R A M E S

Canon EOS Rebel T7
ISO 800, EXP 150.0s
D A R K F R A M E S
• Creates thermal noise gradient frame

Canon EOS Rebel T7
ISO 800, EXP 150.0s
D A R K F R A M E S
• Exposure: matching target frames (light/flat)

Canon EOS Rebel T7
ISO 800, EXP 150.0s
D A R K F R A M E S
• Temperature: matching target frames (light/flat)

Canon EOS Rebel T7
ISO 800, EXP 150.0s
D A R K F R A M E S
• Within 1-2°C (noise doubles every 6°C)

Canon EOS Rebel T7
ISO 800, EXP 150.0s
D A R K F R A M E S
• Regenerate Dark frames monthly/quarterly

D A R K F R A M E S
• Matching:
• Bias: ISO, Month/Season

D A R K F R A M E S
• Matching:
• Flats (optional): ISO, Temp, Exp, Month/
Season

D A R K F R A M E S
• Matching:
• Flats (optional): ISO, Temp, Exp, Month/
Season
• Lights (optional): ISO, Temp, Exp, Month/
Season

D A R K S W O R K F L O W
D ATA C A P T U R E

• Settings:
D ATA C A P T U R E

• Settings:
• Same ISO from imaging session. If using same ISO for
every session, this is much more manageable.
D ATA C A P T U R E

• Settings:
• Exposure matching the lights to calibrate.
D ATA C A P T U R E

• Settings:
• Capture:
D ATA C A P T U R E

• Settings:
• Capture:
• If uncooled, just keep shooting darks overnight until
you have enough at each temperature to integrate a
master.
D ATA C A P T U R E

• Settings:
• Capture:
master.
• Use a freezer or refrigerator as a last resort.
D ATA C A P T U R E

• Settings:
• Capture:
master.
• If shooting in daytime, be sure to cover the scope to
avoid light leak.
D ATA C A P T U R E

• Settings:
• Capture:
master.
• If shooting in daytime, be sure to cover the scope to
avoid light leak.
• Many DSLR will record CCD-TEMP in Exif Data. If
using ASIAir, the temperature can be part of the file
name.
D ATA C A P T U R E

WBPP_Integration Grouping Keywords

O R G A N I Z AT I O N & C A L I B R AT I O N

• Organization assists calibration

• Use Folder naming to create Grouping Keywords

• ISO - must match bias, flats, lights

• EXP - Exposure time as largest unit with integer time

• e.g. 250.0µs, 500.0ms, 150.0s

• e.g. 250.0µs, 500.0ms, 150.0s
• CCD-TEMP - must match target subframes

• e.g. 250.0µs, 500.0ms, 150.0s
• MONTH - YYYY-MM format date

• e.g. 250.0µs, 500.0ms, 150.0s
• Sensors degrade over time, so darks should match target
frames closely by date.

• e.g. 250.0µs, 500.0ms, 150.0s
• Once you have 10-30 darks for a given temperature, use WBPP
to calibrate a master dark with an appropriate master bias.

• e.g. 250.0µs, 500.0ms, 150.0s
• Store master dark in calibration library, sorted by season

• e.g. 250.0µs, 500.0ms, 150.0s
• Store master dark in calibration library, sorted by season
• e.g. 2022-Fall, 2022-Winter, 2023-Spring, etc.

F L AT F R A M E S
Master Flat frame showing dust motes on the image sensor and slight vignetting
Canon EOS Rebel T7
ISO 100, STF Stretch applied to enhance defects

F L AT F R A M E S
• Removes flaws from image train and hardware
configuration
Canon EOS Rebel T7

F L AT F R A M E S
configuration
• Exposure: short (1s - 10s)
Canon EOS Rebel T7

F L AT F R A M E S
configuration
• Generate new flats:
Canon EOS Rebel T7

F L AT F R A M E S
configuration
• For each image train configuration for a session
Canon EOS Rebel T7

F L AT F R A M E S
configuration
• Every session
Canon EOS Rebel T7

F L AT F R A M E S
configuration
• Every session
• Matching:
Canon EOS Rebel T7

F L AT F R A M E S
configuration
• Every session
• Matching:
• Darks (optional): ISO, Temp, Exp, Month/Season
Canon EOS Rebel T7

F L AT F R A M E S
configuration
• Every session
• Matching:
• Lights: ISO, Session, Hardware Configuration
Canon EOS Rebel T7

F L AT F R A M E S
configuration
• Every session
• Matching:
• Lights: ISO, Session, Hardware Configuration
• Telescope, Filter, Sensor Rotation
Canon EOS Rebel T7

F L AT S W O R K F L O W
https://www.thingiverse.com/thing:4757064
ASIAir Autorun plans for calibration frames.
These can be saved and toggled on/off to save time.
D ATA C A P T U R E

• Gear:
D ATA C A P T U R E

• Gear:
• Elumiglow light panel and diffuser with custom printed housing
D ATA C A P T U R E

• Gear:
• Artists tracing panel with sheets of printer paper for diffuser
D ATA C A P T U R E

• Gear:
• Settings:
D ATA C A P T U R E

• Gear:
• Settings:
• Same ISO from imaging session
D ATA C A P T U R E

• Gear:
• Settings:
• Exposure between 1s - 10s, adjusting for ISO instead of adjusting panel
brightness
D ATA C A P T U R E

• Gear:
• Settings:
brightness
• Use consistent exposure time for all flats for a given configuration to
save time and reuse flat-darks.
D ATA C A P T U R E

• Gear:
• Settings:
brightness
• Capture:
D ATA C A P T U R E

• Gear:
• Settings:
brightness
• Capture:
• Capture the morning after each imaging session.
D ATA C A P T U R E

• Gear:
• Settings:
brightness
• Capture:
• Cover the scope to avoid light leakage.
D ATA C A P T U R E

• Gear:
• Settings:
brightness
• Capture:
• Take 30 flats with light panel.
D ATA C A P T U R E

• Gear:
• Settings:
brightness
• Capture:
• Take 30 darks with matching exposure time immediately after the flats.
D ATA C A P T U R E

• Gear:
• Settings:
brightness
• Capture:
• Take 30 darks with matching exposure time immediately after the flats.
• Taken closely together, we can usually assume same temperature.
D ATA C A P T U R E

WBPP_Flats Grouping Keywords

• FLATSET - the date following the imaging session
• Image captured noon-midnight: FLATSET = following day
• Image captured midnight-noon: FLATSET = same day

• EXP
• Time + Unit (150.0s, 2.1s, 650.0ms, 250.0us)
• Ensures matching flat-darks correctly with different filters
• CCD-TEMP

• EXP
• Time + Unit (150.0s, 2.1s, 650.0ms, 250.0us)
• Ensures matching flat-darks correctly with different filters
• CCD-TEMP
• Only if keeping a Flat-Dark library independent of FLATSET
• Not necessary if taken closely with flat-darks
• I don’t use it because I don’t want to think about it

• Keep all raw flats together

• Group Flats and Flat-Darks together

• Use “Folder” button to add to WBPP

• Calibrate as soon as captured

• Keep master flat in Calibration library

• Remove EXP keyword from master
fi
lename

fi
lename
• If EXP is not removed it will not allow matching
lights in WBPP

fi
lename
• If EXP is not removed it will not allow matching
lights in WBPP
• Darks need to match EXP, flats don’t

L I G H T S W O R K F L O W

• TARGET - keyword is implied, but required

• Image captured noon-midnight: FLATSET = following
day

day

day
• EXP

day
• EXP
• Time + Unit (150.0s, 2.1s, 650.0ms, 250.0us)

day
• EXP
• Time + Unit (150.0s, 2.1s, 650.0ms, 250.0us)
• Used to match Darks

day
• EXP
• Time + Unit (150.0s, 2.1s, 650.0ms, 250.0us)
• CCD-TEMP

day
• EXP
• Time + Unit (150.0s, 2.1s, 650.0ms, 250.0us)
• CCD-TEMP
WBPP_Integration Grouping Keywords

• Keep all lights in Lights folder

• Group each set by target name

• Year, month, whatever is covered by
folder name structure

• Year, month, whatever is covered by
folder name structure
• Allows easily mixing old and new
frames down the line

• Use blink after each session to reject
bad subframes

bad subframes
• Bad subs individually get moved to
“reject” folder.

bad subframes
• After bad subs are rejected, move
folder to “accept”.

bad subframes
• After bad subs are rejected, move
folder to “accept”.
• When capture is complete for target,
use “Folder” button to add “accept”
in WBPP.

W B P P I n t e g r a t i o n
WBPP_Integration Grouping Keywords WBPP_Flats Grouping Keywords WBPP_Darks Grouping Keywords

• Use pre-calibrated darks matching temperature range of lights to
be processed.

be processed.
• Use pre-calibrated flats matching FLATSET for the selected lights.

be processed.
• No master bias required

be processed.
• Darks were already calibrated

be processed.
• Flats were calibrated with darks that were calibrated with a
bias. WBPP_Integration Grouping Keywords WBPP_Flats Grouping Keywords WBPP_Darks Grouping Keywords

be processed.
bias.
• Every calibration frame is only needing to match grouping
keywords with light frames.

be processed.
bias.
• Just load the files and click OK.

be processed.
bias.
• Major Benefits

be processed.
bias.
• Major Benefits
• Only calibrate bias, flats, and darks once.

be processed.
bias.
• Major Benefits
• Grouping Keyword matching conflicts are avoided.

be processed.
bias.
• Major Benefits
• Grouping Keyword matching conflicts are avoided.
• Reuse pre-calibrated darks and flats on multiple targets.

- M E , W H E N I M A N U A L LY O R G A N I Z E M Y F I L E S
“This sounds like a lot.”

- A L S O M E
“I don’t want to think about it.”

I W R O T E A S C R I P T T O D O T H I S F O R M E .
https://gist.github.com/shekibobo/b8e853bf70d787b55a87f7a4cf7ed248

Data Organization for Astrophotographers

Recommended

Recommended

More Related Content

Similar to Data Organization for Astrophotographers

Similar to Data Organization for Astrophotographers (20)

Recently uploaded

Recently uploaded (20)

Data Organization for Astrophotographers