YLA seamless GOES project

older version (11-Jan-2011)
19-Jan-2016, Aki Takeda.
1-Mar-2016, statistics added.
28-Mar-2016, link to More about XRS" page added.

1. GOES satellite (not a NASA satellite!)

Information mainly from the GOES website ( http://www.ngdc.noaa.gov/stp/satellite/goes/index.html)
Data center: NGDC(National Geophysical Data Center) under NCEI(National Centers for Environmental Informationl.

GOES : Geostationary Operational Environmental Satellites (No.1 -- 15)
SMS : Synchronous Meteorological Satellites (preceding mission of GOES)
SEM : Space Environment Monitor (instrument subsystem on board all GOES and SMS).

Magnetometer : magnetic field measurements (HP, HE and HN).
Energetic Particle Sensor(EPS) : energetic particle.
X-Ray Sensor(XRS) : soft X-ray flux (0.5-3A, 1-8A).
Extreme Ultraviolet Sensors (EUVS) since GOES-13, 14 & 15.
Soft X-ray Imager : (GOES12)

*** GOES coverage ***
    * SMS-1: 1974-07 to 1974-10
    * SMS-2:: 1975-02 to 1976-04
    * GOES-1 1976-01 to 1978-05
    * GOES-2 1977-08 to 1983-05
    * GOES-3 1978-07 to 1980-08
    * GOES-4 no data
    * GOES-5 1983-01 to 1987-02
    * GOES-6 1983-05 to 1994-11  ### XRS: ?-may-1983 to 30-may-1994
    * GOES-7 1987-03 to 1996-08  ### XRS: 6-mar-1987 to 13-aug-1996
    * GOES-8 1995-03 to 2003-06  ### XRS: 1-mar-1995 to 10-apr-2003  
    * GOES-9 1996-04 to 1998-08  ### XRS: 1-apr-1996 to 27-jul-1998
    * GOES-10 1998-07 to 2009-12 ### XRS: 1-jul-1998 to 30-nov-2009
    * GOES-11 2000-07 to present ### XRS: 1-jun-2006 to 9-feb-2008
    * GOES-12 2003-01 to 2010-09 ### XRS: 10-jan-2003 to 12-apr-2007 
    * GOES-13 2010-04 to present; EUV, XRS, SXI not operational. 
                                 ### launched in apr-2006.
    * GOES-14 2009-12 to present, XRS only, other systems in storage mode.
                                 ### XRS: 26-nov-2009 to 26-oct-2010 
    * GOES-15 2010-09 to present, XRS only, other systems in test mode.
                                 ### XRS: 27-oct-2010 to present 

### : added by AkT

2. The current GOES data archive

Two original data source at NOAA:

(1) Most recent data : http://www.swpc.noaa.gov/ftpmenu/lists/xray.html
daily text files for 1m and 5m average, together with other reports.
(2) NGDC archive : https://ngdc.noaa.gov/stp/satellite/goes/dataaccess.html
monthly text files, 1m- 5m- average (to present) and 3-sec full (-1996, FITS).

Data archive in SolarSoft database:

SDAC GOES data archive: ($SSWDB/goes/xray)

mithra:/sswdb/goes/xray> ls -lrt | head
total 902124
-r--r--r-- 1 software guest 20550 Sep 19  2002 20020919_G8xr_5m.txt
-r--r--r-- 1 software guest 20553 Sep 19  2002 20020919_G10xr_5m.txt
-r--r--r-- 1 software guest 20550 Sep 20  2002 20020920_G8xr_5m.txt
-r--r--r-- 1 software guest 20553 Sep 20  2002 20020920_G10xr_5m.txt
-r--r--r-- 1 software guest 20550 Sep 21  2002 20020921_G8xr_5m.txt
-r--r--r-- 1 software guest 20553 Sep 21  2002 20020921_G10xr_5m.txt
-r--r--r-- 1 software guest 20550 Sep 22  2002 20020922_G8xr_5m.txt
-r--r--r-- 1 software guest 20553 Sep 22  2002 20020922_G10xr_5m.txt
-r--r--r-- 1 software guest 20550 Sep 23  2002 20020923_G8xr_5m.txt

mithra:/sswdb/goes/xray> ls -lt | head
total 902124
-r--r--r-- 1 software guest  7493 Jan 19 00:02 Gp_xr_1m.txt
-r--r--r-- 1 software guest  7493 Jan 19 00:02 Gs_xr_1m.txt
-r--r--r-- 1 software guest 41759 Jan 19 00:01 20160119_Gp_xr_1m.txt
-r--r--r-- 1 software guest 41759 Jan 19 00:01 20160119_Gs_xr_1m.txt
-r--r--r-- 1 software guest 10630 Jan 19 00:00 20160119_Gp_xr_5m.txt
-r--r--r-- 1 software guest 10630 Jan 19 00:00 20160119_Gs_xr_5m.txt
-r--r--r-- 1 software guest  2341 Jan 19 00:00 Gp_xr_5m.txt
-r--r--r-- 1 software guest  2341 Jan 19 00:00 Gs_xr_5m.txt
-r--r--r-- 1 software guest 20566 Jan 18 12:15 20160118_Gp_xr_5m.txt

Yohkoh GOES archive: ($SSWDB/ydb/g??)

mithra:/sswdb/ydb> ls -l
total 2548
...
...
drwxrwxr-x  2 software solar  16384 Feb  1  2011 g01/
drwxrwxr-x  2 software solar  16384 Feb  1  2011 g05/
drwxrwxr-x  2 software solar  16384 Oct  6  2010 g0d/
drwxr-xr-x  2 software cron    4096 Feb  1  2011 g11/
drwxr-xr-x  2 software cron    4096 Feb  1  2011 g15/
drwxr-xr-x  2 software cron    4096 Mar 16  2011 g1d/
drwxr-xr-x  2 software cron    8192 May 24  2009 g21/
drwxr-xr-x  2 software cron    8192 May 24  2009 g25/
drwxr-xr-x  2 software cron    8192 May 24  2009 g2d/
drwxr-xr-x  2 software guest   4096 Feb  1  2011 g41/
drwxr-xr-x  2 software guest   4096 Feb  1  2011 g45/
drwxr-xr-x  2 software guest   4096 Feb  1  2011 g4d/
drwxr-xr-x  2 software guest   8192 Jan 19 05:03 g51/
drwxr-xr-x  2 software guest   8192 Jan 19 05:03 g55/
drwxr-xr-x  2 software guest   8192 Jan 19 05:03 g5d/
drwxrwxr-x  2 software solar   8192 Jun 13  2008 g61/
drwxrwxr-x  2 software solar   4096 Jun 13  2008 g65/
drwxrwxr-x  2 software solar   8192 Jun 13  2008 g6d/
drwxrwxr-x  2 software solar   8192 Jun 13  2008 g71/
drwxrwxr-x  2 software solar   8192 Jun 13  2008 g75/
drwxrwxr-x  2 software solar   8192 Jun 14  2008 g7d/
drwxrwxr-x  2 software solar  16384 Jun  6  2008 g81/
drwxrwxr-x  2 software solar  16384 Jun  6  2008 g85/
drwxrwxr-x  2 software solar  12288 Feb  2  2007 g8d/
drwxrwxr-x  2 software solar   8192 May 24  2009 g91/
drwxrwxr-x  2 software solar  12288 May 24  2009 g95/
drwxrwxr-x  2 software solar   4096 Mar 16  2003 g9d/
...
...

mithra:/sswdb/ydb> ls -l g51/g5116*
-rw-r--r-- 1 software guest  53680 Jan 16 04:50 g51/g5116_01a.01
-rw-r--r-- 1 software guest 168880 Jan 18 04:50 g51/g5116_02a.01
-rw-r--r-- 1 software guest 168880 Jan 18 04:50 g51/g5116_03a.01
-rw-r--r-- 1 software guest  30944 Jan 18 04:50 g51/g5116_04a.01

### problem ###
Both (1) and (2) sometimes have missing records, which often compensate with each other.

Two archives under $SSWDB

$SSWDB/ydb/g** series
- week files since 1991_w01.
- maintained by YLA.
  - 1991-2001 regenerated based on the source (2) and replaced in 2008.
  - 2002-2008 regenerated based on the source (2) in 2009, replaced in 2011.
  - 2009-2010 regenerated based on the source (1) and (2) and replaced in 2011.
- coverage after 2009.
- read with RD_GXD.
$SSWDB/goes/xray/yyyymmdd_G*xr_?m.txt series
- daily files since 19-sep-2002, same files as in the source (1).
- 1 minute or 5 minute avarage text files for GOES 8-15.
- coverage since 2002.
- read with RD_GOESX_ASCII

Comparison of coverage: SDAC vs YDB
GOES X-ray Flux Archive by Andreas Moller
NASA GOES IDL User Guide by Zarro and Tolbert
SDAC GOES FITS archive (1974-present)

GOES primary/secondary defined by SWPC (plus my findings from SolarMonior plots)
https://ngdc.noaa.gov/stp/satellite/goes/doc/GOES_XRS_readme.pdf

[Table 2.] 
Chronology of primary and secondary satellites for XRS measurements since 1986
------------------------------------------------------
Start Date  Time   Primary Secondary      Remarks
------------------------------------------------------
2015-06-09  00:00  GOES-15  GOES-13  (AkT from SolarMonitor plots)     
2015-05-21  00:00  GOES-14  GOES-13  (AkT from SolarMonitor plots)     
2015-01-24  00:00  GOES-15  GOES-13  (AkT from SolarMonitor plots)     
2012-11-19  16:31  GOES-15  None
2012-10-23  16:00  GOES-14  GOES-15
2011-09-01   0:00  GOES-15  GOES-14
2010-10-28   0:00  GOES-15  None
2010-09-01   0:00  GOES-14  GOES-15
2009-12-01   0:00  GOES-14  None     GOES-10 decommissioned
2008-02-10  16:30  GOES-10  None     GOES-11 XRS failure
2007-12-18   0:00  GOES-11  None     GOES-10 not tracked due to antenna problems
2007-12-05   0:00  GOES-11  GOES-10
2007-11-21   0:00  GOES-11  None     GOES-10 not tracked due to antenna problems
2007-04-12   0:00  GOES-11  GOES-10
2007-01-01   0:00  GOES-10  GOES-11
2006-06-28   0:00  GOES-12  GOES-11
2006-06-22  14:00  GOES-12  None
2003-05-15  15:00  GOES-12  GOES-10
2003-04-08   0:00  GOES-10  GOES-12
1998-07-27   0:00  GOES-8   GOES-10
1995-03-01   0:00  GOES-8   GOES-7
1994-12-11   0:00  GOES-7   GOES-8
1988-01-26   0:00  GOES-7   GOES-6
1986-01-01   0:00  GOES-6   GOES-5
------------------------------------------------------

3. YLA seamless GOES data archive

Specification of the product:

Continuous records of GOES/XRS high- and low- channel flux from the Primary satellite.
For the period 1991/01/01 through 2009/12/31 with 1 minute average data.
- 1991 : start year of Yohkoh project
- 2009 : Primary/Secondary filename convention started on 2009/11/26 at $SSWDB/goes/xray files.
The missing records are complemented with the Secondary satellite's data.
Text format consistent with the current SSWDB GOES archive: "$SSWDB/goes/xray/yyyymmdd_Gp_xr_1m_txt".

list of the pairs of Primary/Secondary satellites (from GOES_XRS_readme.pdf with correction in blue by AkT).

year	Primary	Secondary	memo
1991	GOES7	GOES6
1992	GOES7	GOES6
1993	GOES7	GOES6
1994	GOES7	GOES6 none (8/19 00:00- )	GOES 6 data available until 8/19. No GOES 8 data exist in 1994.
1995	GOES7 GOES8 (3/1 00:00- )	GOES8 GOES7 (3/1 00:00- )	No YDB GOES 8 data found for 1/1 -- 6/28, complemented by NGDC new average files.
1996	GOES8	GOES7 GOES9 (8/1 00:00- )	No GOES 7 data exist after 8/1. Secondary replaced with GOES 9.
1997	GOES8	GOES9	Secondary replaced with GOES 9.
1998	GOES8	GOES9 GOES10 (7/27 00:00- )	Secondary replaced with GOES 9 until 7/26.
1999	GOES8	GOES10
2000	GOES8	GOES10
2001	GOES8	GOES10
2002	GOES8	GOES10
2003	GOES8 GOES10 (4/8 00:00- ) GOES12 (5/15 15:00- )	GOES10 GOES12 (4/8 00:00- ) GOES10 (5/15 15:00- )
2004	GOES12	GOES10
2005	GOES12	GOES10
2006	GOES12	GOES10 none (6/22 14:00- ) GOES11 (6/28 00:00- )
2007	GOES10 GOES11 (4/12 00:00- )	GOES11 GOES10 (4/12 00:00- ) none (11/21 00:00- ) GOES10 (12/5 00:00- ) none (12/18 00:00- )
2008	GOES11 GOES10 (2/10 16:30- )	none
2009	GOES10 GOES14 (12/1 00:00- )	none

YLA seamless GOES data preparation steps:

Creation of seamless data set.
- prepare date-time structure array for each year (1991-2009).
- prepare data set only with records from primary satellites.
- prepare data set only with records from secondary satellites.
- make hybrid data set by complementing missing primary records with secondary.
Adjustments/corrections of flux values.
- between primary and complemented secondary records.
- between different primary satellites.
- overall correction, if any (needs to contact GOES experts).
Creation of daily files in the ascii format.
Prepare software to hadle new data base.
- modified version of rd_goesx_ascii.pro.
Inject final products into SSWDB and SDAC and advertize.

Breakdown of seamless GOES data records.

Contributions from primary(P) and secondary(S) satellites.

year	total	rec. from P.	rec. from S.	S-chunks	rec. no data	no data -chunks	satellite (P/S)	daily plot movies
1991	525600	516252 (98.2%)	7687 ( 1.5%)	170	1661 ( 0.3%)	40	7/6	lo & hi -bands
1992	527040	519137 (98.5%)	6347 ( 1.2%)	201	1556 ( 0.3%)	79	7/6	low & high
1993	525600	520301 (99.0%)	4255 ( 0.8%)	173	1044 ( 0.2%)	71	7/6	low & high
1994	525600	516658 (98.3%)	4166 ( 0.8%)	233	4776 ( 0.9%)	246	7/6	low & high
1995 1/1 -- 3/1 --	525600 84960 440640	511420 (97.3%) 84575 (99.5%) 426845 (96.9%)	10603 ( 2.0%) 383 ( 0.5%) 10220 ( 2.3%)	244	3577 ( 0.7%) 2 ( 0.0%) 3575 ( 0.8%)	44	--- 7/8 8/7	low & high
1996 1/1 -- 8/1 --	527040 306720 220320	518494 (98.4%) 302338 (98.6%) 214716 (98.1%)	8127 ( 1.5%) 4347 ( 1.4%) 3780 ( 1.7%)	237	419 ( 0.1%) 35 ( 0.0%) 384 ( 0.2%)	22	--- 8/7 8/9	low & high
1997	525600	507745 (96.6%)	16631 ( 3.2%)	235	1224 ( 0.2%)	76	8/9	low & high
1998 1/1 -- 7/27 --	525600 298080 227520	518430 (98.6%) 294672 (98.9%) 223758 (98.3%)	6975 ( 1.3%) 3228 ( 1.1%) 3747 ( 1.6%)	248	195 ( 0.0%) 180 ( 0.1%) 15 ( 0.0%)	33	--- 8/9 8/10	low & high
1999	525600	518436 (98.6%)	7076 ( 1.3%)	292	88 ( 0.0%)	50	8/10	low & high
2000	527040	519885 (98.6%)	7010 ( 1.3%)	198	145 ( 0.0%)	36	8/10	low & high
2001	525600	517519 (98.4%)	7886 ( 1.5%)	190	195 ( 0.0%)	16	8/10	low & high
2002	525600	514599 (97.9%)	10998 ( 2.1%)	176	3 ( 0.0%)	2	8/10	low & high
2003 1/1 -- 4/8 -- 5/15 --	525600 139680 54180 331740	517102 (98.4%) 136772 (97.9%) 52497 (96.9%) 327833 (98.8%)	8404 ( 1.6%) 2908 ( 2.1%) 1674 ( 3.1%) 3822 ( 1.2%)	261	94 ( 0.0%) 0 ( 0.0%) 9 ( 0.0%) 85 ( 0.0%)	11	--- 8/10 10/12 12/10	low & high
2004	527040	519655 (98.6%)	7147 ( 1.4%)	164	238 ( 0.0%)	10	12/10	low & high
2005	525600	518532 (98.7%)	7018 ( 1.3%)	169	50 ( 0.0%)	6	12/10	low & high
2006 1/1 -- 6/28 --	525600 256320 269280	510167 (97.1%) 249514 (97.3%) 260653 (96.8%)	15296 ( 2.9%) 6721 ( 2.6%) 8575 ( 3.1%)	268	137 ( 0.0%) 85 ( 0.0%) 52 ( 0.0%)	5	--- 12/10 12/11	low & high
2007 1/1 -- 4/12 --	525600 145440 380160	513427 (97.7%) 137575 (94.6%) 375852 (98.9%)	11441 ( 2.2%) 7647 ( 5.3%) 3794 ( 1.0%)	217	732 ( 0.1%) 218 ( 0.1%) 514 ( 0.1%)	34	--- 10/11 11/10	low & high
2008 1/1 -- 2/10 --	527040 58590 468450	518083 (98.3%) 58027 (99.0%) 460056 (98.2%)	--- --- ---	---	8957 ( 1.7%) 563 ( 1.0%) 8394 ( 1.8%)	203	--- 11/- 10/-	low & high
2009 1/1 -- 12/10 --	525600 480960 44640	515661 (98.1%) 471745 (98.1%) 43916 (98.4%)	--- --- ---	---	9939 ( 1.9%) 9215 ( 1.9%) 724 ( 1.6%)	198	--- 10/- 14/-	low & high

Check new hybrid 1992 daily plots

Sample plots : 1 day flux seamless goes performance
Problematic cases: 1 (error in original data) 2 (discrepant P & S int.) 3 (noise in origial data)

Questions to GOES specialists (Rodney@SWPC or Janet@NCEI )

removing scale factor necessary for pre-GOES8 data? Why? :
Because, the scale factor was to adjust GOES-8 value to the same level as GOES-7. Now that GOES-8 measurements are regarded more reliable, we need to remove the GOES-8 factor, and apply the inverse of the original factor to GOES-7 measurements (so that both measurements are consistently connected).
calbration between primary & secondary sat data (gap sometime):
The cross-calibration between new and old XRSs are performed using the peak of flare data, and the background level are not cared much.
GOES-R will be launched in October 2016. The new XRS on-board GOES-R is photo-diode (not ion chamber) and supoused to provide better flux values (W/m^2) than previous series. Assuming it is launched as scheduled in this October, when can we have obtain the first reliable data?
It will be within a couple of months.
is there similar products already exist? I recently learned about a report of Space Envionment Technologies:
Rodney has never seen the products, but he knows Dave Bouwer (SET) in person and he can ask if needed.
isn't GOES-9 the Secondary for the period, 1996 8/1 -- 1998 7/26?
You can use GOES09 data if available and behave no problem.
In 2003 high channel, is it OK to connect GOES8 to GOES12 without using GOES10? The detection limit of GOES10(high channel) is significantly higher than that of GOES8 and GOES12.
If no better method, we should accept.
Different behavior between GOES7 and 8 :
May be because of the "electron contamination" (not completely removed by the front magnet), which affects more GOES 8 (3-axis stabl.) then GOES 7(rotating, not always seeing the Sun).
Are the current (I) data in archive?
No current data in the archive, but for the recent sat. the "count" data exist.
Quantum effect in transition of GOES08 to 10 in 2003 (seen in hi-channel or low count situation):
It is due to the interaction of noise reduction and digital signal sampling employed since GOES10. (Noise makes light curves smooth).