Data Management
For the primary purpose of the DFN network to recover meteorites, only a small fraction of the data captiured by the observatories is needed: only the images containing fireballs! These ‘highlights’ are detected by the algorithms on the observatories, and transmitted to the central server.
However… there are many other potential uses for all the data recorded by the cameras in areas of Astronomy, or for example Space Situational Awareness. Each camera records an image every 30 seconds, so we have an archive of the whole sky, all night, every night since about 2014, every 30 seconds, from multiple sites across Australia. We keep all this data in case of the Australian DFN, currently about 1.5PB of archived images, and growing at about 2TB per day for only the DFN alone. We cannot transmit this back to the central store, so to recover it from the observatories, every 1-2 years or so the sites are visited to service the systems, and to swap hard drives and return these to Perth. There the data is archived in a data store at the Pawsey Supercomputing centre. The Pawsey multi-petabyte data store allows searching of the dataset, using generic and project-custom metadata, and data sharing with other research groups. If you would like access to this store for scientific study, please contact us.
The partner networks within the Global Fireball Observatory produce another ~2TB per day, however, management of this data is partner’s responsibility. The DFN team takes care only for a subset of the data needed for the meteorite recovery.
People
Martin Cupák
Curtin U.
Martin Towner
Curtin U.
Hadrien Devillepoix
Curtin U.