Looking Up to the Heavens Using Big Data Gives Way to New Discoveries

Remember when the sky looks this heavenly? / Photo by: Triff via Shutterstock


Big Data helped astronomers discover that thousands of black holes are likely to be found near the center of the Milky Way, the Earth’s very own galaxy. The data used in making the discovery were obtained not by using the latest state-of-the-art telescope but from 20-year-old x-ray images that have long been archived, according to Eileen Meyer, writing for the Smithsonian.

The advent of big data will make astronomical discoveries such as the one described above more frequent as the amount of data gathered increases exponentially but it will take years to bring to light all the unexamined data kept in archives.

Hubble Space Telescope

For example, the Hubble Space Telescope has made 1.3 million observations since it was launched in 1990 and sends around 20GB of raw data back to Earth each week. But such massive amount of data pales in comparison to the 2 terabytes of data that the Atacama Large Millimeter Array in Chile generates every day.

Impressive by ordinary standards, further technological developments in astronomy will make big data even bigger. Each batch of observatories is 10 times more sensitive than the previous batch because of better technology or the mission is larger.

Large Synoptic Space Telescope

The LSST is an optical telescope that is being constructed in Chile, and this beast will scan the entire sky every few nights upon completion. The LSST is so accurate that it can generate 10 million alerts each night on new or transient astrophysical sources, resulting in a database of 15 petabytes after 10 years.


A Hubble Space Telescope / Photo Credit via Wikimedia Commons


Square Kilometer Array

When finished by 2020, the SKA will be the world’s most sensitive telescope, capable of detecting airport radar stations of alien civilizations up to 50 light years away. In just one year alone, the SKA will produce more data than the Internet.

Such ambitious projects will surely test the limits of how scientists handle data. Transmitted images will have to be immediately processed, with the data reduced to a manageable size. The new observatories will require data processing facilities capable of handling hundreds of terabytes of data daily.

For Dr. Maya Dillon, community manager of Pivigo, the real challenge is how to search and synthesize such an immense amount of data. Dr. Dillon claims there are four big issues associated with the processing of large astronomical data.  These are :

-  Visualization of astronomical data

-  Development of efficient algorithms that will be used in processing astronomical data

-  Efficient development and interaction of the large database

-  Use of machine learning methodologies

These issues are brought about by the characteristics of big data, which are volume or the amount of data, variety or the complexity of data and velocity, the rate of data, and information flow. To make headway in the processing of such massive data, astronomy needs to adopt data mining techniques. This will include using artificial intelligence, machine learning, statistics, and database systems.   



Radio Astronomy

On the other hand, Ray Norris also believes that the use of big data in radio astronomy will lead to awesome discoveries in the said field.

From a few hundred thousand, the number of known radio sources increased to about 2.5 million with the unveiling of four research projects by the turn of the new millennium. These are the Westerbork Northern Sky Survey, the NRAO VLA Sky Survey, the Faint Images of the Radio Sky at Twenty-cm and the Sydney University Molonglo Sky Survey. But in the next two decades hence, there is still no significant increase in the number of known radio sources because nobody could improve on what the four research projects had done.

But a new generation of telescopes being built across the globe is set to generate more knowledge in radio astronomy. This is led by Australia's Evolutionary Map of the Universe project, featuring the Australian Square Kilometre Array Pathfinder telescope, with the revolutionary phased array feed, allowing ASKAP to view large areas of the sky at once.

The EMU project alone will raise the number of radio sources to about 70 million, which dwarfs the 2.5 million radio sources discovered in the entire history of radio astronomy. Such a huge growth in radio astronomy knowledge will have several consequences.


Radio astronomy for heavenly bodies / Photo by geralt via Pixabay


It will provide answers to the major questions in astrophysics, such as the pervasiveness of super-massive black holes in the universe. It will change the practice of radio astronomy because data collected by the EMU and the other up-and-coming research projects will be made available on the Internet. As such, radio astronomy research will be done by searching the Web instead of direct observation.

It will also change the way astronomers do their research at other wavelengths. At the moment, only a small minority of galaxies have been studied at radio wavelengths. From now on, most galaxies will be studied using excellent radio data.

Having such large amounts of data will change the way science is being done. For example, to understand how the gravitational field of nearby galaxies bends light from distant galaxies, one needs to find the best single example possible and spend nights using the telescope to study the process in detail. In the future, by downloading data from the Internet, astronomers will be able to correlate the millions of background galaxies with the millions of foreground galaxies in even greater detail.