Analyzing Large Radar Datasets Using Python

Date and Time: 
Monday 2018 Apr 2nd
CG Auditorium
Robert Jackson

Robert Jackson, Scott Collis, Zach Sherman, Giri Palanisamy, Scott Giangrande, Jitendra Kumar, Joseph Hardin

The Atmospheric Radiation Measurement (ARM) program is on track to collect hundreds of terabytes of scanning radar data per year for use by the scientific community. Such measurements provide a wide variety of data necessary for developing climatologies. This presentation demonstrates the use of Python to analyze up to hundreds of terabytes of radar data on U.S. Department of Energy (DOE) Atmospheric Radiation Measurement (ARM)’s Stratus and Cumulus supercomputers. In particular, Corrected Moments in Antenna Coordinates 2.0 (CMAC2.0), a Py-ART based framework for processing of ARM scanning radar data, is scaled using Dask and PySpark to distribute the task of processing scanning radar data to hundreds of cores. The advantages and disadvantages of using Dask and PySpark will be shown, as well as the performance of each. Both packages are viable solutions for the processing of multi-terabyte radar datasets. A demonstration of the use of Dask’s graphical interface to profile the performance of CMAC2.0 will also be shown

Speaker Description: 

Bobby Jackson is a postdoctoral researcher at Argonne National Laboratory. He received his B.S. in Math and Computer Science and Ph.D. in Atmospheric Sciences at the University of Illinois at Urbana-Champaign. He now specializes in developing algorithms to analyze large scanning radar datasets using HPC platforms.

PDF icon seaconf18_jackson.pdf997.8 KB

Event Category: