Robert Jackson, Scott Collis, Zach Sherman, Giri Palanisamy, Scott Giangrande, Jitendra Kumar, Joseph Hardin

The Atmospheric Radiation Measurement (ARM) program is on track to collect hundreds of terabytes of scanning radar data per year for use by the scientific community. Such measurements provide a wide variety of data necessary for developing climatologies. This presentation demonstrates the use of Python to analyze up to hundreds of terabytes of radar data on U.S. Department of Energy (DOE) Atmospheric Radiation Measurement (ARM)’s Stratus and Cumulus supercomputers. In particular, Corrected Moments in Antenna Coordinates 2.0 (CMAC2.0), a Py-ART based framework for processing of ARM scanning radar data, is scaled using Dask and PySpark to distribute the task of processing scanning radar data to hundreds of cores. The advantages and disadvantages of using Dask and PySpark will be shown, as well as the performance of each. Both packages are viable solutions for the processing of multi-terabyte radar datasets. A demonstration of the use of Dask’s graphical interface to profile the performance of CMAC2.0 will also be shown