Light-weight Parallel Python Tools Within the CESM Workflow

Date and Time: 
2015 April 13 @ 9:30am
FL2-1022 Large Auditorium
Sheri Mickelson and Alice Bertini

Authors: Sheri Mickelson, Kevin Paul, Alice Bertini, Dave Brown, John Dennis

As climate model output increases in both spatial and temporal resolution, it is becoming more and more difficult to post-process these results. For example, it is expected that the data required for the next Intergovernmental Panel on Climate Change (IPCC) Assessment Report (AR6) will increase by more than a factor of 10 to an expected 25 terabytes per model. Experiences from the last Coupled Model Intercomparison Project (CMIP5), which assembled the data used for the last IPCC Assessment Report (AR5), concluded that the processing, archiving, and post-run diagnostic operations required on such large model output took almost as long to complete as the model runs themselves! As a result, we have been developing new light-weight parallel Python-based tools to improve the performance of these post-processing procedures and integrating them into the CESM workflow. We have developed a parallel Python tool (pyReshaper) for converting time-slice model output to time-series format and we have developed a parallel Python tool (pyAverager) to perform fast time-averaging of climate data, an operation needed for many diagnostic computations. These tools are designed to be light-weight, easy to install, with very few dependencies, and can be easily inserted into the climate model workflow with negligible disruption. We have also developed Python control scripts that CESM uses to control the calls to these tools. In this work, we present the motivation, approach, and results of the light-weight parallel Python tools that we have developed, as well as our plans for future research and development

Speaker Description: 

Sheri Mickelson is a software engineer in the Application Scalability and Performance Group and the CESM Software Engineering Group at NCAR. Her main interest is in application performance. Currently, she is helping lead the effort to improve the performance of various CESM post-processing tools in order to get ready for CMIP6.

Alice Bertini is a software engineer at NCAR in the CESM Software Engineering Group (CSEG). She is the CESM release coordinator, database programmer and administrator for the CSEG databases, and python programmer integrating light-weight parallel python tools developed by NCAR CISL ASAP group into the CESM workflow post-processing tools.

File Mickelson-Bertini_SEA2015.pptx1.75 MB
Video recorded: 

If you use a non-flash enabled device, you may download the video here

Event Category: