Practicalities of Running Weather Models on Amazon Web Services

Date and Time: 
Wednesday April 10th 2019
CG Auditorium
David Hahn
Creating plans to use modern cloud computing services can raise many important questions that are difficult to answer, such as their performance, cost, and which cloud resources are appropriate. The Joint Center for Satellite Data Assimilation (JCSDA) has done recent work to assess Amazon Web Services (AWS) for the FV3 model. JCSDA used a global C-768 configuration to benchmark the performance and cost of running FV3; and developed a process to bring up the 1,728-core Linux cluster within minutes and start running the model. The AWS cluster was provisioned with very recently available hardware, including the Elastic Fabric Adapter (EFA, preview in early 2019); C5n instance types (Nov 2018); and FSx for Lustre (Nov 2018). Performance comparisons will be made among the various AWS hardware options, and between AWS and super computers. Finally, the methods used to provision a full cluster and be ready to run within minutes will be reviewed. These methods include using CloudFormation, custom Amazon Machine Images (AMIs), and scripting with Python and the Boto3 API.
Speaker Description: 
David has been working with Amazon Web Services for the last three years for deploying applications ranging from microservices to HPC.
Video recorded: 

Slides available here.

Event Category: