Software Deployment in the Field (Technical Debt and Data-Ops)

Date and Time: 
2015 April 14 @ 11:30am
FL2-1022 Large Auditorium
Gary Granger

In the Earth Observing Laboratory at NCAR, we develop and deploy many different instruments, for platforms ranging from miniature sondes to radars and aircraft, to areas all over the world to observe all kinds of phenomena. Every field deployment is in fact a custom and very complicated configuration of systems, software, instruments, and data streams. Besides the stories about writing software in some interesting situations, field projects present some (exciting) challenges for software engineering. The concept of technical debt gives an interesting perspective on these challenges and on the costs incurred while under the pressure of a field deployment. Some ideas from dev-ops can be borrowed to help deal with these costs and to optimize data collection in general, in that the more automated we can make system deployment, data collection, and data processing in the field, the more we can mitigate the costs of manual customization, intervention, and "real-time development". This talk will present a few approaches to dev-ops and "data-ops" for field deployments, and frame some questions about how to know which approaches to use when. In particular, the talk will describe a python package used for managing streams of data files for the Integrated Sounding System, and the ways this package has evolved to test a data-ops idea of continuous data processing, similar to continuous integration and release for software.



Speaker Description: 

Gary Granger received a Bachelor of Science degree in Computer Engineering from Virginia Tech, then began working for the Atmospheric Technology Division at NCAR in 1992. Over the years he has worked in several software development areas related to field deployment and instrument development, including field operations, visualization, and wind profiling radars. Currently he works in the Software Systems Group of the Earth Observing Laboratory, developing software in C++ and Python for the Integrated Sounding System, and developing LabVIEW software for spectrometers. He also advocates good software engineering practices in EOL and supports related infrastructure, such as subversion, build frameworks, and continuous integration servers.

Video recorded: 

If you use a non-flash enabled device, you may download the video here

Event Category: