The ecProf profiler is capable for performance analysis (including PAPI counters), I/O and memory tracing and has been long time coming from ECMWF. This profiler can analyze any Linux dynamic executables (including Linux commands) without any need to add explicit calls to the application program, or even to have the application source code available. It knows about MPI (if present), and OpenMP (if present) and can also trace presence of any auxiliary pthreads. Also the calling tree based profiles and possibility to embed source code and assembly in the middle of the profile are some advanced features available in ecProf.
It has currently been ported to work with GNU, Intel, Cray and PGI compilers. The IBM XL compiler port is underway on Summit. At present it only traces programs that run on CPUs.
The ecProf gives access to cache and TLB misses as well as floating point and instruction counters for every task and thread through help of PAPI. It also figures out how much time is spent in the libraries (e.g. MPI, LAPACK). Also OpenMP load imbalance can be figured out from the time spent in the OpenMP barriers.