Sr. High-Performance Computing Engineer - 141797

The Sr. High-Performance Computing (HPC) Engineer will be responsible for the technical operation of AWARE, CW3E's high-performance computing system. Working with the CW3E Research Cyberinfrastructure Lead, this person will take the lead on system administration, maintenance, upgrades, deployments, and troubleshooting. AWARE supports both time-sensitive "near realtime" operational forecasting and research, so system availability and careful planning of changes are important parts of the job.

The work covers the full HPC environment, including compute and GPU nodes, Ethernet and InfiniBand networks, parallel storage, cluster management, job scheduling, authentication, monitoring, and data-transfer services. The engineer will install and integrate new hardware and software, maintain system configurations and automation, monitor performance and capacity, and respond to production problems. They will also coordinate with campus teams and vendors when outside support is needed and maintain documentation for system changes, procedures, and recovery.

A significant part of the job involves working directly with researchers and operational workflow managers. The engineer will help troubleshoot jobs, MPI applications, software environments, containers, memory and I/O problems, and CPU/GPU performance. This requires being able to determine whether an issue is coming from the application, software environment, scheduler, network, storage system, or hardware and then work with the appropriate people to resolve it.

The engineer will help the CW3E Research Cyberinfrastructure Lead plan future additions and upgrades to AWARE. This may include gathering requirements, comparing technical options, developing benchmarks, working through procurements, and planning deployments with as little disruption as possible.

As a senior member of the group, the engineer will also be expected to share knowledge with other staff. This includes mentoring team members, explaining HPC concepts, reviewing technical work, and improving the documentation needed to operate and support the system.

The position requires applicable experience supporting HPC, research computing, or large-scale Linux systems. Experience with technologies used at CW3E, such as Slurm, InfiniBand, BeeGFS or Lustre, ZFS, MPI, GPU computing, configuration-management platforms (BCM, xcat), and system-monitoring & metric tools (telegraf, influxdb, nagios), is desired but not required. Experience with comparable systems and the ability to learn and troubleshoot unfamiliar technologies will also be considered.

• Advanced knowledge of systems integration deploying complex systems integration solutions.

• Advanced skills associated with system integration design, modification, implementation and deployment in a complex environment.

• Demonstrated ability to develop complex conversion and system implementation plans.

• Demonstrated ability to understand functional needs of the campus / medical center and / or UC system and how systems can support those needs.

• Demonstrated ability to interface with senior level technical and non-technical management on a regular basis.

• Demonstrated ability to create software specifications.

• Experience with identification and use of code libraries and open-source forums.

• Experience with planning for deployment and creation of feedback mechanisms.

• Experience developing and implementing complex project plans.

• Experience researching and evaluating new technology and solutions for complex environments.

• Ability to impart knowledge to senior level technical and non-technical audiences and to influence outcomes to meet objectives.

• Demonstrated ability to identify software and hardware interdependences in a complex multi-platform environment.

• Demonstrated ability to develop solutions for complex business and systems / custom / ad-hoc solutions.

• Advanced knowledge and experience with monitoring tools, performance, and systems optimization, including intermediate knowledge of security for data, data transfers and systems.

• Experience with complex multi-platform troubleshooting in a multi-platform environment.

• Demonstrated ability to plan and execute the technical aspects of deploying, upgrading, and decommissioning storage systems.

• Advanced knowledge and experience with monitoring tools, performance, and systems optimization, including intermediate knowledge of security for data, data transfers and systems.

• Experience with complex multi-platform troubleshooting in a multi-platform environment.

• Demonstrated knowledge of TCP/IP and Infiniband networking, including sockets, VLANs, and firewalls.

• Proven experience designing, deploying, and operating large-scale (petabyte-class) high-performance parallel and distributed file systems (e.g., Lustre, Ceph, BeeGFS, GPFS), as well as enterprise and local file systems (e.g., NFS, ZFS, ext4, XFS) in Linux-based environments, including troubleshooting and performance tuning.

• Expert knowledge and experience with large data storage arrays (more than100TB), and methods to administer, monitor and upgrade.

• Proven experience administering and supporting large-scale HPC clusters or other distributed POSIX (Linux) systems, including advanced knowledge of Linux system administration, primarily Red Hat and its derivatives (e.g., Rocky Linux).

• In depth experience in cluster management tasks including deployment, configuration, and troubleshooting of compute nodes, management nodes, network switches, and file servers.

• Demonstrated experience with scripting and automation using languages such as Bash and Python; use of configuration management tools (e.g., Ansible, CFEngine); and version control systems (e.g., Git) to manage and maintain system configurations and infrastructure.

• Advanced knowledge of HPC middleware stack including cluster management tools, job schedulers and resources managers. Examples include: Slurm, PBS, HPCM, and Bright Cluster Manager.

• Demonstrated experience evaluating new storage system technologies.

• Job offer is contingent upon satisfactory clearance based on Background Check results.

• Occasional evenings and weekends may be required.

• Possession of, or ability to obtain, a favorable NACI adjudication is preferred but not required.