Advisory Engineer, AI Model Evaluation

Summary:

We are seeking a highly motivated and skilled AI Model Evaluation Engineer to join our rapidly growing AI team. You will play a critical role in assessing the performance, robustness, and safety of large language models (LLMs), large vision models (LVMs), and large multimodal models (LMMs). This is a challenging yet rewarding opportunity to contribute to cutting-edge research and development in generative AI. You'll be working with a collaborative team to push the boundaries of what's possible with AI models and deploy them into innovative products. If you are passionate about making Smarter Technology For All, come help us realize our Hybrid AI vision!

Responsibilities:

• Design, implement, and evaluate comprehensive evaluation pipelines for large generative AI models, encompassing various metrics and methodologies.
• Evaluate the performance of publicly available models, and discuss their relative advantages and disadvantages.
• Establish and maintain benchmarks for evaluating model performance across a range of tasks and datasets.
• Conduct thorough error analysis to identify patterns in model failures and provide actionable insights for improvement.
• Design and implement methods to detect and mitigate biases in model outputs, ensuring fairness and equitable performance.
• Develop and execute robustness tests to assess model resilience against adversarial inputs, noise, and variations in real-world data.
• Assess model safety, including identifying and mitigating harmful or inappropriate outputs.
• Experiment with various evaluation techniques, metrics, and datasets to optimize model quality and reliability.
• Contribute to the development and refinement of evaluation metrics that accurately reflect model performance and desired characteristics.
• Clearly communicate evaluation results and insights to engineers, researchers, and stakeholders.
• Identify potential partnerships with third parties.
• Develop and maintain evaluation tools and infrastructure.
• Monitor and analyze model performance in production environments, identify degradation, and propose solutions.
• Stay up-to-date with the latest advancements in large language and multi-modal models, model evaluation techniques, metrics, and related technologies.
• Contribute to the development of internal tools and infrastructure for model evaluation and monitoring.

Required Qualifications:

• Bachelor's or Master's degree in Computer Science, Machine Learning, or a related field.
• 10+ years of development experience
• Strong programming skills in Python and experience with deep learning frameworks like PyTorch.
• Deep understanding of machine learning evaluation principles, including various metrics (e.g., BLEU, ROUGE, perplexity, F1-score) and methodologies.
• Proven ability to design and conduct rigorous experiments, analyze data, and draw meaningful conclusions.
• Familiarity with large language models, transformer architectures, and related concepts.
• Experience with data processing tools and techniques (e.g., Pandas, NumPy).
• Experience working with Linux systems and/or HPC cluster job scheduling (e.g., Slurm, PBS).

Preferred Qualifications:

• Ph.D. in Computer Science, Machine Learning, or a related field.
• Excellent communication, collaboration, and problem-solving skills.
• Experience with automated model evaluation frameworks and tools.
• Experience with techniques for detecting and mitigating bias in AI models.
• Experience with safety and alignment evaluation methodologies.
• Experience with A/B testing and online evaluation techniques.