Job description

Senior AI/Machine Learning Specialist Report to: Chief Information Technology Officer Our Organization We are a medical specialty certifying board serving anesthesiologists. Since 1938, we have been administering certification exams and today we take an innovative approach to certification and continuous learning. We foster practice standards that instill confidence and trust that board-certified anesthesiologists have the knowledge and skills to provide high-quality patient care. We are dedicated to elevating expertise in an evolving profession. Our mission is to advance the highest standards of the practice of anesthesiology. We work together with physician anesthesiologists to ensure they provide the best care possible for every patient, every day. Position Description The ABA is seeking a experienced Senior AI/Machine Learning Specialist to lead the development and operationalization of machine learning solutions across committees and internal initiatives. This position requires deep technical expertise and strategic advisory capabilities to guide the effective application of advanced ML techniques, ensuring alignment with organizational goals and delivering measurable outcomes. Education • Bachelor’s or Master’s degree in computer science, Artificial Intelligence, Statistics, Mathematics, or a related field. Skills • Proactive self-starter with excellent interpersonal, communication, and customer service skills. • Strong analytical and problem-solving abilities in a dynamic environment. • Ability to manage multiple tasks and projects simultaneously. • Collaborative team player with a focus on achieving common goals. • Meticulous attention to detail. • Quick learner with a passion for staying current with emerging technologies and industry trends. Experience Required Experience • Deep expertise in RAG systems, LLMs, and AI infrastructure • Experience evaluating AI models for different tasks • Strong ability to leverage cloud infrastructure • Excellent collaboration skills and ability to work with non-technical stakeholders • Strong understanding of software engineering best practices and system design. • 5+ years of experience in machine learning engineering, with significant work on production systems • Strong expertise in supervised, unsupervised, reinforcement learning techniques, and statistical modeling. • Proficiency in Python and major ML libraries (e.g., TensorFlow, PyTorch, scikit-learn). • Hands-on experience with cloud-based ML platforms (AWS SageMaker, Azure ML, or Google AI Platform). Valuable Additions • Ability to translate complex business problems into machine learning solutions and communicate technical content to non-technical stakeholders. • Experience working within agile, cross-functional teams. • Experience with educational technology. • Knowledge of AI safety and responsible AI development • Professional certifications such as AWS Certified Machine Learning or TensorFlow Developer Certificate are preferred. Specific Responsibilities • Lead creation of a 3-year AI roadmap aligned with ABA’s strategic plan. Chair an internal AI Governance Council to set policies for model lifecycle, data privacy (HIPAA/FERPA), and bias auditing. • Monitor federal/state regulations and ABMS guidelines to ensure compliance. • Build and deploy predictive and generative models (e.g., adaptive scoring, automated item classification, content gap analysis, chat-based candidate support). Own the end-to-end ML pipeline: data ingestion, feature engineering, model training/validation, MLOps, monitoring. • Partner with Psychometrics to validate model performance against gold-standard measurement theory. Work with Exam Development to pilot large-language model (LLM) workflows for draft item generation with subject-matter expert review. • Support Finance & Operations in forecasting and resource optimization. • Mentor engineers and analysts; establish coding standards and reproducible research templates. Deliver periodic “AI Literacy” workshops for staff, board members, and external diplomates. • Evaluate and negotiate with cloud/AI vendors, ensuring security, cost control, and integration with existing Microsoft/AWS stack. • Represent the ABA in ABMS taskforces and national conferences; publish white papers on ethical AI in physician certification. Special Requirements • Ability to lift up to 10 pounds maximum. • Ability to sit for 6+ hours per day. • Must be able to work onsite in Raleigh, NC, at least two days/week.

Requirements