Director, Psychometrics

We are a medical specialty certifying board serving anesthesiologists. Since 1938, we have been administering certification exams and today we take an innovative approach to certification and continuous learning. We foster practice standards that instill confidence and trust that board-certified anesthesiologists have the knowledge and skills to provide high-quality patient care. We are dedicated to elevating expertise in an evolving profession. Our mission is to advance the highest standards of the practice of anesthesiology. We work together with physician anesthesiologists to ensure they provide the best care possible for every patient, every day.

Position Description

The Director, Psychometrics and Research is a key member of the American Board of Anesthesiology (ABA)’s Assessment Services team. This position provides leadership and management of the Psychometrics and Research staff and is responsible for the overall planning, directing, and overseeing of the psychometric and research activities for the ABA, including but not limited to test development, exam security, development of score interpretation materials and testing program publications, recommendations and advice on measurement issues related to ABA examinations and programs, and development of ABA programs. This position works closely with assessment development staff to ensure that all assessments are built with high reliability, validity, and defensibility. This position is also responsible for overseeing all aspects of data management, statistical inquiries, and report generation, including but not limited to resident tracking and program performance. This position requires working easily and well with all other departments across the ABA as well as with external vendors, including but not limited to our test delivery vendor, item banking vendor and others. This position represents the ABA externally at various events such as testing conferences or the American Board of Medical Specialties (ABMS) and its member board meetings.

Education

• Ph.D. in psychometrics, educational measurement, educational psychology or other closely related quantitative field.

Skills

• Demonstrated commitment to service and continually exhibits a customer service culture.
• Demonstrated dependability for own actions and their consequences, and for the performance of the team functions and duties.
• Demonstrated flexibility in rapidly changing conditions.
• Independent self-starter with strong interpersonal and communication skills, both orally and in writing.
• Ability to work concurrently on multiple tasks and projects.
• Demonstrated willingness to keep technical skills and knowledge current.
• Ability to think logically and analytically in a problem-solving environment.
• Ability to work independently as well as part of a team.
• Proven ability to manage relationships with vendors and consultants.
• Excellent interpersonal and leadership skills, with a strong ability to effectively interface and collaborate with varied constituent groups, including staff, board members, volunteers, and external organizations.

Experience

• At least five years of progressive professional leadership experience.
• Three or more years of direct management of staff.
• Demonstrated expertise in psychometric methods, statistical procedures, and their applications, particularly item response theory (IRT), multi-facet Rasch modeling, equating, and scaling.
• Knowledge of IRT software, equating software, and other key psychometric packages is highly desired.
• Working knowledge of R, SPSS, or SAS.
• Proficiency with Microsoft Office products, particularly Word, Excel, and PowerPoint.
• Knowledge of or experience with ABMS a plus.
• Knowledge of or experience with the Accreditation Council for Graduate Medical Education (ACGME) a plus.

Specific Responsibilities

Psychometrics

• Oversees all scoring, reporting, documentation and psychometric activities for ABA examinations and assessments.
• Oversees the allocation of staff time and other resources needed to perform scoring, reporting, and other psychometric services for the examinations of external organizations to which the ABA provides psychometric services and support.
• Supports and advises on the development, planning and implementation of reliable, valid, and legally defensible in-training, initial certification, continuing certification and subspecialty examinations and assessments.
• Provides expertise and leadership in the development of psychometric theory and activities, as well as the management and monitoring of project activities and workflows.
• Oversees standard-setting studies.
• Supports item writing workshops.

Research and Committee Oversight

• Serves as primary staff liaison to the ABA’s Research Committee, the Foundation for Anesthesia Education and Research and other external researchers on behalf of the ABA’s Research Committee
• Advises and supports the work of the Research Committee by collaborating with the Research Committee Chair and members to define and implement the ABA’s research agenda.
• Oversees the allocation of staff time and other resources to support the Committee’s research initiatives.
• Provides regular communication with the Research Committee Chair and members on the progress and status of research initiatives and facilitates meetings and conference calls.
• Directs the preparation of meeting reports for Committees, the Board, and the Executive Director, Administrative Affairs.
• Supports various research initiatives as directed by the Board and its Committees, including literature reviews, designing, and carrying out data collection procedures, conducting analysis and preparing reports.

Human Resources

• Manages, develops, and recruits, as needed, qualified psychometric and research staff to support the ABA’s certification processes and the execution of its research agenda.
• Establishes and monitors staff performance and development goals, coordinates efforts, sets objectives, establishes priorities, and conducts bi-annual performance reviews.
• Mentors and develops staff using a supportive and collaborative approach on a consistent basis.
• Promotes a culture of high performance and continuous improvement that values learning and a commitment to quality.

Financial Management

• Develops and maintains the annual budget for Psychometric and Research activities.
• Monitors activities and costs of the Psychometrics and Research unit.
• Assists the CAO to develop and maintain the annual budget for the Assessment Services department consistent with the Board’s business plans and established financial guidelines.
• Assists the CAO to monitor budget to actual performance on a quarterly basis.

Management and Leadership

• Maintains broad knowledge of all aspects of examination development, administration, and psychometrics.
• Assures quality assurance practices and measures are implemented.
• Assures that Psychometrics and Research staff understand ABA policies and regulations, and exercise appropriate judgment in the performance of their duties, functions, and responsibilities.
• Provides leadership and direction to the Psychometrics and Research staff.
• Advises the CAO, Committees and Board of Directors regarding IRT models, calibration, and scaling for all examinations, and implementing changes to existing programs and policies.

Special Requirements

• Ability to travel on occasion up to 10% with overnight stays.
• Ability to lift up to 30 pounds maximum.
• Ability to work on feet consistently for extended periods of time.
• Ability to sit for extended periods of time.
• Must be able to work onsite in Raleigh, NC at least two days per week.