Exam Development Manager

SUMMARY

The International Society of Arboriculture (ISA) is a non-profit membership organization that serves the needs of tree care professionals, and promotes the profession of arboriculture around the world. ISA is known as the industry authority on research, education and professional development. Each role in the organization plays an important part in delivering excellent member benefits and customer service.

The Examinations Development Manager is responsible for compliance with documented polices and procedures as it pertains to exam development and maintenance. He/she/they will work with Subject Matter Experts (SMEs), stakeholders, vendors, and staff to develop and maintain examinations at standardized rigor for continued assurance of validity and reliability as well as fairness and impartiality and meeting relevant and required standards (e.g., ISO/ICE 17024, ASTM E2659).

The Examinations Development Manager is charged with meeting the needs of ISA’s current and future credential holders by managing the development and maintenance of all ISA’s credentialing examinations (including both written and performance-based examinations). This is accomplished by utilizing experience in exam development and maintenance, data analysis, psychometrics and testing, volunteer recruitment and engagement, and project management.

ESSENTIAL DUTIES AND RESPONSIBILITIES include the following. Other duties may be assigned.

· Understands ISA’s Mission and adheres to organization and staff core values.

· Contributes to the departmental strategic planning process and collaborates with the Director of Credentialing Services to provide leadership in the implementation of the ISA strategic plan and department priorities.

• Responsible for all credentialing exams including the maintenance, development, and security of the Exam Item Banks.

· Ensures the performance of job task analysis as directed by associated documented policies and procedures.

• Oversees question/item development, maintenance, and translations.

· Ensures exam and psychometric data is collected for performance and delivery of item analyses and maintenance, collusion analyses (and other exam forensic tests, if needed), and exam form creation.

· Lead and serves as main liaison to all Credentialing SME Committees (e.g., ISA Test Committee, ISA BCMA Test Committee).

• Assists the Director of Credentialing Services in supporting the Credentialing Council and maintenance of ISA’s accreditation with ANAB.

• Stays current with and utilizes best management practices for examination development and ensures that the ISA remains compliant with ISO/IEC 17024 standard and any other relevant standards applicable to ISA credentialing programs (e.g., ASTM E2659).

• Ensures a strict divider or “firewall” remains intact to protect test security and confidentiality of examination/test banks during question development and maintenance.

• Recruits, develops and maintains a highly involved contingent of volunteers who provide input and assistance in the development and maintenance of examinations.

• Develops professional relationships with appropriate industry stakeholders, fellow staff members, volunteers, vendors, and others in the credentialing industry.

• Utilizes external services to support and ensure that quality credentialing examinations are developed and maintained in a psychometrically sound, legally defensible manner.

• Maintains a high level of test security and confidentiality to ensure overall program integrity and impartiality.

• Fosters an environment of continuous improvement. Encourages staff to look for ways to improve department policies, procedures, processes and operations that result in improved efficiencies and a positive experience for ISA candidates and credential holders.

• Oversees the daily transfer of appropriate Computer Based Testing and Paper and Pencil exam data and statistics into test banking software according to departmental policies and procedures.

• Gather and analyze exam performance data in order to make recommendations for improvements and initiatives for consideration by the Director of Credentialing.

• Serves as point person for examination publication and as the Computer Based Testing vendor’s main point of contact with ISA.

• Complies with ISO/IEC 17024 to prevent fraudulent examination practices, monitor examination statistics to ensure the integrity of the examination process, and works with vendors (if applicable) to investigate security incidents, breaches, and other concerns related to exam security.

· Ensures examination questions are designed to measure competence based on, and consistent with, the scheme for each given credential.

· Ensures examination forms (including forms associated with performance-based examinations) have comparability of results for each single examination, both in content and difficulty, including validity of fail/pass decisions.

· Oversees the performance of all processes associated with examination design, development, and maintenance.

· Trains Subject Matter Experts (SMEs) on current best management practices for item development and review.

· Serves as main point of contact for translators, editors, and SMEs working on item development, review, and translation.

· Anticipate item and test development needs for all ISA examinations and build multi-year plans to meet those needs.

· Facilitate SME Committee meetings (virtual and in-person) related to item and exam development.

· Manage Committee review and approval of examination items and exam forms in a timely and cyclical fashion according to documented policies and procedures.

· Responsible for item bank management system, ensuring statistical data is available to inform and report on item and exam development activities and needs.

· Ensure exam development process documentation is kept current and accurate.

· Work with Publication and Documentation Lead and other key staff to ensure that relevant policies, procedures and communication is updated and accurate with regard to examinations (e.g., credentialing schemes, exam outlines, cut scores)

Supervisory Responsibilities

Yes; 1 direct report.

Management responsibilities include: interviewing, hiring, and training employees; planning, assigning, and assisting with work; evaluating performance; resolving conflict and promoting positive employee relations in a collaborative work environment.

Clearly defines expectations of staff to meet departmental goals. Coaches and counsels as needed and holds staff accountable to deliver results. Assists with removal of roadblocks and prioritization.

QUALIFICATIONS To perform this job successfully, an individual must be able to perform each essential duty satisfactorily. The requirements listed below are representative of the knowledge, skill, and/or ability required. Reasonable accommodations may be made to enable individuals with disabilities to perform the essential functions.

Education and/or Experience

· 4-year degree from an accredited college or university required. Education, Instructional Design, Organizational Psychology preferred;

· Previous experience in examination development required.

· Previous supervisory or volunteer management experience preferred.

· Prior project management experience exhibiting superior analytical and organizational skills with high levels of attention to detail.

· Involvement and/or experience with professional, high-stakes, accredited credentialing program management is preferred.

Language Skills

· Fluency in English, both verbal and written. Fluency in additional languages is a plus (especially for one of the languages in which our examinations are delivered).

· Excellent written and oral communication skills.

· Ability to read, write, analyze, and interpret complex documents.

· Ability to respond diplomatically to sensitive inquiries or complaints.

· Ability to write articles for various publications and audiences.

· Ability to make effective and persuasive speeches and presentations on controversial or complex topics to top management, public groups, volunteers, and/or boards of directors.

Decision Making/Reasoning Ability

· Ability to define problems, collect data, establish facts, and draw valid conclusions.

· Ability to interpret an extensive variety of technical instructions in mathematical or diagram form and deal with several abstract and concrete variables.

· Ability to demonstrated good judgment in knowing which issues require escalation to director, SMEs, vendors, and other relevant stakeholders.

Additional Competencies

· Ability to assist in the creation of budgets and interpret and analyze financial data.

· Ability to understand and interpret psychometric data associated with managing test bank questions, which requires higher level mathematical operations such as frequency distribution, determination of test reliability and validity, analysis of variance, correlation techniques, sampling theory, and factor analysis.

· Ability to communicate clearly and consistently with key staff members, vendors, and volunteer SMEs concerning the performance of essential duties and responsibilities for the position.

· Ability to inspire and motivate staff and volunteers.

· Ability to coordinate efforts and workflow with key staff members from all levels and departments of the organization.

· Ability to maintain positive attitude under stressful conditions.

Physical Demands

The physical demands described here are representative of those that must be met by an employee to successfully perform the essential functions of this job. Reasonable accommodations may be made to enable individuals with disabilities to perform the essential functions.

• Must be able to remain in a stationary position 80% of the time.

• Moves about inside the office to access file cabinets, office machinery, etc.

• Regularly operates a computer and other office productivity machinery such as calculator, copy machine, and computer printer.

• Frequently communicates with internal and external customers.

• This position requires occasional lifting, under 25 lbs.

Work Environment

The work environment characteristics described here are representative of those an employee encounters while performing the essential functions of this job. Reasonable accommodations may be made to enable individuals with disabilities to perform the essential functions.

• Working conditions are normal for an office environment. Work may require weekend and/or evening work.

• The noise level in the work environment is usually quiet to moderate.

• Domestic and occasional international travel may be required.

Technology and Equipment

· Latest Microsoft Windows/Office Applications; proficient in Word, PowerPoint, Outlook, and Teams.

· Advanced Microsoft Excel skills is a plus.

· Experience with Pearson VUE applications is a plus.

· Experience with examination question banks and other content management systems is a plus.

· Experience using AI for examination development and translation is a plus.

· Database applications, and the Internet.

#J-18808-Ljbffr