Education Evaluations: Why Your Answers Matter
As we approach the 2013 Annual Meeting & OTO EXPOSM in Vancouver, it is important to look at the value of education activity evaluations and the role they play in the overall education program of the Foundation. At the end of each Annual Meeting miniseminar and instruction course, attendees line up at the onsite computer stations to answer a series of questions pertaining to the session they just attended. This is also true for everyone who participates in our more than 150 online courses and lectures, as well as Patient Management Perspectives in Otolaryngology and Home Study Course subscribers. The responses to these questions represent an integral part of what the education leadership considers as they plan education activities that will benefit both our members and their patients. First and foremost, the evaluation process is a requirement for awarding AMA PRA Category 1 Credit™ for the education activity. One of the AMA PRA core requirements for certifying activities specifically states “the activity must evaluate the effectiveness in achieving its educational purpose and/or objectives.” This effectiveness is measured in three ways: knowledge, competence, and performance. A change in knowledge answers the question, “Did I learn new information about the topic at hand?” A change in competence answers, “Did I learn how to apply the new information I learned?” A change in performance answers, “Did I master the new skills I acquired through practice?” With the current emphasis on improving patient outcomes, it is critical that physicians are educated on specific skills that will lead to improved practice and quality of care. All education activity objectives are written as outcome objectives that assess whether the participant has learned a new concept, learned a new skill, or perfected a new technique. The Foundation’s focus is always on measuring not just knowledge, but, more importantly, competence and performance. Other questions found on a typical education activity evaluation include an assessment of the content and format of the activity, the resource materials accompanying the activity, the quality and expertise of the faculty, and if there was any appearance of commercial bias associated with the activity. Annually, staff compiles this evaluation data from all the education activities conducted in a given year in order to get a sense of the overall perceptions of activity participants regarding the quality of our offerings and faculties and whether they are influencing patient care outcomes. The summaries from all this data are used by the eight education committees, who serve as content experts and activity planners, to determine future education resources that will be of value to our members. Recently the Foundation acquired a new education data management system to assist in the reporting and assessment process within education. This database helps us to summarize the evaluation responses from participants in all of our education activities. We are currently inputting evaluation data for all activities conducted during the last three years. This information will be valuable to determine which of our products are succeeding at improving the knowledge, competence, and performance of our members. This new tool will help us better access and utilize this data during future education planning. Needs Assessment Evaluations are just a part of the process of determining the education needs of our members. As mentioned throughout the year, the Foundation has been involved in a membership-wide education needs assessment initiative. One of the components of this initiative has been to perform a gap analysis within the discipline of otolaryngology-head and neck surgery. A gap analysis is a formal study of what the profession is doing currently and where it needs to be in the future. In relation to the practice of otolaryngology-head and neck surgery, this analysis will provide evidence where physician knowledge, competence, and practice may be lacking or best practices are not being utilized. The results of this analysis will help the Foundation determine where best to allocate its time and resources to address these gaps through education and knowledge products. Another component of this initiative was the education needs assessment survey sent to members in August. This survey was another important tool to help paint the overall picture of where the Foundation currently stands in providing valuable education to our members and where it should be going in improving these efforts. While individual education activity evaluations provide insight into one specific education activity, the member-wide survey targets the needs of the entire membership. Results of this initiative will be shared and discussed in a future Bulletin. Combined, all of this evaluation data is critical to help the Foundation create a quality and effective education portfolio that will provide the best service to our members. So the next time you hear “be sure to complete your evaluation,” do so knowing your input is vital to the success of the education program at the Academy.
As we approach the 2013 Annual Meeting & OTO EXPOSM in Vancouver, it is important to look at the value of education activity evaluations and the role they play in the overall education program of the Foundation. At the end of each Annual Meeting miniseminar and instruction course, attendees line up at the onsite computer stations to answer a series of questions pertaining to the session they just attended. This is also true for everyone who participates in our more than 150 online courses and lectures, as well as Patient Management Perspectives in Otolaryngology and Home Study Course subscribers. The responses to these questions represent an integral part of what the education leadership considers as they plan education activities that will benefit both our members and their patients.
First and foremost, the evaluation process is a requirement for awarding AMA PRA Category 1 Credit™ for the education activity. One of the AMA PRA core requirements for certifying activities specifically states “the activity must evaluate the effectiveness in achieving its educational purpose and/or objectives.” This effectiveness is measured in three ways: knowledge, competence, and performance. A change in knowledge answers the question, “Did I learn new information about the topic at hand?” A change in competence answers, “Did I learn how to apply the new information I learned?” A change in performance answers, “Did I master the new skills I acquired through practice?”
With the current emphasis on improving patient outcomes, it is critical that physicians are educated on specific skills that will lead to improved practice and quality of care. All education activity objectives are written as outcome objectives that assess whether the participant has learned a new concept, learned a new skill, or perfected a new technique. The Foundation’s focus is always on measuring not just knowledge, but, more importantly, competence and performance. Other questions found on a typical education activity evaluation include an assessment of the content and format of the activity, the resource materials accompanying the activity, the quality and expertise of the faculty, and if there was any appearance of commercial bias associated with the activity. Annually, staff compiles this evaluation data from all the education activities conducted in a given year in order to get a sense of the overall perceptions of activity participants regarding the quality of our offerings and faculties and whether they are influencing patient care outcomes. The summaries from all this data are used by the eight education committees, who serve as content experts and activity planners, to determine future education resources that will be of value to our members.
Recently the Foundation acquired a new education data management system to assist in the reporting and assessment process within education. This database helps us to summarize the evaluation responses from participants in all of our education activities. We are currently inputting evaluation data for all activities conducted during the last three years. This information will be valuable to determine which of our products are succeeding at improving the knowledge, competence, and performance of our members. This new tool will help us better access and utilize this data during future education planning.
Needs Assessment
Evaluations are just a part of the process of determining the education needs of our members. As mentioned throughout the year, the Foundation has been involved in a membership-wide education needs assessment initiative. One of the components of this initiative has been to perform a gap analysis within the discipline of otolaryngology-head and neck surgery. A gap analysis is a formal study of what the profession is doing currently and where it needs to be in the future. In relation to the practice of otolaryngology-head and neck surgery, this analysis will provide evidence where physician knowledge, competence, and practice may be lacking or best practices are not being utilized. The results of this analysis will help the Foundation determine where best to allocate its time and resources to address these gaps through education and knowledge products.
Another component of this initiative was the education needs assessment survey sent to members in August. This survey was another important tool to help paint the overall picture of where the Foundation currently stands in providing valuable education to our members and where it should be going in improving these efforts. While individual education activity evaluations provide insight into one specific education activity, the member-wide survey targets the needs of the entire membership. Results of this initiative will be shared and discussed in a future Bulletin.
Combined, all of this evaluation data is critical to help the Foundation create a quality and effective education portfolio that will provide the best service to our members. So the next time you hear “be sure to complete your evaluation,” do so knowing your input is vital to the success of the education program at the Academy.