Featured Article
NEJM Knowledge+: Engaging Learning Tool for Residency, Boards, and Beyond
Cutting-edge “adaptive learning” platform delivers personalized practice questions to hone knowledge
January 9, 2018
Take exhaustive editorial rigor, combine it with a sophisticated learning platform, and the result is NEJM Knowledge+. This personalized learning tool for lifelong learning, board preparation, Maintenance of Certification, and CME is now also being embraced by program directors as an excellent resource for residents.
NEJM Knowledge+ contains thousands of case-based questions, delivered in multiple formats to maximize learning and retention. Questions are tied to learning objectives that reflect certification exam blueprints. Underlying it all is a novel “adaptive learning” platform that responds to users’ performance — for example, moving quickly through topics on which the user performs well, while focusing more on questions in areas where the user needs improvement.
“It really supports something I tell my residents: people tend to study what they already know. But NEJM Knowledge+ is an opportunity to make your weaknesses into your strengths,” says David Pizzimenti, DO, FACOI, chair of Graduate Medical Education and director of the Internal Medicine Program at Magnolia Regional Health Center in Corinth, Mississippi. “And what’s different about this is that it is so user friendly.”
Dr. Pizzimenti’s program requires and provides NEJM Knowledge+ Internal Medicine Board Review for all residents. As program director, he can track trainees’ progress and learn where they stumble. Although he became board-certified in Osteopathic Medicine 12 years ago, he recently took and passed his first ABIM certification exam. He used NEJM Knowledge+ to prepare. “The questions were very analogous to the board exam,” he says. “I also tell my residents: of all the board reviews, NEJM Knowledge+ is the most clinically relevant. You can take what you learn in board review and apply it to patient care.”
He adds, “This is a great resource for trainees who don’t have the advantage of being linked to a big institution.”
Programs at large academic medical centers — including Vanderbilt University and Mayo Clinic —also find value in it and subscribe for their entire program. Other programs recommend NEJM Knowledge+ to residents who need remediation.
Matthew O’Rourke, general manager of NEJM Knowledge+, talked with LibraryHub about the origins of this product, its “adaptive learning” platform, and how it supports program directors and learners at all career stages.
Q. Why did NEJM Group choose to get into this education space?
A. We began by asking, “What could we do to make the biggest impact in lifelong learning and training?” We identified self-assessment and exam prep as spaces where we could make a substantial difference. We set out to combine high-quality educational content with technology that uses cutting-edge cognitive science to improve clinicians’ depth of knowledge and self-awareness — all with the ultimate goal of improving the delivery of care.
We knew that we could create content that was not only accurate, but designed to maximize learning. We also knew that in this digital age, you can significantly boost the impact of that learning by using the right kinds of technology, and we identified adaptive learning as the most compelling and sound new approach. From there, it was a natural to partner with Area9, a global leader in adaptive learning technology.
Q. What is unique about NEJM Knowledge+?
A. First is our adaptive learning platform, which is built on adult learning theory for optimal results. How a user answers questions – whether they are correct, how long it takes them to answer, and their confidence levels – are used to generate the delivery of future questions for that user. This makes for extremely time-efficient learning. It forces users to focus on improving areas of weakness, rather than spend time on material they already know.
The questions themselves also make NEJM Knowledge+ unique. Our database of questions is incredibly robust, and updated and expanded regularly. Questions are written carefully and reviewed by experts around the United States, then subjected to many more layers of editorial review. We offer shorter and longer versions of questions to address the same learning point in multiple ways, which helps users consolidate their knowledge. And each question offers extensive, referenced explanations. We quickly and aggressively maintain the content as medicine and medical practice evolves, using the same rigor to review and adjust 40 or so questions most months, based on new evidence or learner feedback. And we constantly add questions.
Finally, NEJM Knowledge+ has many other useful features: A “recharge” function reinforces knowledge against natural erosion that occurs with time. And importantly, NEJM Knowledge+ provides feedback on “metacognition,” which is a learner’s ability to assess his or her own proficiency. This self-awareness is key to successful learning. Lastly, timed practice exams mimic the actual board tests, with additional questions that the learner has not seen before
Q. Describe the evolution of NEJM Knowledge+ and what it currently includes.
A. We launched in 2014 with the Internal Medicine Board Review product. We have added Pediatrics Board Review and Family Medicine Board Review (also used to train Physician Assistants and for PANCE and PANRE exam preparation). Individuals purchase subscriptions for yearly terms, ranging from $599 to $749. This includes two timed practice exams, the ability to earn CME and other credits, robust reporting, and free access to cited NEJM articles. It’s all available on mobile devices, for quick, bite-size learning anytime and anywhere.
Q. What is particularly useful to program directors?
A. Program directors can track the progress of individuals and groups as they work through NEJM Knowledge+ questions and can access detailed reports that show areas of difficulty or mastery. Reports can be generated before and after a rotation, to evaluate the program.
The questions are based on hundreds of specific learning objectives. Program directors can opt to access questions that go deep into a topic – for example, arrhythmia. Or, they can generate and push out customized quizzes. 
Some program directors use these quizzes to assess residents’ general knowledge; others use them as pre- and post-tests around rotations. Still others use them in their teaching. Regardless, they free up educators to focus on hands-on training instead of spending time developing questions.
Program directors also see value for remediation. The most obvious fix previously was to pair a struggling resident with a tutor. That may be helpful, but limited. NEJM Knowledge+ addresses all of internal medicine. If the resident’s knowledge gap is broad, the user can focus widely. If the weakness is in a specific area, the program director can direct him or her to just that module.
Q. Have you made any new additions to NEJM Knowledge+ since it launched?
A. Recently, for residency programs that subscribe, we’ve linked NEJM Knowledge+ with another NEJM Group learning website, NEJM Resident 360. So a resident who struggles with a particular topic on NEJM Knowledge+ may be directed to click through to the relevant NEJM Resident 360 Rotation Prep module, curated content related to 18 common rotations and specialties in internal medicine — one more tool to boost a resident’s knowledge.
Q. How is NEJM Knowledge+ sold to and accessed by institutions? What is available for GMEs and other training programs?
A. NEJM Knowledge+ offers a flexible pricing model with comprehensive content, to meet the needs of training programs of all sizes and types, whether you require a curriculum that is self-directed or faculty-directed. For more information, email knowledgeplussales@nejm.org or request more information to learn more about our cost-effective solution for training and GME programs, certification exam prep, and remediation, plus other program benefits. Our core institutional offering provides all trainees with their own access to NEJM Knowledge+.
Q. How can librarians best help residents or anyone else who is studying — for boards or other review?
A. It’s important to be aware of the different products out there, their features, and the cognitive science and pedagogical models that they are based on. Look at what each set of tools can do for you. We are proud that NEJM Knowledge+ is able to build metacognition and confidence, while at the same time reinforcing clinical skills.
Q. Has the experience of NEJM Knowledge+ inspired any other uses for this learning platform?
A. We’re very pleased that our success has provided the opportunity to create partnerships that offer new products on the NEJM Knowledge+ platform in additional specialties. We’re collaborating with the American Association for Clinical Chemistry on a product for all aspects of laboratory medicine called AACC Learning Lab, and we’re about to launch a new offering in orthopedic surgery with the Journal of Bone and Joint Surgery called JBJS Clinical Classroom.
