KDID Portal

IntraHealth International

International NGO (INGO)

United States

Health

Health workers are vital to achieving the Millennium Development Goals. Yet there is a global shortage of 4.3 million health workers and the World Health Organization has named 57 countries facing a health worker crisis. Addressing this crisis is complex and knowledge intensive. The HRH Global Resource Center (GRC) is a digital library committed to reducing access barriers in developing countries to the best human resources for health (HRH) information currently available. The GRC aggregates free resources, provides search and browse tools, offers personalized librarian support and hosting services, and produces original HRH news and content. IntraHealth International developed and launched the GRC in 2006 under the USAID-funded global HRH project, the Capacity Project. The GRC continues to be a key global resource as a part of the follow-on project, CapacityPlus. Before the GRC was launched, HRH resources were scattered across multiple websites, and there was no single place to access the growing body of tools and resources. The GRC was designed in response to this gap to create and maintain a global exchange of HRH evidence, tools and innovation; foster learning, knowledge sharing and dialogue; and accelerate the implementation of evidence-based HRH solutions. Developed using open source software, the GRC is a user-driven website responsive to the changing needs of its users and the HRH field. The collection, taxonomy and content have evolved over time in response to usage data, user feedback and expert review.

We have used a variety of methods since the GRC was launched to assess the site and measure its success, combining ongoing monitoring with periodic assessment. For ongoing monitoring, we have tracked many indicators including user visits, user requests, number of newsletter subscriptions, number of websites linking to the GRC, number of references to the GRC in other global health resources, number of new resources added in designated priority topics, number of tools and resources created by the GRC team, and number of countries accessing the GRC. The timeframe for ongoing monitoring ranges from monthly (e.g. average number of monthly visits) to annually. Some of these indicators are collected as part of the wider project monitoring plan. We primarily look at trends over time to gauge success. We have also updated these metrics over time. In addition to ongoing monitoring, we assess the GRC periodically. Examples include: • Usage analysis: Analysis of search strings and reads per topic used to inform taxonomy and collection development changes (now conducted annually) • User surveys: Web-based surveys to collect feedback on the collection and features • Expert review: Periodic review by HRH subject experts to ensure quality and breadth of resources • Case studies: Interviews with people who have used the GRC as a key resource in accomplishing their HRH work • Statistically weighted usage mapping: A compelling visualization of GRC usage, showing the number of GRC hits per 100 internet users by country • Targeted feedback requests: Qualitative feedback from users requesting personalized librarian service • Comparative review: Identifying and comparing the GRC to other online HRH collections Our measurements are focused on increasing usage, improving usability as well as ensuring the quality and scope of the collection. The above metrics are compiled and assessed by the implementing team in consultation with project Monitoring and Evaluation staff. The GRC was also included in a third party evaluation of the USAID-funded Capacity Project, where it is noted as “by far, the best resource center for HRH in the world.”

The main motivation for assessing the GRC is the commitment to providing a user-driven site. Without continually monitoring usage and feedback, this would be impossible. We have also done periodic assessments to inform upgrades and improvements to the site. Some indicators are tracked to meet donor requirements.

Usage analysis provided useful and surprising information, revealing how users explore the site and topics. This analysis drove significant improvements. For example, we added Skilled Attendance, Contracting and Job Satisfaction as new subjects based on user demand. Regularly monitoring a set of defined metrics has been successful in demonstrating usage trends. For example, in one quarter in 2009, the GRC logged 29,725 unique visitors. In the same period in 2010, this number increased to 47,715. Periodically assessing these metrics is also valuable. For example, at one time we tracked the number of countries accessing the GRC. Once geographic usage became ubiquitous (e.g. more than 175 countries in 2010), it was no longer a useful measure. As new HRH topics emerge, we must adjust the metrics to ensure the GRC meets changing information needs. Usage statistics, however, only become meaningful when combined with qualitative user feedback. Targeting users who have asked for personalized support has led to some of our best user feedback response rates and quality responses. When we receive a user request, we send feedback questions along with our response and follow up. In response to a user request in 2006, we produced a resource CD to support an HRH strategic planning session in Southern Sudan. Our simple feedback request led to an interview and case study with the General of HR Development and Planning at the Ministry of Health. Online user surveys have not worked particularly well in terms of both the quantity and quality of responses.

We have learned to build time and resources for monitoring and evaluation into our activity and to consult with Monitoring and Evaluation experts from the beginning. It is also important to assess the site and user needs more frequently, even if informally, to avoid missing opportunities to make improvements.

For a long term initiative, we would remind others to expect the assessment process to be iterative. Be open to changing your assessment plans as you learn what works well, what doesn’t and as new needs emerge. Also make sure that you build into your initiative the opportunity and resources to make changes based on the feedback you receive.

How do you assess the impact of KM initiatives on broader outcomes, such as HRH policy or service delivery outcomes? For example, we can show that the GRC and the resources available through the GRC are being accessed (via usage statistics) and being used (via qualitative feedback), and we can describe how they are being used in limited cases. However, it is difficult to measure impact. It is also difficult to assess how an open access model of sharing HRH knowledge compares to the more traditional pay for use models still in place by many online journals and databases.