Introduction

Search methods for systematic reviews and other evidence syntheses should be transparent, reproducible and comprehensive. The PRISMA-S checklist requires full search strategies are reported for each bibliographic database, however, these do not provide any indication of the efficiency of the search strategy, or the usefulness of individual search lines for identifying studies for inclusion.

It is becoming increasingly important to evaluate search strategies for evidence syntheses, particularly for those that require regular updates or for ‘living’ reviews, to ensure search strategies are effective and efficient, and to minimise future screening load.
Aim

To identify a simple method for search strategy evaluation and consider how Information Specialists (ISs) can report and share search strategy evaluations.

Method

Searches were conducted to identify systematic reviews (SRs), randomised controlled trials (RCTs) and economic evaluations for an evidence gap map on peer support interventions. Search strategies included a combination of free-text and controlled vocabulary terms.

A search summary table was created to highlight where included studies were found. This indicated 27 of the 32 included SRs, and 50 of the 61 included RCTs were retrieved by the original Ovid MEDLINE searches.

Test sets were created for included references using PubMed identifiers in Ovid MEDLINE. These were used to evaluate each line of the SR and RCT topic search strategies, in order to identify the lines of the search strategy that retrieved included or unique references, and the simplest combination of search terms that would retrieve all included references.
Results

Initial findings indicate that a simple strategy (using only two search lines) would identify all included SRs, whereas a broader range of terminology (12 search lines) is needed to capture all included RCTs. 18 search lines in the peer support strategy for SRs retrieved at least one included SR. 26 search lines in the RCT search strategy picked up at least one included RCT.

We will present further findings from our evaluation of search strategies conducted for an evidence map.
Conclusion

The conduct and reporting of a search strategy evaluation, in addition to a search summary table, may improve search efficiency and minimise screening load for reviews that require frequent updates. These can be time-consuming tasks, however, search strategy evaluation provides opportunities for ISs to reflect on current practice and gather evidence about the value of different search approaches. Reporting details of search strategy evaluation ensures transparency and reproducibility of search methods, and may also guide ISs working on similar topics to make informed decisions about selection of search terms. The IS community could work together to develop simple and effective methods to evaluate search strategies and consider how this knowledge can best be shared.

The authors intend to conduct further research comparing the performance of ‘evaluated’ search strategies with our original search strategies. We will assess the efficiency and number needed to screen for both strategies to inform updates of the living evidence map of peer support interventions.

ABSTRACT

Background

Identifying and removing reference duplicates when conducting systematic reviews (SRs) remains a major, time-consuming issue for authors who manually check for duplicates using built-in features in citation managers. To address issues related to manual deduplication, we developed an automated, efficient, and rapid artificial intelligence (AI)-based algorithm named Deduklick. Deduklick combines natural language processing (NLP) algorithms with a set of rules created by expert information specialists.

Methods

Deduklick’s deduplication uses a multistep algorithm of data normalization, calculated a similarity score, and identified unique and duplicate references based on metadata fields, such as title, authors, journal, DOI, year, issue, volume, and pages. We measured and compared Deduklick’s capacity to accurately detect duplicates with the information specialists’ standard, manual duplicate removal process using EndNote on eight heterogeneous datasets. Using a sensitivity analysis, the efficiency and noise of both methods were manually cross-compared.

Findings

Following deduplication and comparing performance measurements, Deduklick achieved an average recall of 99·51%, an average precision of 100·00%, and average F1 score of 99·75%. In contrast, the manual deduplication process achieved an average recall of 88·65%, an average precision of 99·95%, and an average F1 Score of 91·98%. Deduklick achieved equal to higher expert-level performance on duplicate removal. It also preserved a high metadata quality, and drastically diminished the time spent on analysis.

Interpretation

Deduklick represents an efficient, transparent, ergonomic, and time-saving solution for searching and removing duplicates in SRs. Deduklick could therefore simplify SRs production and represent important advantages for scientists, including saving time, increasing accuracy, reducing costs, and contributing to quality SRs.

Human Touch (Recommended)

Automated, Reliable and Explainable Deduplication of trials and publications metadata, part of systematic review process.

Introduction

Literature searches to answer clinical and management questions can be time consuming. Our Library and Knowledge Service (LKS) provides 300-400 searches annually: a substantial time investment. Quality Improvement (QI) methodology is a recognised way to address inefficiencies and improve processes. Our organisation recently introduced revised QI practices. The LKS is one team chosen to pilot the new approach. The LKS literature searchers decided to use QI methodology to investigate improvements to the search process.

Aim

To investigate whether the time taken to provide a literature search could be reduced without any impact on quality.

Method

At baseline, we timed the different stages of searches (e.g. correspondence with colleagues/users; searching; peer review; sifting; summarising). As recommended by a QI specialist, we aimed to record data for about 20 searches per searcher per Plan/Do/Study/Act cycle: a large amount of data for a QI project. However, we were advised this was needed given the large variation in complexity of search questions and the time taken to complete them. We have completed 3 Cycles: (1) Introducing reference management software (RMS) and increasing the number of interfaces searched; (2) Returning to a single interface while using RMS; (3) Sharing experiences of RMS to streamline processes (single interface). All other activities which contribute to search quality (e.g. number of databases searched) remained unchanged.

Results

Cycle 1 (presented elsewhere) demonstrated that using RMS offset the increased time taken to search multiple interfaces. Time spent searching databases increased, whilst time was saved on formatting and sifting. We anticipated time savings in Cycle 2 (return to single interface; continued use of RMS). We were surprised this was not the case. In Cycle 2, across all searchers, there was an average time saving of only 2 minutes per search compared to baseline. On investigating, we found considerably differing results for each searcher. This varied from an average increase per search of 139 minutes to an average time saving per search of 62 minutes. Reasons for variation, including human factors, search complexity, and the way individuals’ search processes changed through the cycles, will be discussed. Note: Data collection for cycle 3 will be completed in March 2022, and presented at EAHIL.

Conclusion

This QI project enabled us to demonstrate and understand time savings created by changes to our workflows. It gave us a deeper understanding of search processes and personal variations in technique. Recommendations: (1) Use of QI methodology to evaluate and improve search processes, and understand the impact of changes between database platforms; (2) Use of RMS to streamline literature search delivery.

Human Touch

Our results were unexpected, although we are experienced searchers and had anticipated certain outcomes. This highlights the benefit of QI methodology for improving processes, and the need to question assumptions, even about familiar practices. We would recommend this questioning approach to other Library services to re-evaluate established procedures. It has been professionally rewarding to compare and evaluate our approaches to literature searching. This comparison has allowed us to improve our own practice.

Introduction
Title and abstracts selections form the basis for systematic evidence syntheses which are vital for evidence-based medicine. They are time consuming to perform, but the ever-growing body of scientific literature plus the demand for faster up to date information make the current selection process unsustainable. Machine learning-based screening tools have been suggested as a solution to accelerate the process, but large-scale application in practice stays off. A gap in knowledge about users' expectations and preferences may partially explain this low uptake.
Aim
To create insight in users' expectations and preferences regarding machine learning tools for title and abstract selection to better align tool development and usage in practice.

Method/ Program Description
We use a clinical guideline development setting to derive the expectations and preferences of potential title and abstract tool users. First insights were collected through a survey among guideline developers and clinical specialists. More in-depth expectations were collected through focus groups with guideline developers and clinical specialists.
Results/ Evaluation
The survey was distributed among 335 guideline developers and clinical specialists. 88 responses were obtained of which 79 survey responses could be used for the analysis. Seven guideline developers and 5 clinical specialists were recruited respectively for two separate focus groups. Data is currently being analysed and results will be available at the conference.
Conclusion
As a result of this study, we will provide an overview of users' expectations and preferences regarding machine learning tools for title and abstract selections.

Introduction
Researchers performing systematic reviews often express the desire to limit the search results to a certain study size: "I want to include only studies of more than 100 patients". While we of course can discuss about the validity of such a request, limiting the search results to match the inclusion criteria can reduce the burden of screening for reviewers.

Aim
The aim of our study was to develop a filter in embase.com and Medline Ovid to retrieve references above a certain threshhold of sample size. We compared the effectivenss of our filter in development using existing systematic reviews that report using sample size as an inclusion criteria.

Method/ Program Description
Together with researchers who expressed the desire to limit search results to a certain number of patients we constructed preliminary filters which were tested on the spot by evaluating the patient numbers of relevant references that had not been retrieved. If the patient numbers matched the inclusion criteria, the filter was adapted to retrieve the missed articles. After several rounds of improvement of the filter the filter was then tested against existing systematic reviews that used sample size as inclusion criteria but did not limit their search to a sample size.

Results/ Evaluation

The filter that was developed consists mainly of truncated numbers in proximity with words such as patients, cases, adults, females etc and phrase like "n=". The filter can and should be adapted to the research topic by combining these truncated numbers with specific terms for diseases, interventions or body parts of interest such as melanomas, surgeries, eyes or knees. The sensitivity of the filter as evaluated on existing systematic reviews was at least 94%. The references that were not retrieved were older articles that did not include the study size in their abstract.

Conclusion

The study size filter is a good way to limit search results to a certain number of patients. It is not 100% sensitive, but few filters are. Current guidelines for abstract formats advice authors to include in their abstract the number of patients in their research. We therefore expect the sensitivity of the filters only to improve for newer studies. A limitation is that the filters are only available in the interfaces of embase.com and Ovid and cannot be translated into PubMed, as the filter uses proximity operators which are not available in PubMed.

Human Touch (Recommended)

With the study size filter the burden of screening for systematic reviews can be greatly reduced.

Introduction

Evidence-Based Medicine (EBM) has expanded the role of the librarian beyond identification of the literature only, to be involved in other stages of the evidence review process.
Aim

The aim of this paper is to demonstrate the role of the librarian in conducting evidence synthesis in partnership with clinicians, health care workers, researchers, and policy makers. We will examine a series of rapid reviews conducted during the last five years to support professionals and other decision-makers working in palliative care.
Method

The literature searches were conducted across a range of databases and supplementary sources. In addition to designing and running the literature searches, other tasks included carrying out screening and study selection, developing data extraction forms and carrying out quality assessment of the eligible studies. Final tasks included synthesising evidence and writing the review using reporting templates in collaboration with researchers.
Results

To date, twelve reviews have been conducted using a methodology developed in partnership with the research team. Findings will be presented from the start of the process at the point of partnership working, to development of the review and subsequent follow up to demonstrate impact. The evidence from these reviews impacts directly on palliative care clinicians and other decision makers, and indirectly on patients/carers in receipt of palliative care.

Conclusion

Broadening horizons provides opportunities for information professionals in health care to play an invaluable role. Librarians can be effective partners in supporting researchers to practice evidence-based medicine.

Human Touch (Recommended)

Being integrated into a research team is an invaluable experience and contributing to other aspects of the review process can be rewarding. It provides opportunities to develop our expertise and remain relevant in an ever-changing world.

Introduction: The purpose of systematic reviews is to evaluate and synthesise the best available evidence on a specific question, using rigorous and transparent methods. Generally, most of the included studies are scientific articles published in academic journals. Recently, the number of predatory journals has increased considerably. Predatory journals are often published under the gold open access model. They do not follow good editorial, publishing and transparency practices and use aggressive solicitation methods. As systematic reviews call for exhaustive searches in databases and complementary information sources, articles published in predatory journals might be retrieved. Because of the potential of poor quality, fraud, or erroneous and misleading data in studies published in these journals, their inclusion in a systematic review may affect its results and conclusions.

Aim: To create an automated instrument to help researchers identify articles from predatory journals among the articles retained for inclusion after the selection stage and before the quality appraisal and data extraction steps. The tool analyses a collection of articles and highlights suspicious titles.

Method: The criteria for identifying whether an article is published in a predatory journal should be searchable in an automated way for a set of articles.

Examples of the criteria considered in the initial design:

- Is the publisher a member of COPE (Committee on Publication Ethics)?

- Is the journal listed in DOAJ, or has wrongly claimed to be part of it?

- Is it indexed in Medline or in the Web of Science Core Collection?

- Is the ISSN listed on the ISSN International Centre Portal, and does its name match?

- Is it part of the updated « Beall’s list of Predatory Journals »?

We are still considering more criteria to be added to the instrument. In practical terms, the instrument takes a list of article references exported from EndNote in XML format. A weight is assigned to each criterion in order to calculate a score for each reference. The higher the score, the greater the chance that an article has been published in a predatory journal.

Results: We are currently analysing different sets of articles to improve the report of the results. This report is intended to inform the researchers of the results of the analysis. Researchers will have to check manually suspicious references using complementary criteria, according to a list that we will provide. Finally, they will have to decide how to handle the identified articles. Their strategy as well as the method used for identification should be described in the protocol. No guidance exists yet, but some recommendations have been published.

Conclusion: As a systematic review support service, it is important for us to provide our researchers with a tool to facilitate the identification of articles from predatory journals, which can be time consuming. With this instrument, information specialists are now able to help authors to identify this threat on the quality and validity of their systematic review.

Human Touch: This project emerged from discussions and interrogations within the review teams in which we are involved.