Reciprocal rank fusion (RRF) is a powerful technique to combine results from multiple sub-systems into a single, optimized list. The technical is core to how Carbon's hybrid search works by merging results from both semantic and keyword searches. By taking the reciprocal of each item's rank and giving more weight to top results, this method generates high-quality recommendations that reflect the best of what each source suggests.

Introduction to Reciprocal Rank Fusion

Reciprocal rank fusion is an intuitive and effective technique for merging multiple lists of search results into a single, optimized list. It works by taking the reciprocal (1 divided by the number) of each item's rank in each list, and then summing these reciprocal ranks across all lists to get a final score for each item. This approach gives higher weight to items that are ranked highly in at least one list, even if they aren't ranked as highly in others. The result is a blended list that reflects the best recommendations from all the input lists, with an emphasis on the top-ranked items.

Combining Lists of Search Results

At its core, reciprocal rank fusion combines multiple lists of search results into a single, optimized list. Imagine asking three friends to each rank their top 10 restaurants in your city. Each friend will have a different opinion based on their preferences and experiences. Reciprocal rank fusion provides a way to merge those three lists, considering how highly each restaurant was ranked by each friend. A restaurant ranked #1 by one friend, #3 by the second, and #4 by the third would achieve a higher reciprocal rank than one that was #10 on all three lists.

Valuing Top-Ranked Results

The "reciprocal" in reciprocal rank fusion refers to how the technique assigns more weight to items ranked at the top of each list. It does this by taking the reciprocal (1 divided by the number) of each ranking. For example, the #1 ranked item would receive a score of 1/1 = 1, while the #2 item would get 1/2 = 0.5, and the 10th would only get 1/10 = 0.1. These reciprocal scores are then summed across the different lists to calculate the final, fused score for each item. This reciprocal scoring ensures that items ranked highly by at least one list receive a decent final score, even if not mentioned by others.

Broader Applications and Enhancements

Reciprocal rank fusion has applications beyond just ranking restaurants.

Hybrid search, which has enjoyed a resurgence due to the rise of vector databases and Generative AI, relies heavily on Reciprocal rank fusion. By using RRF to merge results from keyword search, dense vector similarity, and other retrieval methods, hybrid search systems can leverage the strengths of each approach to provide more comprehensive and relevant results to users.

RRF offers several advantages in hybrid search scenarios. It is simple, robust, and requires no tuning of weights for the individual search methods. It can effectively combine results from diverse relevance signals without the need for the signals to be directly comparable. RRF also prioritizes documents that are ranked highly by any of the search methods, ensuring that the most relevant results from each approach are surfaced in the final list. This allows capturing both lexical matches through keyword search and semantic matches using vector embeddings, resulting in an improved search experience.

Search engines also employ RRF-based techniques to combine results from various sub-systems that consider factors such as keyword matching, link analysis, and user engagement metrics. By fusing these different result lists using reciprocal rank scoring, search engines can generate a single, high-quality list that reflects the best recommendations from each sub-system, with extra emphasis on top-ranked pages.

Researchers have also developed variations and enhancements to the basic reciprocal rank fusion method. Some introduce additional weighting factors, while others consider more than just the ranks of the items. However, the core principle remains the same: combining multiple ranked lists in a way that prioritizes top results is a powerful approach for generating optimized, blended recommendations across various domains.