Full-text search with Gatsby and Netlify CMS

Full-text search refers to techniques for searching a single document or a collection in a full-text database. Full-text search is distinguished from searches based on metadata or on parts of the original texts represented in databases (such as titles, abstracts, selected sections, or bibliographical references).

Full-text search allows users to retrieve higher quality results than using other methods. During the full-text search, a search engine examines all of the words in every stored document as it tries to match search criteria.

Implementing a full-text search into Gatsby is quite painless to set up. I’ve experimented with different plugins but decided to use Flexsearch (easiest to implement and set up).

Prerequisites

In this example, I’m presenting an implementation for Gatsby’s website using Netlify CMS to manage content. When using other CMS, the implementation may vary.

First, we need to install gatsby-plugin-flexsearch

$ yarn add gatsby-plugin-flexsearch

After that, we have to initialize the plugin in gatsby-config and configure which attributes we want to include.

{
resolve: 'gatsby-plugin-flexsearch',
options: {
languages: ['en'],
type: 'MarkdownRemark',
fields: [
{
name: 'id',
indexed: false,
resolver: 'id',
store: true,
},
{
name: 'html',
indexed: true,
resolver: 'internal.content',
attributes: {
encode: 'balance',
tokenize: 'strict',
threshold: 0,
resolution: 3,
depth: 3,
},
store: true,
},
{
name: 'title',
indexed: true,
resolver: 'frontmatter.title',
attributes: {
encode: 'extra',
tokenize: 'full',
threshold: 1,
resolution: 3,
},
store: true,
},
{
name: 'description',
indexed: true,
resolver: 'frontmatter.description',
attributes: {
encode: 'icase',
tokenize: 'forward',
threshold: 2,
depth: 3,
},
store: true,
},
{
name: 'type',
indexed: false,
resolver: 'frontmatter.type',
store: true,
},
{
name: 'slug',
indexed: false,
resolver: 'fields.slug',
store: true,
},
{
name: 'layout',
indexed: false,
resolver: 'frontmatter.layout',
store: true,
},
],
},
},

In the example above, I declared custom attributes to be indexed by flex search (id, layout, and slug) that are important for displaying results snippets and have to be available from the search component.

After restarting the development server flex search index and the store will be attached to the window object of your browser.

Search component with snippets

Right now, our Gatsby website will have flex search enabled. So far, so good! Now we have to enable users to perform searches against our content.

To do that, we need a search component! The most important part is the search function itself. If you want to check out the full component take a look at this gist

getSearchResults(query) {
var index = window.__FLEXSEARCH__.en.index
var store = window.__FLEXSEARCH__.en.store
if (!query || !index) {
return []
} else {
var results = []
Object.keys(index).forEach((idx) => {
results.push(...index[idx].values.search(query))
})
      results = Array.from(new Set(results))
      var nodes = store
.filter((node) => (results.includes(node.id) ? node : null))
.map((node) => node.node)
      return nodes
}
}

To search our documents, we can create a simple function that takes the user’s input and performs a search using the Flexsearch index and store.

First, we create an array for results. Then we can check each index for the querying user searched for and return the matched values. Then we’re using the results array to filter Flexsearch store and return objects that are matching our query. The function is returning nodes containing information about documents that matched the query.

These nodes can then be used to render snippets for each search result. You can check out the full implementation on Dionysus website we worked on last month

Psst! If you’re looking for a great Gatsby starter with Tailwind and Netlify CMS you have to check out Henlo


Full-text search with Gatsby and Netlify CMS was originally published in Level Up Coding on Medium, where people are continuing the conversation by highlighting and responding to this story.

Full-text search refers to techniques for searching a single document or a collection in a full-text database. Full-text search is distinguished from searches based on metadata or on parts of the original texts represented in databases (such as titles, abstracts, selected sections, or bibliographical references).

Full-text search allows users to retrieve higher quality results than using other methods. During the full-text search, a search engine examines all of the words in every stored document as it tries to match search criteria.

Implementing a full-text search into Gatsby is quite painless to set up. I’ve experimented with different plugins but decided to use Flexsearch (easiest to implement and set up).

Prerequisites

In this example, I’m presenting an implementation for Gatsby’s website using Netlify CMS to manage content. When using other CMS, the implementation may vary.

First, we need to install gatsby-plugin-flexsearch

$ yarn add gatsby-plugin-flexsearch

After that, we have to initialize the plugin in gatsby-config and configure which attributes we want to include.

{
resolve: 'gatsby-plugin-flexsearch',
options: {
languages: ['en'],
type: 'MarkdownRemark',
fields: [
{
name: 'id',
indexed: false,
resolver: 'id',
store: true,
},
{
name: 'html',
indexed: true,
resolver: 'internal.content',
attributes: {
encode: 'balance',
tokenize: 'strict',
threshold: 0,
resolution: 3,
depth: 3,
},
store: true,
},
{
name: 'title',
indexed: true,
resolver: 'frontmatter.title',
attributes: {
encode: 'extra',
tokenize: 'full',
threshold: 1,
resolution: 3,
},
store: true,
},
{
name: 'description',
indexed: true,
resolver: 'frontmatter.description',
attributes: {
encode: 'icase',
tokenize: 'forward',
threshold: 2,
depth: 3,
},
store: true,
},
{
name: 'type',
indexed: false,
resolver: 'frontmatter.type',
store: true,
},
{
name: 'slug',
indexed: false,
resolver: 'fields.slug',
store: true,
},
{
name: 'layout',
indexed: false,
resolver: 'frontmatter.layout',
store: true,
},
],
},
},

In the example above, I declared custom attributes to be indexed by flex search (id, layout, and slug) that are important for displaying results snippets and have to be available from the search component.

After restarting the development server flex search index and the store will be attached to the window object of your browser.

Search component with snippets

Right now, our Gatsby website will have flex search enabled. So far, so good! Now we have to enable users to perform searches against our content.

To do that, we need a search component! The most important part is the search function itself. If you want to check out the full component take a look at this gist

getSearchResults(query) {
var index = window.__FLEXSEARCH__.en.index
var store = window.__FLEXSEARCH__.en.store
if (!query || !index) {
return []
} else {
var results = []
Object.keys(index).forEach((idx) => {
results.push(...index[idx].values.search(query))
})
      results = Array.from(new Set(results))
      var nodes = store
.filter((node) => (results.includes(node.id) ? node : null))
.map((node) => node.node)
      return nodes
}
}

To search our documents, we can create a simple function that takes the user’s input and performs a search using the Flexsearch index and store.

First, we create an array for results. Then we can check each index for the querying user searched for and return the matched values. Then we’re using the results array to filter Flexsearch store and return objects that are matching our query. The function is returning nodes containing information about documents that matched the query.

These nodes can then be used to render snippets for each search result. You can check out the full implementation on Dionysus website we worked on last month

Psst! If you’re looking for a great Gatsby starter with Tailwind and Netlify CMS you have to check out Henlo


Full-text search with Gatsby and Netlify CMS was originally published in Level Up Coding on Medium, where people are continuing the conversation by highlighting and responding to this story.


Print Share Comment Cite Upload Translate
APA
Wojciech Kałużny | Sciencx (2024-03-28T20:51:08+00:00) » Full-text search with Gatsby and Netlify CMS. Retrieved from https://www.scien.cx/2021/04/06/full-text-search-with-gatsby-and-netlify-cms/.
MLA
" » Full-text search with Gatsby and Netlify CMS." Wojciech Kałużny | Sciencx - Tuesday April 6, 2021, https://www.scien.cx/2021/04/06/full-text-search-with-gatsby-and-netlify-cms/
HARVARD
Wojciech Kałużny | Sciencx Tuesday April 6, 2021 » Full-text search with Gatsby and Netlify CMS., viewed 2024-03-28T20:51:08+00:00,<https://www.scien.cx/2021/04/06/full-text-search-with-gatsby-and-netlify-cms/>
VANCOUVER
Wojciech Kałużny | Sciencx - » Full-text search with Gatsby and Netlify CMS. [Internet]. [Accessed 2024-03-28T20:51:08+00:00]. Available from: https://www.scien.cx/2021/04/06/full-text-search-with-gatsby-and-netlify-cms/
CHICAGO
" » Full-text search with Gatsby and Netlify CMS." Wojciech Kałużny | Sciencx - Accessed 2024-03-28T20:51:08+00:00. https://www.scien.cx/2021/04/06/full-text-search-with-gatsby-and-netlify-cms/
IEEE
" » Full-text search with Gatsby and Netlify CMS." Wojciech Kałużny | Sciencx [Online]. Available: https://www.scien.cx/2021/04/06/full-text-search-with-gatsby-and-netlify-cms/. [Accessed: 2024-03-28T20:51:08+00:00]
rf:citation
» Full-text search with Gatsby and Netlify CMS | Wojciech Kałużny | Sciencx | https://www.scien.cx/2021/04/06/full-text-search-with-gatsby-and-netlify-cms/ | 2024-03-28T20:51:08+00:00
https://github.com/addpipe/simple-recorderjs-demo