What is site search?
Site search (also called on-site search) is a search engine that is confined to a single site, connected sites, or domains. For example, you may want to only search a specific site, or you may have multiple subdomains and a parent www site to search across.
Google, Bing, DuckDuckGo and other search engines are ideal for driving visitors to your site, but for providing a great on-site experience, a site search engine is required.
Unlike general internet search engines, site search engines can do things like:
- Provide personalized search
- Be optimized to outcomes (such as conversions or sales)
- Display custom related results and special promotions
- Be A/B tested to improve search results or relevant product results
- Offer much richer user interfaces
- Have deeper search analytics to understand visitor intent
How important is site search?
Here are just a few interesting facts about the importance of a good site search tool:
- Around 40% of online transactions involve an on-site search, particularly for e-commerce websites, ie, your visitors want to use the search bar on your site.
- Buyers are 90% more likely to have used your site search during a visit and can account for 45% of site revenue.
- Now that more than 50% of web traffic is mobile, mobile-optimized site search is vital. In fact, 50% of total e-commerce revenue comes from mobile and 79% of users restart their Google search if the page they land on isn’t mobile-friendly.
- When one company made their on-site search more visible...
- Site searches climbed by 439%
- Visitors stayed on the site 110% longer when they used search
- Page views increased by 45%
- Goal completions increased by 242%.
- Sites with a semantic-based search engine have a low 2% rate in shopping cart abandonment, compared to as much as 40% on sites with plain text search.
- It's important to optimize search for mobile and desktop — mobile users put items into their shopping carts at almost the same rate as desktop users.
- Speed matters. Page load time affects the purchase decision of almost 70% of shoppers. Amazon found that at 100 millisecond delay can cost millions of dollars.
- Search design is incredibly important. Baymard Institute noted that 57% of e-commerce searchers weren’t sure if they got the results they were looking for.
Who is this buyer's guide for?
Site search is a broad term that could refer to website search, on-site e-commerce search, intranet search, and more. For this guide, we are limiting an overview to features primarily for the following three use cases:
- Visitors to your public site
- Search across one or more domains
- Static or CMS-based websites
- Search for online products and related metadata (price, color, size, etc.)
- Search category pages and collections
- Platform (e.g., Shopify, Magento) or custom stores
- Custom web or mobile-based search applications
What should I look for in a site search solution?
With many open source and hosted solutions to choose from, which site search platform is right for you? We explain the factors that go into selecting a site search product below.
AI and site search
Advanced AI learning models enable site search engines to provide solutions such as:
- AI-based ranking
- Instant relevance
- Query understanding
Let's break each one down to understand the importance of machine learning on search:
Two or more search results on your site can be equally relevant, but one may be more valuable to you than the others. AI-based ranking is a feature that can help you improve results with better ranking can make a huge impact on customer satisfaction, site conversions, and revenue.
Ranking is complex and highly use case specific. It took a team of people at AirBnB over 2 years to get AI based ranking into production! AI ranking is hard, but newer AI models such as reinforcement learning can leverage your clickstream data to deliver better results automatically.
It used to be that companies would need to invest hundreds of hours creating synonyms, writing relevance rules, or stuffing their site with tags and keywords to ensure the right content is found. Fortunately, that odious task is now a relic of the past. AI-based relevance replaces the need to implement complex and costly workarounds.
It also allows companies to enable question or symptom based search (where a customer can type in a question or a list of symptoms and still get good results). Just as importantly, AI relevance helps your visitors and customers find what they're looking for faster.
Query understanding encompasses a large number of technologies, such as natural language processing (NLP), that make it easier for customer to type in just about anything to get good search results. Even when customers misspell words, query on broad categories or exact product names, or entire long multi-word queries, newer query understanding models can help.
Site search engine features
What happens as users type in a query on your site search today? Do they get auto-suggestions (like Google and Amazon’s search) or do they get more visual results (like IMDB or AnnTaylor.com search)?
Does your site search know how to handle typos or misspelled words? Does it search only web pages (HTML) or does it also index documents (PDF, DOC, DOCX, etc)?
Site search user experience experience (UX) can vary a lot. By default, most content management systems (CMS) and legacy solutions offer very limited functionality and poor user experience.
Today there are many features for site search to build a great end-user experience for your audience. Let's look at some of the most important ones.
Crawlers and API indexing
How your site is indexed is important. Most sites can be indexed through search crawlers, but mobile apps or e-commerce sites typically require APIs to connect to SQL and NoSQL data stores. Without getting into all the detail, some questions to ask include: how quickly will this solution index my site? When an edit or change is made, how quickly is that re-indexed? If a new page is added, how does that get added to the search index. If the site search solutions that you’re evaluating cannot provide great answers to those questions, it may be time to look elsewhere.
Connect to business data
What if your site search engine could connect outcomes like signups, conversions, and sales to the search queries that led to that result? Google optimizes ad results in precisely this fashion — displaying ads with the highest conversion rates. It’s available for site search as well, but to do that your search provider needs to be able to connect to business data (ideally in real time) to continually optimize results based on user behavior.
Nearly 1 in 10 search queries are misspelled. That’s a lot of misspelling! Your site search tool needs to know how to handle misspellings or many of your customers will leave your site mistakenly believing you don't offer what they need. Ideally, a search engine should include typo tolerance — a “did you mean?” feature, fuzzy search spell checking, and/or at least display multiple possible results based on similar keyword searches.
In old school search, users type in a query and hit return to see results. Consumers, however, expect at minimum a Google or Amazon-like search with suggestions, or autocomplete or autosuggest, as they type. Dynamic search suggestions not only provide instant gratification, but they also help sort out misspellings faster (see “spell checking” above).
You say “sneakers,” I say “running shoes.” Your users are typing in search terms that may be different from how your site is optimized. Newer search engines use vectors, too, a mathematical approach to understanding language that supersede the need to build synonym libraries. However, until vectors are broadly available, your site search should support a synonym library.
Search suggestions (see above) are a very common search UI feature. Now many companies are using using instant search on their site where search results change in real time as users type in their queries. Instant search works well for more “visual” product sites. It may be worth A/B testing your site search (more on that below) to see what works best for your CTR.
Redirects make it extremely easy to guide your customers when they search. Instead of showing the standard search result page, they redirect the customer to specific pages.
How do you know if your site visitors are finding what they need? Your site search solution needs to have metrics — or at least be able to work with 3rd party solutions like Google Analytics or another business intelligence solution — so you can analyze how users are interacting with search results. This can help you to determine whether certain results need to be boosted, how search trends are changing over time, which queries are returning poor results, and more.
What if, for a given query, you changed results? Would you have better or worse click-throughs, conversions, sales, or user satisfaction? A/B testing can help. Tests could be performed on everything from search terms to how your data has been indexed to the search results design. A modern search solution should feature search A/B testing and provide guidance on what search algorithm helps your company improve its bottom-line results based on whatever criteria you’ve established.
You may want to exclude documents from being indexed. Or you may want to boost specific products or certain types of content (e.g., blogs) and lower ranking for other types of content (e.g., comments). Site search solutions often include rules to enable you to manage priorities to direct users to the right content.
Speed is an invisible feature, but incredibly important. Amazon showed that just .001 second differences in returning results meant huge losses. It’s not just true for online stores. Your users are accustomed to Google-like speed. Anything less is likely to send them elsewhere. Your search tool or search provider should be clocking-in in milliseconds.
Documents (.doc, .docs, .pdf) available for download on your site — whether they’re gated or not — should be indexable as well. If you don’t want them indexed, that’s something you should be able to manage through rules (more on that below!).
Built-in personalization features and/or connecting to third-party personalization solutions should be on the list as well. Search personalization creates contextual profiles for individual visitors to personalize their search results and display relevant content. Personalize results based on user preferences, location, gender, past purchase history, product type, and more.
Filters and facets
Allowing your customers to filter results by price, content type, author, or other factors is very useful especially for sites that have hundreds or thousands of records. Your search platform should support both search facets and filters to help customers narrow down results to find exactly what they need.
Design and SDK
Does your search engine support Latin script languages (think English, French, Spanish, etc) and multi-byte symbol-based queries (Chinese, Japanese, etc.)? Moreover, are the machine learning models your search provider includes multilingual? You may not need it today, but when you do, you’ll want a search engine that can scale to meet the needs of new audiences.
Scaling site search
Companies like Amazon, Reddit, and Aliexpress can afford to hire full-time search engineers and system reliability engineers (SREs) to run their site search (Amazon alone has over 1000 people in their search team!). Most organizations don’t have the luxury to hire a battalion of engineers. Many others would prefer to leverage their engineering talent in other parts of the business.
There’s a spectrum of needs and where your company and use case falls can largely help make this decision. Every use case is different, so take this only as a way to frame thinking.
- Company website: Site search should almost always be hosted for the simple reason that the data is flat, easy to index, and not private.
- Online store: Unless you’re selling highly illicit material or have specific hosting requirements, hosted search will likely meet your needs.
- Sensitive data application: Financial applications or other regulated or highly sensitive data may benefit from self-hosting.
- Internal KB: For corporate wikis or knowledge bases, it’s valuable to have information that’s on-prem or on the public internet secured through SSO.
Most site search tools are priced on the number of monthly queries and/or the total number of records stored in the search index. There are a great many other features that may be marked up (for example, some search providers require a premium payment for machine learning). Questions you will want to answer before selecting a provider:
- How many searches do we anticipate we will have each month?
- How many objects or records need to be indexed?
- Does my provider charge extra for any basic or advanced features like crawling, A/B testing and re-indexing, AI and machine learning, personalization, etc.
- Is set up and onboarding included or separate?
- What other services do you offer?
- What is included or excluded in our set up fee?
Ease of use for technical and non-technical users
Who will be managing your site search software? Will it be developers only, or will your marketers, SEO specialists, product managers, and others also participate? Who will be responsible for ensuring it’s running the way you want and improving user satisfaction?
In-house configuration of open source search software typically requires search engineering, which is difficult even for experienced engineers. For service based delivery the work of maintenance and optimization can potentially fall upon employees without technical skills.
Even after site search is running and optimized, you can’t ignore it. It requires continual optimization to meet the demands of customers on your site. A best practice is to determine your long-term needs and find a solution that will work for your team.
Site search security is important if your index contains sensitive data or you’re limiting access to your content to select users. This is a point that goes beyond the question of how to find the right search solution for your team, but it’s important to identify how important it is for your organization before selecting a search platform.
Ready for a new search engine?
Whew, you made it all the way through. Congratulations. Was this guide helpful? Please drop us a line to let us know.
If you’re considering a new solution, be sure to try Search.io. We offer a fully-functional 14-day free trial, or we would be delighted to provide a live demo to your team.