Search as we know it — keywords, autofill, navigation — is undergoing rapid change.

This is due to the acceleration of generative artificial intelligence (AI), machine learning (ML) and large language models (LLMs) that enable different kinds of search or that augment traditional search.

Still, LLMs are expensive to train, deploy and maintain — and existing models like ChatGPT only have access to publicly available data — thus keeping them out of reach for some enterprises in searching (and securing) company-specific data.

Open-source search engine Elastic aims to upend this with its new Elasticsearch Relevance Engine, available today.

The tool allows companies to take advantage of structured and unstructured data to build custom generative AI apps without exposing that data or having to invest in LLMs.

“LLMs like GPT-3 or Bard are very large, very expensive language models trained extensively with a lot of computation,” said Matt Riley, Elastic’s GM for enterprise search. Also, the info they’re trained on is typically from the internet, “so they don't generally have access to private info.”

Insight engines supporting decision, action

Insight engines like Elastic’s combine search with composite AI to provide context-enriched analysis, according to analyst firm Gartner. They draw data from a wide variety of sources and types — what’s known as “wide data,” including repositories, websites and databases — into a central, queryable index.

“Doing so enables insight engines to serve as a mediator for information to support decision and action, or data to support automation,” Gartner stated.

This allows search within enterprises or on websites; but it also goes beyond that to answer questions, make contextual recommendations and derive insights that can then be taken action on.

The global enterprise search market that Elastic competes in — with the likes of Micro Focus, Squirro, IBM, Microsoft, Sinequa, Coveo and other niche and emerging players — is expected by one estimate to reach $8.8 billion by 2030.

“The growth can be attributed to the increasing need for efficient supervision of large volumes of data among organizations to strengthen their operational capabilities,” according to Grand View Research.

Enterprises are also seeking out tools that enable time-saving data search and enhanced security measures.

Elastic securely leverages company-specific data

With Elasticsearch Relevance Engine, Elastic says it is taking AI-powered search capabilities a step further.

The tool allows enterprises to securely leverage generative AI on private business data, giving users the ability to ask questions without exposing company-specific data to the public internet.

It is powered by unified APIs for vector search and transformer models that help capture meaning and context. A BM25f search ranking function helps estimate document relevance to a given query, and hybrid search combines multiple search algorithms to improve accuracy and relevance. Enterprises can bring their own transformer model and/or integrate with third-party transformer models.

“Enterprises are excited about the potential for generative AI in their applications and workflows, but are all also swamped by the pace of innovation in the field,” said James Governor, cofounder of analyst firm RedMonk.

He pointed out that Elasticsearch Relevance Engine is “designed to ease adoption of transformers, homemade and third-party LLM models, building on the original core strengths of Elastic in search.”

Responding to customer demand

For example, take an ecommerce search engine for a large home improvement store. An employee might ask the platform, “How do you build an irrigation system for a 1 acre backyard in Detroit?” Elastic can then provide instructions, required tools and an equipment list based on a specific company catalog or what's in stock in a specific location, Riley said.

Elasticsearch Relevance Engine is already being used by several Elastic customers, including Relativity, which is experimenting with the tool in conjunction with the Azure OpenAI Service to improve the relevance of results for their e-Discovery product, Riley said.

“Ensuring our customers and partners have industry-leading search capabilities is vital to our mission to help them organize data, discover the truth and act on it,” said Chris Brown, chief product officer at Relativity. “We're experimenting with Elasticsearch Relevance Engine right now and are excited about its potential to deliver powerful, AI-augmented search results to our customers.”

AI is expanding rapidly, enterprises must react

Riley pointed out that when technology like AI expands so rapidly, it quickly changes customer expectations and the software they are looking to interact with. The popularization of ChatGPT and its ability to provide very descriptive, detailed answers has generated “an enormous amount of excitement — not just in the tech community, but in the broader consumer community,” he said.

At the same time, “the amount of data enterprises are creating is exploding and will continue to grow,” he said. So, it is incumbent on companies to invest in smarter search technology based on natural human language, not just keywords.

Elastic began investing in vector search and transformer models about two years ago, Riley explained. This decision was based on feedback from its open-source community.

“We listen to the community and act on trends that are emerging as early as we can,” he said.

Traditional search not going away

Despite these new capabilities, though, Riley said search as we know it — keyword search, navigational search, autocomplete — isn’t going to go away anytime soon.

“I don't think AI is going to entirely eliminate what we've thought of as search for a long time,” he said. “I don't expect everybody to suddenly begin talking to their computer and have it read answers back.”

Instead, LLMs offer the opportunity to augment existing capabilities to find a good balance between the two and use each when appropriate.

Still, it’s early in the game and the technology is moving quickly, he said, so “it’s hard to know exactly where we will be a year from now in terms of those capabilities.”