Scalable Enterprise Search
A growing SaaS solutions provider had outgrown its Indexing Server Search serviceThe Challenge
A major supplier of a multi-tenant, enterprise SaaS system had an Enterprise Search system built by combining the output of a file system produced by Microsoft Indexing Service and results from a SQL Server relational DBMS. This hybrid approach was adequate in the beginning, but as the SaaS system has grown by adding more tenants and as each tenant has increased its data under storage, the Enterprise Search solution failed to scale up.
Microsoft’s Indexing Service was created to search within local files. However, this SaaS provider’s use of the Indexing Service pushed the sheer number and size of files beyond the original design limits. For example, they found it became impossible to regenerate the catalog.
Furthermore, the Indexing Service provides a file search based on indexing text within the files with relevance based mainly on text search plus some additional pre-written rules. Users expectations for relevant results similar to what they see online in Web Search Engines like Google are unattainable in Enterprise Search.
Each of the SaaS provider’s clients has a virtual SQL Server instance. Thus, each tenant has its own data that remains logically isolated from all other tenants’. Metadata associates each database with the correct tenant, and database security prevents any tenant from accidentally or maliciously accessing other tenants’ data.
The Solution
Looi Consulting recommended several approaches and solutions. First, as a preventative measure, we urged immediately action on a tested back-up plan to address the possibility of a catastrophic failure of the Indexing Service. In fact, the SaaS provider eventually did experience a hard failure and could not rebuild the index. The availability of a backup allowed end user operations to continue certain levels of service, albeit with an old index. This bought time for more durable solutions to be implemented.
We also recommended moving the file system to a cloud or distributed infrastructure, rather than staying on their SAN. A cloud solution would scale better, cost less and most importantly, eliminate a single point of failure.
We advised that several open source Enterprise Search platforms be considered, such as ElasticSearch, SOLR and others. We then engaged with the client in several Agile Spikes to rapidly test the capabilities of these technologies and how well they addressed the needs of the SaaS provider. In the end, we jointly determined that a solution based on ElasticSearch was most appropriate and then proceeded to rebuilt their Enterprise Search service from the ground up.
Finally, we recommended investing in the creation of more metadata. While attaining Search relevance similar to Google’s is not possible within the enterprise, the right metadata for each document can help improve relevance. By using various techniques to extract metadata from documents or to prompt users for metadata when they interact with documents, the Enterprise Search service can facilitate the harvesting and injection of metadata over a period of time.
Capabilities
Looi Consulting demonstrated the ability to rapidly understand the complexities of a client’s specialized domain, apply its considerable experience and deliver actionable recommendations. Following up, we implemented the solution in a timely manner to avoid loss of end-customer service. Some of the capabilities demonstrated include:
- Metadata Management: Accessing and building metadata to improve search results.
- Semi-structured data: Leveraging different types of data from various sources and combining them to provide better BI insights.
- Data Quality and Data Cleansing: Improving completeness and consistency of data.
Timeline
Looi Consulting was engaged for a 4 week feasibility and research phase. This was followed by several months of prototype development and productization of the eventual solution using agile development methodologies, including paired programming. The solution has been successfully deployed in production.
%
Stability
%
Scalability
%
Relevance
New Search Solution
Scalable, Stable Enterprise Solution
Document Size and Quantity
Stablilty Greatly Improved
Relevance Improved
- Stability 100%
- Relevance 40%
Remarkable End Results
We started with a largely unmaintainable and unscaleable Enterprise Search that had already begun to fail. The change was remarkable. A best-of-breed platform was customized for the SaaS product and the obsolete system was replaced, tenant by tenant. In addition, the introduction of more metadata contributed to the improved relevance. Perception of better relevance was enhanced by redesigning the UI and improving page load performance; this seemingly unrelated improvement in fact is often perceived as better relevance.
Related Case Studies
