Page Comparison

...

...

Overview

The following shows a schematic overview of the XperienCentral search engine:

Image RemovedImage Added

Filling the index with documents is done in three steps: crawling (1)-(2), parsing (3) and indexing (4). Retrieving search results from the index is done from a client such as a page in XperienCentral with a search element (5), from the Setup Tool, or from a command line client. 

Back to top 

...

Anchor
crawling crawling

Crawling

Retrieving the documents starts with retrieving the URLs of all documents and pages. XperienCentral provides links to all items that should be indexed on the indexer page. This page contains references to:

• Pages of all web initiativeschannels
• Content Repository items created in the last 5 days
• Documents (uploaded to a Download element)
• Special content types

The URL of the indexer page is configured in the properties.txt file (metaurl parameter) and is the default start page in the Setup Tool. The search engine crawler retrieves all the URLs from the indexer page and creates requests for each of the documents. Requests are always sent to the front-end frontend in order to benefit from existing caching and to take authorization and personalization into account. With the default configuration, the crawler only indexes links from the indexer page but does not follow links in the retrieved documents. This means the default index depth is 1. An index depth of 1 ensures that the indexing process is efficient and that no outbound links (links outside the web initiativechannel) are found and indexed.

A request for a document is not only a direct request for the document - additional meta information is requested as well. This extra meta information is provided by the indexer page by requesting the indexer page with an additional document= parameter. For example, to index the homepage on local XperienCentral installation, the crawler requests the URL http://localhost:8080/web/webmanager?id=39016&document= http%3A%2F%2F127.0.0.1%3A8080%2Fweb%2Fshow%2Fid%3D26111. The underlined value is the URL-encoded URL of the homepage. When this URL is requested an XML result is returned which looks similar to the following:

<document>
  <langid>42</langid>
  <contenttype>page</contenttype>
  <date>2007-12-24</date>
<webid>26098</webid>
<pagepath>p26111</pagepath>
  <pagepath_00_name>Home</pagepath_00_name>
  <pagepath_00_url>http://127.0.0.1:8080/web/Home.htm</pagepath_00_url>
</document>

...

Back to top

...

Parsing

All retrieved documents are parsed before they are stored in the index. All documents are converted to a plain text format. Office (Word, Excel) and PDF documents are converted with external programs. These programs are usually executable files and they are configured in <searchengine-directory>/conf/properties.txt. The mapping between document type and converter is configured in <searchengine-directory>/conf/parser.txt. The mapping can be based on both file extension and content-type, which is retrieved from the HTTP header. For more information about the parser.txt file see Parser Configuration. 

Back to top 

...

Indexing

The index stores the indexed information. In fact, it is a small database that is tuned for fast information retrieval. It is also similar to a database because it contains a set of documents (one indexed document is one document in the database) and every document has a set of fields. Notice the similarity between tables and table fields. Before documents are stored in the index they go through three processes. The first is the normalizing process. In this process characters are converted to normal and lowercase equivalents. For example "ç" is converted to "c", and "C" is converted to "c". By doing this a user will get search results when searching for "barcelona" instead of "Barçelona".

...

The title and the keyword are the most important fields.

Back to top

...

Searching

Once a website has been successfully indexed, you can perform tests on the search index. The XperienCentral search engine is a customized version of the popular open source search engine Lucene. The syntax for queries is almost the same as Lucene. The entire syntax is described here: https://lucene.apache.org/core/2_9_4/queryparsersyntax.html. For a detailed overview of the scoring algorithm of the Lucene search engine see https://lucene.apache.org/core/3_6_0/scoring.html

Back to top 

...

Index structure

The XperienCentral search engine has a variable set of fields depending on the type of documents that havebeen indexed. Below is a list of the most import fields that are always part of the search index:

...

Other fields that can be part of the index are:

...

To view the contents of a search engine index see Analyze the Search Index.

Back to top