Elasticsearch QPL - Getting Started

From wiki.searchtechnologies.com
Jump to: navigation, search

For Information on Aspire 3.1 Click Here

This Page is Locked - Content no longer maintained - See QPL for the latest information.
Enterprise Add-On Feature

What is a QPL Script?

Scripting in Elasticsearch is completely different than scripting in QPL. It is also different from Query DSL

Scripting in Elasticsearch:

  • Done for every document searched
  • Is for returning new relevancy scoring algorithms
  • Is very slow

Scripting in QPL

  • Done on the query before it is submitted to ES
  • Actually produces DSL
  • Is very fast

They are two, entirely different types of scripting and should not be confused with each other.

QPL and DSL go together like PL/SQL and SQL. Or like a seed spreader and seeds. Or like a nail gun and nails.

Version Matrix

Version of QPL Version of ES Download link for plugin
elasticsearch-qpl-0.1 Elasticsearch 1.4.x elasticsearch-qpl-0.2.zip
elasticsearch-qpl-2.2 Elasticsearch 2.2.x elasticsearch-qpl-2.2.zip

Installing QPL for Elasticsearch

Up to version 2.2

  1. Download and install the Elasticsearch server (please see above table for supported versions)
  2. Download the corresponding qpl version from the above Version Matrix links. The latest version can be found here
  3. Navigate the home directory for the Elasticsearch server and execute the following command to install the Elasticsearch QPL plugin by replacing the path to the file with the location of where the plugin zip file was downloaded: bin/plugin --url file:///path/to/plugin --install elasticsearch-qpl

Version 2.4.3 onwards

  1. Download and install the Elasticsearch server (please see above table for supported versions)
  2. Download the corresponding qpl version from the above Version Matrix links. The latest version can be found XXXXXX
  3. Navigate the home directory for the Elasticsearch server and execute the following command to install the Elasticsearch QPL plugin by replacing the path to the file with the location of where the plugin zip file was downloaded: bin/plugin install file:///path/to/plugin

Configure Java Security Policy

In version 2.4.3 onwards of Elasticsearch, you need to configure the Java security policy to allow execution of the scripts. You can find information on Java security policies here and their use in Elasticsearch here.

In short, you need to add the following permissions to the java.policy file:

 // QPL in ES 2.4.3
 permission java.lang.RuntimePermission "createClassLoader";
 permission java.lang.RuntimePermission "getClassLoader";
 permission groovy.security.GroovyCodeSourcePermission "/groovy/shell";
 permission groovy.security.GroovyCodeSourcePermission "/groovy/script";

The java.policy file can be found in the following location:

  • JRE
    • $JAVA_HOME/lib/security/java.policy
  • JDK
    • $JAVA_HOME/jre/lib/security/java.policy

How to Use QPL for Elasticsearch

QPL Scripts can be sent for execution to Elasticsearch in three ways:

  • Dynamically as a query parameter
  • Store a script in an ES index to be reused/executed at a later time
  • Store QPL scripts in files
  • The QPL scripts can not be named "es", that because the plugin uses a binding variable named "es" and this generate a missing methods for "es" variable.

1. Dynamically as a query parameter

QPL scripts can be executed through an HTTP GET call at this endpoint: http//{ES domain}:{ES port}/{index}/_qpl and passing the text of the QPL script as a value to a query parameter called: script. e.g. - http://localhost:9300/myindex/_qpl?script=return%20term('peace')

The {index} represents a comma delimited list of indices against which the search is supposed to be executed. A single index can also be specified in this path variable.

2. Store a script in an ES index to be reused/executed at a later time

QPL scripts can be stored in an Elasticsearch index (QPL scripts get stored in an index called qpl_scripts) and later reused. To store a QPL script execute a REST POST request against this endpoint: http//{ES domain}:{ES port}/{index}/_qpladmin/{id} where the id the unique id to be given to the script. The content/text of the script should be supplied in the body of the http post request.

Scripts stored in the ES index can be retrieved and deleted by issuing HTTP GET and DELETE requests, respectively, against the same endpoint as described above: http//{ES domain}:{ES port}/{index}/_qpladmin/{id}

At query time a script stored ahead of time can be reused to execute a search by doing an HTTP GET against this endpoint: http//{ES domain}:{ES port}/{index}/_qpl/{scriptId} where {scriptId} would hold the same id value as what was used when the script was stored.

3. Store QPL scripts in files

QPL scripts can be stored in files which have to reside on a folder accessible from all ES nodes in the cluster. Two settings are supported which need to be added to the elasticsearch.yml on each node:

  • qpl.allow.execution - to allow execution of QPL scripts this entry must be present and have a value of Y
  • qpl.cache.scripts - to allow caching of compiled QPL scripts this entry must be present and have a value of Y
  • qpl.files.location - the path to where the QPL script files are located

To execute a QPL script from a file the caller needs to pass in the http request a parameter called fileName which should hold the name of the of file containing the QPL script. So your GET request should look like: http//{ES domain}:{ES port}/{index}/_qpl/file/{scriptFileName} or http//{ES domain}:{ES port}/{index}/_qpl?fileName={scriptFileName}

Use of script files in Elasticsearch 2.4.3 onwards

If you're using script files in a version of Elasticsearch after version 2.4.3, you need to explicitly grant access to every script file you use in the Java policy file. See [#Configure_Java_Security_Policy Configure Java Security Policy] above for details on the location of the file. For every script file you wish to access you'll need to add a line similar to the following:

 // QPL scripts
 permission java.io.FilePermission "c:\\elastic\\qpl\\script.qpl", "read";

Additional parameters that can be passed with a QPL script

The search term can be passed through a query parameter in the http request.

QPL scripts have access to a special "es" variable. The es.q and es.query will give access to the value of the query parameter passed in the http request.

The following is a list of additional parameters that can be passed alongside with a QPL script. These parameters will perform the exact same functions as the equivalent query parameters part of the main Elasticsearch product, as described here. In addition to these parameters any other application specific parameters can be passed which can be accessed inside the QPL scripts as described below.

The es variable will give access inside the QPL script to the values of all http parameters through the following methods (all accepting as a parameter the name of the http parameter): get, getBoolean, getInteger, getFloat, getDouble, getAll (retrieves a parameter value that consists of an array of Strings). e.g: es.get("size") will retrieve the value of the size parameter as a String. Please note that es.get(parameterName, parameterDefaulValue) works also for any custom http parameter passed in, other then the parameters listed below.

  • size
  • from
  • explain
  • version
  • timeout
  • terminate_after
  • sort
  • fields
  • track_scores
  • indices_boost
  • stats
  • suggest_field
  • suggest_text
  • suggest_size
  • suggest_mode
  • aggs (aggregations) - please note that to use aggregations the search request needs to executed as a POST request and the aggregations need to be passed into the request body, just like when executing a regular ES query.