Elasticsearch

Connections

Connection types:

Elasticsearch

Elasticsearch

Elasticsearch is a distributed, RESTful search and analytics engine capable of addressing a growing number of use cases. As the heart of the Elastic Stack, it centrally stores your data for lightning fast search, fine‑tuned relevancy, and powerful analytics that scale with ease.

The Elasticsearch connection can be used to search for documents in your Elasticsearch cluster. Lowdefy integrates with Elasticsearch using the official Node.js client from Elastic Co., and provides direct access to the full query DSL supported by Elasticsearch.

Properties

node: string|string[]: Required - Elasticsearch node(s) to connect to.
index: string: The default index to query. You can also provide a per-request index to override this setting.
auth: BasicAuth|CloudAuth: The authentication data as an object, containing either basic auth credentials (as username and password), an API key (as apiKey), or Elastic Cloud credentials.
maxRetries: number: Max number of retries for each request.
requestTimeout: number: Max request timeout in milliseconds for each request.

The connection accepts many more advanced configuration options. They will be passed to the Elasticsearch client verbatim, so check out the configuration instructions provided by Elastic.

Examples

Authenticated connection with reads and writes:

connections:
  - id: elasticsearch
    type: Elasticsearch
    properties:
      write: true
      node:
        _secret: ELASTICSEARCH_HOST
      auth:
        username:
          _secret: ELASTICSEARCH_USER
        password:
          _secret: ELASTICSEARCH_PASSWORD

Environment variables:

LOWDEFY_SECRET_ELASTICSEARCH_HOST = http://localhost:9200
LOWDEFY_SECRET_ELASTICSEARCH_USER = es
LOWDEFY_SECRET_ELASTICSEARCH_PASSWORD = password

Unauthenticated connection with default index:

connections:
  - id: elasticsearch
    type: Elasticsearch
    properties:
      write: true
      node:
        _secret: ELASTICSEARCH_HOST
      index:
        _secret: ELASTICSEARCH_INDEX

Environment variables:

LOWDEFY_SECRET_ELASTICSEARCH_HOST = http://localhost:9200
LOWDEFY_SECRET_ELASTICSEARCH_INDEX = articles

Requests

Request types:

ElasticsearchDelete
ElasticsearchDeleteByQuery
ElasticsearchIndex
ElasticsearchSearch
ElasticsearchUpdate
ElasticsearchUpdateByQuery

ElasticsearchDelete

The ElasticsearchDelete request removes a JSON document from the specified index.

Properties

id: string|number: Required - Unique identifier for the document to be deleted.
if_seq_no: number Only perform the operation if the document has this sequence number. See Optimistic concurrency control.
if_primary_term: number Only perform the operation if the document has this primary term. See Optimistic concurrency control.
refresh: string|boolean If true, Elasticsearch refreshes the affected shards to make this operation visible to search, if wait_for then wait for a refresh to make this operation visible to search, if false do nothing with refreshes. Valid values: true, false, wait_for. Default: false.
routing: string Custom value used to route operations to a specific shard.
timeout: string Period to wait for active shards. Defaults to 1m (one minute).
version: integer Explicit version number for concurrency control. The specified version must match the current version of the document for the request to succeed.
version_type Specific version type: external, external_gte.
wait_for_active_shards: string The number of shard copies that must be active before proceeding with the operation. Set to all or any positive integer up to the total number of shards in the index (number_of_replicas+1). Default: 1, the primary shard. See Active Shards.

Examples

Delete a document:

requests:
  - id: delete_document
    type:  ElasticsearchDelete
    properties:
      id:
        _state: selected_id

ElasticsearchDeleteByQuery

The ElasticsearchDeleteByQuery request deletes documents that match the specified query.

Properties

body: object The Elasticsearch query body, expressed in the JSON Query DSL.
- query: object: The Elasticsearch query.
_source: string|string[]|boolean: True or false to return the _source field or not, or a list of fields to return.
_source_excludes: string|string[]: A list of fields to exclude from the returned _source field.
_source_includes: string|string[]: A list of fields to extract and return from the _source field.
allow_no_indices: boolean: Whether to ignore if a wildcard indices expression resolves into no concrete indices. (This includes _all string or when no indices have been specified).
max_docs: number Maximum number of documents to process. Defaults to all documents.
refresh: string|boolean If true, Elasticsearch refreshes the affected shards to make this operation visible to search, if wait_for then wait for a refresh to make this operation visible to search, if false do nothing with refreshes. Valid values: true, false, wait_for. Default: false.
requests_per_second: number The throttle for this request in sub-requests per second. Defaults to -1 (no throttle).
analyzer: string: The analyzer to use for the query string
default_operator: string: The default operator for query string query (AND or OR).
df: string: The field to use as default where no field prefix is given in the query string.
from: number: Starting offset.
size: number: Number of hits to return.
index: string|string[]: A comma-separated list of index names to search; use _all or empty string to perform the operation on all indices.
lenient: boolean: Specify whether format-based query failures (such as providing text to a numeric field) should be ignored.
scroll: string: Specify how long a consistent view of the index should be maintained for scrolled search.
sort: string|string[]: A comma-separated list of <field>:<direction> pairs.

The request accepts many more advanced configuration options. They will be passed to the Elasticsearch client verbatim, so check out the available options provided by Elastic.

Examples

Delete all comments by a user:

requests:
  - id: delete_multiple
    type:  ElasticsearchUpdate
    properties:
      body:
        query:
          term:
            user.id:
              _state: selected_user

ElasticsearchIndex

The ElasticsearchIndex request adds a JSON document to the specified data stream or index and makes it searchable. If the target is an index and the document already exists, the request updates the document and increments its version.

Properties

body: object The Elasticsearch request body contains the JSON source for the document data.
if_seq_no: number Only perform the operation if the document has this sequence number. See Optimistic concurrency control.
if_primary_term: number Only perform the operation if the document has this primary term. See Optimistic concurrency control.
op_type: 'create'|'index' Set to create to only index the document if it does not already exist (put if absent). If a document with the specified _id already exists, the indexing operation will fail. Same as using the <index>/_create endpoint. Valid values: index, create. If document id is specified, it defaults to index. Otherwise, it defaults to create.
pipeline: string ID of the pipeline to use to preprocess incoming documents.
refresh: string|boolean If true, Elasticsearch refreshes the affected shards to make this operation visible to search, if wait_for then wait for a refresh to make this operation visible to search, if false do nothing with refreshes. Valid values: true, false, wait_for. Default: false.
routing: string Custom value used to route operations to a specific shard.
timeout: string Period the request waits for the following operations:
Defaults to 1m (one minute). This guarantees Elasticsearch waits for at least the timeout before failing. The actual wait time could be longer, particularly when multiple waits occur.
version: integer Explicit version number for concurrency control. The specified version must match the current version of the document for the request to succeed.
version_type Specific version type: external, external_gte.
wait_for_active_shards: string The number of shard copies that must be active before proceeding with the operation. Set to all or any positive integer up to the total number of shards in the index (number_of_replicas+1). Default: 1, the primary shard. See Active Shards.
require_alias: boolean If true, the destination must be an index alias. Defaults to false.

Examples

Index a new document.

requests:
  - id: insert_new_comment
    type:  Elasticsearch
    connectionId: elasticsearch
    properties:
      body:
        comment:
          _state: comment_input
        user_id:
          _user: id
        timestamp:
          _date: now

ElasticsearchSearch

The ElasticsearchSearch request searches the Elasticsearch index.

Properties

body: object The Elasticsearch query body, expressed in the JSON Query DSL.
- query: object: The Elasticsearch query.
- aggs: object: The aggregations to request from the server.
_source: string|string[]|boolean: True or false to return the _source field or not, or a list of fields to return.
_source_excludes: string|string[]: A list of fields to exclude from the returned _source field.
_source_includes: string|string[]: A list of fields to extract and return from the _source field.
allow_no_indices: boolean: Whether to ignore if a wildcard indices expression resolves into no concrete indices. (This includes _all string or when no indices have been specified).
allow_partial_search_results: boolean: Indicate if an error should be returned if there is a partial search failure or timeout.
analyzer: string: The analyzer to use for the query string
default_operator: string: The default operator for query string query (AND or OR).
df: string: The field to use as default where no field prefix is given in the query string.
from: number: Starting offset.
size: number: Number of hits to return.
index: string|string[]: A comma-separated list of index names to search; use _all or empty string to perform the operation on all indices.
lenient: boolean: Specify whether format-based query failures (such as providing text to a numeric field) should be ignored.
scroll: string: Specify how long a consistent view of the index should be maintained for scrolled search.
sort: string|string[]: A comma-separated list of <field>:<direction> pairs.

The request accepts many more advanced configuration options. They will be passed to the Elasticsearch client verbatim, so check out the available options provided by Elastic.

Examples

Fetch the first 10 matching documents.

requests:
  - id: fetch_docs
    type:  Elasticsearch
    connectionId: elasticsearch
    properties:
      size: 10
      body:
        query:
          match_all: {}

Find a document with ID 42.

requests:
  - id: fetch_docs
    type:  Elasticsearch
    connectionId: elasticsearch
    properties:
      size: 1
      body:
        query:
          term:
            _id: 42

Fetch articles between yesterday and today, aggregate by author.

requests:
  - id: fetch_docs
    type:  Elasticsearch
    connectionId: elasticsearch
    properties:
      size: 1
      type: article
      body:
        query:
          range:
            timestamp:
              gte: now-1d/d
              lt: now/d
        aggs:
          author_aggregation:
            terms:
              field: author

ElasticsearchUpdate

The ElasticsearchUpdate request updates a document using a script or partial document.

Properties

id: string|number Unique identifier for the document to be updated.
body: object The Elasticsearch request body contains either a script or a doc property with fields to update.
if_seq_no: number Only perform the operation if the document has this sequence number. See Optimistic concurrency control.
if_primary_term: number Only perform the operation if the document has this primary term. See Optimistic concurrency control.
require_alias: boolean If true, the destination must be an index alias. Defaults to false.
refresh: string|boolean If true, Elasticsearch refreshes the affected shards to make this operation visible to search, if wait_for then wait for a refresh to make this operation visible to search, if false do nothing with refreshes. Valid values: true, false, wait_for. Default: false.
retry_on_conflict: boolean Specify how many times should the operation be retried when a conflict occurs. Default: 0.
routing: string Custom value used to route operations to a specific shard.
_source: string[]|boolean Set to false to disable source retrieval (default: true). You can also specify a comma-separated list of the fields you want to retrieve.
_source_excludes: string[] Specify the source fields you want to exclude.
_source_includes: string[] Specify the source fields you want to include.
timeout: string Period the request waits for the following operations:
- Dynamic mapping updates
- Waiting for active shards
Defaults to 1m (one minute). This guarantees Elasticsearch waits for at least the timeout before failing. The actual wait time could be longer, particularly when multiple waits occur.
wait_for_active_shards: string The number of shard copies that must be active before proceeding with the operation. Set to all or any positive integer up to the total number of shards in the index (number_of_replicas+1). Default: 1, the primary shard. See Active Shards.

Examples

Update a document:

requests:
  - id: update
    type:  ElasticsearchUpdate
    properties:
      id:
        _state: _id
      body:
        doc:
          _state: name

Increase the like counter using a script:

requests:
  - id: increase_like_counter
    type:  ElasticsearchUpdate
    properties:
      id:
        _state: _id
      body:
        script:
          source: |
            ctx._source.likes++

ElasticsearchUpdateByQuery

The ElasticsearchUpdateByQuery request updates documents that match the specified query. If no query is specified, performs an update on every document in the data stream or index without modifying the source, which is useful for picking up mapping changes.

Properties

body: object The Elasticsearch query body, expressed in the JSON Query DSL.
- query: object: The Elasticsearch query.
- script: object A script to update documents.
_source: string|string[]|boolean: True or false to return the _source field or not, or a list of fields to return.
_source_excludes: string|string[]: A list of fields to exclude from the returned _source field.
_source_includes: string|string[]: A list of fields to extract and return from the _source field.
allow_no_indices: boolean: Whether to ignore if a wildcard indices expression resolves into no concrete indices. (This includes _all string or when no indices have been specified).
max_docs: number Maximum number of documents to process. Defaults to all documents.
refresh: string|boolean If true, Elasticsearch refreshes the affected shards to make this operation visible to search, if wait_for then wait for a refresh to make this operation visible to search, if false do nothing with refreshes. Valid values: true, false, wait_for. Default: false.
requests_per_second: number The throttle for this request in sub-requests per second. Defaults to -1 (no throttle).
analyzer: string: The analyzer to use for the query string
default_operator: string: The default operator for query string query (AND or OR).
df: string: The field to use as default where no field prefix is given in the query string.
from: number: Starting offset.
size: number: Number of hits to return.
index: string|string[]: A comma-separated list of index names to search; use _all or empty string to perform the operation on all indices.
lenient: boolean: Specify whether format-based query failures (such as providing text to a numeric field) should be ignored.
scroll: string: Specify how long a consistent view of the index should be maintained for scrolled search.
sort: string|string[]: A comma-separated list of <field>:<direction> pairs.

The request accepts many more advanced configuration options. They will be passed to the Elasticsearch client verbatim, so check out the available options provided by Elastic.

Examples

Update all red cars:

requests:
  - id: update_red_cars
    type:  ElasticsearchUpdateByQuery
    properties:
      body:
        query:
          term:
            color:
              value: red
        script:
          source: 'ctx._source.out_of_stock=true'

Elasticsearch

Connections

Elasticsearch

Properties

Examples

Authenticated connection with reads and writes:

Unauthenticated connection with default index:

Requests

ElasticsearchDelete

Properties

Examples

Delete a document:

ElasticsearchDeleteByQuery

Properties

Examples

Delete all comments by a user:

ElasticsearchIndex

Properties

Examples

Index a new document.

ElasticsearchSearch

Properties

Examples

Fetch the first 10 matching documents.

Find a document with ID 42.

Fetch articles between yesterday and today, aggregate by author.

ElasticsearchUpdate

Properties

Examples

Update a document:

Increase the like counter using a script:

ElasticsearchUpdateByQuery

Properties

Examples

Update all red cars:

Subscribe to our newsletter