Introduction

Elastic, also known as Elasticsearch, is a powerful search and analytics engine that can be used to index, search, and analyze large volumes of data quickly and in near real-time. It is open-source and built on top of the Lucene library. In this blog post, we will go over the basics of Elastic and how to get started using it.

  1. Installation

    The first step to using Elastic is to install it on your system. You can download the latest version of Elastic from the official website (https://www.elastic.co/downloads/elasticsearch) and install it according to the instructions provided. Once installed, you can start the Elastic service on your machine and access it through a web interface at http://localhost:9200.

  2. Indexing Data

    In Elastic, data is stored in indices, which are similar to tables in a relational database. To index data, you need to create an index and then add documents to it. You can create an index using the Elastic API, for example by sending a PUT request to the following endpoint: http://localhost:9200/index_name. Once the index is created, you can add documents to it by sending a POST request to the same endpoint with your JSON data.

  3. Searching Data

    Once you have indexed your data, you can search for it using the Elastic API. The most basic search is a match query, which returns all documents that match a specified term. You can perform a match query by sending a GET request to the following endpoint: http://localhost:9200/index_name/_search?q=your_search_term. You can also use more advanced search options such as filtering, aggregation, and highlighting.

  4. Analyzing Data

    Elastic also provides a variety of tools for analyzing data, such as the ability to create visualizations and dashboards using Kibana. Kibana is a separate product from Elastic but it can be easily integrated with it. With Kibana, you can create charts, tables, and maps to visualize your data and gain insights from it.

Indexing Data

  1. Creating an Index

    The most basic way to create an index in Elasticsearch is to send a PUT request to the Elasticsearch API endpoint for the index you want to create. For example, to create an index named “customers”, you can use the following command:
    curl -XPUT http://localhost:9200/customers
    This will create an index with the default settings and mapping.

  2. Customizing Mappings

    When you create an index, Elasticsearch automatically creates a mapping for it, which defines how the data in the index should be treated. You can customize the mapping for an index by sending a PUT request with a JSON document that defines the mappings you want to use. For example, to create an index named “customers” with mappings for the fields “name”, “age”, and “address”, you can use the following command:
    curl -XPUT http://localhost:9200/customers -H 'Content-Type: application/json' -d '
{
     "mappings": {
         "properties": {
             "name": { "type": "text" },
             "age": { "type": "integer" },
             "address": { "type": "text" }
         }
     }
 }'

  3. Update Mappings

    If you have already created an index and want to update its mappings, you can use the _mapping endpoint and send a PUT request with the updated mappings. For example, to update the mappings for the “customers” index, you can use the following command:
    curl -XPUT http://localhost:9200/customers/_mapping -H 'Content-Type: application/json' -d '
{
     "properties": {
         "name": { "type": "text" },
         "age": { "type": "integer" },
         "address": { "type": "text" }
     }
 }'

  4. Deleting an Index

    If you no longer need an index, you can delete it by sending a DELETE request to the Elasticsearch API endpoint for the index. For example, to delete the “customers” index, you can use the following command:
    curl -XDELETE http://localhost:9200/customers

Searching Data

  1. The most basic search in Elasticsearch is the match query, which returns all documents that match a specified term. For example, to search for all documents in the “customers” index that have the term “John” in the “name” field, you can use the following command:
    curl -XGET http://localhost:9200/customers/_search -H 'Content-Type: application/json' -d '
{
     "query": {
         "match": {
             "name": "John"
         }
     }
 }'
  2. Elasticsearch also offers a wide range of search options to customize your search queries, such as filtering, aggregation, and highlighting. For example, to search for all documents in the “customers” index that have the term “John” in the “name” field and are over the age of 30, you can use the following command:
    curl -XGET http://localhost:9200/customers/_search -H 'Content-Type: application/json' -d '
{
     "query": {
         "bool": {
             "must": {
                 "match": {
                     "name": "John"
                 }
             },
             "filter": {
                 "range": {
                     "age": {
                         "gt": 30
                     }
                 }
             }
         }
     }
 }'

  3. Searching with Aggregations

    You can also perform data aggregations on your search results to gain insights into your data. For example, to search for all documents in the “customers” index and group them by the “age” field, you can use the following command:
    curl -XGET http://localhost:9200/customers/_search -H 'Content-Type: application/json' -d '
     {
         "query": {
             "match_all": {}
         },
         "aggs": {
             "group_by_age": {
                 "terms": {
                     "field": "age"
                 }
             }
         }
     }'

  4. Searching with Highlighting

    You can also use highlighting to highlight the matched search terms in the search results. For example, to search for all documents in the “customers” index that have the term “John” in the “name” field and highlight the matched term, you can use the following command:
    curl -XGET http://localhost:9200/customers/_search -H 'Content-Type: application/json' -d '
{
     "query": {
         "match": {
             "name": "John"
         }
     },
     "highlight": {
         "fields": {
             "name": {}
         }
     }
 }'

Aggregated Indexes

Sometimes you need to perform aggregation and never use the “raw data”. In this cases, creating an aggregated index could improve the performance. An aggregation is a way to group, filter or calculate the statistics of your data. There are different types of aggregations in Elasticsearch, such as terms, range, date histogram, and more. For example, you can use the term aggregation to group your data by a specific field, like age, or use range aggregation to filter your data based on a specific range of values.

  1. Creating Aggregated Indexes

    You can create an index and perform aggregations on it by sending a PUT request to the Elasticsearch API endpoint for the index you want to create, and including the aggregations in the request body. For example, to create an index named “sales” with an aggregation on the “price” field, you can use the following command:

    curl -XPUT http://localhost:9200/sales -H 'Content-Type: application/json' -d '
{
     "mappings": {
         "properties": {
             "price": { "type": "double" },
             "date": { "type": "date" }
         }
     },
     "aggs": {
         "avg_price": {
             "avg": {
                 "field": "price"
             }
         }
     }
 }'

  2. Using Aggregations

    Once you have created an aggregated index, you can use the Elasticsearch API to perform search queries on it and retrieve the aggregated results. For example, to retrieve the average price of all documents in the “sales” index, you can use the following command:

    curl -XGET http://localhost:9200/sales/_search -H 'Content-Type: application/json' -d '
{
     "aggs": {
         "avg_price": {
             "avg": {
                 "field": "price"
             }
         }
     }
 }'

  3. Combining Aggregations

    You can also combine multiple aggregations to gain more insights from your data. For example, you can use a terms aggregation to group your data by a specific field, like product type, and then use a range aggregation to filter your data based on a specific range of values.

     curl -XGET http://localhost:9200/sales/_search -H 'Content-Type: application/json' -d '
 {
     "aggs": {
         "product_type": {
             "terms": {
                 "field": "product_type"
             },
             "aggs": {
                 "price_range": {
                     "range": {
                         "field": "price",
                         "ranges": [
                             { "to": 10 },
                             { "from": 10, "to": 20 },
                             { "from": 20 }
                         ]
                     }
                 }
             }
         }
     }
 }'

    In this example, the query is grouping the documents by “product_type” field and then filters them by price range. This way you can have a better understanding of how different product types are performing based on the price range.

    Another example could be:

    curl -XGET http://localhost:9200/sales/_search -H 'Content-Type: application/json' -d '
{
    "aggs": {
        "by_customer_id": {
            "terms": {
                "field": "customer_id"
            },
            "aggs": {
                "avg_price": {
                    "avg": {
                        "field": "price"
                    }
                }
            }
        }
    }
}'

    This way you can have a better understanding of how different customers are performing based on the average price of their purchases.

In conclusion, Elastic is a powerful tool for indexing, searching, and analyzing large volumes of data. It is easy to get started with, and the possibilities are endless with the wide range of features it offers. This is just the tip of the iceberg, and there is a lot more to explore. Give Elastic a try and see how it can help you with your big data needs."