I’ve heard of Elasticsearch quite a while ago but started using around 6-7 months ago. It’s a very fine piece of technology, allowing you to create blazingly fast queries with very complex structure. Comming from a SQL background, I spent quality time reading through the official docs to write even the most basic queries. The purpose of this article is to save you that time and get you straight to work.
I planned writing this article a long time ago, never finding time or motivation to do it. Additionally, in this time span, I haven’t worked much with Elastic. Still, I believe it’s quite needed as I couldn’t find something similar while building the queries for our application.
Queries in Elastic differ quite a lot from standard (No)SQL ones. Even though the end result is pretty much the same (get all from table A, order by column B,C desc where column D like ‘E’) and you could easily draw a line between those two, the complexity of queries in Elastic is way much higher compared to SQL ones.
Our needs weren’t too complex, so the examples below will cover the most basic scenarios only. I’m willing to accept both requests (how to write a query performing a single task) and responses (code samples) in order to further improve this post.
To perform queries on our Elastic server, we used Olivere’s amazing Elastic library (version 5), although most of the queries below should work on other versions too.
MultiMatchQuery - Querying by multiple fields
Querying multiple fields (column) by a single term, using ‘like’
SELECT * FROM COMPANIES WHERE NAME LIKE %TERM%
OR ABBREVIATION LIKE %TERM%func (ci *CompanyIndex) SeachCompaniesByName(ctx context.Context, term string) ([]model.Company, error) {
var companies []model.Company
q := elastic.NewMultiMatchQuery(term, "name", "abbreviation").Type("phrase_prefix")
index := path[os.Getenv("DATASTORE\_PROJECT\_ID")] // dev/prod taken from a map
searchResult, err := ci.client.Search().
Index(index). // name of Index (dev / prod)
Type("company"). // type of Index
Query(q).
Do(ctx)
if err != nil {
return nil, err
}
for _, hit := range searchResult.Hits.Hits {
var cmp model.Company
err := json.Unmarshal(*hit.Source, &cmp)
if err != nil {
return nil, err
}
companies = append(companies, cmp)
}
return companies, nil
}MatchQuery & BoolQuery
Match query allows you to do exact queries, e.g. (id = $key). In this example, I used company_id extracted from user’s session, to query only data from his company.
Bool(ean)Query serves as an ‘AND’ (or AND NOT), joining two or more queries together with an AND (NOT).
Offset and Limit are used for pagination purposes.
SELECT * FROM COMPANIES WHERE ID = $KEY AND
(NAME LIKE %TERM% OR ADDRESS LIKE %TERM% OR LOCATION LIKE %TERM% OR EMAIL LIKE %TERM% OR PHONE_NUMBER LIKE %TERM% OR PLACE LIKE %TERM%)
LIMIT $LIMIT OFFSET $OFFSETfunc (ci *CompanyIndex) SearchCompanies(c context.Context, key int64, term string, offset, limit int) ([]model.Company, error) {
var cmps []model.Company
index := path[os.Getenv("DATASTORE\_PROJECT\_ID")]
multiQuery := elastic.NewMultiMatchQuery(
term,
"name", "address", "location", "email", "phone_number", "place", "postcode",
).Type("phrase_prefix")
matchQuery := elastic.NewMatchQuery("id", key)
query := elastic.NewBoolQuery().Must(multiQuery, matchQuery)
searchResult, err := ci.client.Search().
Index(index).
Type("company"). // search in type
Query(query).
From(offset). // Starting from this result
Size(limit). // Limit of responds
Do(c) // execute
if err != nil {
return nil, err
}
for _, hit := range searchResult.Hits.Hits {
var cmp model.Company
err := json.Unmarshal(*hit.Source, &cmp)
if err != nil {
return nil, err
}
cmps = append(cmps, cmp)
}
return cmps, nil
}Since all of the query functions are returning pointers, you can easily include only non-empty parameters in query:
func (ci *CustomerIndexer) SearchCustomers(c context.Context, age, departmentID, limit, offset int, email, gender string) ([]model.Customer, error) {
var csts []model.Customer
q := elastic.NewMatchQuery("departmentId", departmentID)
bq := elastic.NewBoolQuery().Must(q)
if age > 0 {
bq.Must(elastic.NewMatchQuery("age", age))
}
if email != "" {
bq.Must(elastic.NewMatchQuery("email", email))
}
if gender != "" {
bq.Must(elastic.NewMatchQuery("gender", gender))
}
sr, err := t.cl.Search().
Index(getEnv()).
Type("customer"). // search in type
Query(bq).
From(offset). // Starting from this result
Size(limit). // Limit of responds
Do(c) // execute
if err != nil {
return nil, err
}
for _, hit := range sr.Hits.Hits {
var cst model.Customer
err := json.Unmarshal(*hit.Source, &cst)
if err != nil {
return nil, err
}
csts = append(csts, cst)
}
return csts, nil
}And since all query functions implement the Query interface, you can easily substitute them. We used the query below to return search through all companies if the user making the query has admin role (replacing the key with -1):
func (ci *CompanyIndex) SearchCompanies(c context.Context, key int64, term string, offset, limit int) ([]model.Company, error) {
var cmps []model.Company
index := path[os.Getenv("DATASTORE\_PROJECT\_ID")]
multiQuery := elastic.NewMultiMatchQuery(
term,
"name", "address", "location", "email", "phone_number", "place", "postcode",
).Type("phrase_prefix")
matchQuery := elastic.NewMatchQuery("id", key)
query := elastic.NewBoolQuery().Must(multiQuery, matchQuery)
searchResult, err := ci.client.Search().
Index(index).
Type("company"). // search in type
Query(query).
From(offset). // Starting from this result
Size(limit). // Limit of responds
Do(c) // execute
if err != nil {
return nil, err
}
for _, hit := range searchResult.Hits.Hits {
var cmp model.Company
err := json.Unmarshal(*hit.Source, &cmp)
if err != nil {
return nil, err
}
cmps = append(cmps, cmp)
}
return cmps, nil
}
func getQuery(key int64, bq *elastic.BoolQuery, mq *elastic.MultiMatchQuery) elastic.Query {
if key == -1 {
return mq
}
return bq
}Aggregation - Distinct
Using TermsAggregation lets you create a query returning distinct values only:
SELECT DISTINCT(DEPARTMENT) FROM CUSTOMERS
ORDER BY DEPARTMENT ASC
LIMIT $LIMIT OFFSET $OFFSETfunc (ci *CustomerIndexer) GetDistinctDepartments(c context.Context, offset, limit int) ([]string, error) {
var deps []string
sr, err := ci.cl.Search().
Aggregation("departments", elastic.NewTermsAggregation().Field("department").OrderByTermAsc()).
Index("dev").
Type("customer"). // search in type
From(offset). // Starting from this result
Size(limit). // Limit of responds
Do(c) // execute
if err != nil {
return nil, err
}
departments, found := sr.Aggregations.Terms("departments")
if found {
for _, b := range departments.Buckets {
deps = append(deps, b.Key.(string))
}
}
return deps, nil
}These are some of the most common queries you may need to use when querying Elastic. As mentioned earlier, our requirements from Elastic weren’t too complex so far, and these queries satisfied our needs. I’d gladly accept more complex examples from others as well as questions on how to build certain queries.