Update: July 2020: I am now building lab exercises to help people prepare for the exam. You can see the list of available labs here
I find lab-based exercises to be the best way to cement information as knowledge. While I am going back and forth on how to deliver training material, I wanted to publish a draft of some Elasticsearch exercises I have put together and solicit some feedback.
I’m not releasing data at this stage but would really appreciate some feedback on the format and writing style. Would a series of blog posts be suitable, or is a PDF a better approach?
The environment used here is similar to that of the Certification but the exercises aren’t meant to be part of a mock Certification exam; they’re for someone learning Elasticsearch but aren’t a million miles from the type of questions in the exam. I also don’t want to spoon-feed which API to use in each scenario, as would happen in a traditional training course. I’m trying to strike a balance between the two.
You will need a machine capable of running a small, single-node Elasticsearch v7.2 cluster and Kibana. A modern laptop with 8 or 16GB of RAM should suffice for our purposes. Later exercises will require a multi-node cluster and we will discuss how best to create a suitable lab environment nearer the time, or read how to do this using Vagrant.
The environment used in the Certification exam will be reproduced as closely as possible, so Elasticsearch will need be run directly on a host, virtual machine or cloud instance where you can access the shell directly; ideally over SSH. Elasticsearch and Kibana distributions (extracted from the
.zip archive) are required on the node, as well as the data files used in these exercises.
All REST calls to Elasticsearch in these exercises will assume that Elasticsearch is running on
localhost. You will need to modify those addresses with the host of your cluster if it is different.
- Creating indices
- Defining mappings
- Ingest pipelines
- Delete by query
Configure Elasticsearch with the following criteria and start Elasticsearch:
The Bulk API may be covered in the exam but, for now, we’re only going to use it to get data into the cluster. There will be exercises later to ensure you can craft a suitable
_bulk request body.
The volume of data being passed to the Bulk API here is far more than you would normally post in one batch. A more efficient and mechanically sympathetic strategy would be to split the file into batches and post each batch individually. I am sacrificing more than efficiency for the sake of platform portability here.
olympic-events.ndjson file contains all the data for these exercises, formatted for use with the Bulk API. The file contains 271116 documents. Run the following command from the same location as the
ndjson file to import this data into a new index called
curl -X POST 'http://localhost:9200/olympic-events/_bulk' -H "Content-Type: application/x-ndjson" --data-binary @olympic-events.ndjson > /dev/null
Validate that the data was imported correctly by using a single API call to show the index name, index health, number of documents, and the size of the primary store. The details in the response must be in that order, with headers, and for the new index only.
The cluster health is yellow. Use a cluster API that can explain the problem.
Change the cluster or index settings as required to get the cluster to a green status.
Look at how Elasticsearch has applied very general-purpose mappings to the data. Why has it chosen to use a
text type for the
Age field? Find all unique values for the
Age field; there are less than 100 unique values for the
Age field. Look for any suspicious values.
We will be deleting data in the next exercise; making a backup is always prudent. Without making any changes to the data, reindex the
olympic-events index into a new index called
Weight fields suffer from the same problem as the
Age field. Later exercises will require numeric-type queries for these fields so we want to exclude any document we can’t use in our analyses. In a single request, delete all documents from the
olympic-events index that have a value of
NA for either the
Notice how the
Games field contains both the Olympic year and season. Create an ingest pipeline called
split_games that will split this field into two new fields -
season - and remove the original
Ensure your new pipeline is working correctly by simulating it with these values:
We’ll now start to clean up the mappings. Create a new index called
olympic-events-fixed with 1 shard, 0 replicas, and the following mapping: