Creating a Custom Elasticsearch Beat

The Beats documentation, specifically the Beats Developer Guide, is not in great shape. If you want to write your own custom Beat and follow the docs, you won’t be able to create a new custom Beat. A lot of questions are being posted on the Elastic Discuss forum by people struggling to get started.

As well as the documentation being incorrect in places, the prerequisites are in different areas and some steps are missing altogether.

I have opened pull requests to fix some of the problems but they’re not getting any traction so it doesn’t appear that the situation will improve any time soon.

In the meantime, if you’re looking to create a custom Elasticsearch Beat, I’ll describe what you need to do to get the base code generated.

I have tested these steps on macOS (Catalina), CentOS (in a Vagrant VM) and an AWS EC2 instance running Amazon Linux 2. I can’t test the process on Windows, unfortunately.

Install exactly Go 1.13.10

The documentation states that 1.13.10 is the minimum version required but it’s actually the exact version required. Running 1.14 will result in errors during the code generation:

There is an open issue to add support for 1.14 but it’ll be a while before it makes it into master.

Simply follow the guide on the Golang website; not all package managers will have 1.13.10 available.

Your PATH needs to be updated after installing but we’ll do that in a later step.

Install gcc

gcc is required by the Beats build system. Depending on how you installed Go, you may not have it installed.

Your favourite package manager will be able to locate, download and install it for you.

Configure environment

Create the GOPATH environment variable. This is usually ~/go/.

export GOPATH=$HOME/go

Add the $GOPATH/bin directory to your $PATH. Some Go packages - including Mage, which we’ll install shortly - will add binaries to this folder and we want to be able to run them from the terminal. While we’re here, we’ll also add Go to the $PATH; I’m assuming it’s installed in the default /usr/local/go/.

export PATH=$PATH:/usr/local/go/bin:$GOPATH/bin

Install Git

You will likely have this installed already but, for the sake of completeness, it is a required step.

sudo yum install git

Install Mage

This step isn’t mentioned in the documentation but it’s absolutely required.

Mage is used to build Beats. It’s also used to run the code generator that builds the base of our new Beat.

mkdir -p ${GOPATH}/src/github.com/magefile && cd $_
git clone https://github.com/magefile/mage
cd mage
go run bootstrap.go

Install Python 3

If you don’t have Python 3 installed, go ahead and install that. While there’s ongoing work to remove the dependency on Python, parts of the Beats platform still use it for various tasks.

Check which version you have as the default by running python --version. If that shows Python 2, also try python3 --version. If python3 isn’t installed, you’ll need to install it. You can use yum, homebrew, or apt for that.

sudo yum install python3

Create your path

You will need your own directory in your GOPATH. Assuming your GitHub username is beatsAuthor, you’d run:

mkdir ${GOPATH}/src/github.com/beatsauthor

Clone the Beats repository

Create a directory in your $GOPATH for the Beats repository:

mkdir ${GOPATH}/src/github.com/elastic && cd $_
git clone https://github.com/elastic/beats
cd beats

Create your Beat

This is another spot where the documentation is wrong; it tells you to cd into your beatsauthor directory and run the next command from there. Doing so will give you the following error:

No .go files marked with the mage build tag in this directory.

You need to stay in the $GOPATH/src/github.com/elastic/beats directory, then run mage GenerateCustomBeat:

[ec2-user@ip-111-111-111-111 beats]$ mage GenerateCustomBeat
Enter the beat name [examplebeat]: firstbeat
Enter your github name [your-github-name]: beatsauthor
Enter the beat path [github.com/beatsauthor/firstbeat]: 
Enter your full name [Firstname Lastname]: Beats Author
Enter the beat type [beat]: 
Enter the github.com/elastic/beats revision [master]: 
go: creating new go.mod: module github.com/beatsauthor/firstbeat
go: finding github.com master
go: finding github.com/elastic/beats/v7 master
go: finding github.com/elastic/beats master
go: finding github.com/elastic master
go: finding golang.org/x/tools latest
...
...
=======================
Your custom beat is now available as /home/ec2-user/go/src/github.com/beatsauthor/firstbeat
=======================

Follow the prompts and the base code for your new Beat will be generated.

Generate dependencies

Now cd into your new firstbeat directory.

cd $GOPATH/src/github.com/beatsauthor/firstbeat

Another error with the documentation is that says to run make setup; doing so will error with the following message:

make: *** No rule to make target `setup'.  Stop.

Instead, run:

make update

This generates the Beat config and fields files.

Build your Beat

Now you’re ready to build the default Beat:

mage build

Learn Go!

You’re now in a position to flesh out the Beat and set it to work. You can fetch data from anywhere you need to, modify it as required, and index it in Elasticsearch.

If you’re not familiar with Go, this is a great learning exercise. I hadn’t used it prior to writing my first Beat but there are a lot of resources available to help you get to grips with it.

One big frustration I had was serialising and deserialising JSON. A strongly typed language requires more groundwork than Python or Javascript. There are plenty of other reasons to really like Go but the main draw is likely the reason Go was chosen as the language for Beats; it allows you to write highly concurrent code.