Serverless semantic search with ELSER in Python: Exploring Summer Olympic games history

Want to get Elastic certified? Find out when the next Elasticsearch Engineer training is running!

Elasticsearch is packed with new features to help you build the best search solutions for your use case. Dive into our sample notebooks to learn more, start a free cloud trial, or try Elastic on your local machine now.

This blog shows how to fetch information from an Elasticsearch index, in a natural language expression, using semantic search. We will create a serverless Elasticsearch project, load previous olympic games data set into an index, generate inferred data (in a sparse vector field) using the inference processor along with ELSER model, and finally search for historical olympic competition information in a natural language expression, thanks to text expansion query.

The tools and the data set

For this project we will use an Elasticsearch serverless project, and the serverless Python client (elasticsearch_serverless) for interactions with Elasticsearch. To create a serverless project, simply follow the get started with serverless guide. More information on serverless including pricing can be found here.

When setting up a serverless project, be sure to select the option for Elasticsearch and the general purpose option for working this tutorial.

The data set used is that of summer olympic games competitors from 1896 to 2020, obtained from Kaggle (Athletes_summer_games.csv). It contains information about the competition year, the type of competition, the name of the participant, whether they won a medal or not and which medal eventually, along with other information.

For the data set manipulation, we will use Eland, a Python client and toolkit for DataFrames and machine learning in Elasticsearch.

Finally the natural language processing (NLP) model used is Elastic Learned Sparse EncodeR (ELSER), a retrieval model trained by Elastic that allows to retrieve more relevant search results through semantic search.

Before following the steps below, please make sure you have installed the serverless Python client and Eland.

Please note the versions I used below. If you are not using the same versions, you might need to adjust the code to any eventual syntax change in the versions you are using.

Download and deploy ELSER model

We will use the Python client to download and deploy the ELSER model. Before doing that, let's first confirm that we can connect to our serverless project. The URL and API key below are read from environment variables; you need to use the appropriate values in your case, or use whichever method you prefer for reading credentials.

If everything is properly configured, you should get an output like below:

Now that we've confirmed that the Python client is successfully connecting to the serverless Elasticsearch project, let’s download and deploy the ELSER model. We will check if the model was previously deployed and delete it in order to perform a fresh install. Also, as the deploy phase could take a few minutes, we will continuously check the model configuration information to make sure that the model definition is present before moving to the next phase. For more information check the Get trained models API.

Once we get the confirmation that the model is downloaded and ready to be deployed, we can go ahead and start ELSER. It can take a little while to fully be ready to be deployed.

Load the data set into Elasticsearch using Eland

eland.csv_to_eland allows reading a comma-separated values (csv) file into a data frame stored in an Elasticsearch index. We will use it to load the Olympics data (Athletes_summer_games.csv) into Elasticsearch. The es_type_overrides allows to override default mappings.

After executing the lines above, the data will be written in the index elser-olympic-games. You can also retrieve the resulting dataframe (eland.DataFrame) into a variable for further manipulations.

Create an ingest pipeline for inference based on ELSER

The next step in our journey to explore past Olympic competition data using semantic search is to create an ingest pipeline containing an inference processor that runs the ELSER model. A set of fields has been selected and concatenated into a single field on which the inference processor will work. Depending on your use case, you might want to use another strategy.

The concatenation is done using the script processor. The inference processor uses the previously deployed ELSER model, taking as input the concatenated field, and storing the output in a sparse vector type field (see following point).

Preparing the index

This is the last stage before being able to query past Olympic competition data using natural language expressions. We will update the previously created index’s mapping adding a sparse vector type field.

Update the mapping: add a sparse vector field

We will update the index mapping by adding a field that will hold the concatenated data, and a sparse vector field that will hold the inferred information computed by the inference processor using the ELSER model.

Populate the sparse vector field

We will run an update by query to call the previously created ingest pipeline in order to populate the sparse vector field in each document.

The request will take a few moments depending on the number of documents, and the number of allocations and threads per allocation used for deploying ELSER. Once this step is completed, we can now start exploring past olympic data set using semantic search.

Let's explore the Olympic data set using semantic search

Now we will use text expansion queries to retrieve information about past Olympic game competitions using natural language expressions. Before going to the demonstration, let's create a function to retrieve and format the search results.

The function above will receive a question about past Olympic games competition winners, performing a semantic search using Elastic’s text expansion query. The retrieved results are formatted and printed. Notice that we force the existence of medals in the query, as we are only interested in the winners. We also limited the size of the result to 3 as we expect three winners (gold, silver, bronze). Again, based on your use case, you might not necessarily do the same thing.

🏌️‍♂️ “Who won the Golf competition in 1900?”

Request:

Output:

🏃‍♀️ “2004 Women's Marathon winners”

Request:

Output:

🏹 “Women archery winners of 1908”

Request:

Output:

🚴‍♂️ “Who won the individual cycling competition in 1972?”

Request:

Output:

Conclusion

This blog showed how you can perform semantic search with the Elastic Learned Sparse EncodeR (ELSER) NLP model, in Python programming language using Serverless. You will want to make sure you turn off severless after running this tutorial to avoid any extra charges. To go further, feel free to check out our Elasticsearch Relevance Engine (ESRE) Engineer course where you can learn how to leverage the Elasticsearch Relevance Engine (ESRE) and large language models (LLMs) to build advanced RAG (Retrieval-Augmented Generation) applications that combine the storage, processing, and search features of Elasticsearch with the generative power of an LLM.

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.

Report an issue

Serverless semantic search with ELSER in Python: Exploring Summer Olympic games history

The tools and the data set

Download and deploy ELSER model

Load the data set into Elasticsearch using Eland

Create an ingest pipeline for inference based on ELSER

Preparing the index