Monitor smart cities with Grafana, Timescale and Sentilo

Integrating open source into Sentilo: Improvements for smart cities

In the company, we build and maintain Sentilo, an open-source sensor and actuator platform for smart cities. As part of that project, we are always looking for ways to integrate open-source tools that can save time and money while increasing the efficiency of smart cities.

As if it were a puzzle, we strive to incorporate the "Open source" tool that best fits each use case as new needs arise. Initially, we developed Sentilo in 2012 as a small IoT project in collaboration with the Barcelona City Council and today, we have a mature product supported by a diverse community of public and private entities. As a reference, the Barcelona Sentilo instance manages more than 7.5 million messages per day. SEIDOR maintains Sentilo for several clients, both through "on-premises" deployments as well as SaaS service through its spin-off Thingtia. Last but not least, we lead the Sentilo Community with the financial support of the Barcelona City Council.

As part of that growth, and with the aim of expanding the set of modules that support the platform, we recently incorporated Grafana and TimescaleDB, which have become important pieces in our smart city solution. In this article, I will explain why a client request led us to create new customizable visualizations with Grafana that helped improve a variety of processes in smart cities.

Integration of Grafana and Timescale

One of the main objectives of Sentilo is to deliver information to its subscribed modules as quickly as possible, these modules are usually other external systems or data sources, such as an energy monitoring system or a corporate data lake. To achieve this, the data collected by the sensors of the smart city (from vehicles, electrical panels, and more) is stored in memory, which gives us high performance... but it is important to keep in mind that memory is expensive. This data repository is only temporary; we needed long-term storage for historical data.

For some of our Sentilo implementations, users have their own data lakes, so we have a set of connectors for different target repositories, such as Kafka, Elasticsearch, and the most common relational databases in the market. However, this type of configuration does not work for all scenarios due to their volume and/or cost. For this reason, when one of our clients asked us to create their historical repository from scratch and integrate it into their Sentilo platform, we had to evaluate several solutions to provide the appropriate solution for the use case.

We needed something to help us process information in real time and allow us to explore and discover historical data through an interactive visual web-based tool. We had some previous experience using Grafana for system monitoring and were familiar with its excellent features and the multiple integrable data sources. Therefore, after analyzing other market options and conducting several PoCs, we decided to opt for the following combination:

• Timescale. This open-source time-series database, which is built on PostgreSQL, provides high performance and scalability even when working with large volumes of data. It also offers a familiar query language like SQL.
• Grafana. This open-source visualization tool has a great set of powerful panels. Additionally, it provides interesting features for our use cases such as allowing content embedding, multi-user support, and dashboard sharing.

As can be seen in the architecture diagram, our smart city platform is based on a combination of databases, sensors, connectors, and tools. Here is a brief description of how these pieces interact:

• Sentilo **stores data** from sensors in two different databases: Redis for real-time data and MongoDB for sensor catalog metadata.
• The Sentilo API **provides services** to interact with the catalog and to publish real-time data (1). Subsequently, the data is stored in the corresponding repository (2).
• A custom Sentilo connector, based on a common agent architecture that receives the data stream from Redis, **receives and stores the information** in TimescaleDB (3).
• **Sensor metadata is periodically synchronized** with the Timescale database using another custom module (4).
• Sentilo provides a web interface for various administrative purposes (5). Administrators can **manage their catalog and visualize real-time data**. And through Grafana, users can explore the historical database and create specific dashboards (6).
• Sentilo also includes a web interface focused on citizenship, where any citizen can **explore the city map, visualize sensors and their publicly accessible data**.

Through Grafana, we can offer richer and more interactive visualizations tailored to specific sectors or typologies that can be seamlessly integrated into the public platform website.

This solution has been implemented for Sentilo instances of the Barcelona Smart Region and for the City Council of Eivissa (Spain) through Thingtia. It provides an interactive framework for searching and discovering large volumes of data (millions) and broad time frames (years) with optimal performance and response time.

The Grafana panels are integrated into the standard Sentilo public viewer and are linked to different sectors: noise monitoring, air quality, irrigation, water sources, and energy monitoring. Our clients can create their own powerful dashboards autonomously and share them with other members of the Sentilo community.

Using Grafana to extract value from Big Data

Of course, it is always important to keep in mind that when working with large volumes of data and applying filters, you must optimize the queries, add indexes, or limit the data that the panel can handle, especially if those functions are open to the public. This is crucial if you want to keep costs low and provide quick results when running your queries.

Nevertheless, the solution has proven to be a good approach for working with large amounts of data. It has helped us explore and discover data, and we can extract valuable information by combining different sources. It is also a great way to create powerful dashboards that can be easily integrated into other city websites.

If you would like more information on how Sentilo works or how you can incorporate it into your own smart city plans, you can visit the Sentilo community. You can download the code for free, use it as SaaS, or contact us to build a custom solution.