Mobile devices and IoT devices with GPS features have been widely used in recent years. GPS technology has been applied in various functions like real...

by Jonah Wang, Solution Architect Alibaba Cloud Indonesia

Introduction

Mobile devices and IoT devices with GPS features have been widely used in recent years. GPS technology has been applied in various functions like real time information processing, historical location data in transportation apps, and mobility apps.

This article explains the architecture design and solutions to develop a GPS Info Processing system to collect, store and query GPS data efficiently on Alibaba Cloud.

Overview

Designing a GPS Info Processing system involves several important considerations:
1) Data Collection, including appropriate GPS devices or client application for real time information processing, and a data ingestion pipeline to receive data from these devices;
2) Data Storage, a database or storage solution optimized for storing GPS data efficiently, and proper data schema and partition & indexing design to optimize the query performance;
3) Data Processing, to implement a stream processing framework to handle incoming data in real-time and an data integration process to standardize and aggregate the data, for storage and analytics;
4) Data Access, to provide the APIs or query interfaces to retrieve GPS data for location-based queries and geographic visualization;

The architecture diagram below shows the design and solution components of the system we are going to demonstrate in the article.

Here is an overview of the key components and their functions in the design:

1) Data Collection: GPS client (OwnTracks), Data ingestion pipeline (IoT Platform with MQTT);
2) Data Storage: ApsaraDB for MongoDB;
3) Data Processing: Data forwarder (IoT Platform rule engine), Message Queue (RabbitMQ), data integration (Function Compute);
4) Data Access: API (Function Compute), Geo view (Grafana)

In the sections below, we will explain the functionalities and setup details of each components.

Data Collection

GPS Client (OwnTracks)

OwnTracks is a free mobile app allows you to keep track of your own location from your mobile phone and upload the location information to the servers. To see the details of app and download the app, please check the URL: https://owntracks.org/

OwnTracks support location data uploading through HTTP and MQTT protocols. In this article, we will use OwnTracks to collect the location data of mobile phone to simulate GPS Info Processing.

IoT Platform

On Alibaba Cloud, you can setup MQTT platform using open source solutions (e.g. ThingsBoard) or cloud-native solution, which is Alibaba Cloud IoT Platform, the details of setting up the platform on Alibaba Cloud can be found in the link: https://www.alibabacloud.com/blog/600293

For different IoT Platforms, the communication configuration of GPS client will be slightly different.

Communicate using ThingsBoard

In this section, we will demonstrate how to connect OwnTracks app to ThingsBoard Platform. OwnTracks will leverage on MQTT basic authentication to authenticate the device endpoints.

• Device Configuration
To configure the MQTT connection from the mobile phone with OwnTracks installed, you can go to OwnTracks set Preference->Configuration management, using the parameters below:

Annotation_2023_09_25_101423

• IoT Platform Setup
Set up the ThingsBoard IoT Platform based on the instruction from https://www.alibabacloud.com/blog/600293, and make sure the topic setup on the ThingsBoard is following the format of **owntracks/[username]/[deviceId]/waypoints
**

• Data Forwarding & Message Queue
By default, ThingsBoard will keep the collected data to the local log file and PostgreSQL database. In order to forward the data out to get integrated with external systems, we will use a message queue to store and process the incoming data.

We can add a RabbitMQ (External -> rabbitmq) in the Rule Chain of ThingsBoard, after the save timeseries node, see image below:

• GPS data upload
After setting up everything, you can turn on OwnTracks, go to home page, and enable “Move” as the “Monitoring Mode” to start to upload the GPS data.

Communicate using Alibaba Cloud IoT Platform

To connect OwnTracks app to Alibaba Cloud IoT Platform, the connection code and configuration is almost the same, except some of the parameters will be replaced by attributes of IoT Platform:

• Device Configuration (Basic Authentication)

Annotation_2023_09_25_101857

In this demonstration, we will create a custom topic “gps” on IoT Platform for the data collection. This topic will be used in data collection and data forwarding. To create custom topic, you can refer to the link below:
https://www.alibabacloud.com/help/en/iot/user-guide/use-custom-topics

• Data forwarding
The data forwarding configuration of IoT Platform will be simpler. You can forward the collected data to Message Queue, Function Compute, or external databases directly from the IoT Platform. In this demonstration, we will configure the data forwarding to forward the data to Alibaba Cloud Function Compute for further processing, the user guide can be found in the link below:
https://www.alibabacloud.com/help/en/iot/user-guide/forward-data-to-function-compute-1

When you are configuring the data source of the data forwarding rule, you can configure as “/[product_key]/+/user/gps” to make the forwarding rule applicable for all devices under the same product.

After that, you can follow the same way as the previous section to upload the GPS data.

Data Storage

The data collected from IoT devices are classified as time series data. In Alibaba Cloud, there are various cloud-native databases to keep the time series data, such as Lindorm, Tablestore, and MongoDB. In this demonstration, we will use ApsaraDB for MongoDB to store the GPS data collected.

• ApsaraDB for MongoDB
ApsaraDB for MongoDB is a MongoDB-compatible database service that is developed based on the distributed system and high-reliability storage engine of Apsara. You can refer to the link below to create a MongoDB instance on Alibaba Cloud:
https://www.alibabacloud.com/help/en/mongodb/getting-started/quick-start-2

After creating the MongoDB instance, we will use the schema below to store the GPS time series data.

Field Name	Data Type	Description
_id	String	The object id of the aggregated GPS data, in the format of *[deviceid-timestamp]*, where the timestamp is aggregated in the group of 10^6 seconds. E.g. phone1-loc-1684 is referring to the data collection between Sat May 13 2023 17:46:40 GMT+0000 and Thu May 25 2023 07:33:20 GMT+0000
data	Object	The aggregated GPS data table, as a list of *[timestamp]:* *[location]*
data.[timestamp]	Object	The location data collected at the timestamp, in the format of *{lat: [latitude], lon: [longitude]}*

The sample data:

To elaborate the data schema using Mongo syntax:

{"_id": "phone1-loc-1684"}, {"$set": {"data.1684129199": {"lat": -6.2015, "lon": 106.8195}}}

Data Processing

In previous sections, we have completed the data collection of GPS data from devices and real-time forward the raw data from the IoT Platform to backend services. Now, we can start to implement the data integration process to parse and process the raw data.

The data integration process will be handled by Function Compute. Function Compute is a Function-as-a-Service, it is serverless, event-driven, and fully managed cloud service on Alibaba Cloud. To use Function Compute, you need to only upload your code, and execute the applications without worrying about the deployment and server resources. Function Compute allows your application to integrate with various Alibaba Cloud product and services.

In this demonstration, if you are using ThingsBoard for data collection, you can use Function Compute to listen to the RabbitMQ for data integration, while if you are using IoT Platform for data collection, the data can be directly forwarded to Function Compute for processing.

In Function Compute, we will write the script to process the raw data and convert into standard format and save into MongoDB.

• MongoDB Client
To save the time series data to MongoDB in Function Compute, you have to install the MongoDB client on Function Compute environment. In this demonstration, we will use python 3.7 code to be executed in the Function Compute:

1) Go to Functions -> Function Details, click the tab of “Code”;
2) In the “Code” panel, click “Terminal” -> “New Terminal”, run command to install the library under the folder of the Function Compute code:

pip install -t . pymongo

You can refer to the link for the sample code to insert data into MongoDB using Python:
https://www.w3schools.com/python/python_mongodb_insert.asp

• Event Triggered Function Compute for Data Integration
There are 2 types of Functions in Alibaba Cloud Function Compute: event triggered and HTTP triggered. In the data integration process, the function will be triggered by data collection from IoT Platform, so we will implement an event triggered function to listen to the event passed from IoT Platform, process the data and store the time series data into the MongoDB. The sample code of python to be written in Function Compute to extract GPS data from the raw data collected from client (OwnTracks):

data = request.stream.read().decode()
input = json.loads(data) 
## latitude: input['lat'], longitude: input['lon'], velocity: input['vel'], timestamp: input['tst']
value = {'lat': input['lat'], 'lon': input['lon']}
timestamp = datetime.datetime.fromtimestamp(int(input['tst']))

Data Access

We can use the other type of Functions in Function Compute - HTTP Triggered, to develop RESTful API for Data Access. The instructions of developing a HTTP handlers Function can be found here: https://www.alibabacloud.com/help/en/fc/http-handlers-1

You can use the sample code below to deploy RESTful to retrieve the latest GPS data from MongoDB using Function Compute.

def read_latest(asset_oid):
    doc = col.find_one({"_id": {"$lt": asset_oid + "-900000000000"}}, sort=[("_id", pymongo.DESCENDING)])
return to_plot_data(doc)

def to_plot_data(tsdb_json):
    tsdata = []
    data_table = tsdb_json["data"]
    myKeys = list(data_table.keys())
    myKeys.sort()
    for k in myKeys:
        dt = datetime.fromtimestamp(int(k)).strftime("%Y/%m/%d %H:%M:%S")
        reading = data_table[k]
        tsdata.append({"x": dt, "y": reading, "s": 1})
    return tsdata

Sample output of RESTful API of GPS data:

[{"s":1,"x":"2023/07/26 02:15:14","y":{"lat":-6.2324056,"lon":106.820457}}]

With the RESTful API of GPS data, you can use a data visualization application with capability of geographic view or map view to visualize the data, such as Geomap panel of Grafana:
https://grafana.com/docs/grafana/latest/panels-visualizations/visualizations/geomap/

Summary

The objective of this article is to demonstrate the capability Alibaba Cloud to facilitate the customers to deploy an IoT application in a Serverless and Cloud-native approach, which has the significant advantages on: scalability to handle the fluctuating workloads, cost-efficiency with pay-as-you-go pricing model, managed services to reduce O&M effort, build-in high availability, and faster time-to-market.

Community

GPS Info Processing and Analysis on Alibaba Cloud

Introduction

Overview

Data Collection

GPS Client (OwnTracks)

IoT Platform

Communicate using ThingsBoard

Communicate using Alibaba Cloud IoT Platform

Data Storage

Data Processing

Data Access

Summary

Read previous post:

Read next post:

Alibaba Cloud Indonesia

You may also like

Comments

Alibaba Cloud Indonesia

Related Products

Realtime Compute for Apache Flink

MaxCompute

Data Lake Storage Solution

Big Data Consulting for Data Technology Solution