Shodan and Python API

By Colince Azeye, Alibaba Cloud Tech Share Author. Tech Share is Alibaba Cloud's incentive program to encourage the sharing of technical knowledge and best practices within the cloud community.

If you want to know about all your connected devices and what is going on with these devices, then you can consider using Shodan. Shodan (Sentient Hyper-Optimised Data Access Network) is a search engine designed to map and gather information about internet-connected device and systems.

It is sometimes referred to as a search engine for the internet of things (IoT) and also for search engine for pentesters. Typical application of Shodan include market search, vulnerability analysis, and penetration testing. In this article, we are going to talk about Shodan, how it works, and how to integrate it in your tools or script.

What Can Shodan Do

Shodan detects devices that are connected to the internet at any given time, the location of those devices and their current users. Such devices could be in almost any type of system, including business networks, surveillance cameras, industrial control system (ICS) and smart homes. Shodan attempts to gram the system's banner directly, witch is first key step for penetration testing as it helps identify vulnerable systems. He can even do more by supporting Boolean operators and provides filters to improve the efficiency of searching. The search engine provides 50 results for free and offers paid subscriptions for more extensive results.

How It Works

There are three ways to use Shodan:

Web interface
CLI
In your tools or script

We are going to see how to use all this three methods to use Shodan.

We will be using Alibaba Cloud Elastic Compute Service (ECS) for this tutorial. To ensure our instance is secure, we'll have to configure it accordingly.

Create rule a accept in and out going traffic in your Security Group
Create a pen testing authorization

We will not cover the server setup here, but you can refer to the ECS documentation for more details.

After performing the two setups above, you can install Apache on your ECS and test it by accessing it on a web browser.

Shodan Web Interface

Shodan can be accessed like most other search engines, by navigating to shodan.io in a web browser.

Ok let say you want to know how many server's around the world are using Apache. You will just enter the key word "apache"

As you can see in the image,the result tell us that, there are 25,544,783 servers running on the internet.

There are many more features available on Shodan, but you need to register to use them. As shown below, Shodan prompted an error went trying our a deep analysis without an account.

OK, now let's assume that you have successfully created an account and have logged in. One of Shodan's feature is filter, let's see what we can do with it.

Filters are special keywords that Shodan uses to let you narrow search results based on the meta-data of a service or device. The format for entering filters is:

filtername:value

For example:

product: "apache tomcat"
country: "filter to a specific country
city: filter to a specific city
geo: filter by coordinates
hostname: look for a matching hostname
net: limit to an IP
os: filter based on operating system
port: filter base on open port

Let's say we want to do some precise search by looking for all Apache servers with Tomcat.

The keyword will be 'product:"apache tomcat"'. The result is below inside the screenshot.

The result states that there are 1,445,150 servers running Apache Tomcat in the world

We can further refine our research by looking for all Apache Tomcat servers in China.

Product:"apache tomcat" country:cn

What we did is that we asked for Shodan to search for Apache servers with Tomcat in China.

The result say that there are 409,609 Apache servers with Tomcat in China.

Now let say you have your services running on one of your ECS and you want to see what is going on it. In my case, as said above I'm going to use on of my Alibaba ECS for example.

To do the task we just have to use net filter and the values will be our ip_address

net:ip_address

Considering that we have not yet created a rule in our security group to allow traffic coming in and out, but let's say you have installed Apache. If you try to access it by entering you IP address inside a navigator, you will not be able to see something because Alibaba Cloud restricts traffic by default. This is to ensure that your ECS server is secure even if you haven't created rules on your security groups.

As you can see in this image, the server is running on Alibaba and the IPS is Aliyun Computing. There are two services running on the server because we are saying two ports (80,443) , the web technology used is jQuery. This is some of the information that a hacker would gather about our server to see if there are any vulnerabilities, such as the version of the SSL.

Now, let's see how we can use Shodan through command line interface (CLI).

Shodan Command Line Interface

The Shodan command line interface (CLI) is packaged with the official Python library for Shodan, which means if you're running the latest version of the library you already have access to the CLI. To install the new tool simply execute:

$ easy_install shodan

Once the tool is installed it has to be initialized with your API key:

$ shodan init YOUR_API_KEY

Visit https://account.shodan.io to retrieve the API key for your account. You will see at the right above My Account button. Once the API_KEY is initialized you can now start using Shodan commands.

We can check for information about our account.

Now imagine you are in your ECS instance and you want to know what is you public IP address. If you use ifconfig, you will only have your private IP address because Alibaba Cloud protects your services by default. If you want to know your public IP address, just enter the command Shodan myip.

$ shodan myip

The result is showing my public IP 47.89.249.0

Now if we want to have information about a host, such as where it's located, what ports are open, and which organization owns the IP, we can use Shodan. Let's say that 42.120.226.13 is our server and we want to test it and see what is running on it.

$ shodan host 42.120.226.13

Our server is located in China, the last update is 2019-01-21 and the is two ports running on it (80, 443), the version of SSL has to be verified because the result gave us more than one version.

There are other functions on the CLI like networks analysis, maltego add-on, browser plug-ins, but to use them, you have to upgrade your plan account.

Shodan in Your Script

Shodan provides a developer API for programmatic access to the information that is collected. All of the websites and tools, including the main Shodan website, are powered by the API. Everything that can be done via the website can be accomplished from your own code.

The API is divided into 2 parts: REST API and Streaming API. The REST API provides methods to search Shodan, look up hosts, get summary information on queries and a variety of utility methods to make developing easier. The Streaming API provides a raw, real-time feed of the data that Shodan is currently collecting. There are several feeds that can be subscribed to, but the data can't be searched or otherwise interacted with; it's a live feed of data meant for large-scale consumption of Shodan's information.

Usage Limits

There are 3 methods of the API that get limited depending on your API plan:

Searching: To limit the number of searches that can be performed per month Shodan uses query credits. 1 query credits is used when you perform a search containing filters or go past the 1st page. For example, if you search for "apache" that doesn't use any query credits. If you search for "apache country:US" that would use 1 query credit. Likewise, if you searched for the 2nd page of results for "apache" that would use 1 query credit. Finally, a search query for the 2nd page of "apache country:US" would also use up 1 query credit.
Scanning: The on-demand scanning API uses scan credits to limit the number of hosts that you can request Shodan to scan every month. For every host that you request a scan of Shodan deducts 1 scan credit.
Network Alerts: The number of IPs that can be monitored using alerts is limited based on your API subscription. Only paid customers have access to this feature. And you can't create more than 100 alerts on your account.

Note: Query and scan credits are reset at the start of every month

To install the Shodan library for Python run the following command:

$ easy_install shodan

If you already have it installed and want to upgrade to the latest version:

$ easy_install -U shodan

The first thing that always has to be done is initializing the Shodan API object:

import shodan
api = shodan.Shodan('YOUR API KEY')

Where YOUR API KEY is the API key for you account which you can obtain from:
https://account.shodan.io

Now that we have all ours API in place we can perform some search with a small script.

As you can see in this picture we write a small script that is going to scan the internet to search for server or query that you we will submit as a parameter. The code is written in python3.

If you execute this script with a parameter Apache, Shodan will scan the internet and give you all information on Apache server.

Let execute the script and see if is working and what will be the result.

The script result is shown above. As you can see, we have ip address running Apache servers.

Conclusion

The evolution of the technology forces us to keep ourselves constantly updated in order to not jeopardize our security. Especially for IoT applications where multiple devices are exposed to the web, security is crucial not only for ensuring the proper functioning the assets but also for protecting our privacy. Shodan is one of the most powerful tools available used to arrive to that end. But like every good tool, Shodan can be a double-edged sword; it can be easily exploited by hackers but can also help you understand your network better. The danger is not with the tool itself but rather the one who uses it.

Community

Shodan and Python API

What Can Shodan Do

How It Works

Shodan Web Interface

Shodan Command Line Interface

Shodan in Your Script

Conclusion

Read previous post:

Read next post:

Alibaba Clouder

You may also like

Comments

Alibaba Clouder

Related Products

IoT Platform

EDAS

CloudMonitor