Maltego Part 5: Your face betrayed you

Hi, guys! Today we'll talk about the heavier artillery in the OSINT framework, namely, the application of face recognition technology from Social Links for Maltego.

During testing the functionality of searching information on various social networks using Maltego (see Part 4), I was stunned that just by the name and photo from the account on Instagram, Maltego immediately found my LinkedIn account.

This is how, I suppose, the Social Links Face Recognition mechanism worked. You can't just go and get past such a topic! You need to make sure how accurate the functionality is and whether it can be fully used in OSINT or my case was unique and the output will be clogged with similar people.

We will test on celebrities. They have accounts on social networks as well as fakes with their photos. We can determine if there is a recognition error. At least Donald Trump can be distinguished by anyone!

In the beginning, we will take a detailed look at the procedure according to the instructions from Social Links. For other cases, we will only analyze the results. The claimed percentage of accuracy from Social Links is 75% recognition, which is not enough.

Author’s note: in the mechanisms of Maltego there is a one not very pleasant parameter. It is called "Timeout" and is equal to 2 minutes. It determines how much Maltego will expect a response from a third-party API when executing Transforms.

Here lies an unpleasant moment. The user can increase this parameter in the settings, but there is no mechanism for the Transform developers to find out what he has installed. This means that when developing third-party Transform, companies like Social Links are forced to rely on the default value. This is so important that you either guessed it or will find it out in the conclusions at the end of the article.

Donald Trump

We go into the Social Links account and create a person's profile there. We indicate the name and surname, country, and gender. There is still a large "note" field for notes.

Add a photo. We choose both ordinary and provoking ones. Everything is ready. You can start!

For search, we will use Entity Search Profile by Face. We uploaded the photo into the Social Links account to get a convenient direct link to it. But if you already have it elsewhere, then you can use it.

After starting the search for the first photo, it became more apparent how the system works. The person responsible for facial recognition in the Social Links neural network makes an API request to the social network to search for all accounts by the given name and surname. After which, already from the photos of these accounts, it compares through face recognition with the photo that we indicated as a reference.

In overall, according to Trump's first photograph, we have:

• Facebook - 1 account (not enough)
• LinkedIn - 1 account (not Trump's, but together with the account owner)
• Instagram - 8 accounts
• VKontakte - 8 accounts
• Twitter - 7 accounts
• MySpace - 3 accounts
• Foursquare - 6 accounts

It is also worth noting that the system could determine Trump’s face even on photo collages and joint photos. As a result of working with the first photo, misses on accounts with random images were not noticed. After starting the search for 2 and 3 photos, several additional accounts appeared. There are still no pictures left. All accounts placed on the graph contain the image of Trump in one form or another. Even Photoshop art did not slip past.

Total, after working out Transforms for all the reference photos, we have:

• Facebook - 2 accounts (still not enough)
• LinkedIn - 1 account
• Instagram - 10 accounts
• VKontakte - 8 accounts
• Twitter - 10 accounts
• MySpace - 7 accounts
• Foursquare - 7 accounts

In this case, only the LinkedIn account should be considered a blunder, since it belongs to Blaine Kelly, but in the photo, it is captured together with Trump, so it's 30 to 70. Trump is still there. The resulting graph on the screenshot below.

Zemfira Ramazanova

Now let's try to recognize a female face. There was a playing the song "Cuckoo" in the office, so Zemfira got into the article. Honestly, no more bias. Test photos below.

Zemfira's fakes, of course, are smaller than Donald Trump's frauds. Respectively, the information output is cleaner. Total managed to find:

• Facebook - 5 accounts
• Instagram - 3 accounts
• VKontakte - 2 accounts

No random photos of people like Zemfira were found. I believe that this is a consequence because the primary sampling is based on the full name. And if you get an account with the name of Zemfira and the image of the left person or a picture, then the face recognition system will already filter him out.

Jackie Chan

Let's complicate the task. We'll check how the system works with an actor with a pronounced Asian appearance. I primarily chose both the pictures of an aged artist and his photo in the prime of his career.

Total managed to find:

• Facebook - 9 accounts
• Instagram - 8 accounts
• VKontakte - 6 accounts
• Twitter - 4 accounts

There are no random pictures of old Asians. The information output is entirely accurate except for many fakes with the name and photo of Jackie Chan. As you can see, we can find accounts with pictures of Jackie at any age equally well.

Thandie Newton

As you can see, accounts with photos of Jackie at any age are searched equally well. Finally, let's check how the system works when looking for pictures of people of the black race.

For the next search, I chose the actress Thandie Newton, who is familiar to all of us from her roles in such films as "The Chronicles of Riddick,” "RocknRolla,” "Solo: A Star Wars Story" and, of course, "Westworld.”

Total managed to find:

• Facebook - 2 accounts
• Instagram - 3 accounts
• Twitter - 6 accounts
• MySpace - 5 accounts

There are no mistakes in recognition.

Summary

According to the results of my tests and reading the documentation on the work of Transforms with the Face Recognition mechanism from Social Links I can easily say that they work great. While writing this article, I even managed to close one OSINT project with their application. However, there are several BUT that you should remember when working with them:

• Due to the large number of photos in social networks and the presence of the "Timeout" parameter in the Maltego settings, we can't just take and scan all the photos on the selected site for a match. Firstly, it will take too much time, and secondly, we are limited to a response window of 2 minutes from the Social Links servers, otherwise Maltego will disconnect and complete Transforms.

All of the above leads us to the fact that before the phase of direct recognition of the face of the search object, we have to do the preparatory stage. It is expressed in the preliminary selection of social network accounts for some parameters. In our case, it was automated in Transform Search Profile by Face and Name in the form of a selection by the person's first and last name. This stage can be performed in different ways, and, for example, geolocation (specific location, hometown, work address, etc.) can be taken as a sampling parameter.

• If there is any apparent discrepancy between the faces in the photo, the account is cut off from the selection. This can be caused not only by the fact that the person in the photo is not the one in the sample.Recognition is seriously hindered by the presence of glasses and hats.

• Another interfering factor for the correct operation of the face recognition mechanism is the degree of "inhibition" of the photo. Both the original sample and the photo on the account from the sample.

If you have the original photo with a low resolution or such a photo on your account, do not expect great results. The resolution of the sample should be better, the preferred file format * .jpg, but if there is PNG, then the network will not be offended.

That's all for today. I hope you enjoyed this article. In the next one, we will talk about how you can apply geolocation information in OSINT using Maltego.