Playing with AI in Azure 2 Photo with rectangles

Playing with AI in Azure

When I was learning about Azure AI, I asked myself the question: “how far is AI? What can it determine – and what is impossible (yet)?”

A nice tool that can give (partly) answers to that, is the website that Azure offers to scan images, it uses Azure Cognitive Services (computer vision): ( ). It’s a nice website: you simply insert the photo url and then the result is shown.

Playing with AI in Azure 1 Insert photo URL

Police car

I tested it on some images of the news site I got the link from one of their web pages (F12, then search for jpg):

When you paste this link and then press “Submit”, you will see rectangles around the main objects on the photo:

Playing with AI in Azure 2 Photo with rectangles

On the right side, you see the conclusions that are drawn:

Objects[ { “rectangle”: { “x”: 671, “y”: 407, “w”: 124, “h”: 76 }, “object”: “Vehicle registration plate”, “confidence”: 0.555 }, { “rectangle”: { “x”: 224, “y”: 159, “w”: 671, “h”: 507 }, “object”: “car”, “parent”: { “object”: “Land vehicle”, “parent”: { “object”: “Vehicle”, “confidence”: 0.926 }, “confidence”: 0.924 }, “confidence”: 0.876 } ]
Tags[ { “name”: “car”, “confidence”: 0.9668071 }, { “name”: “land vehicle”, “confidence”: 0.964869 }, { “name”: “vehicle”, “confidence”: 0.950910568 }, { “name”: “wheel”, “confidence”: 0.702893734 }, { “name”: “vehicle registration plate”, “confidence”: 0.685587 }, { “name”: “full-size car”, “confidence”: 0.682034552 }, { “name”: “stage”, “confidence”: 0.6383412 }, { “name”: “text”, “confidence”: 0.6330378 }, { “name”: “mid-size car”, “confidence”: 0.5292624 }, { “name”: “blue”, “confidence”: 0.4259651 }, { “name”: “light”, “confidence”: 0.175804853 } ]
Description{ “tags”: [ “stage”, “table”, “sitting”, “car”, “blue”, “small”, “light”, “parked”, “truck”, “room”, “airplane”, “laptop”, “computer”, “night”, “street”, “bus”, “standing”, “man”, “bed”, “plane” ], “captions”: [ { “text”: “a blue car on a stage”, “confidence”: 0.6204072 } ] }

What we see on the photo, is obvious for everyone in the Netherlands: it’s a police car in action. Azure did understand the vehicle registration plate, the fact that it is a car, and that it is a land vehicle. The meaning of the stripes on the car and the meaning of the blue light were missed.

Donald Trump

It’s nice to play around a little bit with different photos. Let’s take this one:

It does recognize Donald Trump, it also recognizes the smile on his face. Trump is wearing a suit and a tie is visible (correct). It also sees a phone (where?!) and as humans we see that he is waving (the recognition software doesn’t recognize this).


My conclusion: sometimes it’s impressive, sometimes it misses the clue.

Have fun with your own photos!

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.