I’ve been using Gemini Live with camera sharing a lot and I’m gobsmacked

In case you were unaware, Google’s primary AI model – Gemini – now has eyes to see things in the real world. Created under the ‘Project Astra’ working title, Gemini’s new ability to utilize your phone’s camera to see and react to the physical world is mind-blowing to say the least. And to show you what I’m talking about, let me tell you a quick story from a recent family get-together where Gemini blew everyone away.

It starts with an obscure card game

Have you heard of a game called Skyjo? Probably not, but if I’m the first to tell you about it, you are welcome. It’s a great mix of luck and strategy and is a game that can be learned quickly and played with up to 8 people. We love it, but also realize it isn’t exactly UNO or Spades – most people have never seen this deck of cards in their lives.

Last weekend, I was at my Mom’s house and we were playing Skyjo with a group of 6 of us. And at one point, a rules question came up that I didn’t know the answer to. For this particular issue, a move that is generally great (removing lots of points from your hand) was working against one player. In Skyjo, if you get 3 matching cards in one column, you get to remove the whole column from your hand.

With low scores being the object of the game, this is generally something you would do without question. However, in this instance, the column was full of -2 point cards, thus actually adding to the player’s point total. So, the question was – do you have to remove the cards in this situation, or is it an option?

Gemini Live to the rescue

It was at this point (while my wife was getting the game rules out of the box) that I thought Gemini’s new vision abilities might be worth a shot. I brought up Gemini Live, pointed the camera at the cards on the table, and without naming the game, I simply asked if Gemini knew what we were playing. It of course answered with “Skyjo.”

I then laid out the issue and asked if that column had to be removed or if it was an option, and I was blown away by how fast it responded with the answer. Without missing a beat, we learned that 3 like cards in a column must be removed, regardless of their value. And everyone at the table was a bit slack-jawed.

Not only did Gemini quickly discern the game we were playing – it also understood the context of where we were in the game and offered up accurate rules within seconds that matched up directly with the rule book. It was beyond impressive, and I still think about the scenario and how wild it is that we’ve come this far with AI and mobile tech.

Gemini Live rarely misses

That’s an extreme story, but I’ve had other instances that – to me, at least – are equally impressive. For instance, I was walking my dog downtown the other day and asked Gemini what a particularly-fragrant plant was by just pointing the camera at it. I then had a full-blown conversation with Gemini about replacing my knockout rose bushes at home with this viburnum bush for the rest of my walk and was amazed by how much I could learn from Gemini by just conversing with it.

Of course, I’ve also just spent a lot of time pointing the camera at random stuff and asking, “What is this?” over and over again. And I’ve found that 95% of the time, Gemini can understand and identify objects without fail. From random coffee mugs to our Zero Water filter to my dog and our specific golf launch monitor, Gemini rarely misses, and that gives me a lot of confidence when asking about things I really want to know.

With such a broad contextual awareness and access to limitless knowledge, asking Gemini about all sorts of things is becoming something I lean into more and more as time passes. And I still feel like we’re just scratching the surface of what Google’s AI will be capable of.

I imagine this sort of ability in smart glasses that I can wear day to day and what it will be like to ask questions and get audible and visual feedback on all sorts of things in the physical world on a regular basis. Those actual devices are being developed as we speak, and something that felt like utter science fiction just a couple years ago could be on our faces as early as next year. It’s a wild time to be alive, and the more I use Gemini in this way, the more convinced I’m becoming that nothing will ever be the same as AI continues to progress. Here’s hoping we – as a species – can continue using it in the right ways.