Others have found that CalAI likely uses Gemini 1.5 (not sure which variant) which is used because it's very cheap, not for its accuracy.
I understand the fundamental issues with calorie counting via photo analysis/LLM. However these articles would be more interesting if they included comparison with results from actual cutting-edge LLMs, rather than the budget models. I acknowledge that apps don't use these models though to protect margins and lower cost.
Also:
> Cameras capture two-dimensional images of three-dimensional objects.
iPhone camera captures depth, too. I don't think it's being used in CalAI though. This article is more about current commercialized solutions rather than analysis of state of the art capability.
They don't, but that doesn't stop them from lying about it on their homepage.
> Snap a photo with Cal AI, and your phone's depth sensor calculates food volume. Our AI then analyzes and breaks down your meal to determine calories, protein, carbs, and fat.
Oh interesting. I wouldn't put it past them that they include that data in the input though. Why are you certain they don't? (I guess that they don't, too, because someone reported getting identical calorie readings from giving photos to Gemini 1.5 directly)
They’re using some off the shelf multimodal LLM API, perhaps Gemini 1.5 or something else. And no off the shelf VLM API is capable of handling 4 channel images (RGB+D).
You can take crude measurements from the depth map and provide as text. Or perhaps provide the depth map as an image next to the actual image as those are certainly in training data. I haven’t tested these
idk tbh asking an llm to estimate calories seems like such a -bad idea- on the first place, that doesnt seem that having a better model would improve results much.
I understand the fundamental issues with calorie counting via photo analysis/LLM. However these articles would be more interesting if they included comparison with results from actual cutting-edge LLMs, rather than the budget models. I acknowledge that apps don't use these models though to protect margins and lower cost.
Also:
> Cameras capture two-dimensional images of three-dimensional objects.
iPhone camera captures depth, too. I don't think it's being used in CalAI though. This article is more about current commercialized solutions rather than analysis of state of the art capability.