Qwen3.5-4B-SoraEyeAI-GGUF

SoraEyeAI の「目」— 歩きながら周りの様子を短い日本語で伝える、端末内で動く視覚言語モデル The "eye" of SoraEyeAI — an on-device vision-language model that describes a pedestrian's surroundings in short Japanese sentences.

This model is fine-tuned from Qwen/Qwen3.5-4B. The name "SoraEyeAI" is the name of our app; the base model is shown in the repository name out of respect for the Qwen Team's work.

App Store

SoraEyeAI は App Store で公開されています(無料)。 SoraEyeAI is now available on the App Store (free).

👉 https://apps.apple.com/jp/app/soraeyeai/id6818048876

アプリの説明 (App Store より)

胸ポケットの iPhone が、歩きながら周りの様子を短い言葉で声にしてお伝えします。iPhone の中だけで動き、写真を外へ送りません。

SoraEyeAI は、目の不自由な方が歩くとき、胸元の iPhone が周りの景色を短い日本語の声でお伝えする「一緒に歩くお友達」です。

■ 大切なお約束 ・AI を使っており、間違えることがあります。 ・白杖や盲導犬の代わりではありません。白杖では分からない「景色」を足すためのものです。 ・写真を撮ってから声になるまで数秒かかります。聞こえた景色は、少し前の景色です。 ・見間違えることがあります。足元や車の確かめ、道を渡るなどの安全の判断は、いつもどおり白杖・盲導犬・耳と、周りの方の助けで行ってください。 初めて開いたときにこのお約束を読み上げ、「同意する」を押すと使い始められます。

■ ウォーキングアイ (歩く目) 胸元やポケットの iPhone が写真を撮り、「正面は交差点です」「右に点字ブロックがあります」のように、見えたことを短くお伝えします。 ・まん中の大きな丸で実況を始め、もう一度押すと止まります。 ・写す間隔を「連続・20秒・30秒・1分」から選べます。 ・画面をロックすると、胸ポケットで服が触れても動きません。 ・写真を撮るたびに、短い「ポーン」という音でお知らせします。聞こえた景色が、どのくらい前のものかの目安になります (設定で切れます)。

■ 書類読み取り 書類やメニューにカメラを向けると、書いてある文字を読み上げます。文字は iPhone に備わっている文字の読み取りで読み、AI が言葉を作ることはありません。右の縦長のバーで、読み上げる場所を選べます。

■ 写真と個人情報 写真はすべて iPhone の中で読み取り、外へ送りません。保存もしません。マイクは使いません。個人情報は集めません。

■ 使える iPhone ・iPhone 15 Pro 以降 (iPhone の中で「目」を動かすために 8 ギガバイトのメモリを使います) ・空き容量 5 ギガバイトほど ・ウォーキングアイを使うには、設定画面から「目」のデータ (約 4.1 ギガバイト) を取り込みます。Wi-Fi につないで、充電しながら行ってください。書類読み取りは、取り込まなくてもすぐに使えます。

■ 「目」のもとになったモデル SoraEyeAI の「目」は、Qwen チーム (Alibaba Cloud) が公開している Qwen3.5-4B をもとに、歩きながらの景色を伝えられるよう、株式会社ソフトウエア開発が追加で学習したものです (Apache License 2.0)。

■ お問い合わせ https://soraeyeai.computing.jp/

日本語の概要

• 何をするモデルか: 胸元やポケットの iPhone が撮った写真を見て、周りの様子を短い日本語で言います。
  • 歩いているとき: 「正面は交差点です。」「右に点字ブロックがあります。」「正面 後ろ向きに人。」のように短く。
  • 立ち止まったとき (第2版から): 人や車・構造物、お店や建物の名前、雰囲気の順に、少し長めに (120 文字ほど)。お店の名前は、看板の文字や色ではっきり分かるときだけ言うよう学ばせています。
  • 第3版: 側溝の金属のふた (グレーチング) を「点字ブロック」と言わないように学ばせました。
  • 第3.1版: いつも歩く道の側溝の写真を増やし、画像編集で電柱を消した側溝の写真も足して学び直しました。学習に使っていない写真では第3版との差は小さく、電柱を言う回数がやや増えています。
• 誰のためか: 白杖を持って歩く、目の不自由な方のためのアプリ SoraEyeAI(しゃべるお友達)で使います。
• どこで動くか: iPhone の中だけで動きます。写真を外へ送りません。
• 位置づけ: 白杖や盲導犬の代わりではありません。 白杖では得られない「景色」を足すためのものです。
• 安全のための決まり: 見えた事実だけを言います。「渡れます」「車はいません」のように安全だと言い切る文は、教材に入れていません。
• 開発: 株式会社ソフトウエア開発(仙台)

Model Description

Versions

The app downloads a fixed tag (resolve/v3.1/..., resolve/v3/..., resolve/v2/... or resolve/v1/...), so each version stays available even after main moves.

Files

Both files are required to read images. (Earlier model files: tag v3 SHA-256 5c321be98a7a0d595882622f31d951481fc9dac7f37e8823dd26b308ed0ac43e, tag v2 SHA-256 966b17aa6d32fc530b33a68044eb4c800c4b6f83f4d04904224a4bd3f50a8c22, tag v1 SHA-256 073b3fb62bb962ececb6135819f3ab185cc29ba9229b04ca0efc7a2a26464e22.)

Intended Use

• Narration for blind and low-vision pedestrians inside the SoraEyeAI iPhone app.
• While the user walks, the app asks for one to three short sentences. When the user stands still, the app asks once for a longer description of the surroundings.
• The iPhone's built-in voice reads the text aloud.

Out of scope: navigation decisions, judging whether it is safe to cross, reading signal colors, or any use as a replacement for a white cane, guide dog, or human assistance.

Output Style

Walking (short, fact-only, in a fixed order of importance):

• 「正面は交差点です。」 (an intersection ahead)
• 「右に点字ブロックがあります。」 (tactile paving on the right)
• 「左に工事の柵があります。」 (a construction fence on the left)
• 「正面 後ろ向きに人。」 (a person ahead, facing away) — people are described without guessing their movement
• 「この写真では周囲の状況を説明できません。」 (when the lens is covered or the image is unusable)

Standing still (v2): about 120 Japanese characters — first people, cars and structures, then names of shops and buildings when a sign makes them clear, then the atmosphere.

How to Use (llama.cpp)

Walking instruction (Japanese):

今、あなたは歩いています。
この写真から、周囲の状況を80文字で説明してください。
まず最初に人や車、電柱などの構造物について説明し、その後に雰囲気や景色について説明してください。
ハレーションを起こさず、事実のみ伝えてください。

Standing-still instruction (Japanese, v2):

今、あなたは立ち止まっています。
この写真から、周囲の様子を120文字ほどで説明してください。
まず最初に人や車、電柱などの構造物について説明し、次にお店や建物の名前、その後に雰囲気や景色について説明してください。
お店の名前は、看板の文字や色ではっきり分かるときだけ伝えてください。
ハレーションを起こさず、事実のみ伝えてください。

Example:

llama-mtmd-cli -m SoraEyeAI-Qwen3.5-4B-Q6_K.gguf --mmproj mmproj-F16.gguf \
  --image photo.jpg -p "<instruction above>" --temp 0.2 -n 120

Notes:

• The app sends walking images with the long side at 560 px ("標準", default) or 672 px ("詳細"), and standing-still images at 672 px.
• The app asks for up to 120 tokens while walking and up to 220 tokens when standing still.
• Thinking is not used: the app opens the assistant turn with an empty <think></think> block.

Training Data

v3.1 uses 1,437 image–text pairs. They are v3's 1,394 pairs (v2's 1,351 plus 43 new pairs, listed below) with the changes for v3.1 at the end of this list:

• 1,023 pairs from v1, unchanged (walking instruction). Photos taken by the developer while walking in Sendai and Tokyo, Japan (sidewalks, crossings, arcades, stations, indoor corridors). Labels were written or corrected by the developer, who set the narration rules; written by an AI assistant and checked against those rules; or produced by a larger locally-run 27B-class vision-language model and tidied by rules. Real "covered lens" photos and their darkened/brightened copies are labeled as unusable.
• 114 shop pairs with the walking instruction and 92 shop pairs with the standing-still instruction. Photos of Japanese convenience stores and post offices, collected through ordinary web image search under the developer's direction. Stock-photo sites and sites marking their content as not for AI training were excluded. Some walking pairs are lightly altered copies (crop, brightness, blur, slight tilt; never mirrored, so sign text stays readable). Labels were produced by a locally-run 27B-class vision-language model and filtered by the same rules as above.
• 122 pairs with the standing-still instruction on the developer's own photos without shops, so that the model does not name a shop where none is visible.
• 43 pairs from the developer's own walk on 2026-10-03 (new in v3): photos where v2 had called metal drain covers (gratings) "tactile paving", labeled without mentioning either, plus photos of convenience-store signs seen from the road. Each photo also has a few lightly altered copies. Another walk on the same day was kept out of training and used as a test.
• Changes in v3.1: the 35 drain-cover pairs of v3 were replaced by 40 pairs on 11 drain-cover photos from two walks on 2026-10-03 (three wordings each, plus copies with evening light), still without mentioning tactile paving or drain covers. One older label was corrected by the developer. 38 pairs on 19 drain-cover photos whose utility poles were removed with an image-editing model (Qwen/Qwen-Image-Edit-2509, run locally) were added, labeled without mentioning utility poles, so that the model does not tie drain covers to utility poles. Edited images that gained objects that were not in the photo (signals, cars, drain covers) were discarded.
• The training images are not released, because they contain passers-by and third-party photos.

Training

• Unsloth, QLoRA (4-bit base), LoRA r=16, alpha=32, language layers only
• Learning rate 2.9e-4, 4 epochs, batch size 12, warmup ratio 0.05, linear schedule (same settings for v3 and v3.1; 468 steps for v3). The learning rate and number of epochs were chosen after comparing eight settings (learning rate 2e-4 to 3.3e-4, 3 to 6 epochs)
• Merged, converted with llama.cpp convert_hf_to_gguf.py, quantized to Q6_K
• The multi-token-prediction layers of the base model were set to 0 before conversion (mtp_num_hidden_layers = 0)

Evaluation

Single-developer checks on images not used for training (long side 672 px, temperature 0.2, seed 1234):

60 walking photos from a held-out walk on 2026-10-03 (long side 560 px): v2 said "tactile paving" where there was none (a drain cover, and curb stones) in 2 photos, v3 in 0, v3.1 in 0. v2 mixed English into 1 reply, v3 and v3.1 in 0.

70 photos from an evening walk on the same day (long side 560 px), excluding the 4 photos that were added to v3.1's training: v2 said "tactile paving" in 2 photos, v3 in 0, v3.1 in 0. The two evening drain-cover photos that v3 still called tactile paving were added to v3.1's training, so on photos not used for training v3.1 is not clearly better than v3 on this point.

v3.1 mentions utility poles (電柱) more often: in 13 of the 60 photos (v3: 8) and in 18 of the 70 evening photos (v3: 6). Most of these mentions were correct, but in these tests v3.1 once said a utility pole was ahead in an indoor photo, and once said a crosswalk was ahead where there was none.

Sign-size test (v2): when a held-out shop sign was pasted onto shop-free photos at smaller and smaller sizes, the model named the shop less often (44, 31, 17, 10 and 2 of 45 at relative widths 1.0, 0.6, 0.4, 0.25 and 0.15) and mostly stayed silent rather than naming the wrong shop.

This is a small evaluation by one developer, not a benchmark.

Limitations

• It can be wrong, and it describes a moment that has already passed. On an iPhone a description takes several seconds; the scene may have changed.
• Shop names are learned mainly for Japanese convenience stores and post offices. Small or distant signs are often not named. Other shop names may be misread.
• Large flat surfaces (walls, shop windows, shutters) are sometimes called an "obstacle".
• v3.1 mentions utility poles more often than v3, and occasionally names one where there is none.
• It sometimes repeats a sentence or mentions the same person twice.
• Objects that are not in the training labels may be named poorly or not at all (for example, a fire extinguisher in a corridor may be described only as "a corridor continues").
• It cannot reliably judge height differences (for example, whether a curb exists).
• It is not trained to read documents. In the SoraEyeAI app, document reading uses the iPhone's built-in text recognition, not this model.
• Japanese output only. 4B-class capacity.
• Requires a recent iPhone with enough memory (we target 8 GB models such as iPhone 15 Pro and later).

Safety

• The model is trained to state only what it sees. It is not trained to say a place is safe.
• None of the 1,023 v1 labels says that a place is safe (checked on 2026-10-01). The shop and standing-still labels added in v2 were filtered by rules that drop sentences claiming safety or guessing movement (for example 渡れ / 大丈夫 / 安全 / 来ます / 近づ).
• Please always rely on a white cane, a guide dog, or human help for safety decisions.

Acknowledgements

This model is built upon Qwen/Qwen3.5-4B by the Qwen Team, Alibaba Cloud. Some v3.1 training images were edited with Qwen/Qwen-Image-Edit-2509 by the Qwen Team. Training used Unsloth, and GGUF conversion, quantization, and on-device inference use llama.cpp. We gratefully acknowledge these teams for their contributions to the open LLM community.

License Notice

This model is a modified version of Qwen/Qwen3.5-4B, which is released under the Apache License 2.0. Modifications: LoRA fine-tuning of the language layers for Japanese walking and standing-still narration, merging, removal of the multi-token-prediction layers, and GGUF quantization. The vision projector is redistributed unmodified.

This model is therefore also released under the Apache License 2.0. See the LICENSE file in this repository for the full terms.