NEW Open Source Qwen? BEATS GLM 5.2 & Claude? 🧐 Ornith 397B TESTED
Keywords
Summary
137 words
Critical Evaluation
Value of the Information & Strength of the Argument
The video’s primary value lies in its hands-on, independent testing of a new model, offering real-world examples of outputs and runtime behavior. The host provides detailed observations on token usage, generation speed, and visual quality, which are useful for practitioners considering local deployment. The argumentation is largely evidence-based, but relies on single-run qualitative assessments without statistical controls. The host is transparent about failures and does not hesitate to criticize the model despite positive examples. The comparison against the base Qwen 3.5 is beneficial, revealing that fine-tuning can degrade certain capabilities. However, the lack of a standardized test protocol and reliance on subjective visual judgments weakens the overall argument.
Scientific Rigor, Source Quality, Title Accuracy
Scientific rigor is moderate: the host is transparent about his hardware (Mac Studio M3 Ultra) and quantization (9-bit), but the testing is anecdotal and not reproducible without exact prompts and seeds. The use of benchmarks from the model card is discussed critically, noting omissions and potential cherry-picking. Sources cited include only the HuggingFace model page and the inferencer.com tool; no external peer-reviewed literature is referenced. The title overstates the model’s performance, as the content shows mixed results and the claim of beating GLM and Claude is not substantiated by the tests. No comments were provided, so public reception could not be assessed.
225 words
Title / Content Match
The title suggests Ornith decisively beats GLM and Claude, but the actual tests show mixed results, frequent loops, and overall parity or inferiority to the original Qwen, making the title somewhat overhyped.
Quality & Reliability
6/10
The video provides a hands-on empirical evaluation of the Ornith 397B model across multiple coding and vision tasks, but the testing methodology is anecdotal (single seeds, no statistical rigor) and the benchmarks shown are selectively presented, reducing overall reliability.
Key Moments
Markers derived by PSI from the transcript: the creator did not define chapters.
- Introduction to Ornith 397B and its benchmark claims against GLM 5.2 and Claude.
- First test: 3D voxel cat image, where Ornith loops with thinking disabled but succeeds with thinking enabled.
- Math problem (even numbers) where Ornith loops at 21k tokens without thinking, then solves with thinking enabled.
- Photorealistic WebGL render comparison; Ornith produces a distorted face, while Qwen gives a basic but stable result.
- Minecraft clone test: Ornith makes a bouncy, deformable world, whereas Qwen creates a functional basic version.
- Flappy Bird generation: Ornith produces a decent version but with no collisions, while Nex N2 Pro loops.
- MS Word clone: Ornith looks better but font change doesn't work; Qwen basic but functional.
- C++ sorting visualization comparison: Ornith generates a nice HTML visualization, with and without thinking enabled.
Cited Sources
- Ornith-1.0-397B on Hugging Face — Model card and download page referenced in the video.
- Inferencer App — Tool used for running and testing the model locally.
External References
Contribution & Novelties
The video contributes an independent evaluation of a newly released open-source model, Ornith-1.0-397B, based on Qwen 3.5. It highlights potential pitfalls of fine-tuning, such as overfitting and degradation of reasoning on certain tasks, while showcasing strengths in creative visual generation. The practical insights on token usage and loop behavior are valuable for local deployment decisions.
Pour aller plus loin :
- Large language model — Overview of LLMs, relevant to understanding the base model and fine-tuning context.
- Fine-tuning (deep learning) — Explains the process of adapting pre-trained models, directly related to Ornith’s development.
- Reinforcement learning from human feedback — A technique often used to improve LLM alignment, possibly relevant to Ornith’s training.
- Qwen — Background on the original Qwen model family that Ornith is based on.
125 words
Radar Profile
The radar profile shows high scores in quantity of information and technical depth, but lower scores in information quality and reliability. This indicates a content-rich but methodologically loose review, typical of rapid hands-on testing without rigorous experimental design.
💬 Sur les 30 commentaires analysés, l'orientation est équilibrée : les spectateurs apprécient l'effort de test et demandent d'autres essais, mais expriment leur déception quant aux boucles de génération et à la performance inférieure du modèle comparé à la version originale. Certains suggèrent que l'orchestration et l'outillage comptent plus que le modèle lui-même.