#Benchmarks
200
2026-10-01
Show HN: Open-source model routing for coding agents at Astra-level performance
S
A few months ago we started building a model router for coding agents because we thought we could outperform any single model with an ensemble approach. Recently we’ve achieved that milestone and I want to talk about how we did it. First of all, a quick explanation: the Weave Router ( https://github.com/weave-os/router) plugs into any coding agent (e.g. Claude Code or Codex) and intelligently switches between LLMs. So, for example, Astra handles tricky debugging or complex system design tasks, and Deepseek v4 Flash handles simple frontend updates. What we’re announcing today is our new routing model, which we’re calling Weave Router 2.0. We benchmarked 2.0 against GPT-6 Astra on Terminal Bench 4.0 and SWE Atlas. On both benchmarks, the router had equivalent pass rates. On Terminal Bench, the router hit 52% of Astra’s cost, and completed tasks 2.2x faster. On SWE Atlas, the router cost 54% as much as Astra and ran 2.5x faster. (Full results on our website at https://weaveos.com/router!) It turns out training a model to route effectively - taking into consideration model capabilities, costs, cache awareness, and more - is a really hard problem! I want to talk about three ways we were able to improve so much over the last few months: 1) a new architecture, 2) larger training data set size, and 3) smarter cache-eviction impact calculation. 1) a new architecture. Our initial approach used an RL model without many priors. While RL is still an important part of the story, the cost of fully exploring the space of routing decisions is very high, so we’ve taken some shortcuts that have significantly improved performance. Consider how large the search space for the routing problem is. Take a typical coding agent session, with ~100 agent turns (i.e. 100 LLM API calls). Technically there are 100 chances to select a model. If we assume a roster of ~10 models (of course there are lots more but we can remove any that are Pareto dominated), then there are 10^100 possible paths through that session. We simply cannot explore all of them! So that's why clever tricks to shrink this space are so important. In particular: we trained a hidden Markov model to trace the session state, then a classifier maps the session to one of a few buckets of similar models. Using the HMM allows us to evaluate not just where a session is currently, but how it got there. We've gotten significantly better performance on bucket selection by incorporating that information - we believe this is because two sessions that might look quite similar to a naive classifier are much better distinguished by this HMM approach. Using this HMM + classifier to select a bucket first significantly shrinks the space to explore, by throwing out most models that could not reasonably serve the given session. This rearchitecture was the single biggest performance unlock! 2) larger training data set size (much less technically interesting but still an important part of the story). By using frontier LLMs to help us label a larger and more diverse set of coding agent sessions, we were able to bootstrap the two models discussed in 1) to a better state, while also providing even richer reward signals for RL. 3) smarter cache-eviction impact calculation. One of the hardest parts of routing well (if you care about saving money) is using the model caches intelligently. We built a subsystem that can calculate the expected value of switching models (and thus paying a high one-time cost to fill up a different cache) much more accurately, helping us avoid costly and unnecessary switches in more cases, while still switching when the benefit outweighs the cost. This is where most of our improvement on cost has come from. We still have a lot of room to continue to improve (we won’t rest until we’re consistently beating Astra/Fable, not just tying!) but matching frontier model performance was a huge milestone for our routing model, and in my opinion validates our initial hypothesis that an ensemble of models can do better than any single model ever could. Our router is open source ( https://github.com/weave-os/router) so anyone can try it out. Or if you prefer you can use our hosted version ( https://weaveos.com/router).
2026-09-17
Show HN: Swift-Qwen3.8-27B, -58.3% thinking, x1.95 speed, accuracy of xhigh
S
Hi everybody, we post-trained Qwen 3.8 27B to be more efficient by figuring out which tokens were linked to overthinking and penalizing them without "attacking" the reasoning length directly then fixed the accuracy with a bit of secret sauce (hint On-Policy Distillation) and achieved great results (-58% length, 1.95x speed up, <1% accuracy loss) so we wanted to open-source it and hear the feedback of the community. This is the link to the model: https://huggingface.co/ukisai/Swift-Qwen3.8-27b / It's at 80k downloads in 3 days with independent evals here: https://www.reddit.com/r/LocalLLaMA/comments/1wg7dd5/ukisai_... We are also providing a Free Research Purpose API (OpenAI compatible), courtesy of Nvidia who were kind enough to provide us with the GPUs. It's limited at 5RPM. https://ukisai.com/api/swift/v1/models We also made a GGUF (Q1-Q8): https://huggingface.co/ukisai/Swift-Qwen3.8-27B-GGUF and there's also a few nice community quants with even lower/higher precision (Bartowski: https://huggingface.co/bartowski/ukisai_Swift-Qwen3.8-27b-GG...). The community also created amazing MLX, NVFP4, W4A16 and Uncensored versions you can find on Huggingface. IMPORTANT: Our training approach is not a replacement for the reasoning effort settings, chat templates or token caps but is complementary and targets a completely separate issue (overthinking and "anxiety-like" reasoning loops prior seen in PTQ, but as far as we identified also prominent in BF16 of this size class LLMs as well). Contrary to popular belief, these specific patterns do not contribute to answer quality when properly targeted. (our thesis being: reasoning length IS extremely important and should NOT be shortened by force, but rather optimized). This is also demonstrated bellow in our xhigh vs medium effort benchmark. The goal is to keep xhigh accuracy while reducing only the unnecessary part of thinking. The TLDR of our thought process, research, training and a link to the Meta paper that inspired us is in the first comment. The benchmarks: Swift-27B vs Qwen3.8-27B (BF16, all benchmarks ran x5, thinking effort xhigh) GPQA-Diamond: 88.4% -> 88.3%, 58% fewer median tokens LiveCodeBench v6: 76.8% -> 81.6% (+4.8pp, due to default truncation in LCB it is not performance gain), 46% fewer median thinking tokens Terminal-Bench 2.1: 66.7% -> 65.8%, 39% fewer median tokens MMLU-Pro: 85.5% -> 85.0%, 28% fewer median tokens C-Eval: 90.0% -> 90.6%, 19% fewer median tokens IFBench: 73.5% -> 71.8%, 51% fewer median tokens AIME 2026: 98.7% -> 94.0%, 50% fewer median tokens HMMT (Nov 2025): 99.3% -> 96.0%, 46% fewer median tokens ERQA (vision): 67.5% -> 66.3%, 55% fewer median tokens Token savings hold at every reasoning effort (mean thinking reduction): xhigh 41%, medium 23%, low 26% (albeit with accuracy loses of 1-4% on medium and 1-2% on low which we need further testing for) Swift at xhigh vs the base's own effort settings on GPQA-Diamond (198 questions x 5 seeds): Base xhigh: 88.4%, 6,642 median tokens Swift xhigh: 88.3%, 2,771 median tokens Base medium: 84.1%, 1,753 median tokens So Swift keeps xhigh accuracy at under half the tokens, and beats base-medium by 4pp at roughly 1.6x its tokens. End note: While we are keen on complete open-source, we still need to keep a part of our training and data private, being a new lab. The license is not Apache 2.0, but it only affects companies >$1M. We hope this does not pose a problem for the community and are open to feedback on it. We want to contribute as much as possible to the community and would really appreciate feedback on our work, quantization or Swift model requests. We are working on Swift 3.8 Flash Next and have so far gotten up to -53% thinking token usage. We have strong indicators our methodology is reproducible on other model families as well and are asking the community which ones you want us to optimize next.
2026-08-27
UditAkhourii/cdaf
S
CDAF (Cached Descriptive Asset Files) - open sidecar format for video so AI agents stop re-analyzing the same footage. Spec, CLI, agent skill, reproducible benchmark.
2026-09-19
bespokelabsai/nimble
A
Local typed decisions, contrastive data curation, and model evaluation.
2026-09-12
Hona/openeval
A
Typed prompt-and-judge SDK, CLI, and evidence viewer for agent evaluations
2026-10-14
composo-ai/DishwasherBench
S
A dishwasher-loading game on a physical simulation of one real machine, built to benchmark language models and playable in a browser. The game inside is called Spotless.
2026-10-21
browser-use/rl-env-use
A
Build RL environments for any domain from real cases that Browser Use agents mine from the web
2026-10-17
Show HN: Gemini 4 Argon plays StarCraft
A
I've refreshed StarSkirmish Bench, a benchmark where LLMs play StarCraft BW by writing code, with results from the flagship models of OpenAI, Anthropic, Google, and Muse
2026-10-25
alleyxd/quantpicker
A
Measures which GGUF quant of a model to run on your GPU: real speed and quality loss (KL divergence), not estimates.
2026-08-28
christophhdesign/open-design-system-bench
A
How AI-ready is your design system? A plug-in benchmark that runs coding agents against your component library and grades the output. Built against a production design system and generalized for any React + TypeScript component library.
2026-10-28
bjarneo/velo
A
A speed test for the terminal with a cockpit dashboard: latency, download, upload, and bufferbloat.
2026-09-28
MaximeRivest/tiny-classifiers
A
Train a 17M-parameter text classifier in a minute, with labels from a big AI model, and know when it beats an API. Recipe, tutorial, results on 10 tasks, phone trainer.
2026-10-22
ScottDuncanAI/HistorianRCA
A
HistorianRCA: a root cause analysis benchmark. 90 days of simulated plant historian data from a steam-heated hold tank, with hidden faults to find and diagnose.
2026-09-28
Show HN: TinyAIArena watch AI agents battle it out
B
Did you ever click on an “AI Arena” expecting glorious battle and instead get a boring benchmark? If so, this project is for you: proper life-or-death fights between four models on a picturesque 8×8 grid. May the most intelligent one win! Click on any of the matches to spectate them. Code: https://github.com/hp6/ai-arena
2026-10-09
UniPat-AI/PaperBenchX
A
A benchmark for scientific research agents, evaluating end-to-end paper reproduction across 12 research areas.
2026-09-28
orlp/polyxor
A
A 128-bit universal hash function designed for fast file/data stream integrity checks with cryptographic strength, written in Rust.
2026-09-07
FeiZhuLulu/real-api-pricing
B
Real API pricing: subscription fee ÷ usable tokens, with allowance, unit-price, and leaderboard Pareto charts.
2026-09-07
pcm469768-droid/llm-api-speedtest
A
Benchmark LLM APIs, OpenAI-compatible gateways, and API proxies by token speed, TTFB, latency, and success rate. Compare multiple providers with reproducible JSON reports and scheduled GitHub Actions.
2026-08-29
SesameAILabs/turnbench
A
TurnBench: a multi-domain benchmark for end-of-turn and interruption detection in spoken dialogue
2026-08-23
GamePhanes/GamePhanes
B
An open-source game coding agent environment and benchmark for Godot.
2026-10-07
bottomless/agent-duel
A
Blinded battles between coding agents: two agents, one prompt, you pick the winner. Desktop app, runs locally.
2026-09-25
rudratoshs/buried-injections
A
🛡️ Regex catches 0%, Meta's Prompt Guard 2 catches 1% of 629 realistic AgentDojo injection attacks when they're buried in tool output. Reproducible benchmark.
2026-08-27
VeryVale-RT/quantfeature
B
road feature coverage with explicit compatibility contracts and causal defaults. Formula parity is tested family by family.
2026-10-15
Yukinodot/ScaleScout
A
Explainable budgeted zoom for YOLO small-object detection. Equal-budget experiments, crop-edge filtering, per-image evidence, and an offline viewer.
2026-10-07
nsodhi-cmu/MinigamesBenchmark
A
A benchmark of seventy minigames for computer use agents, testing a variety of skills. Think your agent can beat them all?
2026-09-29
B0yko/booking-truth
A
Tests any AI appointment-setting agent on the calendar's end state, not its transcript: fault injection, pass^k, false-success rate. Ships a guarded Cal.com / Google Calendar / HubSpot booking agent.
2026-09-03
dreamers-laboratory/ai-evaluation-framework
B
Accuracy, latency p95, and cost benchmarking for model-based solutions, with per-field ground-truth scoring.
2026-10-14
Danielxu0208/rsi_eval
A
A lightweight Python SDK and CLI for evaluating recursive self-improvement (RSI) methods across agent benchmarks.
2026-09-09
openaiotlab/CUHK-X
B
[MobiSys 2026] A large-scale, multimodal dataset and benchmark for Human Action Recognition, Understanding and Reasoning
2026-09-07
Datura-ai/lium-localmaxxing
A
Reproduce Lium's LocalMaxxing leaderboard runs (7 boards, 20 Verified) on a rented Lium GPU in one script — vLLM/llama.cpp recipes, measured cost per million tokens
2026-09-01
KamaiEnterprises/aec-geometric-bench
A
AEC-Geometric-Bench: geometric and symbolic data extraction from PDF construction drawings. 312 sheets scored, 15 released with ground truth and a runnable scorer.
2026-09-23
Show HN: JevBench, a reproducible benchmark for typed decision models
B
Hi HN! I built JevBench because Jev kicks ass, and the world deserves to know how the serious open source and fake lookalike projects really perform in comparison. Jev-class models return bounded choices and probabilities instead of text, and are disruptively faster and cheaper than LLMs, while being similarly intelligent on the text input they operate on. JevBench allows looking at accuracy, latency and price all at once, in a weighted way - you can even configure the weighting. A full run asks 534 English decisions. The v1.3 score combines chance-corrected Intelligence, Calibration, Speed and Cost. Leaderboard right now: #1 - Jev 74.4 #2 - SemIf 73.1 #3 - djev 73.0 #4 - Winnow-12B Q8 71.2 #5 reflex 4B 70.3. MIT harness, public items, frozen artifacts, scoring code and public per-task outcomes: https://github.com/fstandhartinger/jevbench Two no-signup demos: https://who-is-right.app.mintapis.com https://is-it-ai-slop.app.mintapis.com Limitations: English-only; latency from one German server; local/demo latency gets a disclosed ×2 adjustment (+150 ms on my servers) which is an informed assumption; held-out prompts still reach evaluated services; ~1-point gaps can be noise. Wdyt?
2026-10-21
openai/MultiLangBench
B
An eval based on SQLite that benchmarks language models in multiple languages.
2026-10-16
FirmwareVersion/choravox
B
Offline multi-speaker meeting-speech toolkit: planted-ground-truth synthetic meeting factory, VAD, speaker embeddings, diarization (AHC/spectral/VB-HMM), overlap detection, CTC speaker-attributed ASR, cpWER/DER/JER evaluation, online streaming mode, deterministic reports. NumPy core, CPU-only torch extra.
2026-10-14
Show HN: Parsimony – an open-source benchmark for how much code AI models add
B
Existing benchmarks measure solve rate and token output/cost, but don't punish excess code, which makes a codebase harder to work with in the long term. I wanted to measure this since it matters most for long-term projects. Method: it takes the patches that benchmarks already publish (DeepSWE, SWE-bench Verified, SWE-bench Live) and counts "coding units": syntax elements like statements, calls, names and literals. I didn't use lines of code, so cramming code isn't rewarded. Deleted code is subtracted, so it measures net units. GPT-6 and Claude 5.5 have no public patches yet, so I ran them myself on 3–10 tasks on my own subscriptions (shown as faded circles, not ranked). Any feedback or model contributions are welcome, especially public patch exports for newer models. GitHub: https://github.com/La5u/Parsimony Not affiliated with any of the benchmarks.
2026-10-03
chen-006/meow-ai-arena
B
Open-sourcing the top three entries of an AI algorithm competition: territory capture, Bomberman, and capture the flag. Problems, judges, each model's original code, per-round results, replays, and one-click challenge scripts
2026-09-24
DaVVinCi/llm-multi-agent-game-evaluation
B
A dynamic multi-agent game framework for evaluating LLM strategic reasoning, communication-driven decision-making, deception, and social-norm behavior across RPS, Hawk-Dove, and Liars Bar environments.
2026-09-18
liyupi/ai-model-world
B
AI large model world, a visualization site that anthropomorphizes 556 large models into pixel characters. Come in and see who is smartest right now, who writes code best, who is cheapest, and who just launched, then scroll down to vendor plazas split into domestic and international sections, a complete release timeline, and multi-dimensional leaderboards. Search recognizes model names, vendors, and capabilities; typing "multimodal" directly lists all multimodal models. Data comes from Epoch AI, models.dev, LiveBench, and Hugging Face, auto-synced hourly; all copy is generated from real data without calling any LLM. Next.js static export, zero backend.
2026-09-05
mpaymenremora/CacheCraft
B
Semantic cache strategy bench: replay recorded request traces through candidate caching policies and measure which one actually pays off.
2026-08-29
hastylmao/Hasty-CR
B
Deterministic Clash Royale simulator and a reinforcement-learning agent for 2.6 Hog Cycle. Integer-exact physics from client data, 1,600+ tests.
2026-10-22
TianqiLiu1/PACE-Bench
B
Official code and project page for PACE-Bench: When Coding Agents Outpace Review: Evaluating PRD-Driven Development in Mature Enterprise Repositories.
2026-09-30
B0yko/local-enough
B
Measure whether a back-office AI task can run on your own hardware: accuracy vs gold, p50/p95 latency, $/1k tasks, amortised hardware cost and break-even volume, plus an OpenAI-compatible router that sends each task to the cheapest model meeting your quality bar.
2026-09-29
Show HN: Pac-Bench – How well can models one-shot a Pac-Man game?
B
Benchmarks how well Harness+models can create a Pac-Man game from a single prompt: “Create a Pac-Man game in a single HTML page” Each model gets one shot — no follow-up prompts or fixes.
2026-10-10
swirl-uk/BiGym2
B
BiGym 2.0: Benchmarking Learned and Agent-Developed Policies for Humanoid Household Manipulation (arXiv:2610.07594)
2026-09-25
cookieyes/cookiebannerbench
B
An open benchmark for what consent banners cost the pages they sit on — time to banner, payload, viewport blocked and page stability, measured on identical pages.
2026-09-16
AlchemyInCode/plumloom-autoeval-oss
B
Open-source CLI and MCP server for evaluating AI applications and gating releases with reliability evidence.
2026-09-06
PillCrew/deedchain
B
Benchmark of report fidelity for browser agents: does the agent's final report tell the truth about what it actually did?
2026-09-05
MiaAI-Lab/Qwen3.8-Flash-Next-Single-DGX-Spark
B
Qwen3.8-Flash-Next on ONE DGX Spark (TP=1)
2026-09-05
michailfragkiskos/EdgeMosaic
B
Rapid on-device AI evaluation lab: manifest validation, benchmark plans, latency/memory/thermal aggregation, JSON+Markdown reports.
2026-11-02
rmr2024/finextract
B
Benchmark and fix LLM hallucinations when extracting figures from SEC 10-K filings
2026-10-09
LINs-lab/UltraText_Bench
B
UltraText Bench: A Comprehensive Bilingual Benchmark for Evaluating Visual Text Rendering in Image Generation
2026-10-09
Yu-ChangCheng/manufacturing-forecasting-framework
B
Reproducible monthly demand forecasting with Random Forest, chronological backtests, transparent baselines, and a guided local app.
2026-09-24
OpenDCAI/DataFlow-AgentMM
B
A framework for collecting and evaluating multimodal agent trajectories in visual environments.
2026-09-22
buddy/minesweeper
B
A Minesweeper benchmark for LLM agents. One identical board, up to nine models in parallel, one clock, one tool layer.
2026-09-11
tonyd2wild/DeepSeek-V4.1-Flash-vLLM-DGX-Spark
C
DeepSeek-V4.1-Flash (552B MoE, MXFP4 experts, 1M ctx) on four NVIDIA DGX Sparks with vLLM TP4: Engram-on-disk patch, sm121 kernel build, launchers, measured numbers
2026-09-04
shtjww/llm-inference-capacity-handbook
B
A desk reference for LLM inference (open-source edition): given GPU compute, how many QPS can a model handle? Three-layer model × three walls × queueing theory × open-loop load testing
2026-10-17
gilnett/learned_index_lab
B
High-performance Piecewise Geometric Model (PGM) learned index implementation in pure Rust with strict epsilon error guarantees
2026-09-18
Daniel-rmc/ManiLoop
B
An open framework for closed-loop evaluation of VLA, VLM, and other robot policies across benchmarks, designed for fair and reproducible comparisons. Current integrations: MuJoCo tasks, LIBERO, GPT, and LeRobot.
2026-09-18
rorshopping/jev-on-a-laptop
B
Unofficial study: Jev-style parallel typed decisions on stock 1.5B-8B models on an Apple Silicon laptop. Benchmarks, research notes, and a Hugging Face Space demo.
2026-09-09
openai/NavierStokesAndEuler
C
Lean certificates accompanying Navier-Stokes and Euler results
2026-10-24
theorem-labs/oss-verify-proofs
C
Theorem statements about OSS-Fuzz fuzz targets as compiled RISC-V binaries, and machine-checked Rocq proofs of them
2026-10-17
plotarmordev/drama-tensorfold
B
Fish Audio Drama 3 text-to-speech on the TensorFold native CUDA engine for DGX Spark (GB10): recipe, engine patches, benchmarks. Built with Fish Audio.
2026-10-12
final-bench/quantum-bench-harness
B
Open Quantum Challenge (OQC) - open evaluation harness + frozen problems for the FINAL-Bench Quantum leaderboard
2026-09-25
yoheinakajima/glance-speedlab
B
A measurement-first lab for low-latency local camera inference with Glance.
2026-09-17
dollarwong/ninfer-v100
B
NInfer on Tesla V100 (sm_70): Qwen3.8-27B deployment, build/convert scripts, systemd service, and cross-engine benchmark (vs llama.cpp). Fork of Neroued/ninfer with sm70 vision-encoding speedup.
2026-09-17
gensyn-ai/open-transformers
B
Reproducible LLM pretraining and audit-replay harness behind the OPEN model collection
2026-09-02
Instituto-IA-LNCC/manaca-1b-base
B
LLM aberto e reprodutível para o Português do Brasil An Open, Reproducible Brazilian-Portuguese Language Model
2026-10-28
yuiseki/map-from-scratch
B
For each request like "I want to do X, so make me a map," build a minimal, fastest map renderer from almost scratch that satisfies only that purpose, and measure and verify its performance and correctness — a skill and harness for AI agents
2026-10-19
repoboost-hq/github-star-benchmarks
B
AI research dataset on GitHub stars: 32.6M starred repositories, star thresholds (100/1k/10k/100k), language and age cohorts, plus star-buying and fake-star research context from ICSE 2026. Open CSVs and reproducible queries for empirical software engineering, scientometrics and LLM evaluation.
2026-10-12
DarkHater01/slopbench
B
Measures how much code LLM agents waste. 5 tasks, sandboxed, weekly snapshots, contestable references.
2026-09-16
zj-unicom-ai/uni-llm-bench
B
A self-hosted lightweight LLM API benchmarking platform — model fingerprint comparison + model quality evaluation.
2026-09-02
needsbuilder/bside
B
Benchmarks for Aside — do AI browsers actually beat existing automation on the real web? Five Korean-web errands, three setups, everything published.
2026-08-28
open-daimax/daimax-appbench
B
An automated benchmark platform for AI-generated apps — built-in evaluation rules, standard sample sets, and layered test cases across a multi-dimensional metric system. One pipeline runs build, install, E2E tests, and scoring into a shareable report.
2026-09-23
DaVVinCi/FactorForge
B
LLM-assisted quantitative factor research with safe DSL validation, benchmark comparisons, and cost-aware Top-K backtesting
2026-08-25
hongyuanyu/JANUS
B
a plug-and-play, training-free sampler that mines local Jacobian geometry from the search trace to guide black-box optimization — one Gauss–Newton Jacobian drives both exploitation and exploration, improving CMA-ES / NSGA-II hosts on BBOB, high-dimensional UAV planning, and multi-objective tasks.
2026-10-13
Fengzdadi/memory-bench-registry
B
Structured registry of agent memory benchmarks, with every field traced to its source
2026-10-12
Caldalis/MiniCPM-Math-harness
C
built around a 2.5B math model that fits on one GPU or a laptop.
2026-09-27
Junjue-Wang/DORA
B
[NeurIPS 2026 Oral] DORA: Can LLM Agents Respond to Disasters? Benchmarking Heterogeneous Geospatial Reasoning in Emergency Operations
2026-09-10
HongyeYangGT/DepthBenchCAD
B
A three-level benchmark for evaluating audit depth, evidence allocation, and reliability in generative CAD systems.
2026-10-29
denizguneey/invariant-twins
B
Exact finite graph-invariant fingerprints with reproducible non-isomorphic witnesses.
2026-10-22
ChungKit-WAN/Quant-research-lab-v0.1.0
B
Python framework for monthly sector ETF allocation and multifactor stock backtesting, with transaction-cost accounting and separate Chinese and English research reports.
2026-10-09
Swapnil-jain/integer-mult-kappa
B
Conditional witness kappa = 249/(5*10^10) for integer multiplication, building on OpenAI #109 and Colkitt's integer-mult-bounds
2026-09-20
Giulio2002/bend-sha256
B
SHA-256 in Bend with a machine-checked functional correctness proof against a separate FIPS 180-4 specification.
2026-09-16
kschweig/transductARC
B
Training transductive autoregressive transformers for ARC-AGI 1 & 2
2026-09-15
just-a-go/SOVA
B
Official implementation of SOVA: Strict Outcome-Conditioned Virtual Advantages for Video Reasoning. Correctness-oriented calibration for TW-GRPO without extra rollouts.
2026-09-05
Thinking-Space/One-Shot-OPD
C
Rethinking On-Policy Distillation of Large Language Models II: One Training Example
2026-08-31
PixelML/club-170hx
B
Community-tested recipes, diagnostics, and benchmarks for CMP 170HX AI workloads
2026-08-26
jsdhwfmax/EvalForge
B
Open-source quality and security evaluation platform for RAG applications and AI assistants.
2026-10-25
msaroufim/openkernelchallenge-audit
B
Evidence, limitations, and reproducible static analysis of Open Kernel Challenge and KernelBot similarities
2026-10-09
mcpdelta/mcpdelta
B
Delta MCP is a free app that sits between your AI apps and their MCP servers: one program per task instead of one tool call per step, up to 24.1× fewer tokens in our benchmarks. Every change is recorded and undoable. Coming soon for macOS, Windows and Linux.
2026-09-21
masonlee39/Multi-Agent
B
One orchestration engine for multi-agent work: warm persistent sessions, a durable mailbox, no manager LLM in the loop — with per-task token accounting so speed and cost are measured, not assumed.
2026-08-27
tonyd2wild/Qwen3.8-Flash-Next-NVFP4-DGX-Spark
B
Day-0 deployment of Qwen3.8-Flash-Next (NVFP4) on 2x DGX Spark TP2 via SGLang. Includes the SM121 QSA kernel-guard fix, launcher, benchmarks, and full report.
2026-10-02
Tensward/tensward
B
Profile and diagnose LLM serving on your own GPU. Measures your vLLM setup on your real workload and suggests what to change.
2026-09-27
ai-jiaqian/cannbench-operator-solutions
C
Official CANNBench best operator sources with Job provenance and synchronization
2026-09-21
intikhab49/open-jev-typed-decision-engine
B
Open reproduction of TypeSafe Jev: a 150M typed decision engine (noul/choice/score in one non-autoregressive pass, calibrated confidence). 0.697 vs Jev's 0.727, 2.5x better calibrated, 4x faster, free. Trains on a Colab T4 in 30 min.
2026-10-09
Show HN: Jevman – AI decision models play Pac-Man
C
Openai just launched their decisions endpoint, cloudflare launched clef the other week, and many more jev alternatives are out there. We wanted to put the popular ones to the test and thought Pac-Man is a good benchmark for simple and fast decision making. So we let jev 1.13, kev, clef, clef flash, GPT-6 Luna and Laya play Pac-Man against bot ghosts. The low latency of these models allows for real time play. We had each model play 100 games, published a leader board and open-sourced the repo so anyone can run their own model and join the ranking. Link to repo: https://github.com/opper-ai/jevman-benchmark/blob/main/CONTR... You can also join the game and play as Pac-Man yourself, and the ghosts are the models, either a mix of models or all jev, kev, clef etc. A game costs about 2 cent, all models are running via my startup opper, and we added free credits for everyone to try. It's pretty fun to play and surprisingly difficult to beat jev's highscore. Any feedback is more than welcome!
2026-10-05
arjaythedev/twelve-dollar-server-challenge
C
Implement one small API on SQLite in any language; we benchmark it on a $12 server (1 vCPU, 2 GB)
2026-10-02
kingofspace0wzz/cua-swe
C
CUA-SWE: When Computer-Use Agents Meet Visual Software Engineering. 105 tasks across Web, Game, Mobile, and DevOps.
2026-09-30
0xSero/local-ai-recipe-kit
C
Run a big model on your own GPU, measure it with one protocol, and submit the result. Intake for local-ai-registry recipes.
2026-09-16
GraafHenk/numberwang
C
A small neural network that decides whether a number is Numberwang.
2026-09-15
yijieyuan/kaggle-pokemon-tcg
C
1st-place solution for the Kaggle Pokémon TCG AI Battle Challenge
2026-09-25
Simreal-AI/MathmoBench
C
[Public preview] An advanced AI Benchmark for math, produced by mathmo at St John's College, Cambridge.
2026-09-22
wdobry/laya-playground
C
A website, two games, a benchmark and an agent skill for Laya, the open-source decision model. Runs on your machine.
2026-09-15
Ikaleio/lm-detector
C
Model fingerprint detection with a shared Web and Bun CLI reference bank
2026-09-07
matsuo-koya/minimax-h3-notes
C
Measured speed-ups for MiniMax H3 on one RTX 5090 (ComfyUI): INT8 attention, FastH3 LoRA conversion, distributed segments, and what didn't work
2026-10-14
Acm1011/ProgramGym
C
Scaling Training Environments for Long-Horizon Program Reproduction Agents
2026-09-30
alex-ybuild/humanbench
C
How many B are you? How many B are you? A meme quiz where you play the LLM, open-sourced with the full creative trail: every prompt, briefs, and a 3-part retro.
2026-09-25
terryds/bestvaluemodel
C
Which LLM is worth it: Artificial Analysis Intelligence Index vs API price, refreshed daily
2026-09-04
dreamers-laboratory/timeseries-atlas
C
A map of modern time-series forecasting architectures, each with a short README and runnable code.
2026-08-29
ukpkhkk/SAO_and_CompactionRL
C
This project implements asynchronous pipelines, SAO algorithm and CompactionRL compression strategy based on the Slime framework, and provides reproducible Qwen3-4B experimental scripts for Tool-Intergreated Reasoning.
2026-10-17
apple-aiml-research/ml-rsi-inference-bench
C
A Long-Horizon Benchmark for Self-Improving Inference Systems
2026-09-16
QLNI/SHADOW-50M-Instruct
C
SHADOW 50M Instruct: a proof of concept. 44M ternary parameters, exact circuits inside the model, memory on disk, 19.8 MB, offline on a CPU.
2026-08-26
Trae1ounG/PostTrainBench0
C
[Blog] PostTrainBench⁰ — Can LLM agents automate LLM post-training without gradients?
2026-09-30
KickItLikeShika/composimplex
C
Composable LLM decoding on the probability simplex: library and paper code.
2026-09-24
Mocas-12/ai-model-rankings
C
Real-time AI model power rankings (水墨战力榜) — usage, benchmarks, price-performance & speed from OpenRouter, as an ink-wash Streamlit dashboard.
2026-09-18
YunahJang/AdaptiveRAG-Robustness
C
Official repository for "How You Ask Matters! Adaptive RAG Robustness to Query Variations".
2026-10-26
Kaigen-Technologies/net-crowd-stress-test
C
A plaza of networked players on one server: a netcode stress test for the Kaigen Engine
2026-10-05
PhysicalAIBenchmarks/Text2WetLab
C
Anthropic Science Lead: the gap in the next 2 years, will be the orchestration layer of taking experiments described in natural language and orchestrating them in the real world. The two orchestration layers we will test L1 and L2 is liquid-handling and fine-grained instructions.
2026-09-30
ThomasK1018/TabAlign
C
Official code for "From Table to Cell: Attention for Better Reasoning with TabAlign" (NeurIPS 2026)
2026-09-19
r-ms/mini-jev
C
mini-Jev: what a Jev-style typed-decision interface looks like on a frozen Qwen3-4B — read the option letter's logits instead of generating JSON. Preregistered experiment, results, teaching bench.
2026-10-24
EgorBo/BlazingFastDotnet
D
Tips & tricks to tune .NET workloads for better perf (mostly, targeting benchmarks).
2026-10-24
JackMin7/supermimic_whole_body_tracking
C
On the same static standing task, compares MBRL (SuperMimic) with PPO (original BeyondMimic), with each group trained for about 1 hour cumulatively.
2026-10-03
zsxm1998/awesome-perception-aware-rlvr
C
Unified reproductions, benchmarking & evaluation for perception-aware RLVR in VLMs. 11 methods · 25+ benchmarks · Official code of CGPO (ACM MM 2026 Oral).
2026-10-02
Random8756/Bench2Drive-RLinfra
C
Scalable reinforcement learning infrastructure for autonomous driving, supporting privileged planning and end-to-end policy RL post-training in CARLA.
2026-09-22
rlaope/jeval
C
Measures what your Jev classifier's confidence is really worth, and sets the human hand-off line from what a mistake costs.
2026-09-20
Heman10x-NGU/openJev-verdict-2.0
D
Calibrated 151M Non-Autoregressive Decision Engine beating TypeSafe Jev & Laya on LocalLLaMA/typed-decisions (77.10% acc, 0.0636 Brier, 0.0144 ECE)
2026-09-30
Show HN: TurboGPT: train 22KiB transformer in 13s
D
Inspired by minGPT for home experiments. Requires CUDA 13.4; build script is Windows only.
2026-09-22
smkrv/jev-calibrate
C
Calibrate Jev questions against your own labels: tune criteria on labelled examples, confirm on a held-out set, get a verdict per question. Unofficial.
2026-10-23
cloudrift-ai/emmy-hackathon-2026
C
Emmy Hackathon 2026: write a compiler that turns Emmy's Kernel IR into PTX
2026-09-19
giuliosmall/pg_typesafe
C
Pre-alpha PostgreSQL extension for TypeSafe AI (Jev) categorical classification
2026-09-03
pangoleen/qwen3.8-27b-dgx-spark-dflash2
C
Qwen3.8-27B on one DGX Spark: SGLang + DFlash2 recipe, engine image, benchmarks, data and charts. By @redp314
2026-10-14
huacnlee/gpui-shell-benchs
C
A collection of gpui-shell applications generated from scratch by AI coding agents.
2026-10-14
yeyuxin777/baoyan-agent-benchmark
D
Sprix AI at 屿智同行 × 保研岛 — an open-source benchmark for AI agents in Baoyan advising and application planning.
2026-09-25
SilentFleetKK/qingming-bianjing
C
Same prompt, two cities — a real-time 3D Bianjing benchmark for LLMs.
2026-09-24
allebee/jevk5
C
JevK5: open-weight alternative to TypeSafe Jev. Typed decisions with probabilities in one forward pass; Apache-2.0 weights and code.
2026-09-22
virajbhartiya/laya-vs-jev
C
Laya vs Jev: local MLX and hosted AI decisions playing T-Rex side by side, with live metrics and replay recording
2026-10-21
Show HN: Jev Cards, Can you beat a vision model at a picture quiz?
D
I built a quiz game around TypeSafe's new Jev model, which has vision capabilities. Jev looks at a picture and a question and picks an answer (or taps a place) in about a quarter of a second. I've always loved quiz games and thought this would be a fun way to see what it can do and how fast it is. I made the cards from scratch, and Jev gets exactly the picture you see. More cards will come over time. Play against Jev on your own (there's a leaderboard for each mode), or start a live match and challenge your friends, with Jev as one more player. The code is open source, so feel free to build your own game or make more cards that I can publish. Play: https://jevcards.com Code: https://github.com/Jose-Sabater/jevcards (the newest cards stay private for a while) Disclaimer: I ran Jev through both TypeSafe and Opper, and made the cards through Opper's vision models and Claude Code. I work at Opper, but jevcards is open source and free to play.
2026-09-22
openlayer-ai/jevals
C
Agent evals and guardrails in one request. Built on Jev, Kev and Laya.
2026-09-30
LiXin97/DN-MOPD
C
Beyond Teacher Assignment: Domain-Normalized Multi-Teacher On-Policy Distillation (DN-MOPD)
2026-09-20
Zaious/jev-capability-atlas
C
Independent, evidence-based map of when TypeSafe's Jev actually holds up vs. breaks down — real API-call receipts, not a leaderboard. A primarily Chinese bilingual repo.
2026-09-05
MikeCaldera/intel-arc-pro-b70-qwen38-vllm
C
Reproducible Qwen3.8-27B GPTQ INT4 + MTP4 inference on Intel Arc Pro B70 using vLLM XPU — 84.65 tok/s median.
2026-08-30
unclebob/pi-two-pack
C
Arbitrary-precision pi via the Gauss–Legendre (Brent–Salamin) algorithm
2026-09-24
shapsider/OmniJev
C
OmniJev — multimodal finite-choice decision interface and MuJoCo embodied workbench: trajectory replays, decision probes, benchmark panels, 60s walkthrough.
2026-10-07
AetherLabsAI/Video2World
C
Benchmarking Coding Agents for Interactive World Modeling from Embodied Videos - Rethinking Embodied Real2Sim from a Software Engineering Perspective
2026-09-15
Show HN: Pelican-bicycle alternatives
D
In November and December 2025, inspired by Simon Willison’s pelican-riding-a-bicycle benchmark, I had some then-current LLMs create SVGs from thirty similar prompts, such as “Generate an SVG of an octopus operating a pipe organ.” Simon mentioned that experiment on his blog [1]. Nine months have passed and much stronger models have been released, so I tried the experiment again today. The linked site shows the results. Running ten of the prompts through six models at OpenRouter cost about twenty dollars, so I stopped there for now. [1] https://simonwillison.net/2025/Nov/25/
2026-09-04
gitcommit90/glm-5.3-one-spark
C
GLM-5.3-Flash at 64 tok/s on one DGX Spark — TP1 EXL3 2.05 + DFlash2 K7
2026-08-29
kadenball/qwen38-27b-rtx3060-dcfr
C
Qwen3.8-27B at 64K on one RTX 3060 12GB: D-CFR llama.cpp research patch and benchmarks
2026-08-28
drowzeys/keys-vLLm.0.27.1-GLM-5.3-Flash-NVFP4-NVFP4KV-1M-Context-Abliterated
C
GLM-5.3-Flash (Z.ai) NVFP4 (LibertAI) + NVFP4 KV, 1M context on 2x DGX Spark GB10 — vLLM TP2 recipe, walls ledger, benchmarks, needle to 1.04M; abliteration with blackforest_AI next
2026-10-24
Show HN: Searching for optimal strategies in a puzzle game with chance (Drop7)
D
Sixteen years ago, I wrote a Java program which tried to pick the ideal move in a puzzle game called Drop7. On each turn, the player only has seven choices for their next move, but there is also an element of chance which turns this simple game into a surprisingly difficult problem involving search, hidden information, long-term planning, and chance. I renewed my investigation this weekend, into whether there is an autonomous strategy that averages more than one million points per game. An experienced human player can score a million points with long-term strategic thinking, but the best automated strategies I could come up with are currently averaging around 300k-400k points. One of the easiest ways to contribute to advancing this research is to compete on the global leaderboard and submit your human strategy - https://drop7.dev/compete. This project has also been an enlightening experiment into using agents as research collaborators and evidence collectors. Repository: https://github.com/keshavsaharia/drop7-bench I’d especially appreciate feedback from people who are knowledgeable in game theory, stochastic planning, learned value functions, or heavy-tailed evaluation. How would you approach long-horizon action ranking here? I’d also love to hear whether the research makes sense to someone encountering the project for the first time.
2026-09-15
imMamdouhaboammar/satisfy-your-agent
C
Give your coding agent a short break after real work, then measure what happens next. Universal agent skill & experiment harness.
2026-10-09
Spydonny/s6e8-lookup-transformer
D
My Lookup-Transformer solution for Kaggle S6E8: public LB ROC AUC 0.97041. Adopted in a 14th-place solution; 11 verified Kaggle references.
2026-09-19
luxingcom/LuZ-0.1.7-DeepSeek-v4.1-Flash-DGXspark-TP4-Ring
D
DeepSeek-V4.1-Flash on 4x DGX Spark TP4 switchless RoCE ring - SGLang production recipe, 600K context, DSpark MTP, benchmark archive
2026-09-16
kaka86mm/dsv41-flash-pp4-170hx
D
DeepSeek-V4.1-Flash (764B) on 4×CMP 170HX: PP4 + shadow KV + EXL3 2bpw + DSpark + 512k ctx + native vision — patches, recipes, acceptance bench, honest findings
2026-09-06
majiayu000/stealthprint
D
Fingerprint analysis of the omen-alpha stealth model (GLM-5 tokenizer, ~1M context, Rust gateway) + reusable test toolkit for any stealth model
2026-11-02
Interactive-Active-AI/NextTransformer
D
:sunrise: A comprehensive, cutting-edge open-source framework for next-generation Transformers, with compute-matched benchmarking and a platform for autonomous architecture research.
2026-09-25
Reza2kn/Bev
D
Ternary decision scoring with Jevfire-style inference, typed APIs, and reproducible Persian evaluation.
2026-09-19
AbdelStark/jev-benchmarks
D
Probability-aware evaluation for typed decision models: calibration, selective risk, latency, and reproducible benchmarks.
2026-09-18
JoshuaSP/open-jev
D
Typed JSON inference with DiffusionGemma, with Every and Jev benchmark results
2026-10-19
Elevatormusic/DeepSeek-V4.1-Flash-1017-EXL3-TensorFold-2x-DGX-Spark
D
DeepSeek-V4.1-Flash-1017 EXL3 2.9bpw on 2x NVIDIA DGX Spark (GB10) with TensorFold: 2.1-2.3x the vLLM kit's decode speed.
2026-09-21
NanmiCoder/jev-arena
D
Jev model introduction and hands-on test: converts natural language into typed judgments and probabilities via Choice / Score / Noul, for classification, scoring, and routing; supports comparison with models such as DeepSeek on comment labeling, speed, and results, with CSV/Excel import, original-speed replay, and offline reports.
2026-09-21
erendikmenn/jev-rag-benchmark
D
Reproducible benchmark for measuring Jev reranking quality, latency, and cost in RAG
2026-09-21
fstandhartinger/jevbench
D
JevBench v1 - a benchmark for Jev-class typed decision models: smart, cheap, fast, reliable, open.
2026-11-01
CrossCommissioner/CrystalDiskMark-Portable
D
CrystalDiskMark Portable – pre‑packaged, password‑protected portable edition of the industry‑standard disk benchmark. Fast, reliable, zero‑install. For Windows 10/11.
2026-09-18
iammrduncan/typesafe-ai-benchmark
D
This is a LLM Gateway that mimics typesafe ai structured output. Like an imposter Jev.
2026-10-19
Show HN: 500 clocks in one 6.3 MB HTML file by Opus 5.5
D
Lately I've been testing frontier models by having them create self-contained web pages that display clocks--sort of my personal version of Simon Willison’s pelican-on-a-bicycle metric. I have been particularly impressed by the variety, interest, and quality of the clocks created by Claude Opus 5.5. Over the course of several sessions in the Claude chat interface, I had it build a total of 500 clocks. Claude’s summary of the prompting process and page design is here: https://github.com/tkgally/500-clocks/tree/main