Random Learning
← The journal

July 25, 2026

3 things I learned

last30days v3.3.2 · synced 2026-07-25

Note on evidence: this 30-day window is heavy on tooling docs, calibration guides, and fresh papers but thin on first-person "our judge lied to us" postmortems (YouTube returned zero videos, and most Reddit hits in the window were open-weights politics rather than eval practice). The strongest practitioner signal is a small number of high-engagement X posts plus one honest r/mlops thread, so the "where it quietly breaks" read below leans on calibration methodology and the July paper crop more than on a large body of war stories.

What I learned:

The 30-day delta is that the argument moved off "is model grading good enough" and onto "show me the kappa number." - The most honest artifact in the window is r/mlops asking flat out "How are you all actually evaluating LLM/agent systems in prod? LLM-as-judge feels shaky" - which is the question the tooling side is now answering with a specific statistic rather than a vibe. Arize's July 22 methodology post spells out the ritual: have multiple annotators apply a draft rubric, calculate agreement, inspect the recurring disagreements, revise, then "freeze the rubric," annotate the full dataset, adjudicate, and only then run the judge and compare its labels against both the individual human judgments and the aggregated reference. The number teams quote as the pass mark is Cohen's kappa above 0.6 on the target slice, piloted on a couple hundred dual-labeled examples and re-sampled monthly to catch judge drift. The headline "80-90% agreement with human" that Langfuse's docs cite for GPT-5-class judges is real but is the ceiling for a well-specified task, not the default you get for free.

Where the match holds is narrower than the marketing: frozen rubric, coarse scale, reference answer, one narrow question at a time. - TMLS's "Scaling Laws for LLM-as-Judge Agreement" lands the most useful concrete finding of the month - when you cannot avoid an ordinal scale, "a zero-to-five scale gives the highest human-LLM alignment, a sweet spot between too coarse to discriminate and too fine to be consistent," and supplying a reference answer moves alignment further. The same instinct shows up in Ask, Don't Judge, a June 25 paper arguing for decomposing a quality score into binary questions instead of a Likert rating, precisely because the Likert judges "remain susceptible to position, verbosity, and self-enhancement biases." The practitioner version of that advice is buildevals.com's course spine: golden cases first, then "scaling grading with LLM-as-judge," then "rubrics that turn 'good' into something checkable." Nobody credible in this window claims a judge reproduces holistic human taste; they claim it reproduces a checkable rubric.

Where it quietly breaks is the important word - the failures are not noisy, they are systematically optimistic, and the newest one is the hardest to see. - The classic three are catalogued plainly on the LLM-as-a-Judge Wikipedia page: position (order) bias, verbosity (length) bias, and self-preference. Galtea's guide gives the operational read - self-preference is "the tendency for a judge to favour outputs from the same model family it belongs to," and the fix for length is not a smarter model but putting "length neutrality into the rubric explicitly." The worked disaster case circulating in calibration guides is a team grading GPT-4 outputs with a GPT-4 judge and discovering, once a domain expert labeled 50 production traces, a judge-expert kappa of 0.31 - the judge had been over-rewarding its own family and under-penalizing fluent hallucinations. The genuinely new failure mode is Preference Leakage, accepted to ICLR 2026, which shows the evaluator's preferences leak into the model being graded through synthetic training data whenever generator and judge are the same model, inherit from each other, or share a family - and the authors' own framing is that it is "harder to detect compared to previously identified biases." Even the retrieval-side papers agree the gap does not close: As It Was reports its judges' correlations with human relevance judgments "remain moderate, leaving a substantial gap to perfect agreement with human annotators."

The sharpest practitioner verdict this month is that automated eval tooling catches what humans miss and still cannot carry taste. - @HamelHusain put real production data through the new crop of AI-trace-analysis tools and published the split in Do Automated Evals Work?: "Where they shine - They often spot issues human miss - Integrate into your workflow: viewing traces, creating LLM judges etc. Where they fall short - They miss problems that require domain expertise and taste - Don't have great mechanisms to learn from human feedback." That last clause is the load-bearing one and it maps directly onto why kappa decays - a judge with no feedback loop back to human labels is a frozen snapshot of last quarter's taste. The adjacent pain is visible on r/AI_Agents asking how long it takes anyone to understand why a production agent failed - scoring the output is the easy half.

The architecture teams are converging on is judge-as-one-scorer-among-several, with the same scoring logic running before ship and in production. - The highest-engagement eval post in the window is @4rblaber's breakdown of an Anthropic Applied AI deep dive on eval harnesses, and its chapter list is the whole design pattern: "Moving from text output to agent outcome evaluation," "Converting production failures into test tasks," and - tellingly - a dedicated segment on "Code-based checks vs LLM-as-a-judge calibration." The same split shows up in Chier Hu's teardown of a production eval system as three parts - versioned golden datasets, an eval runner that executes the system-under-test, and "scorer/grader abstractions - a common interface spanning code-based" and model-based graders - and in Harness's July 21 launch selling the continuity directly: "The scoring logic you use to test an agent before it ships is the exact same logic that scores it in production. That means no gap between what you validated and what you're actually measuring." The tooling weight behind this is real and live: langfuse at 32K stars, promptfoo at 24K, and deepeval at 17K all ship judge primitives as a first-class abstraction rather than the whole product.

The contrarian move gaining ground is to delete the judge where determinism is available, and to optimize rather than hand-write it where it is not. - A Deterministic Replacement for LLM-as-Judge in Stateful Agent Evaluation surfaced on HN on July 3 making the case that for agents with inspectable end-states you should assert on the state, not ask a model how it felt about the transcript - which is the same instinct behind the Anthropic harness talk's "verifying system end-states" framing. Where a judge is unavoidable, the emerging discipline is to stop writing the judge prompt by hand: @BestBlogsDev flagged Dropbox using DSPy to tune its LLM judges, treating the rubric as parameters to optimize against human labels rather than prose to argue about. The background hum under all of it is the r/mlops line that generalizes past inference to grading: "Your LLM inference framework won its benchmark. Your production traffic didn't care."

KEY PATTERNS from the research:

  1. The pass/fail question is now a statistic, not an opinion - compute Cohen's kappa against a human-labeled sample, target above 0.6 on your slice, and re-sample monthly for drift, per Arize's alignment methodology.
  2. Agreement holds where the task is narrow and the scale is coarse - a frozen rubric with a reference answer and a zero-to-five (or binary) scale, per TMLS's judge scaling laws and Ask, Don't Judge.
  3. The breakage is optimistic, not random - position, verbosity, and self-preference bias push scores up, and the ICLR 2026 preference-leakage paper adds a contamination channel its authors call harder to detect than any of them.
  4. Never let the generator and the judge share a family - the canonical worked failure is a GPT-4 judge on GPT-4 outputs hitting 0.31 expert kappa from family bias plus under-penalized fluent hallucinations.
  5. The shipping pattern is a hybrid harness, not a judge - code-based checks and end-state assertions where determinism exists, a calibrated judge for the reasoning-shaped residue, humans on the flagged tail, and the same scorer running pre-ship and in prod, per @4rblaber's Anthropic harness breakdown and Harness. @HamelHusain is the check on the whole thing: the tools "miss problems that require domain expertise and taste."
last30days v3.3.2 · synced 2026-07-25

Note on evidence: no single "I photographed every house on my street" project went viral inside this 30-day window. YouTube returned zero videos, Reddit surfaced no typology-specific threads, and every engine item landed with an entity-miss demotion. What the window does contain is three separable layers: a major Becher retrospective mid-run, the method's own vocabulary circulating on X one frame at a time, and the highest-engagement architecture-adjacent Reddit communities using the deadpan-grid grammar without ever naming it. The read below is built from those layers rather than from one viral artifact, and the "keeps going viral" premise is treated as a question rather than assumed.

What I learned:

The live anchor in this window is a retrospective that sells the grid as a method, not as a body of work. - Fondazione MAST in Bologna is halfway through "Bernd & Hilla Becher. History of a Method," 350+ original black-and-white prints across 10 sections plus drawings, books and posters, free admission, running April 23 to September 27 2026, with a companion Düsseldorf School show in the MAST.Foyer. FOTO Cult states the pitch plainly: the water towers, silos and gas holders matter because the Bechers turned them into "a method that continues to guide contemporary photography." A second show, Blast Furnaces at Die Photographische Sammlung in Cologne, was running in the same window. The title is doing the work. Institutions are currently marketing a reproducible procedure, which is precisely the thing an amateur with a phone can copy on their own street this weekend.

The mechanic behind the virality is that a grid hands the viewer a sorting task a single photo cannot. - The sharpest line in the whole corpus comes from Elle Decor Italia's coverage of the Bologna show: Hilla Becher compared typologies to biological systematics, where (translating the Italian) "every specimen belongs to a family, a species, a subspecies." One photo of a house is a picture. Twelve houses in a 3x4 grid is a taxonomy, and the viewer cannot stop themselves ranking, matching and hunting for the odd one out. The academic framing in Landscapes of Repair reduces the entire genre to that one mechanic: "series of black-and-white photographs organized in grids." The engagement is participatory, not aesthetic, which is why the format survives translation to a phone screen.

On X this month the typology circulates one frame at a time, and the most-quoted line is the anti-art one. - The Becher posts in the window are all single images from architecture accounts rather than grids: @Bertrom's Verviers water tower from 1983 was the highest-engagement item at 44 likes and 13 reposts, and @archinerds ran the Lübeck-Herrenwyk factory as a daily-dose architecture post. The most telling repost is @EvergreenDazed quoting the Bechers directly: "The question if this is a work of art or not is not very interesting for us." That disclaimer is the permission slip. It is what lets a hobbyist shooting bungalows say they are documenting rather than making art. A Japanese thread from @room402g is the only post in the window that explains the method itself, walking through the neutral objective gaze and citing "Gas Tanks" (2007) alongside "Framework Houses" (1973), which is the literal ancestor of the shoot-every-house-on-a-street idea and the one Becher series that was houses rather than industry.

The living descendants are subreddits running the deadpan grammar at enormous scale while never using the word "typology." - r/AccidentalWesAnderson is the format's mass-market form: flat, centered, frontal, one ordinary building per frame, with a floating convenience store at 8,417 upvotes and a kid's drawing at 13,211 upvotes. r/UrbanHell is the same grammar aimed at judgment instead of delight, with a Toronto waterfront shot at 15,419 upvotes and 1,270 comments and a 420-comment argument literally titled "Why are commie blocks so hated?". The repeat-form vernacular post also carries r/architecture this month, where "Another Collection of Contemporary Brick Architecture in Iran" pulled 2,673 upvotes and "1980s Iraq's Neo-Traditional Architecture" pulled 615. The demand for the format is real and large; the label is not what travels.

Contemporary typology has migrated off the gallery wall and into the numbered edition. - The most direct modern application in the window is not a house project at all: @glitchmarfa's notes on DRIVE // LAP 2, a 333-photograph collection of vehicles by @dave_krugman anchored to Ethereum, where the artist "shoots each image from the same lateral angle" so a decade of unposed street encounters resolves into one archive. That is the Becher rule set exactly, with the grid replaced by an edition count. It also explains the drift: the typology's real payload was always the series as an object, and a mint list preserves that better than a feed does.

The format keeps reproducing mostly because it is trivially teachable, which is also its ceiling. - The rule set is short enough to be a worksheet: Digital Photography School's mini-typology guide makes consistency "the golden rule" (same angle, same distance, same time of day, 6 to 12 frames), and Fstoppers reduces deadpan to flat lighting, neutral palette, low contrast and centered framing instead of rule of thirds. It has hardened into curriculum: PhotoPedagogy runs a typologies module for UK A-level students and Kansas 4-H ships a dated "Typology Photography - 2026" project sheet. The counter-current worth holding onto is that the grid is not sacred even to the people closest to it: per Wikipedia, when Hilla Becher curated "August Sander / Bernd and Hilla Becher: A Dialogue" in 2014, she inverted the convention, hanging the Becher buildings as singular "portraits" while Sander's photographs of people were the ones arranged as typological grids.

KEY PATTERNS from the research:

  1. The 30-day driver is institutional, not viral: the MAST Bologna retrospective is explicitly titled "History of a Method" and runs free through September 27, which is what keeps putting a copyable procedure in front of people.
  2. Grids go viral because they convert looking into sorting - Hilla Becher's own framing was biological systematics, family/species/subspecies, per Elle Decor Italia.
  3. The permission slip is the Bechers' own disclaimer that whether it is art "is not very interesting for us," which is the single most-reposted Becher line in the window, per @EvergreenDazed.
  4. The format's real mass audience calls it something else entirely - r/AccidentalWesAnderson and r/UrbanHell run the deadpan frontal vernacular frame at five-figure upvote scale without the vocabulary.
  5. The live evolution is series-as-edition rather than series-as-grid, exemplified by the 333-frame same-lateral-angle DRIVE // LAP 2 archive, per @glitchmarfa.

Provenance — 2026-07-25

Redacted by design: this records the funnel shape, not the private source links or personal capture notes. Raw self URLs and why? text are never written here.

Source entries (3 picked, topic-level only)

  • A saved CLI test runner that evaluates agent skills by grading the same prompts with and without the skill loaded (tags: ai, agent-skills, evaluation, cli). Genuine personal pull: it maps onto a standing plan to add skill evaluation to an existing personal tool, which makes "can you trust a model to grade this?" the load-bearing question.
  • A saved MIT CSAIL three-sided Y-zipper (tags: zipper, 3d-printing, flexible-technology, robotics, wearables, durability) — one of the freshest captures in the pool. Pull: admiration for the mechanical design, with the interesting substance being how a 3D-printed elastic structure survives 18,000 zip cycles.
  • A saved interactive split-house photo mixer that recombines halves of real house photographs (tags: interactive, photography, mixer, composite, digital-art) — also a same-day capture. Pull: enthusiasm for the art project.

Chosen for domain spread (AI evaluation · mechanical design · photography) and to break out of the AI-infrastructure cluster that has dominated recent days.

Fan-out — 12 adjacent candidates (all cleared the near-dup guard)

From the skill-eval entry: 1. LLM-as-a-judge reliability ✅ picked 2. Ablation-style A/B evals for prompts and agent skills 3. Regression testing non-deterministic LLM output 4. Agent skill versioning and iteration workflow

From the Y-zipper entry: 5. Compliant mechanisms: flexures instead of hinges ✅ picked (merged with print-in-place) 6. 3D-printed mechanical metamaterials 7. Rapidly deployable shelters and structures 8. Soft robotics actuators and wearables 9. Fatigue and durability cycle testing for consumer hardware

From the photo-mixer entry: 10. Combinatorial generative art from photo archives 11. Photographic typology and vernacular architecture documentation ✅ picked 12. Print-to-web companion interactive art objects

Near-dup guard: all 12 scored below the dedup threshold against the published index (highest ≈ 0.16 — "LLM-as-a-judge reliability" vs the prior Context-rot day, and "combinatorial generative art" vs the prior Generative-art-after-AI day). None flagged.

Narrowing to the top 3

  • #1 LLM-as-a-judge reliability — strongest curiosity (ties directly to the standing skill-eval plan), heavy live 30-day discussion, and concretely learnable: there are real numbers (agreement thresholds, documented failure cases) rather than vibes. Picked over #2/#4, which are the same question one level less fundamental.
  • #5 Compliant mechanisms, reframed as print-in-place flexure hinges — this is the actual engineering substance behind the source entry's durability claim, and the reframing is what gives it a live 30-day maker corpus to mine. Picked over #6 (thinner practitioner discussion) and #9 (too generic).
  • #11 Typology photography of vernacular architecture — the method underneath the source entry's house-recombination premise, and the most learnable of the three art candidates. Picked over #10 (closest to the already-published generative-art day) and #12 (narrower).

Final three span three distinct domains and do not overlap.

Evidence quality notes

All three briefs carry an explicit italic evidence note where recall was uneven, per the engine's honesty convention:

  • The eval brief is thin on first-person postmortems (Reddit search degraded to a keyless listing fallback; YouTube returned zero), so it leans on calibration methodology and the July paper crop.
  • The 3D-printing brief hit keyword collisions on "print" across X and a Reddit fallback to front-page content; the on-topic signal concentrated in high-view YouTube design guides and 2026 tolerance guidance.
  • The typology brief found that its own premise did not hold — no "every house on my street" artifact went viral in the window — and says so up front, rebuilding the read from an institutional retrospective, the method's vocabulary on X, and the subreddits running the format without naming it.