I Made 6 Local LLMs Write a Holiday-Aware Calendar Script — This Time With US Holidays, and One Finally Got Everything Right

Last week I ran an experiment that ended in a massacre: I asked six local Qwen models (running on a single RTX 3090) to write a Google Apps Script that generates a yearly calendar with Japanese holidays highlighted and named. All six failed. The only script that ran without errors printed holidays from a law that was repealed ~30 years ago. (Japanese write-up here.)

A natural follow-up question: was the task hard, or was Japan hard?

So I reran the exact same experiment with one change — the prompt is in English and asks for US federal holidays. Same machine (RTX 3090, 24GB VRAM), same Ollama 0.32.6, same settings (8192 context, temperature 0.3, thinking off), one generation per model, no retries. Every promising script got executed in a real Apps Script project, driven by browser automation, and I verified the output spreadsheets by hand.

The answer: Japan was hard.

Scoreboard: Japan round vs. US round

US round, model by model

The winner: qwen3.6:27b

In the Japanese round, 27b had the smartest design and died on a hallucinated API call. In the US round it delivered the complete package: correct getNthWeekday helpers (3rd Monday of January is actually computed as the 3rd Monday), Juneteenth included, MLK Day on Jan 19, Presidents' Day on Feb 16, Thanksgiving on Nov 26 — all verified in the generated sheet.

This is the first fully correct output across both experiments: 12 attempts (6 models × 2 countries), 1 perfect score.

The instructive failure: qwen3-coder:30b

The coding-specialist model produced the cleanest-running code again — and a wonderfully systematic bug. Its floating-holiday formula anchors the calculation to the weekday of January 1st instead of the target date, so MLK Day, Presidents' Day, Memorial Day, Labor Day, and Columbus Day all land exactly one day early — on Sundays:

A calendar where every federal holiday falls on a Sunday is the kind of output that looks fine at a glance and quietly poisons whatever consumes it. (It also skipped Juneteenth, a 2021 addition — training-data vintage strikes again.)

The overachiever: qwen3.6:35b

The strangest result. In the Japanese round, 35b annotated its own holiday data with "(wrong)" and shipped it anyway. In the US round it hardcoded every 2026 federal holiday correctly from memory — MLK Jan 19, Memorial May 25, Thanksgiving Nov 26, Juneteenth included. Then it crashed on the very first cell write, because it wrapped an array inside a 1×1 setValues call. Right data, dead code — the exact mirror image of coder:30b.

What the "correct" solution looks like

Same as the Japanese version: don't hardcode holidays, ask Google. Apps Script can read Google's US Holidays calendar directly:

const HOLIDAY_CALENDAR_ID = 'en.usa#[email protected]';

function fetchHolidays(year) {
  let cal = CalendarApp.getCalendarById(HOLIDAY_CALENDAR_ID);
  if (!cal) cal = CalendarApp.subscribeToCalendar(HOLIDAY_CALENDAR_ID); // see note below
  const events = cal.getEvents(new Date(year, 0, 1), new Date(year + 1, 0, 1));
  // filter observances (Valentine's Day etc.) against the federal holiday list...
}

Two pitfalls I hit while verifying, which explain why no model found this route:

1. On a non-US Google account, getCalendarById returns null for the US holiday calendar — you have to subscribeToCalendar first. (Interestingly, the reverse route for Japan's "official holidays only" calendar is blocked entirely: subscribing throws "Action not allowed.")
2. Google's holiday calendars mix observances (Valentine's Day, Halloween...) with actual federal holidays, and the event description field that should distinguish them is empty. You still need a name whitelist on top.

The reference implementation with those fixes generates the 11 federal holidays of 2026, named and colored, in one run:

The takeaway: your local LLM is only as good as its training-data neighborhood

Same models, same task, same prompt structure. Change the country and the results flip:

• US civic knowledge is abundantly represented in training data. Three models could recite or compute federal holidays; one got everything right. Nth-weekday rules ("4th Thursday of November") are well-trodden textbook problems in English.
• Japanese civic knowledge is sparse and full of stale revisions. Holiday laws changed in 2000, 2003, 2016, 2020 — and the models serve a random vintage of that history with full confidence.

If you're deploying local LLMs for coding in a non-English locale, the failure mode is not "the code doesn't run." It's this: the code runs, the output looks plausible, and the domain facts inside it are quietly wrong. The only defense is executing the code and checking the output against ground truth — which is exactly the step I automated here (Puppeteer driving the Apps Script editor; the same pipeline from the Japanese article):

Total cloud API cost for both experiments: $0. Everything ran on one RTX 3090 next to my desk.