tag:civilytics.com,2026:social2026-10-28T12:01:21ZCivilytics Consultingtag:civilytics.com,2026:social/2026-10-07-vibecode-sessions2026-10-07T07:00:00-04:002026-10-07T07:00:00-04:00

On our Census of Governments API, AI agents caught nearly two-thirds of their own errors before anything shipped. Here's what I learned from the errors agents missed: I caught 12, tests caught 11, and both API outages came after I pushed an agent past its plan.

On our Census of Governments API, AI agents caught nearly two-thirds of their own errors before anything shipped. Here's what I learned from the errors agents missed: I caught 12, tests caught 11, and both API outages came after I pushed an agent past its plan. { "append_url": false, "append_url_if_truncated": true, "append_url_label": "🔗", "append_url_spacer": "\n\n", "attach_link": false, "content": "On our Census of Governments API, AI agents caught nearly two-thirds of their own errors before anything shipped. Here's what I learned from the errors agents missed: I caught 12, tests caught 11, and both API outages came after I pushed an agent past its plan.", "format_string": "{{content}}", "media": [ { "alt": "Human turns per interactive session, in order. 74 bars from April 16 to September 4 2026; half of the sessions take between five and 15 turns and the tallest takes 38. Filled circles mark the 27 sessions where the agent admitted at least one of the 75 errors, open squares sessions with an unclear prompt of mine. Both kinds are spread across the whole project.", "bytes": 107890, "height": 919, "mime": "image/png", "type": "image", "url": "https://civilytics.com/social/2026-10-07-vibecode-sessions/session-turns.png", "width": 1880 } ], "platform_overrides": { "bsky": { "append_url": true, "syndicate": true }, "linkedin": { "append_url": true, "content": "On our Census of Governments API, AI agents caught nearly two-thirds of their own errors before anything shipped. Here's what I learned from the errors agents missed.\n\nI went back through 74 recorded sessions and found 75 errors the agent admitted, spread across the whole project. The circles in this chart mark the 27 sessions they came from. Of the errors agents missed, I caught 12, tests and build checks caught 11, and four reached users.\n\nThe ones I caught came from domain knowledge or a direct question, so ask the agent to explain its designs before they ship. Both API outages came after the agent had finished its plan and I told it to keep going without writing a new one, so when the plan runs out, write the next one.\n\nAnd if you are doing software development of any kind, ask your agent to set up a strong set of test and build checks and review the test plan. Agents will happily build hundreds of inconsequential tests, so it is important to guide the agent to build tests around the cases that matter to your project. Sometimes simply saying something like \"I'd really like to validate this number against a published result.\" is enough for the agent to write a test that will help you out a lot later.\n\nWhat patterns are you using to catch errors before they reach production?\n\n#PublicInterestTech #OpenData #CapacityBuilding", "format_string": "{{content}}", "syndicate": true }, "mastodon": { "append_url": true, "syndicate": true } }, "syndicate": false, "truncate": true, "url": "https://civilytics.com/newsletter/vibecode-or-collaboration/" } tag:civilytics.com,2026:social/2026-10-07-vibecode-launch2026-10-07T07:00:00-04:002026-10-07T07:00:00-04:00

Getting good work from an AI agent on our Census of Governments API had more to do with planning, review and documentation than with how I worded prompts. The longer I kept steering a session past its opening instruction, the more often the agent admitted an error.

Getting good work from an AI agent on our Census of Governments API had more to do with planning, review and documentation than with how I worded prompts. The longer I kept steering a session past its opening instruction, the more often the agent admitted an error. { "append_url": false, "append_url_if_truncated": true, "append_url_label": "🔗", "append_url_spacer": "\n\n", "attach_link": false, "content": "Getting good work from an AI agent on our Census of Governments API had more to do with planning, review and documentation than with how I worded prompts. The longer I kept steering a session past its opening instruction, the more often the agent admitted an error.", "format_string": "{{content}}", "media": [ { "alt": "Share of the agent's work and of its admissions, for sessions grouped by how many messages I sent after the opening instruction's run: 0–3 messages (15 sessions), 4–9 (27) and 10 or more (26). The share of work rises from 15% to 35% to 50%; the share of admissions rises faster, from 2% to 23% to 74%.", "bytes": 101531, "height": 1000, "mime": "image/png", "type": "image", "url": "https://civilytics.com/social/2026-10-07-vibecode-launch/session-length.png", "width": 1600 } ], "platform_overrides": { "bsky": { "append_url": true, "syndicate": true }, "linkedin": { "append_url": true, "content": "There is a lot of discussion about prompt engineering that makes it seem like there are these magic incantations you need to learn and practice to get high-quality LLM results. The record from this project suggests a different lesson — the importance of review and engagement. I read back through 74 recorded sessions of building the Census of Governments API with an AI agent to find how often the model made mistakes and how they were caught. The answers are planning each session, questioning the agent's designs, grounding the work in published documentation, and careful review.\n\nThe longer I kept steering a session after its opening instruction, the more often the agent admitted an error. Sessions where I sent 10 or more follow-up messages hold half the agent's work and 74% of its admissions. And the consequential errors, the ones that took the API down, both have to be attributed to me.\n\nBoth times I took the API down, it happened after the agent had finished its plan and I had told it to keep going without writing a new one. I have learned from reviewing this project to better identify when the agent has stretched beyond the plan, but this can often be a very subtle transition. When in doubt, tell your agent to hand off and start a new session!\n\nOr in other words: when the plan runs out, stop and write the next one.\n\nWhere are you finding yourself getting into trouble using AI? What patterns have you seen in the sessions that seem to just go off the rails?\n\n#PublicInterestTech #OpenData #CapacityBuilding", "format_string": "{{content}}", "syndicate": true }, "mastodon": { "append_url": true, "syndicate": true } }, "syndicate": false, "truncate": true, "url": "https://civilytics.com/newsletter/vibecode-or-collaboration/" } tag:civilytics.com,2026:social/2026-10-07-case-study-leverage2026-10-07T07:00:00-04:002026-10-07T07:00:00-04:00

We built a free API for 56 years of state and local government finance data with an AI agent, and I studied the transcripts. Here's what I learned: the agent put in 8.4 to 10.9 hours for each hour of mine during build weeks, and that held as work got harder.

We built a free API for 56 years of state and local government finance data with an AI agent, and I studied the transcripts. Here's what I learned: the agent put in 8.4 to 10.9 hours for each hour of mine during build weeks, and that held as work got harder. { "append_url": false, "append_url_if_truncated": true, "append_url_label": "🔗", "append_url_spacer": "\n\n", "attach_link": false, "content": "We built a free API for 56 years of state and local government finance data with an AI agent, and I studied the transcripts. Here's what I learned: the agent put in 8.4 to 10.9 hours for each hour of mine during build weeks, and that held as work got harder.", "format_string": "{{content}}", "media": [ { "alt": "Line chart of agent hours per hour spent in the coding harness, by week. The line starts at 5.8 in the week of 6 July, jumps to 10.9 the following week, and then runs 10.7, 9.3, 8.4 and 10.0 through the rest of the build against a dashed reference line at the build average of 9.0. The first week is annotated \"still working out how to hand work over\". The line breaks across two shaded weeks labelled vacation and conference, which have no measurable sessions, and resumes at 4.9 in the communication week.", "bytes": 119947, "height": 940, "mime": "image/png", "type": "image", "url": "https://civilytics.com/social/2026-10-07-case-study-leverage/leverage.png", "width": 1839 } ], "platform_overrides": { "bsky": { "append_url": true, "syndicate": true }, "linkedin": { "append_url": true, "content": "We built a free API for 56 years of state and local government finance data with an AI agent, and I kept a record of every session. Here's what I learned.\n\nThe work got harder every week, but the ratio of the agent's hours to my own held steady. For each hour I spent in the coding harness, the agent put in 8.4 to 10.9 hours during build weeks. That first week, at 5.8, was me learning how to hand work over.\n\nMy hypothesis for how I sustained this leverage? Tooling. I iterated on the tools I gave the agent in order to facilitate better collaboration. First, version control and an issue tracker to organize the work and keep a record of what was done and what we still needed to do. Agents have superpowers at using git and with access to it they will understand the history of the project so much better in each fresh session.\n\nSecond, I added automated review, so every change an agent makes is checked by another agent. This slows things down, which gives me time to read and review, and it catches a lot of small errors before they cause confusion or break things.\n\nThird, I kept all the transcripts and gathered telemetry on agent usage. Agents and I can search the record of past sessions if we need, and I can review if I am letting sessions run too long or grouping too many tasks into a single session.\n\nIf you use agents, set these tools up before you scale the work.\n\nWhat agent-adjacent tools are you using in your work to get better results?\n\n#PublicInterestTech #OpenData #CapacityBuilding", "format_string": "{{content}}", "syndicate": true }, "mastodon": { "append_url": true, "syndicate": true } }, "syndicate": false, "truncate": true, "url": "https://civilytics.com/newsletter/case-study-agentic-data-science-cog-api/" } tag:civilytics.com,2026:social/2026-10-07-case-study-components2026-10-07T07:00:00-04:002026-10-07T07:00:00-04:00

What does it take to get good work from an AI agent on a real data project? Let's look at the transcripts! Building our Census of Governments API, the agent spent nearly as long on plans, notes and documentation (69.8 hours) as on code (76.2).

What does it take to get good work from an AI agent on a real data project? Let's look at the transcripts! Building our Census of Governments API, the agent spent nearly as long on plans, notes and documentation (69.8 hours) as on code (76.2). { "append_url": false, "append_url_if_truncated": true, "append_url_label": "🔗", "append_url_spacer": "\n\n", "attach_link": false, "content": "What does it take to get good work from an AI agent on a real data project? Let's look at the transcripts! Building our Census of Governments API, the agent spent nearly as long on plans, notes and documentation (69.8 hours) as on code (76.2).", "format_string": "{{content}}", "media": [ { "alt": "Horizontal bars of agent hours by component. Pipeline 50.0 hours or 32%; plans and working notes 49.0 or 32%; documentation 20.8 or 14%; R package 10.5 or 7%; API 9.6 or 6%; launch and planning 6.8 or 4%; demo app 6.1 or 4%. The bars are colored in three groups: code, writing about the work, and launch materials.", "bytes": 112425, "height": 860, "mime": "image/png", "type": "image", "url": "https://civilytics.com/social/2026-10-07-case-study-components/components.png", "width": 1839 } ], "platform_overrides": { "bsky": { "append_url": true, "syndicate": true }, "linkedin": { "append_url": true, "content": "What does it take to get good work from an AI agent on a real data project? Let's look at the transcripts!\n\nBuilding our Census of Governments API, the agent spent nearly as long writing about the work as writing the code: 69.8 agent hours on plans, working notes and documentation, against 76.2 on the pipeline, R package, API and demo app.\n\nMost of that writing never leaves the project: a design document before each phase, a progress file while it runs, a findings file at the end. It is how I set the agent up to work for long stretches on its own.\n\nThe longer I worked and the more complex the Census of Governments API project became, the more time I spent writing and reviewing plans. The plans became longer and more detailed. The reviews took me more time. But this process allowed me to push past that point where many AI vibecoding projects start to fizzle and stall.\n\nOne key takeaway: when you plan agent work, budget for the time for writing and reviewing, not just the tokens to generate the code.\n\nWhere are you finding the rough edges in your own AI agent work? Are you using plans and specs to get more done?\n\n#PublicInterestTech #OpenData #CapacityBuilding", "format_string": "{{content}}", "syndicate": true }, "mastodon": { "append_url": true, "syndicate": true } }, "syndicate": false, "truncate": true, "url": "https://civilytics.com/newsletter/case-study-agentic-data-science-cog-api/" } tag:civilytics.com,2026:social/2026-10-06-cog-chat-spokane2026-10-06T07:00:00-04:002026-10-06T07:00:00-04:00

A preview of what we have cooking up in the lab: COG Chat. Pick a city or county, ask about its finances in plain language, and get a chart, the figures and a summary from the Census of Governments. Here I ask about Spokane's police spending, with the waits sped up. COG Chat isn't public yet.

A preview of what we have cooking up in the lab: COG Chat. Pick a city or county, ask about its finances in plain language, and get a chart, the figures and a summary from the Census of Governments. Here I ask about Spokane's police spending, with the waits sped up. COG Chat isn't public yet. { "append_url": false, "append_url_if_truncated": true, "append_url_label": "🔗", "append_url_spacer": "\n\n", "attach_link": false, "content": "A preview of what we have cooking up in the lab: COG Chat. Pick a city or county, ask about its finances in plain language, and get a chart, the figures and a summary from the Census of Governments. Here I ask about Spokane's police spending, with the waits sped up. COG Chat isn't public yet.", "format_string": "{{content}}", "media": [ { "alt": "A screen recording of COG Chat. Someone picks Spokane, Washington, asks how its police spending per resident has changed since 1990, and gets a line chart and a written summary; then asks how that compares with similar-sized cities and gets a bar chart of Spokane against 10 peers.", "bytes": 4137219, "height": 810, "mime": "video/mp4", "poster_url": "https://civilytics.com/social/2026-10-06-cog-chat-spokane/spokane-police.jpg", "presentation": "gif", "type": "video", "url": "https://civilytics.com/social/2026-10-06-cog-chat-spokane/spokane-police.mp4", "width": 1080 } ], "platform_overrides": { "bsky": { "append_url": true, "syndicate": true }, "linkedin": { "append_url": true, "content": "A quick preview of what we have cooking up in the lab right now: COG Chat, which answers plain-language questions about a city or county's finances.\n\nWe already publish the Census of Governments (COG) finance record as a dashboard, the COG Place Explorer. I've come to think a conversation is a better way in for most people, so I've been building one.\n\nA dashboard asks you to learn it: which filters exist and which chart answers which question. That rewards knowing what you're looking for. I think most people who care about a city or county budget start with one specific question, and they shouldn't have to learn an interface to get it answered.\n\nIn COG Chat you pick your government and ask in plain language. The answer comes back as a chart, the figures and a short summary, and then you ask the follow-up you actually have. In the clip I ask how Spokane's police spending has changed and how it compares with similar-sized cities.\n\nCOG Chat runs on a language model on our own hardware, against the same Census data behind the dashboard. It isn't public yet, and the model's waits are sped up in the clip. The data behind both is open through our API.\n\nIf you work on a city or county budget, what's the first question you'd ask it?\n\n#PublicFinance #LocalGovernment #OpenData", "format_string": "{{content}}", "media": [ { "alt": "A screen recording of COG Chat. Someone picks Spokane, Washington, asks how its police spending per resident has changed since 1990, and gets a line chart and a written summary; then asks how that compares with similar-sized cities and gets a bar chart of Spokane against 10 peers.", "bytes": 4521644, "height": 810, "mime": "image/gif", "type": "image", "url": "https://civilytics.com/social/2026-10-06-cog-chat-spokane/spokane-police.gif", "width": 1080 } ], "syndicate": true }, "mastodon": { "append_url": true, "content": "A preview of what we have cooking up in the lab: COG Chat, a chat interface to the Census of Governments finance record. Pick a government, ask about its finances in plain language, and get a chart, the figures and a short summary.\n\nIn the clip we ask about Spokane's police spending and how it compares with similar-sized cities. The model's waits are sped up.\n\nCOG Chat runs on a language model on our own hardware and isn't public yet. Its data is open through our API:", "format_string": "{{content}}", "media": [ { "alt": "A screen recording of COG Chat. Someone picks Spokane, Washington, asks how its police spending per resident has changed since 1990, and gets a line chart and a written summary; then asks how that compares with similar-sized cities and gets a bar chart of Spokane against 10 peers.", "bytes": 4521644, "height": 810, "mime": "image/gif", "type": "image", "url": "https://civilytics.com/social/2026-10-06-cog-chat-spokane/spokane-police.gif", "width": 1080 } ], "syndicate": true } }, "syndicate": false, "truncate": true, "url": "https://civilytics.org/cog/" }