Explore every episode of the podcast Ship It Weekly - DevOps, SRE, Platform and Cloud Engineering News
| Title | Pub. Date | Duration | |
|---|---|---|---|
| Ship It Conversations: Ted Julian of Flux on the AI Code Visibility Gap, the Code Review Bottleneck, Measuring What AI Changes, Who Owns AI-Generated Code in Production, Guardrails Before Loops, and Why Disruption Is a Career Opportunity | 18 Nov 2026 | 00:39:53 | |
This is a guest conversation episode of Ship It Weekly, separate from the weekly news recaps. In this Ship It Conversations episode, I talk with Ted Julian about the gap between AI-generated code getting written and teams trusting it enough to ship it, why code review is the first thing to break, and who should own AI-generated code once it's running in production. Ted is the founder and CEO of Flux (askflux.ai), an engineering intelligence platform, not Flux CD the GitOps tool. He has spent more than twenty years building security and infrastructure companies, including Resilient Systems, Arbor Networks and @stake. We start with Flux's AI Code Generation Reality Check: 44.7% of the organizations surveyed already run AI-generated code in production, and 35% write it but don't ship it. What holds them back is trust: complexity that could spiral into an outage, a new library with a vulnerability, regulated data nobody has really scrutinized. Ted calls code review act two of an AI transformation, and he says it's a people problem as much as a tooling problem. His once-in-a-career hack: measure features versus maintenance versus bug fixes now, backfill it, and show the before and after. At one customer, feature work went from 9% to 43%, and it became a chart in their board meeting. We also get into why engineering has to be the yes squad, with guardrails, the bill that comes due when someone has to understand code nobody really wrote, who feeds the cat, what loops mean for visibility, and why tech disruption is the best career opportunity you'll get. Highlights • Written but not shipped: 44.7% in production, 35% holding back • Code review breaks first, and it's a people problem • Measure features vs. maintenance vs. bug fixes, then backfill • From 9% to 43% feature work, as a board chart • The yes squad, with guardrails • The bill for code nobody really wrote • Who feeds the cat: one owner for AI-generated code • Loops, dark factories and a control plane • Why disruption is your best career move Links Ted on LinkedIn · Flux · AI Code Generation Reality Check · Cobalt case study · More episodes · On Call Brief · LMGT Awards | |||
| Ship It Conversations: Tristan Handy of Fivetran + dbt on AI Agents as Data Consumers, Why Dashboards Stop Being the Product, Context Layers, Evals, Operating on the System, Software Factories, and Why Text-to-SQL Isn't the Hard Part | 16 Nov 2026 | 00:37:44 | |
This is a guest conversation episode of Ship It Weekly, separate from the weekly news recaps. In this Ship It Conversations episode, I talk with Tristan Handy about what changes when AI agents, not people, become the main consumers of your company's data, why the hard part isn't text-to-SQL, and why he thinks the most interesting work is operating on the system instead of inside it. Tristan is the founder of dbt Labs and president and co-founder of Fivetran. If you've worked anywhere near a data team, you've probably run into dbt. His team likes to call it Terraform for data. Correction: Tristan's title changed after we recorded. He is now the founder of dbt Labs and president and co-founder of Fivetran. We start with his line that dashboards are a hack for the human visual cortex. As agents take over the pattern-finding, the asset moves down the stack: a well-modeled gold layer, a semantic layer that captures how your organization calculates its metrics, and a context store that helps the agent make sense of all of it. On reliability, Tristan says agents already write SQL that compiles and runs almost every time. The hard part is whether they understood the question and the business behind it. In the benchmarks he's seen, an agent turned loose on raw enterprise data gets about 30% of its answers semantically right. Give it context and a semantic layer, and it's north of 90%. We also get into where autonomy should stop once agents can act on data, why analytics engineers (and SREs) should be operating on the system rather than inside it, how dbt earned practitioners' trust, what he got wrong selling to enterprises, and where dbt draws its open source line after the Fivetran merger. His first step for a platform or data team is to eat your spinach: a well-governed data lake, well-modeled data and a semantic layer, then a context layer and evals. And his team paused dbt 2 work for two months to build an agent loop that took conformance from about 45% to 93%. Highlights • Why dashboards stop being the product when agents read the data • Gold layer, semantic layer and context store • Text-to-SQL vs semantic correctness: about 30% vs north of 90% • Where useful autonomy ends and regret begins • Operating on the system instead of inside it • How dbt earned trust, and what it got wrong selling to enterprises • State and UI as the open source line • Eat your spinach, then a context layer and evals • A software factory that got dbt 2 conformance from 45% to 93% Links Tristan on LinkedIn · dbt · Fivetran · The Analytics Engineering Roundup · The Analytics Engineering Podcast · Spider 2.0 benchmark · More episodes · On Call Brief · LMGT Awards | |||
| AWS Roles Trusting Qubole Abused to Steal Secrets, Claude Code's Don't Ask Again File Leaks Credentials, KubeCon Agent Sandboxes for Kubernetes & WorkOS Breach Traced to an AI Agent | 13 Nov 2026 | 00:15:41 | |
This week on Ship It Weekly, an attacker used Qubole's trusted access into customer AWS accounts to read secrets and pull whole S3 buckets. Claude Code's "don't ask again" button has been saving passwords and API keys into .claude/settings.local.json, a file that keeps turning up on GitHub and Docker Hub. At KubeCon, the CNCF took in Agent Substrate and OpenShell as Sandbox projects, and AWS launched Nitro Sandboxes for EKS. And WorkOS traced a breach of an internal sales tool to a security researcher's AI agent that escaped its test environment. The thread: what can it touch? A vendor role nobody removed, an approval that saved the secret with the command, agents running unreviewed code, and an internet-facing tool with a guessable dev credential. In the lightning round: an exploited Outlook bug (CVE-2026-124194) and a 9.9 Azure DevOps Server RCE in Patch Tuesday, N-able patching the Able2Shell exploits in N-central, KubeVirt's CNCF graduation, and OpenShift 5 going GA. The human closer comes from Brent Chapman: incident resolution time isn't something you should target. Target what you can change, and track the rest. Ship It Weekly is a 2026 DevOps Dozen finalist for Best DevOps Community Program or Event, and I'm a finalist for Enterprise DevOps Champion of the Year. The public vote is open through December 31. Highlights • Why the External ID didn't stop a stolen vendor credential • Find every role that trusts an outside account, and delete the idle ones • settings.local.json: rotate, untrack, and add it to .dockerignore • Moving an agent's boundary off the shared kernel • WorkOS's own list: internet-facing API, dev credential, no rate limit • Target the steps you control, not resolution time Links Qubole cross-account access abused (Wiz) Claude Code's "don't ask again" file leaks secrets (The Hacker News) settings.local.json secrets on GitHub and Docker Hub (GitGuardian) Agent Substrate and OpenShell join the CNCF Nitro Sandboxes for Amazon EKS (AWS) November 2026 Patch Tuesday (Rapid7) N-central and Able2Shell (Huntress) Red Hat OpenShift 5 (Virtualization Review) Why incident resolution time isn't something you should target (Brent Chapman) | |||
| Ship It Conversations: Mallory Haigh of PlatformEngineering.org on Why Context Isn't Enough for AI Agents, Harness and Governance, Probabilistic and Hybrid Paths, Agent Identity, and Building Platform Trust Before Agents | 11 Nov 2026 | 00:39:20 | |
This is a guest conversation episode of Ship It Weekly, separate from the weekly news recaps. In this Ship It Conversations episode, I talk with Mallory Haigh about why context alone isn't enough for AI agents, the harness and governance layers that actually run them, and why she tells some platform teams to pump the brakes before adding agents at all. Mallory is Head of Platform Education and Advocacy at PlatformEngineering.org. She teaches in Platform Engineering University and coaches platform teams through Platform Engineering Consulting, where the job gets jokingly called "platform therapist". We start with her New York City analogy. Drop someone in the middle of the city with no directions and they'll get lost, and an agents.md file on its own does the same to an agent. It also needs to know what to do, how to do it, how it'll know it's done, and what it's allowed to touch. Mallory splits that into the harness, which is how the agent does the work, and governance: who and what is allowed, what's safe, and how it's watched. Then we get into IDPs as context layers. Mallory says that's partly right, because the IDP and its golden paths are the foundation. What's missing are path specifications. Probabilistic paths are agent-driven, like a read-only PR review. Deterministic paths are your pipelines, like a CI build. Hybrid paths pair an agent with a gate: the agent opens the PR, CI validates it, the agent retries, and a human makes the final call. Cap those retries at three to five, and if an agent always does the same thing with the same outcome, ask whether it should be an agent at all. On governance, Mallory says platform engineering is a practice about building trust, and that in the last three weeks alone she's told multiple platform teams to pump the brakes on agents because they can't get human governance right yet. We cover agent identity (specialists, not generalists, built on the RBAC you already have), agents watching terabytes of logs a day for SRE teams, and why every team now has to write down the tribal knowledge it never documented. Her advice on where to start: ask why. Then try a read-only PR review agent, move to a hybrid validate-change path, and remember the line from one of her PlatformCon New York workshops: if we can't even write a README for our repos, how are we going to write one for our agents? Highlights • Why context delivery alone isn't enough for AI agents • The harness and governance layers that make up agent infrastructure • Observability for agents: token spend, time to completion and retry loops • Probabilistic, deterministic and hybrid paths • Retry limits, and when an agent should be a checklist instead • Pumping the brakes when human governance isn't there yet • Agent identity, RBAC and specialist agents • Where a platform team should start: ask why, then a read-only PR review agent Links Mallory on LinkedIn · PlatformEngineering.org · Platform Engineering University · Platform Engineering Consulting · Thinking in Platforms · More episodes · On Call Brief · LMGT Awards | |||
| Ship It Conversations: Ryan Packer of BambooHR on Why CTOs Are Coding Again, Building a Software Factory, AI Straining CI/CD, Protecting HR Data, Mid-Career and Junior Engineers, and Why Every Engineer Gets a Team of Agents | 09 Nov 2026 | 00:36:49 | |
This is a guest conversation episode of Ship It Weekly, separate from the weekly news recaps. In this Ship It Conversations episode, I talk with Ryan Packer about why a CTO is writing code again, what happens to CI/CD when every engineer runs a fleet of agents, and who trains junior engineers when AI can fix the bugs. Ryan is the CTO at BambooHR and a former VP of Engineering at Adobe. He's watched engineering organizations change from Omniture to Adobe to BambooHR, and he says he's spent more time coding this year than in the rest of his career combined. We start with why. Ryan will never write production code for his team again, but he wants to use the same tools and learn the same techniques they're learning, and to show an idea in working code instead of a PRD or a Slack message. He's also honest about the limits: his sandbox serves maybe a hundred people, and his team's software serves millions. Then we get into the software factory. In Ryan's own version, a customer files a bug, an agent opens the PR, tests it for security and performance, scores it, and ships it if it meets the bar. At BambooHR, pull requests are up almost 80% since the start of the year, and quality has improved by the metrics they track. The part he's most excited about is more at-bats with customers: an idea to a working prototype in a day. For platform teams, Ryan says AI has been more of a challenge than a help for CI/CD. BambooHR was already releasing hundreds of times a day, but agents do their best work when they can run and test what they build. Ten, twenty or a hundred agents per engineer means environments that spin up and down fast, with real data. We also cover protecting HR and payroll data: terms with the frontier labs, data retention, and an AI council. On careers, Ryan thinks mid-career engineers face the hardest change, and that the ones who keep learning have little to worry about. On junior engineers, he doesn't claim a perfect answer. He no longer needs them to fix bugs, so he suggests making the LLM explain its choices and using an adversarial agent that lists five things it would have done differently. We finish with every engineer managing a team of agents, small pods doing what 10 to 15 people used to, and why he expects the same or more engineers a year from now. The big takeaway: if every project in your portfolio has better than a 50% chance of success, you're not being ambitious enough. Put something in there with a 10% chance. Highlights • Why a CTO is coding again, and why his sandbox isn't production • The software factory: a customer bug to an agent PR to a tested release • Pull requests up almost 80%, with quality improving • More at-bats with customers: idea to working prototype in a day • Why AI is straining CI/CD and test environments • Protecting HR and payroll data while adopting AI • Junior engineers, adversarial agents and making the LLM explain itself • Every engineer with a team of agents, and the 10% project Links Ryan on LinkedIn · BambooHR · BambooHR careers · More episodes · On Call Brief · LMGT Awards | |||
| GitHub Blocks pull_request_target by Default, WordPress Update Race Exploited, Polymarket Drained via Hijacked DebugBear Script & Alpine's GitLab Upgrade Missed the Fix | 06 Nov 2026 | 00:16:07 | |
This week on Ship It Weekly, GitHub will block pull_request_target in public repos by default from December 2, with brownouts on Mondays, November 9 and 16, 11 AM to 1 PM ET. A WordPress core update race (CVE-2026-119977) is on CISA's exploited list: during a full-package update, a fresh installer can sit for a moment at a public URL under wp-content/upgrade. A hijacked DebugBear script drained about $2.7 million from Polymarket users after DebugBear's DNS was compromised through its registrar. And Alpine Linux explains why upgrading GitLab didn't fix CVE-2026-97229 on its own image: the fix was a patch to Ruby. The thread: what actually ran, as opposed to what you think ran. A trigger that runs a fork's code with your secrets, an update that opens its own window, a script served by whoever controls the DNS, and a version number that said "fixed". In the lightning round: OpenSSH 10.6p2 fixes a password login bypass on musl builds with PAM off, which is how Alpine builds openssh-server; Debian and Ubuntu patch an APT package swap (CVE-2026-109518); and .NET 8 and 9 reach end of support on Tuesday, November 10. The human closer comes back to Alpine's "we cannot rule it out." If an attack hit you this week, could your logs prove it didn't work? Highlights • pull_request_target: what it runs, and what to check before Monday • WordPress: when keeping current is what opens the window • DNS, HTTPS and subresource integrity for third-party scripts • Registrar lock, MFA and alerts on DNS changes • Alpine: GitLab from your own image needs a fixed Ruby too • "No evidence" is only as good as the logs you kept Links pull_request_target blocked by default (GitHub): https://tsn.io/KwebM WordPress advisory GHSA-4w3m-c66w-69mg: https://tsn.io/1HGiK WordPress upgrade-window RCE (Wordfence): https://tsn.io/Pc6X2 Polymarket postmortem: https://tsn.io/rOc2M DebugBear on the registrar DNS compromise: https://tsn.io/4BjIg Polymarket losses and refunds (Hackread): https://tsn.io/qNhoe Alpine's GitLab security incident: https://tsn.io/GpNoR OpenSSH 10.6p2 release notes: https://tsn.io/dQHbG Alpine openssh-server advisory: https://tsn.io/lExh2 APT CVE-2026-109518 (Debian DSA-6607-1): https://tsn.io/LSjTb .NET 8 and 9 end of support: https://tsn.io/7NC5R This week's On Call Brief: https://tsn.io/XJLrm More episodes and show notes: https://tsn.io/NqkdP On Call Brief: https://tsn.io/Gpz2d LMGT Awards: https://tsn.io/lAERA | |||
| Ship It Conversations: The AI Agent That Ran Terraform Apply, With Davlet Dzhakishev of Cloudgeni on Read-Only Cloud Access, Sandboxes Beyond Docker, Agentic PR Reviews, the Real Cost of Running Agents, FinOps for AI, and Permissions Over Prompts | 04 Nov 2026 | 00:37:24 | |
This is a guest conversation episode of Ship It Weekly, separate from the weekly news recaps. In this Ship It Conversations episode, I talk with Davlet Dzhakishev about what it takes to let AI agents work on your cloud infrastructure, why guardrails belong in the architecture and not the prompt, and what running your own agents really costs. Davlet is the CEO and co-founder of Cloudgeni, which builds AI agents for cloud infrastructure. He got into this after working through SOC 2 and ISO 27001 compliance prep mostly by hand, and asking what agents would need to fix those problems reliably without making things worse. We start with the incident that changed how Cloudgeni builds. An agent hit a snag on a simple Terraform change and decided the fix was to run apply. On staging. Resources got recreated, and Davlet calls it his wake-up call. Since then, guardrails come first: sandboxed runtimes, scoped credentials, read-only cloud access, and changes that go through Git, CI/CD and tests. At the architecture level, the agent shouldn't be able to do what you don't want it to do. Then we get into agentic PR reviews, which check human and agent work the same way, and why a good reviewer catches more than the obvious errors. Davlet's example is a private network around a key vault: great for security, until your own apps lose access. We also cover drift, and why you need to reconcile it before you can trust agent PRs. On cost, Davlet's view is that AI subscriptions are heavily subsidized, and running your own agents on API tokens can cost ten to twenty times more. Most agent usage also has no audit trail, so a stuck loop can poll a dead API server every day and no one will know. He thinks FinOps for AI is going to be big, orphaned agents and all. That need for audits is part of why Cloudgeni open-sourced its foundation as OpenGeni. We finish with what he still wouldn't hand to an agent, and where a platform team should start without building a monster: PR reviews first, then agents writing IaC from your templates, then automation. His starter kit is a real sandbox, ideally in the cloud, because Docker doesn't count. Give it read-only access to an AWS account or Azure subscription, one IaC repo, and permission to open PRs. Ask it to delete a resource, and the worst it can do is open a PR. The big takeaway: a safe agent is mostly a permissions problem, not a prompting problem. Give it read-only access and PR-only changes, and it can't make that mistake however you word the request. Highlights • The Terraform apply on staging that became his wake-up call • Guardrails at the architecture level, not in the prompt • Read-only access, with every change through Git, CI/CD and tests • Agentic PR reviews that catch operational impact • Why drift has to be reconciled before you can trust agent PRs • Why API tokens can cost 10 to 20 times more • FinOps for AI, stuck loops and orphaned agents • A starter kit: a real sandbox, read-only cloud access, one IaC repo, PRs only Links Davlet on LinkedIn · Davlet on X · Cloudgeni · OpenGeni · Agents guide · More episodes · On Call Brief · LMGT Awards | |||
| Ship It Conversations: The AI Hallucination That Looks Right, With Devansh of Irys on Legal AI, Three-Tier Error Isolation, Models That Change Under You, Overhyped Reinforcement Learning, and Optimizing for Maintenance Over Generation | 02 Nov 2026 | 00:33:12 | |
This is a guest conversation episode of Ship It Weekly, separate from the weekly news recaps. In this Ship It Conversations episode, I talk with Devansh about why the trickiest legal AI hallucination is a real case used the wrong way, how his team splits their system into three tiers to find failures fast, and why he'd rather optimize for maintenance than for generation. Devansh is the Founder and Head of AI at Irys. He builds reasoning and verification systems for legal AI, a tough place for hallucinations and confident wrong answers. He also publishes open source AI research on Substack. We start with why AI for professionals breaks down. Devansh thinks models are getting good at doing a lot of things somewhat well, but they fall short on the last mile to reliable. So his team engineers structure back in instead of hoping a better model fixes it. In legal, fake citations are the failure everyone has heard about. The more interesting one is a real case that doesn't apply. So much of law is interpretation, and a model built to predict the most probable token pulls toward the average. Then he walks through the three tiers behind Irys One. Ingestion maps messy input, like PDFs with handwritten notes and email chains that refer to them, into a knowledge graph so the AI knows where to look. Reasoning navigates that graph. Delivery applies how a firm or a lawyer likes to work. The payoff is error isolation: if a lawyer says something is missing, two or three targeted queries tell them whether it was ingestion or reasoning. We also get into why chunk-based retrieval misses connections, why going from 10 docs to 10 gigabytes is mostly about finding the few documents that matter, and how "answer contracts" handle "it depends." Then I take a DevOps angle. Devansh says the hardest part of running AI in production is tracking everything. A fully agentic system might perform 20% better, but you never know how it breaks until it breaks. The first thing his team tracks is whether the system behaves as expected, because labs change models behind the scenes with post-training, quantization and tokenizer changes. We also cover why he calls reinforcement learning the most overhyped technique in AI: it works when the output is truly verifiable, and falls apart when "better" has no real signal. The big takeaway: you can't fix what you can't find. Engineer for error isolation first, and optimize for maintenance, not generation. Coding agents will churn out heaps of code, but maintenance is where a system lives or dies. Highlights • Why a real case applied wrong is worse than a fake citation • Why AI's pull toward the average struggles with legal judgment • Ingestion, reasoning and delivery as three separate tiers • Knowledge graphs vs chunking for finding what matters • Isolating failures with two or three targeted queries • Models that change under you, and tracking expected behavior • Why reinforcement learning is overhyped • Optimize for maintenance, not generation Links Devansh on LinkedIn: https://tsn.io/7Hjfm Irys: https://tsn.io/ytG1A Artificial Intelligence Made Simple (Devansh's Substack): https://tsn.io/ZM6eS More episodes and show notes: https://tsn.io/NqkdP On Call Brief: https://tsn.io/Gpz2d LMGT Awards: https://tsn.io/lAERA | |||
| GitHub Actions Jobs Move to AWS, Brotli RCE Exploited, Gheist Steals Actions Secrets & AGENTS.md Hijacks AI Coding Agents | 30 Oct 2026 | 00:16:53 | |
This week on Ship It Weekly, GitHub Actions can now run eligible Linux jobs on AWS as well as Azure, so allowlists and Entra Conditional Access policies that only trust Microsoft ranges can break. The Brotli decoder bug (CVE-2026-111497) landed on CISA's exploited list, and it reaches past Node.js to Python Brotli, Bun, Envoy, Istio, a Go port and .NET. A campaign called Gheist is using stolen GitHub tokens to push ci-sync branches that print Actions secrets past log masking, then delete the run. And attackers are dropping AGENTS.md files into Downloads folders to hijack AI coding agents. The thread: each one is an inventory question. Which services decompress Brotli, which allowlists assume CI runs on Azure, which OIDC trust policies accept any branch, and which laptops have started an agent in Downloads? In the lightning round: GitHub Actions workflow lockfiles are generally available, Railway disclosed a breach of its internal systems, Tailscale 1.104.4 fixes four security bulletins, and Datadog moved its US1 status page to its own Status Pages product. The human closer, from Bill Duncan: "Twenty minutes in, you haven't started diagnosing anything. You've been doing archaeology." Write down who owns it before the pager asks. Highlights • GitHub Actions on AWS: why the same job can pass, then fail • Sync runner IP ranges from GitHub's meta API, not one cloud's • Brotli: a decoder bug your HTTP stack may run for you • Gheist: what private repos and OIDC trust let through • AGENTS.md in Downloads: an instruction file is code now • Service catalogs and the "what depends on me?" question Links GitHub Actions multi-cloud compute: https://tsn.io/fJkLJ Node.js October security releases (Brotli): https://tsn.io/BvZBU Brotli CVE-2026-111497, affected libraries (Wiz): https://tsn.io/HJuKS .NET out-of-band security updates: https://tsn.io/j0Xwf Gheist steals GitHub Actions secrets (Wiz): https://tsn.io/nunEs AI agents hijacked via the Downloads folder (Forever Security): https://tsn.io/qnwuc Actions workflow lockfiles GA: https://tsn.io/M84uA Railway security disclosure: https://tsn.io/21t0j Tailscale security bulletins: https://tsn.io/GvZxR Who Owns This? (Bill Duncan): https://tsn.io/LFrzc SRE Weekly #540: https://tsn.io/O8tpO This week's On Call Brief: https://tsn.io/HqfCP More episodes and show notes: https://tsn.io/NqkdP On Call Brief: https://tsn.io/Gpz2d LMGT Awards: https://tsn.io/lAERA | |||
| Ship It Conversations: AI Coding Outruns Software Delivery, With Darko Fabijan of SuperPlane on What Breaks After the Pull Request, Change Risk Scores, Onboarding AI Agents Like Engineers, Read-Only Ops Agents, and Why the Answer Is More Automation | 28 Oct 2026 | 00:35:49 | |
This is a guest conversation episode of Ship It Weekly, separate from the weekly news recaps. In this Ship It Conversations episode, I talk with Darko Fabijan about what starts breaking after the pull request when AI writes more of the code, what should sit between an AI agent and production, and why his answer is more automation. Darko is the CEO of SuperPlane and co-founder of Semaphore, the CI/CD platform. At SuperPlane, he's working on how to give AI agents write access to production in a controlled way. We start with what AI-generated code does to delivery. Pull requests are bigger and more frequent, reviewers can't keep up, and Darko points to a Faros AI report that shows defects and incidents going up. He describes AI as alien technology that dropped from the sky, landing in processes built for people who review code eight hours a day. His main hypothesis is a deterministic layer between AI agents and production, with scoped access that goes through approval. I bring up the rubber stamping problem with human approvals, and his reply is that humans hallucinate too. So he wants a risk score on every change. Renaming a bucket scores low. Deleting an S3 bucket or an RDS instance scores red. Then we get into the glued-together ops tooling most teams run on. Darko doesn't think anything is wrong with it. When I ask how SuperPlane differs from Argo Workflows or n8n, he says he wants agents and humans on the same plane, and he'd onboard an agent like a new engineer: give it an identity, let it look around, and show it what it can do. We also cover team-scoped agents, compliance at AI speed, and docs where every line traces back to what produced it. We finish on where to start. Darko says read-only first: he pretends to be super stupid, asks a lot of dumb questions, and says "onboard me." For the first write operations, he'd pick loops you can measure with a number, like latency, GitHub Actions speed or CVE count, tested on a copy of production or staging first. The big takeaway: AI didn't break software delivery. It showed how much of it still runs at human speed. Darko's answer is more automation, with a deterministic layer and a risk score between AI and production. Highlights • What breaks after the pull request when AI writes more of the code • A deterministic layer between AI agents and production • Rubber stamping, and a risk score for every change • Onboarding an AI agent like a new engineer • The dam and the river: compliance at AI speed Links Darko Fabijan on LinkedIn: https://tsn.io/DySr5 Darko Fabijan on X: https://tsn.io/xJGhN SuperPlane: https://tsn.io/o08yM Semaphore: https://tsn.io/jiHmd Faros AI's Acceleration Whiplash report: https://tsn.io/5ZCR3 Argo Workflows: https://tsn.io/oyTYF n8n: https://tsn.io/nloFF More episodes and show notes: https://tsn.io/NqkdP On Call Brief: https://tsn.io/Gpz2d LMGT Awards: https://tsn.io/lAERA | |||
| Ship It Conversations: Running Retros as Tabletop RPGs, With Jinger Meilani of Mountain on Chapters and Characters, Technical Archetypes, Reading the Room, Following Through, and Why Silliness Gets You Honest Answers | 26 Oct 2026 | 00:37:23 | |
This is a guest conversation episode of Ship It Weekly, separate from the weekly news recaps. In this Ship It Conversations episode, I talk with Jinger Meilani about Tabletop Retro, her way of turning an engineering retrospective into a tabletop role-playing game, and why a little silliness gets teams to give honest answers. Jinger is a Senior Software Engineer in DevOps at Mountain. She says Tabletop Retro brings together three things she loves: world building, psychological safety on teams, and engineering enablement. We start with why she reinvented the retro at all. She was bored of them, and even the themed ones got rote. At a previous company, the team rotated who ran the retro, and her turn landed around Halloween, so she built a story around the team. She calls the result retro plus plus plus. The usual formats and outcomes are all still there, inside a story. A lot of it comes down to silliness. Jinger compares it to hiding vegetables in a fruit smoothie: it disarms people, and she gets honest answers, sometimes while everyone thinks they're talking about something else. That's also why she won't run one during a fire. People have to feel comfortable before they'll speak their truth, so the flame-breathing dragon can wait. Then we get into how one is built. A Tabletop Retro is a series of chapters, and each chapter holds part of the story and at least one activity. Players fill out character sheets, and each in-game race maps to a technical archetype, like a Goliath for a high-level systems thinker or a gnome for a tinkering automation engineer. Afterward the team talks through what those picks say about their strengths and weaknesses. She presents it like a storybook, in Lucidchart or Canva over Zoom. Characters cut both ways. For some people, an orc is permission to say what they'd never put on a retro board, because it's not really how they feel. They're just an orc. Others find role-play intimidating, so she reads the room: improv for an outgoing teammate, a line on screen for someone else, or no speaking part at all. If you want to try one, start with the questions, not the world. Where is the team unaligned? When does communication break down? Then keep the setting small. Kids trick-or-treating is plenty. The big takeaway: a Tabletop Retro is still a retro, and Jinger holds it to a higher bar. If everyone has fun and tells you something honest, and then nothing changes, it failed. Someone still has to do the work. Highlights • Why she calls it retro plus plus plus • Hiding vegetables in a fruit smoothie • Why you don't run one during a fire • Chapters, character sheets and technical archetypes • "I'm just an orc," and why role-play makes some people nervous • Reading the room: improv, on-screen lines, or no lines at all • Start with the questions, then keep the world small • Fun with no follow-through is a failed retro Links Tabletop Retro, Jinger's framework: https://tsn.io/wXdtL Jinger Meilani on LinkedIn: https://tsn.io/xn8Jr Jinger's IaCConf talk, Designing IaC Interfaces That Work for Humans, AI Agents, and Whatever Comes Next: https://tsn.io/5b4DM More episodes and show notes: https://tsn.io/NqkdP On Call Brief: https://tsn.io/Gpz2d LMGT Awards: https://tsn.io/lAERA | |||
| LiteLLM Release Pipeline Hijacked Again, GitHub Package Firewall, PHP2Shell Core RCEs & eMudhra CA Keys Compromised | 24 Oct 2026 | 00:16:57 | |
This week on Ship It Weekly, LiteLLM's release pipeline was hijacked again: stolen GitHub CLI tokens from three engineers changed the pipeline, kicked off the release and approved it, and a credential-stealing 1.107.0 went out to PyPI and Docker Hub. GitHub put a package firewall into public preview, with advisory and supply chain checks and a cooldown for new releases. PHP fixed PHP2Shell, two critical core runtime bugs (CVE-2026-109618 and CVE-2026-109619) that someone tried to exploit four days before disclosure. And eMudhra, a certificate authority in India, found out the private keys for eight intermediate certificates could be worked out from its own public certificates. The thread: most of this came through things you depend on but don't run. A library's release pipeline, the registry you pull from, the runtime under your app, and the trust store in your images. In the lightning round: another GitLab CVSS 10 (CVE-2026-110264), Google's report on the europe-west2 London outage, AWS taking its European Sovereign Cloud off the global backbone on October 24, and a GitHub audit log change that can quietly break SIEM rules. The human closer, from Brent Chapman: page on what users actually do. Your users don't care how your machines feel. They care whether they can log in. On Monday, Jinger Meilani joins me to talk about running retros like a tabletop RPG. Highlights • LiteLLM 1.107.0: two stolen identities passed a two-person approval • GitHub package firewall: advisories, attestations and a cooldown • PHP2Shell: an exploit attempt four days before disclosure • Patching PHP is easy, finding all of it is hard • eMudhra: CA private keys worked out from public certificates • Page on what users feel, not how the machine feels Links LiteLLM 1.107.0 compromise (Wiz): https://tsn.io/yQtmu GitHub package firewall in public preview: https://tsn.io/QjofJ PHP2Shell, CVE-2026-109618 and CVE-2026-109619 (Help Net Security): https://tsn.io/GUyop Eight eMudhra CA keys compromised (The Hacker News): https://tsn.io/K2jqM GitLab 19.5.1 patch release: https://tsn.io/0wsPX Google Cloud europe-west2 incident report: https://tsn.io/nVk2N AWS European Sovereign Cloud independent operation: https://tsn.io/gKnQl GitHub secret scanning updates: https://tsn.io/JDGmT Stop paging on how your machines feel (Brent Chapman): https://tsn.io/MAItM SRE Weekly #539: https://tsn.io/XAsKz This week's On Call Brief: https://tsn.io/nBAVg More episodes and show notes: https://tsn.io/NqkdP On Call Brief: https://tsn.io/Gpz2d LMGT Awards: https://tsn.io/lAERA | |||
| Ship It Conversations: Rob Koch of Slalom on Accessible Cloud Native, Sign Language Recognition with Kubeflow, Overhyped AI, Conference Accessibility, and Why Teams Should Hire Deaf Engineers | 19 Oct 2026 | 00:45:21 | |
This is a guest conversation episode of Ship It Weekly, separate from the weekly news recaps. In this Ship It Conversations episode, I talk with Rob Koch about what accessible cloud native actually looks like, why sign language is still one of the hardest problems in AI, and why the fastest way to make your tooling accessible is to hire deaf engineers. Rob is a senior principal at Slalom, an AWS Hero, and co-chair of the CNCF Deaf and Hard of Hearing Working Group. Rob is Deaf, and his ASL interpreter, Amelia Fruehsamer, voices his answers. Special thanks to Amelia for interpreting the conversation. We start with an analogy Rob comes back to all episode: cloud native is the toolbox, and platform engineering is the workbench. Then we get into what's overhyped about AI, the guardrails it still needs, and what auditable AI observability looks like. A big part of the conversation is accessibility in practice. Rob explains how captions pushed for deaf attendees ended up helping neurodivergent folks too, and what conferences get wrong: accessibility planned at the very end instead of the start, and the real cost of interpreters versus captions. Then we get technical. Rob walks through using Kubernetes and Kubeflow to train models that recognize sign language, and why it's so hard: regional signs, lighting, skin tone and backgrounds, and a camera that can't tell which way a hand is facing. His joke: we don't have AGI until AI understands sign language. The big takeaway: start with people, not a checklist. Hire deaf and hard of hearing engineers, ask them how they want to work, and your tools get more accessible because they're in the room. Highlights • Cloud native as the toolbox, platform engineering as the workbench • What's overhyped about AI, and the guardrails it still needs • What accessible cloud native means in practice • Captions that help far more people than you planned for • What conferences get wrong about accessibility, and what it costs • Kubernetes and Kubeflow for sign language recognition • Gesture-based computing and why speech-to-text had a 30-year head start • Where to start: hire deaf engineers Links Rob Koch on LinkedIn: https://tsn.io/4qmFo Rob Koch on GitHub: https://tsn.io/Mbxrs Amelia Fruehsamer (Rob's ASL interpreter) on LinkedIn: https://tsn.io/UJB88 CNCF Deaf and Hard of Hearing Working Group: https://tsn.io/tB1lS CNCF conference accessibility best practices: https://tsn.io/BZvpv CNCF Merge Forward: https://tsn.io/QPVNQ Kubeflow: https://tsn.io/4OVKf More episodes and show notes: https://tsn.io/NqkdP On Call Brief: https://tsn.io/Gpz2d LMGT Awards: https://tsn.io/lAERA | |||
| GitHub Actions Lockfiles, GitLab CVSS 10 CI Flaw, Google Cloud Storage Outage & Torvalds Builds Kernel Patches With Claude Code | 17 Oct 2026 | 00:14:02 | |
This week on Ship It Weekly, GitHub Actions workflow lockfiles land in public preview, pinning actions and their dependencies to exact commits. GitLab shipped a critical patch release for CVE-2026-97229, a CVSS 10 CI/CD regex flaw that could let an unauthenticated attacker run code on the server. Google published its final report on two Cloud Storage incidents in us-central1, including a recurrence caused by an incomplete rollback. And Linus Torvalds pushed an experimental kernel branch with page-cache patches built using Claude Code, as maintainers debate dropping the mandatory AI attribution tag. The thread this week: know what you're actually running, from the commit you reviewed to the state a rollback really left behind and code someone can still explain. In the lightning round: a record Patch Tuesday with more than 1,400 CVEs and an actively exploited Windows Shell bypass, GitHub AI Scan for pull requests and agentic autofix going GA, AWS Strands Box, an open-source agent sandbox that keeps real credentials out of the agent's environment, and a Microsoft 365 outage through Global Secure Access. The human closer comes from Paul LaPosta's “Your error budgets don't know AI exists.” AI has split shipping a change from understanding it, and SLOs don't measure comprehension. Your SLO can tell you the service is up. It cannot tell you whether anyone still understands it. Programming note: Ship It Weekly turns one next month, so expect more conversations over the next couple of months, usually two a week, starting Monday with Rob Koch. Links GitHub Actions workflow lockfiles GitLab 19.4.2 / 19.3.5 / 19.2.8 patch release Google Cloud Storage incident report Linus Torvalds and Claude Code (Phoronix) Linux drops the Assisted-by requirement (Phoronix) GitHub AI Scan for pull requests GA Microsoft 365 Global Secure Access outage | |||
| Ship It Conversations: Erik Osterman of Cloud Posse on Terraform Modules, IaC Frameworks, Atmos, Internal Developer Platforms, AI Agents, and Why Conventions Matter More Than Ever | 12 Oct 2026 | 00:39:31 | |
This is a guest conversation episode of Ship It Weekly, separate from the weekly news recaps. In this Ship It Conversations episode, I talk with Erik Osterman of Cloud Posse about Terraform modules, infrastructure as code frameworks, internal developer platforms, and why AI makes conventions more valuable, not less. Erik is the founder and CEO of Cloud Posse, which has published more than 180 open source Terraform modules, and he hosts Cloud Posse's weekly DevOps Office Hours. We start with what AI has done to open source infrastructure projects, then get into what makes a good Terraform module: consistent naming, a consistent interface, not so small that it wraps a single resource, and not so large that it has a hundred inputs nobody can test. We also talk about CloudFormation, Pulumi, and why HCL's constraints helped build the module ecosystem. Erik explains where Crossplane fits, and why accounts, IAM, networking, DNS and certificates all come before the cluster. A big part of the conversation is frameworks. Erik argues that nobody actually runs "vanilla Terraform," because Terraform alone doesn't handle state layout, promotion, authentication, tool installation or repository structure. Every team ends up building its own framework, usually undocumented. We talk about Terragrunt, Atmos, naming conventions, brownfield adoption, and breaking a monolithic root module into services. Near the end, we get into AI and self-service. Erik's team now generates most of its infrastructure work with AI, and it works because the conventions, agent skills, MCPs and pipelines are already in place. Self-service doesn't have to mean another no-code platform. The big takeaway: AI can generate infrastructure code faster than ever, but without conventions it is generating into chaos. Highlights • What makes a good Terraform module • Root modules vs. child modules • CloudFormation, Terraform, Pulumi, and why constraints helped HCL win • Where Crossplane fits, and what has to exist before the cluster • Why internal developer platforms like Backstage come after zero-to-one • Why infrastructure as code needs frameworks, not just modules • Agent skills, MCPs, and self-service without a no-code platform Links Cloud Posse: https://cloudposse.com Atmos: https://atmos.tools Cloud Posse on GitHub: https://github.com/cloudposse DevOps Office Hours: https://tsn.io/2gq7v Erik Osterman on LinkedIn: https://www.linkedin.com/in/osterman More episodes and show notes: https://tsn.io/NqkdP On Call Brief: https://tsn.io/Gpz2d LMGT Awards: https://lmgt.org | |||
| Critical Atlassian CVE Hits Jira, Confluence & Bitbucket, GitHub Copilot Sandboxing Goes GA, AWS Lambda SnapStart for Containers & Google Spanner Omni | 10 Oct 2026 | 00:17:56 | |
This week on Ship It Weekly: Atlassian disclosed a critical arbitrary file access vulnerability affecting eight Data Center products, including Jira, Confluence, Bitbucket, and Bamboo. GitHub Copilot local sandboxing is now generally available, giving teams enforceable controls around filesystem, network, credentials, and local tooling. AWS published a deeper look at Lambda SnapStart for container images, including the operational risks of restoring initialized state. And Google Spanner Omni is now generally available, bringing Spanner outside Google Cloud and putting more of the availability responsibility back on the customer. The bigger theme this week is control. Developer tools are production infrastructure. AI agents need real execution boundaries, not just instructions. Faster startup through snapshots means understanding exactly what state is being restored. And moving a managed database onto your own infrastructure gives you portability, but also gives you the pager. In the lightning round: Citrix has a NetScaler SAML vulnerability under active exploitation. Cloudflare is working toward becoming a publicly trusted certificate authority. And attackers are actively scanning for Rejetto HFS servers vulnerable to a flaw that can lead to account takeover and remote code execution. And the human closer comes from Lorin Hochstein and a recent Google Cloud networking incident. One part of the outage involved someone correctly following an incorrect procedure, while another involved someone incorrectly following a correct one. Training matters, but systems also need guardrails that keep normal human mistakes from turning into large outages. If your system only works when nobody makes a mistake, the system is the problem. Links Atlassian CVE-2026-21589 GitHub Copilot Local Sandboxing AWS Lambda SnapStart for Container Images Google Spanner Omni GA Citrix NetScaler CVE-2026-88779 Cloudflare Certificate Authority Rejetto HFS CVE-2026-61500 Google Cloud Networking Incident https://www.tellerstech.com/go/s-dc989ad2/ There Was a Light That Never Went Out https://www.tellerstech.com/go/s-cb0bc459/ This Week’s On Call Brief Ship It Weekly On Call Brief | |||
| AWS Retires DevOps Guru: What the End of Support Means, Kubernetes Cross-Namespace CVE-2026-2270, Node.js Undici WebSocket DoS & Cloudflare’s New CLI for AI Agents | 01 Oct 2026 | 00:16:41 | |
This week on Ship It Weekly: AWS is retiring Amazon DevOps Guru and pointing customers toward CloudWatch and the newer Amazon DevOps Agent. Kubernetes disclosed a vulnerability where StatefulSet and ControllerRevision permissions can allow cross-namespace pod creation under specific conditions. A vulnerability in Undici can let a malicious WebSocket server crash a Node.js process through compressed data. And Cloudflare launched a new CLI as AI agents grow from 25 percent to 48 percent of Wrangler usage. The bigger theme this week is how the systems around our infrastructure are changing. Managed cloud services still have lifecycles that eventually become migration work. Kubernetes authorization can depend on what controllers do with the resources users are allowed to manipulate. Applications acting as clients still process untrusted data. And infrastructure tooling is starting to treat AI agents as first-class users rather than humans who happen to automate commands. In the lightning round: another Kubernetes vulnerability affecting Windows nodes can expose NetNTLMv2 credentials through NTLM coercion. GitHub now supports custom runners for Dependabot version and security updates. And external systems like a CMDB or internal developer portal can push repository properties into GitHub while remaining the source of truth. And the human closer comes from Lorin Hochstein and SRE Weekly. Some availability risks are probably never going away. Resources are finite, networks fail, security controls can affect availability, and production systems have to change. Preventing individual failures still matters, but incident response is part of reliability engineering too. Sometimes improving reliability means getting better at handling the failures you cannot eliminate. Links Amazon DevOps Guru End of Support - https://tsn.io/GQHN8 Kubernetes CVE-2026-2270: Cross-Namespace Pod Creation - https://tsn.io/BsNs8 Undici CVE-2026-85024: WebSocket Denial of Service - https://tsn.io/LncLd Cloudflare: Introducing the cf CLI - https://tsn.io/Wk3ma Cloudflare Forge - https://tsn.io/bAPJu Lightning Round Kubernetes CVE-2026-76654: Windows NTLM Coercion - https://tsn.io/36Kc3 GitHub: Custom Runners for Dependabot - https://tsn.io/tYl9K GitHub: External Custom Properties - https://www.tellerstech.com/go/s-1fd1396d/ Human Closer Omnipresent Availability Risks in Cloud Software - https://www.tellerstech.com/go/s-076db9dd/ Our Links This Week’s On Call Brief - https://tsn.io/fKB9V Ship It Weekly - https://tsn.io/NqkdP On Call Brief - https://tsn.io/Gpz2d | |||
| AWS Puts Elastic Beanstalk on EKS, CrowdSec Supply-Chain Breach, Critical Next.js RCE, Microsoft Disrupts EvilTokens & Why Fixing the Initial Compromise Isn’t Enough | 25 Sep 2026 | 00:17:18 | |
This week on Ship It Weekly: AWS introduced Elastic Beanstalk Cluster Mode, allowing multiple applications to run on shared EKS infrastructure while AWS handles much of the Kubernetes complexity. CrowdSec published how a software supply-chain compromise led to attackers copying roughly 170 private repositories using a stolen OAuth token. A critical Next.js vulnerability in ImageResponse can lead to remote code execution through attacker-controlled SVG data. And Microsoft disrupted EvilTokens, a cybercrime platform linked to more than 12,000 compromised inboxes across 10,000 organizations. The bigger theme this week is what happens after trust has been established. Elastic Beanstalk Cluster Mode puts more infrastructure behind a managed abstraction, but shared infrastructure still means understanding isolation and blast radius. CrowdSec shows how an initial compromise can become a credential problem long after the malicious code is gone. Next.js shows how something as ordinary as generating a social preview image can expose a server-side execution path. And EvilTokens shows how attackers can use valid access to move faster once inside an account. In the lightning round: F5 has a critical BIG-IP APM vulnerability under active exploitation. GitHub Enterprise Cloud can now export an inventory of credentials with enterprise access, including PATs, SSH keys, OAuth tokens, and GitHub App credentials. Zyxel patched a vulnerability affecting GS1900 switches. And Veeam Agent for Microsoft Windows has a privilege-escalation vulnerability that can lead to SYSTEM access. And the human closer comes back to CrowdSec. Removing the malicious package, patching the server, or reimaging the workstation does not necessarily end the incident. If an attacker already stole an OAuth token, cloud credential, SSH key, session, or registry credential, that access can survive long after the original compromise is gone. Containment means understanding not only how the attacker got in, but what they took with them Links AWS Elastic Beanstalk Cluster Mode CrowdSec Supply-Chain Attack Analysis Next.js ImageResponse Security Advisory Microsoft: Disrupting EvilTokens Microsoft: EvilTokens and Device-Code Phishing F5 BIG-IP APM CVE-2026-94127 GitHub Enterprise Credential Inventory Zyxel GS1900 Security Advisory https://www.tellerstech.com/go/s-b2595852/ Veeam Agent for Microsoft Windows Vulnerability https://www.tellerstech.com/go/s-166d3119/ This Week’s On Call Brief Ship It Weekly On Call Brief | |||
| GitHub Actions Security, Cisco Email Gateway RCE, Helm 3 End-of-Life, Ubuntu 26.04 Runners & Why “Nothing Changed” Is Never the Whole Story | 19 Sep 2026 | 00:15:33 | |
This week on Ship It Weekly: GitHub Actions workflow execution protections are now generally available, giving organizations more control over who and what can trigger individual workflows. Cisco is patching critical vulnerabilities in Secure Email Gateway, including an actively exploited issue that can lead to remote command execution as root. Helm 3 has reached its final minor release and is heading toward end-of-life in February 2027. And GitHub’s ubuntu-latest Actions runner is preparing to move from Ubuntu 24.04 to 26.04. The bigger theme this week is infrastructure that changes even when your code does not. GitHub is making CI execution permissions more explicit, Helm teams now have a defined migration deadline, and the ubuntu-latest transition is a good example of how a completely unchanged workflow can suddenly be running in a different environment. Pinning everything forever is not necessarily the answer. The important part is knowing which dependencies are allowed to move and testing those changes deliberately. In the lightning round: GitHub Actions checks, workflow runs, and statuses will begin following your configured retention period on October 1. GitHub Advanced Security can now enforce configurations from the enterprise level. GitHub added API support for tracking when self-hosted Actions runner versions lose support. And AI Scan for pull requests can now be used without requiring CodeQL default setup. And the human closer starts with a sentence almost every infrastructure engineer has heard during an incident: “But nothing changed.” Maybe nothing changed in the application, but the runner image changed, a dependency moved, a certificate expired, DNS changed, or an external service behaved differently. Latest tags, loose version constraints, external APIs, and even support windows are dependencies. The goal is not to freeze everything forever. It is to avoid accidental mutability, where something can change without the team realizing it was ever allowed to change. Links GitHub Actions Workflow Execution Protections Cisco Secure Email Gateway Security Advisory Helm 3 End of Life Ubuntu 26.04 GitHub Actions Runners and ubuntu-latest Migration GitHub Actions Retention Changes GitHub Advanced Security Configuration Enforcement GitHub Actions Self-Hosted Runner Lifecycle API GitHub Code Scanning AI Scan This Week’s On Call Brief https://www.tellerstech.com/go/26w38/ Ship It Weekly On Call Brief | |||
| Amazon Linux 2027, GitHub Actions Cache Security, Secret-Scanning Merge Blocks, N-central CVSS 10 RCE, Karmada Graduation, ShieldCrash, CodeQL ARM64 & When Observability Fails Too | 12 Sep 2026 | 00:14:47 | |
This week on Ship It Weekly: Amazon Linux 2027 enters public preview with kernel 7.1+, SELinux enforcing by default, DNF5, newer language runtimes, AWS-LC, and an x86-64-v3 baseline. GitHub Actions adds explicit cache permissions to reduce cache-poisoning risk. GitHub can now block pull requests from merging when they introduce exposed secrets. And N-able N-central has a critical pre-auth RCE that Huntress says is being actively exploited in the wild. The bigger theme this week is catching problems before they turn into incidents. Amazon Linux 2027 gives teams time to test AMIs, bootstrap scripts, agents, Terraform, CloudFormation, and CI/CD before the next platform generation becomes production reality. GitHub’s new cache controls make workflow trust boundaries explicit instead of leaving them implied. And secret-scanning rulesets move credential detection directly into the merge path, where developers can actually act on it. In the lightning round: Karmada graduates from the CNCF as multi-cluster and distributed AI scheduling grow, ShieldCrash research claims another Microsoft Defender patch bypass with SYSTEM-level access, CodeQL 2.27 adds native Linux ARM64 support, and Dependabot can now read private GitHub Packages without another personal access token. And the human closer is about what happens when observability shares the same failure domain as the thing it is watching. A full disk is bad enough. It gets worse when logs stop writing, monitoring data disappears, and the tools used to diagnose the outage start failing too. The takeaway is not that every monitoring component needs total isolation. It is that you should know what can blind you, and make sure at least one useful signal survives the failures you care about most. Links Amazon Linux 2027 Public Preview Amazon Linux 2027 Overview and Preview Details Amazon Linux 2027 Known Issues and Preview Limitations GitHub Actions Cache Permissions with cache-mode Block Pull Requests with Exposed Secrets from Merging N-able N-central 2026.3 Hotfix 4 Huntress: N-able N-central Vulnerability and Active Exploitation Karmada Graduates from the CNCF Microsoft Defender ShieldCrash Zero-Day Research CodeQL 2.27 Adds Linux ARM64 Support Automatic Dependabot Access to GitHub-Hosted Registries Ship It Weekly https://www.tellerstech.com/go/siw/ On Call Brief | |||
| AWS GWLB TCP Reset, Azure DevOps Live Migrations to GitHub, GitHub Runner Enforcement, Docker Root Risk, Lambda IAM Updates, PostgreSQL Upgrade Traps, SonicWall Zero-Days & Better Incident Reviews | 04 Sep 2026 | 00:17:26 | |
This week on Ship It Weekly: AWS Gateway Load Balancer gets TCP Reset, giving applications a faster way to recover when firewalls or other inline appliances fail instead of waiting minutes for TCP retries to time out. Microsoft puts Enterprise Live Migrations into public preview for moving Azure DevOps repositories to GitHub Enterprise Cloud with data residency while developers keep working. GitHub is beginning enforcement against outdated self-hosted Actions runners. And Omarchy fixes a Docker configuration that effectively gave normal desktop processes a path to root. The bigger theme this week is failure modes hiding inside infrastructure we already trust. A dead network path can look like a slow application. A repository migration involves far more than copying Git history. A self-hosted runner can quietly become unsupported while it continues looking healthy. And giving a developer access to the Docker socket may sound like convenience until you remember that the Docker group is effectively a root-level privilege. In the lightning round: Lambda gets full IAM resource-based policies, AWS warns that circular PostgreSQL role memberships can stall major RDS and Aurora upgrades, a researcher releases the FalconFlank CrowdStrike privilege-escalation PoC while CrowdStrike investigates, and SonicWall patches two SMA1000 zero-days after confirming active exploitation. Links AWS Gateway Load Balancer TCP Reset https://www.tellerstech.com/go/s-d7e609ab/ Azure DevOps Enterprise Live Migrations Public Preview https://www.tellerstech.com/go/s-ea05aff9/ GitHub Actions Self-Hosted Runner Minimum Version Enforcement https://www.tellerstech.com/go/s-6e8540c4/ Omarchy: Any User Process Can Escalate to Root https://www.tellerstech.com/go/s-d22971c3/ AWS Lambda Full IAM Resource-Based Policies https://www.tellerstech.com/go/s-ff2a04b5/ Fix Circular Role Dependencies Before Upgrading RDS and Aurora PostgreSQL https://www.tellerstech.com/go/s-e4578f52/ FalconFlank CrowdStrike Privilege Escalation PoC https://www.tellerstech.com/go/s-8c21b00b/ SonicWall SMA1000 Zero-Day Advisory https://www.tellerstech.com/go/s-559ffc8b/ Remote Incident Reviews: Async First, Live Later? https://www.tellerstech.com/go/s-68ca9f5e/ This Week’s On Call Brief Ship It Weekly https://www.tellerstech.com/go/siw/ On Call Brief | |||
| Cloudflare Saves 100TB of RAM, AI Drives Server Prices Up, AWS Adds a Fourth London AZ, Route 53 DNS Self-Service, AKS eBPF Routing, Go 1.27, and the Danger of Hidden Infrastructure Assumptions | 29 Aug 2026 | 00:16:48 | |
This week on Ship It Weekly: Cloudflare explains how five low-level optimizations to the cache behind 1.1.1.1 freed roughly 100 terabytes of RAM while also improving performance. OVHcloud is raising infrastructure prices as AI demand reshapes the memory supply chain. AWS adds a fourth Availability Zone to London, exposing automation that quietly assumed there would always be three. And Route 53 Global Resolver gets a cleaner cross-account model for DNS self-service. The bigger theme this week is assumptions. A few wasted bytes do not matter until you have 250 billion cache entries. A Region having three Availability Zones feels permanent until AWS adds a fourth. And centralized DNS governance works fine until every application team needs a networking ticket just to make a private zone resolvable. In the lightning round: new research looks at manipulating DRAM controller translation registers and the assumptions that creates for memory isolation, AKS eBPF Host Routing reaches general availability, CloudFront Functions can now put custom context directly into access logs, and Go 1.27 lands generic methods along with runtime, tooling, and standard-library improvements. And the human closer looks at an easy Kubernetes mistake: running kubectl against the wrong cluster. Because the active context belongs to the kubeconfig rather than a terminal tab, changing it in one shell can silently affect another. It is a good reminder that some friction is worth keeping around production, and that the safest guardrails live somewhere stronger than operator memory. Links Cloudflare: How We Saved 100 Terabytes of Memory by Optimizing 1.1.1.1’s DNS Cache https://www.tellerstech.com/go/s-9d6c2943/ OVHcloud Raises Prices as AI Memory Demand Reprices Non-AI Infrastructure https://tsn.io/tnaYj AWS Adds a Fourth Availability Zone to Europe (London) https://tsn.io/YfGxa Shared DNS Views with Amazon Route 53 Global Resolver https://tsn.io/WEyig DRAM Controller Register Manipulation Breaks CPU Memory Isolation https://tsn.io/QKr1x AKS eBPF Host Routing https://tsn.io/T3PMh CloudFront Functions Unified Logging https://tsn.io/nTXLn Go 1.27 https://tsn.io/vfXIT kubectl Ran on the Wrong Cluster? Fix Your Context Switching https://tsn.io/6LxfG This Week’s On Call Brief https://tsn.io/064QE Ship It Weekly https://www.tellerstech.com/go/siw/ On Call Brief https://www.tellerstech.com/go/ocb/ | |||
| Ship It Conversations: Justin Garrison of Sidero Labs on Kubernetes, Platform Engineering, AI, Golden Paths, and Knowing What to Say No To | 24 Aug 2026 | 00:41:10 | |
This is a guest conversation episode of Ship It Weekly, separate from the weekly news recaps. In this Ship It Conversations episode, I talk with Justin Garrison of Sidero Labs about Kubernetes, platform engineering, bare metal, AI, golden paths, and why knowing what to say no to may be one of the most important skills a platform team can develop. Justin is Field CTO at Sidero Labs, the company behind Talos Linux, and co-host of Fork Around and Find Out. We start with the evolution of Kubernetes and how managed services like EKS and GKE made Kubernetes easier to consume while also pulling teams deeper into proprietary cloud ecosystems. Justin explains why on-prem and bare metal are getting renewed attention, especially as teams look at cloud costs, data sovereignty, and the operational overhead that comes with constantly optimizing cloud environments. We also get into where Kubernetes helps and where it becomes self-inflicted pain. Justin talks about abstraction, cognitive load, and why teams tend to use familiar tools for problems they were never really designed to solve. A big part of the conversation is platform engineering and golden paths. Justin argues that every organization needs its own path, but platforms become dangerous when they try to centralize everything. He shares why one of the best decisions his team made at Disney Plus was simply saying no to stateful workloads. We also talk about what really belongs in a platform: security controls, logging, monitoring, software supply chain visibility, and cost management. Justin explains why centralization can help in those areas, but can become a bottleneck when applied too broadly. Near the end, we get into AI, security, tooling dependency, and engineering culture. Justin makes the point that people have always formed strong attachments to tools, and AI is another version of that. The challenge is knowing where AI actually helps versus where it becomes another dependency teams stop questioning. The big takeaway: good platform engineering is not about supporting everything. It is about understanding what should be standardized, what should stay flexible, and what your team should explicitly refuse to own. Highlights • Why Kubernetes has become increasingly productized • Why some teams are moving back toward on-prem and bare metal • Where cloud cost optimization starts to become its own operational burden • Why Kubernetes helps with abstraction and cognitive load • Why familiar tools often get used for the wrong workloads • What golden paths actually represent inside an organization • Why platform teams need to know what to say no to • What should and should not be centralized • How AI changes engineering workflows without changing the need for judgment • Why finding work you actually enjoy matters for avoiding burnout Links Sidero Labs: https://www.siderolabs.com Talos Linux: https://www.talos.dev Justin Garrison: https://justingarrison.com Fork Around and Find Out: https://www.forkaroundandfindout.com More episodes and show notes: https://shipitweekly.fm On Call Brief: https://oncallbrief.com LMGT Awards: https://lmgt.org | |||
| GitHub Outage, PleaseFix Agentic Browser Vulnerability, AWS Certificate Manager Drops Email Validation, Cloudflare TypeScript CI Workflows, AI Observability Consolidation, and the Hidden Cost of “Simple” Platform Changes | 21 Aug 2026 | 00:17:39 | |
This week on Ship It Weekly: GitHub suffers another widespread outage affecting the web interface, APIs, Actions, authentication, Copilot, and other critical developer workflows. Zenity Labs demonstrates PleaseFix attacks against agentic browsers, where malicious content can influence agents with access to authenticated sessions and privileged tools. AWS Certificate Manager is moving away from email validation, and Cloudflare is experimenting with CI pipelines defined as TypeScript instead of YAML. The bigger theme this week is dependencies and boundaries we tend to ignore until something breaks. GitHub is no longer just where the code lives. Agentic browsers are no longer just displaying webpages. Certificate renewal is not something you want depending on someone checking an inbox. And CI pipelines have become software systems of their own. Links GitHub Hit by Widespread Outage https://devops.com/github-hit-by-widespread-outage-halting-work-for-global-developers/ Zenity Labs: PleaseFix in Agentic Browsers https://zenity.io/company-overview/newsroom/company-news/zenity-labs-exposes-the-full-scope-of-pleasefix AWS Certificate Manager Ending Email Validation https://aws.amazon.com/blogs/security/aws-certificate-manager-will-discontinue-email-validation-to-prove-domain-validation-for-certificates/ Certificate Expiry Is Still Taking Down Major Platforms https://tokentimer.ch/blog/tls-certificate-expiry-outages Cloudflare Turns CI Pipelines into TypeScript Workflows https://www.infoq.com/news/2026/08/cloudflare-ci-code-workflows/ Dynatrace Acquires Arize https://devops.com/dynatrace-acquires-arize-as-ai-agents-deepen-the-observability-challenge/ AWS Open-Sources Dogwood https://www.infoq.com/news/2026/08/aws-dogwood-agent-policy/ Pulumi v3.258.0 https://github.com/pulumi/pulumi/releases/tag/v3.258.0 AWS Key Breach and Data-Transfer Signal https://assets.theregister.com/2026/08/13/20267/ Mario Saved the EU but Broke My System https://www.uptimelabs.io/articles/hamed-2012-outage-reflections This week’s On Call Brief https://www.tellerstech.com/on-call-brief-news/2026-W34/ Ship It Weekly https://shipitweekly.fm/ | |||
| Ship It Conversations: Ned Bellavance of Ned in the Cloud on DevOps Beyond the Buzzwords, Terraform, AI, the Future of Infrastructure as Code, and Why Fundamentals Still Matter | 16 Aug 2026 | 00:36:12 | |
This is a guest conversation episode of Ship It Weekly, separate from the weekly news recaps. In this Ship It Conversations episode, I talk with Ned Bellavance of Ned in the Cloud about DevOps beyond the buzzwords, platform engineering, infrastructure as code, AI, and why fundamentals still matter even as the tools change. Ned is the founder of Ned in the Cloud and host of the Day 2 DevOps podcast, with more than 20 years in IT across systems administration, cloud, architecture, automation, and technical education. We start with a problem a lot of teams run into: adopting the ceremonies of DevOps without actually adopting the principles. Standups, sprints, pipelines, and tooling can make an organization look mature, but the real goal is better communication, faster feedback loops, and delivery tied to actual outcomes. We also talk about how teams decide what to prioritize next. Security, reliability, performance, FinOps, and platform work can all matter, but chasing whatever is newest does not help if the basics are still broken. A big part of the conversation is where infrastructure as code goes from here. We get into Terraform's state and scaling model, API rate limits, the Terraform/OpenTofu split, Terragrunt, and newer approaches like Swamp from System Initiative. AI is making infrastructure code cheaper to produce, but understanding the architecture behind that code is becoming more valuable. That leads into learning and career development. We talk about why networking, Linux, databases, security, cloud architecture, and troubleshooting still matter, even if an LLM writes most of the syntax. Build things, get them wrong in controlled environments, troubleshoot them, and learn what is happening underneath the abstraction. The big takeaway: tools will keep changing. Judgment, architecture, troubleshooting, and understanding the systems underneath them are much harder to automate away. Highlights • Why DevOps ceremony is not the same as DevOps principles • How teams should decide what platform work actually matters • Why basic security hygiene still matters • Where Terraform's current state model starts to hit limits • Terraform, OpenTofu, Terragrunt, and the future of infrastructure as code • How AI changes the value of writing infrastructure code • Why architecture and troubleshooting skills become more important • Why breaking things in controlled environments is one of the best ways to learn Links Ned in the Cloud: https://nedinthecloud.com Day 2 DevOps: https://day2devops.com Ned in the Cloud on YouTube: https://www.youtube.com/c/NedintheCloud/ Ned Bellavance on LinkedIn: https://www.linkedin.com/in/ned-bellavance/ Swamp: https://swamp.club Terraform: https://developer.hashicorp.com/terraform OpenTofu: https://opentofu.org More episodes and show notes: https://shipitweekly.fm On Call Brief: https://oncallbrief.com | |||
| Railway US East Outage, Stripe’s Graph-Based Database Recovery, Kata Containers Host Escape, DynamoDB Vector Search, AWS Network Firewall Proxy, Gateway API 1.6, and containerd 2.4 | 14 Aug 2026 | 00:15:52 | |
This week on Ship It Weekly: Railway explains how an upstream network problem turned into a much larger US East outage, including storage traffic falling back onto the management network and stale connections continuing to cause problems after routing recovered. Stripe shares how graph search and state machines helped cut database pager volume by about 30 percent. Kata Containers patches a critical guest-to-host escape, and DynamoDB adds native vector search. The bigger theme this week is what happens after the obvious failure. Fixing the route does not necessarily clear the connections created while it was broken. Automating recovery does not have to mean handing an AI agent unrestricted production access. And stronger isolation does not eliminate the components that still cross the guest-host boundary. In the lightning round: AWS brings explicit forward proxy functionality back through Network Firewall, Gateway API 1.6 moves TCPRoute and UDPRoute to stable, and containerd 2.4 enters beta with new functionality alongside breaking changes worth finding before your next runtime upgrade. Links Railway: July 2, 2026 US East Services Outage https://blog.railway.com/p/incident-report-july-2-2026-us-east-services-outage Stripe: How Stripe uses graph search and state machines to auto-remediate a global database fleet https://stripe.dev/blog/how-stripe-uses-graph-search-and-state-machines-to-auto-remediate-a-global-database-fleet Kata Containers: Guest-root to host-root escape via virtiofs https://github.com/kata-containers/kata-containers/security/advisories/GHSA-2gv2-cffp-j227 Amazon DynamoDB now supports real-time vector search https://aws.amazon.com/about-aws/whats-new/2026/08/amazon-dynamodb-vector-search/ AWS Network Firewall forward proxy preview https://aws.amazon.com/about-aws/whats-new/2026/08/aws-network-firewall-forward-proxy-preview/ Gateway API v1.6: TCPRoute and UDPRoute Graduate to Standard https://kubernetes.io/blog/2026/08/03/gateway-api-v1-6-release/ containerd 2.4 beta https://github.com/containerd/containerd/releases CNCF: Learning Cloud-Native Engineering Beyond Tutorials Through LFX https://www.cncf.io/blog/2026/08/10/learning-cloud-native-engineering-beyond-tutorials-through-lfx/ This week’s On Call Brief https://www.tellerstech.com/on-call-brief-news/2026-W33/ Ship It Weekly https://shipitweekly.fm/ | |||
| AI Agents Target GitHub, Kubernetes 1.37 Deprecations, AWS Transit Gateway Policy Routing, IAM Identity Center Multi-Region, Cloudflare Meerkat, AMOS Mac Malware, and the Risk of Half-Migrated Production Systems | 07 Aug 2026 | 00:18:19 | |
This week on Ship It Weekly: AI agents from Anthropic and OpenAI took unsanctioned actions on the real internet during UK government cyber testing, including an attempt to push malicious code into a real GitHub project. Kubernetes 1.37 starts retiring IPVS mode, pushes cgroup v1 closer to removal, and brings an SELinux volume change worth testing before upgrades. AWS Transit Gateway gets policy-based routing, and IAM Identity Center expands multi-Region support to organizations using AWS’s built-in directory. The bigger theme: access is not the same thing as authority, and availability is not just about whether your application is running. Agents need boundaries around the actions they can take. Routing policies need enough visibility to explain why traffic went where it did. And regional resilience does not help much if the people responding to the outage cannot authenticate. In the lightning round: Cloudflare’s Meerkat consensus system, AMOS macOS malware, N-able’s incomplete N-central fix, and an AWS CLI bug that disabled SSH host-key verification. Links AISI: Unsanctioned agent behaviour during cyber testing https://www.aisi.gov.uk/blog/incident-report-unsanctioned-agent-behaviour-during-cyber-testing Kubernetes v1.37 Sneak Peek https://kubernetes.io/blog/2026/07/31/kubernetes-v1-37-sneak-peek/ AWS Transit Gateway Policy-Based Routing https://aws.amazon.com/about-aws/whats-new/2026/07/aws-transit-gateway-policy-based-routing/ AWS IAM Identity Center multi-Region directory support https://aws.amazon.com/about-aws/whats-new/2026/07/aws-iam-identity-center-extends-multi-region-support-to-identity-center-directory Cloudflare: Introducing Meerkat https://blog.cloudflare.com/meerkat-introduction/ Atomic macOS / AMOS stealer infection https://isc.sans.edu/diary/rss/33208 N-able N-central exploitation after incomplete fix https://thehackernews.com/2026/08/n-able-says-attackers-take-over-n.html CVE-2026-18654: AWS CLI EMR SSH host-key verification https://aws.amazon.com/security/security-bulletins/rss/2026-071-aws/ CloudFront VPC Origins half-migrated incident https://www.reddit.com/r/devops/comments/1vdovj6/the_cloudfront_vpc_origins_outage_caught_me/ This week’s On Call Brief https://www.tellerstech.com/on-call-brief-news/2026-W32/ More episodes https://shipitweekly.fm/ | |||
| Telstra’s Time Sync Outage, DoorDash’s 1.5M RPS Cache, Stateless MCP, GitHub and PyPI Supply Chain Delays, and Why the Quietest Infrastructure Often Has the Biggest Blast Radius | 31 Jul 2026 | 00:16:57 | |
This week on Ship It Weekly: Telstra’s mobile network jumped back to 2006 after a timing device restarted with the wrong date, disrupting calls, data sessions, and hundreds of emergency calls. DoorDash explains how Entity Cache, built with Envoy and Valkey, handles more than 1.5 million requests per second and uses stale-data policies, invalidation, and fallback behavior as a reliability layer. The latest MCP release candidate removes protocol-level sessions, making servers easier to scale behind ordinary load balancers while leaving teams responsible for authentication, tracing, retries, rate limits, and application state. GitHub and PyPI are also adding friction to package automation. Dependabot now delays routine updates by three days, while PyPI blocks new files from releases older than 14 days. Links Telstra outage https://www.telstra.com.au/exchange/our-mobile-network-outage-has-been-resolved-heres-what-happened DoorDash Entity Cache https://careersatdoordash.com/blog/high-performance-proxy-cache-for-doordash-services/ MCP specification release candidate https://blog.modelcontextprotocol.io/posts/2026-07-28-release-candidate/ Dependabot package cooldown https://github.blog/changelog/2026-07-14-dependabot-version-updates-introduce-default-package-cooldown/ PyPI release-file restrictions https://blog.pypi.org/posts/2026-07-22-releases-now-reject-new-files-after-14-days/ Amazon ECS Action Logs https://aws.amazon.com/about-aws/whats-new/2026/07/amazon-ecs-action-logs/ Network Load Balancer listener rules https://aws.amazon.com/about-aws/whats-new/2026/07/aws-network-load-balancer-supports-listener-rules/ Amazon Managed Prometheus limits https://aws.amazon.com/about-aws/whats-new/2026/07/amazon-managed-service-prometheus-1500m-metrics-workspace/ PixelSmash in FFmpeg https://jfrog.com/blog/pixelsmash-critical-ffmpeg-vulnerability-turns-media-files-into-weapons/ SRE Weekly Issue 527 https://sreweekly.com/sre-weekly-issue-527/ This week’s On Call Brief https://www.tellerstech.com/on-call-brief-news/2026-W31/ More episodes https://shipitweekly.fm/ | |||
| Ship It Conversations: Jay Lark of Hookbridge on Webhook Reliability, Retries, Idempotency, Replay, HMAC Security, Local Testing, and Operating Webhooks in Production | 27 Jul 2026 | 00:31:26 | |
This is a guest conversation episode of Ship It Weekly, separate from the weekly news recaps. In this Ship It Conversations episode, I talk with Jay Lark of Hookbridge about webhook reliability, retries, idempotency, replay, security, local development, and what happens when a simple HTTP POST becomes production infrastructure. Jay is a Principal DevOps Engineer and the founder of Hookbridge, a service focused on making webhook delivery more reliable and easier to operate. We start with the basic model: one service sends an HTTP POST to another when something happens. The happy path is easy. The problems begin when an endpoint is unavailable, responds too slowly, receives duplicate events, or gets them out of order. Jay explains what “at least once delivery” means and why receivers must expect duplicates. We talk about idempotency, event IDs, retry behavior, availability during deployments, and why returning a 200 does not prove downstream processing succeeded. We also dig into observability and security. Teams need enough visibility to know whether a webhook arrived, whether signature verification passed, what response was returned, and where processing failed. Jay breaks down HMAC signatures, timestamps, replay protection, and why a valid signature still does not replace normal business-logic validation. Local development is another source of friction. External providers cannot send events directly to localhost, so developers often rely on temporary tunnels, staging deployments, copied payloads, or mocks. Jay explains how Hookbridge uses a fixed URL and local client to forward real webhook traffic to a developer’s machine. We also talk about n8n, self-hosted OpenClaw systems, Hookbridge pull endpoints, and when polling may be simpler or safer than exposing another inbound endpoint. The big takeaway: design the failure path before a webhook becomes business-critical. Verify the sender, expect duplicate and out-of-order events, build enough visibility to debug failures, and have a replay strategy before the first incident. Highlights • Why webhooks are harder than “just an HTTP POST” • What at-least-once delivery means for receivers • Why idempotency, retries, and event ordering matter • What teams need for webhook observability and debugging • Where HMAC signatures, timestamps, and replay protection fit • Why local webhook development is still awkward • When polling or pull-based delivery may be a better fit • When teams should stop building webhook infrastructure themselves Links Hookbridge: https://hookbridge.io Hookbridge local development CLI: https://www.hookbridge.io/cli.html Hookbridge pull endpoints: https://www.hookbridge.io/pull.html Jay Lark on LinkedIn: https://www.linkedin.com/in/jay-lark-ba7a3b5/ OpenClaw: https://openclaw.ai More episodes and show notes: https://shipitweekly.fm On Call Brief: https://oncallbrief.com | |||
| AWS CloudFormation Express Mode, Spark 4.2 Vector Search, AI Speeds Coding but Not Delivery, Platform Governance Developers Won’t Hate, and Why Could Is Not the Same as Should | 24 Jul 2026 | 00:18:34 | |
This week on Ship It Weekly: AWS CloudFormation Express mode promises faster infrastructure feedback by reporting deployments complete before extended resource stabilization finishes. Apache Spark 4.2 adds native vector operations and nearest-neighbor joins, giving some teams a way to keep AI data workloads closer to the platforms they already run. GitLab’s latest research says AI is helping developers generate and commit code faster, but review, testing, governance, and deployment are not accelerating at the same pace. Then, a platform engineering case study from Sevdesk shows how minimum viable governance, useful feedback, and progressive enforcement can improve compliance without turning the platform team into another approval queue. The theme this week: making one part of the system faster does not automatically improve the whole system. In the lightning round: OpenShift 4.22.5 receives an important security update, outdated autoscaling thresholds disrupt GitHub Actions, Cloudflare experiences an incident where some POST requests fail to reach customer origins, and GitHub Code Quality becomes generally available—with automatic billing attached. The episode closes with Reid Savage reflecting on their first year managing Honeycomb’s SRE team and the difference between what a capable team could do and what it should do. Links CloudFormation Express mode https://aws.amazon.com/blogs/aws/accelerate-your-infrastructure-deployments-by-up-to-4x-with-aws-cloudformation-express-mode/ Apache Spark 4.2 https://spark.apache.org/releases/spark-release-4-2-0.html GitLab AI Accountability Report https://about.gitlab.com/resources/ai-accountability-survey-2026/ Platform governance at Sevdesk https://www.infoq.com/presentations/platform-engineering-team-compliance/ OpenShift 4.22.5 security update https://access.redhat.com/errata/RHSA-2026%3A37585 GitHub Actions incident https://github.com/orgs/community/discussions/201795 Cloudflare status https://www.cloudflarestatus.com/ GitHub Code Quality GA https://github.blog/changelog/2026-07-20-github-code-quality-is-now-generally-available/ Could vs. Should — Reid Savage https://www.honeycomb.io/blog/could-should-first-year-managing-sre-team This week’s On Call Brief https://www.tellerstech.com/on-call-brief-news/2026-W30/ More episodes and full show notes https://shipitweekly.fm/ | |||
| Ship It Conversations: Mat Ryer of Grafana Labs on AI Observability, Agents, Evals, and Operating AI in Production | 20 Jul 2026 | 00:45:06 | |
This is a guest conversation episode of Ship It Weekly, separate from the weekly news recaps. In this Ship It Conversations episode, I talk with Mat Ryer of Grafana Labs about AI observability, production agents, evals, telemetry cost, guardrails, and what changes once AI moves beyond demos and into systems teams actually depend on. Mat is Senior Director of AI at Grafana Labs, where he focuses on how AI fits into observability and production systems. We talk about Grafana Assistant, why AI observability is not just logs, latency, and HTTP 200s, and how teams can measure whether agents are actually helping. Mat gets into evals, LLM-as-judge patterns, traces as a way to think about conversations, user feedback, tool choice, model changes, and the cost of collecting new telemetry. We also dig into UX and trust. If an AI assistant gives you a wall of text, you still have to decide whether to believe it. If it can show the graph, deep link into Grafana, apply filters, and expose the source data, that becomes a much more useful operating experience. The big takeaway: start small, enhance workflows you already have, build feedback loops, and treat production AI like something you actually have to operate. Highlights • Why AI demos are easy, but production AI is harder • Why agents need observability, evals, and guardrails • How Grafana thinks about AI Assistant and AI observability • Where LLM-as-judge patterns, traces, tool calls, and feedback fit • Why telemetry cost problems may repeat with AI workloads • Why UX matters when operators need to trust the answer • Where AI can help SRE and platform teams today Links Mat Ryer on LinkedIn: https://www.linkedin.com/in/matryer/ Mat Ryer on GitHub: https://github.com/matryer Grafana Labs: https://grafana.com Grafana Assistant: https://grafana.com/products/cloud/ai-assistant/ Grafana AI Observability: https://grafana.com/docs/grafana-cloud/machine-learning/ai-observability/ Grafana Adaptive Telemetry: https://grafana.com/products/cloud/adaptive-telemetry/ Grafana MCP server: https://github.com/grafana/mcp-grafana OpenTelemetry: https://opentelemetry.io Prometheus: https://prometheus.io Grafana Loki: https://grafana.com/docs/loki/latest/ More episodes and show notes: https://shipitweekly.fm On Call Brief: https://oncallbrief.com | |||
| GitHub API Enumeration, Grok Build CLI Data Exposure, AWS Security Hub Network Scanning, AI-Powered Patch Pressure, and Why Visibility Is Not Ownership | 18 Jul 2026 | 00:19:16 | |
This week on Ship It Weekly: Datadog tracked coordinated GitHub API enumeration, xAI’s Grok Build CLI reportedly uploaded repo data without redaction, AWS Security Hub added Network Scanning and exposure impact analysis, and Microsoft says AI-powered vulnerability discovery is changing patch pressure. The theme: visibility is not ownership. A GitHub API map does not revoke a token. An exposure finding does not close a port. A patch bulletin does not patch the fleet. And an AI coding tool reading your repo is still access. Brian covers GitHub as a production surface, AI coding tool data boundaries, cloud exposure based on reachability and blast radius, and why patching needs to look more like production operations than spreadsheet theater. Also, the Ship It Weekly shop is open at shop.tellerstech.com with Ship It Weekly t-shirt designs. Use coupon code SHIPTHESTORE for 20% off your order for the next few weeks. Links Datadog: Coordinated GitHub API enumeration https://securitylabs.datadoghq.com/articles/coordinated-github-api-enumeration/ The Verge: Grok Build CLI repository upload report https://www.theverge.com/ai-artificial-intelligence/965600/spacexai-grok-build-repository-upload AWS Security Hub Network Scanning https://aws.amazon.com/about-aws/whats-new/2026/07/aws-security-hub-network-scanning/ AWS Security Hub impact analysis for exposure findings https://aws.amazon.com/about-aws/whats-new/2026/07/impact-analysis-aws-security-hub/ Microsoft: Windows vulnerability management and AI-powered discovery https://blogs.windows.com/windowsexperience/2026/07/09/evolving-windows-vulnerability-management-to-meet-the-speed-of-ai-powered-discovery/ SRE Weekly Issue 525 https://sreweekly.com/sre-weekly-issue-525/ HalluSquatting / hallucinated package risk https://www.endorlabs.com/learn/slopsquatting-when-ai-agents-hallucinate-malicious-packages AWS Lambda Managed Instances for Java cold starts https://aws.amazon.com/blogs/compute/eliminating-java-cold-starts-with-aws-lambda-managed-instances/ This week’s On Call Brief https://www.tellerstech.com/on-call-brief-news/2026-W29/ Ship It Weekly shop https://shop.tellerstech.com/ More episodes and full show notes https://shipitweekly.fm/ | |||
| EKS Rollbacks, GitHub Actions Supply Chain Attacks, AI Agentjacking, CloudWatch Log Alarms, and Why Safety Nets Don’t Replace Ownership | 10 Jul 2026 | 00:19:13 | |
This week on Ship It Weekly: Amazon EKS added Kubernetes version rollbacks, Novee Security published Cordyceps research on GitHub Actions supply chain risk, Tenet Security showed how fake telemetry can hijack AI coding agents, and Amazon CloudWatch added alarms directly from log queries. The theme: safety nets are getting better, but the blast radius is getting wider. Rollback buttons, log alarms, zone-aware routing, secret scanning, and AI agent workflows all help, but they do not replace ownership. Brian covers why EKS rollbacks are useful but not a substitute for real upgrade discipline, why GitHub Actions YAML is production code with credentials, how fake Sentry telemetry can become hostile agent context, and why easier log-based alarms can also mean easier pager noise. In the lightning round: ECS Service Connect zone-aware routing, etcd 3.7, GitHub innersource advisories, secret scanning metadata improvements, and CloudWatch Application Signals service events. Links Amazon EKS Kubernetes version rollbacks https://aws.amazon.com/blogs/aws/upgrade-amazon-eks-clusters-with-confidence-using-kubernetes-version-rollbacks/ Novee Security: Cordyceps supply chain research https://novee.security/blog/cordyceps/ Tenet Security: Agentjacking through fake Sentry errors https://tenetsecurity.ai/blog/agentjacking-coding-agents-with-fake-sentry-errors/ Amazon CloudWatch log query alarms https://aws.amazon.com/about-aws/whats-new/2026/07/amazon-cloudwatch-log-alarms/ ECS Service Connect zone-aware routing https://aws.amazon.com/about-aws/whats-new/2026/07/ecs-service-connect-zone-aware/ etcd 3.7 announcement https://etcd.io/blog/2026/announcing-etcd-3.7/ GitHub innersource security advisories https://github.blog/changelog/2026-07-08-innersource-security-advisories-are-generally-available/ GitHub secret scanning extended metadata and multipart validation https://github.blog/changelog/2026-07-07-secret-scanning-extended-metadata-and-multipart-validation/ CloudWatch Application Signals service events https://aws.amazon.com/about-aws/whats-new/2026/06/cloudwatch-service-events/ Our Links This week’s On Call Brief https://www.tellerstech.com/on-call-brief-news/2026-W28/ More episodes and full show notes https://shipitweekly.fm/ | |||
| Ship It Conversations: Evan Phoenix of Miren on Deployment Pain, Terraform, Waypoint, and Better Defaults for Small Teams | 06 Jul 2026 | 00:41:52 | |
This is a guest conversation episode of Ship It Weekly, separate from the weekly news recaps. In this Ship It: Conversations episode, I talk with Evan Phoenix of Miren about why deployment is still painful, what teams keep getting wrong when they try to simplify it, and why small teams may need better defaults more than more platform knobs. Evan is the CEO of Miren. He previously worked on Terraform Enterprise and Waypoint at HashiCorp, and he also built Puma and Rubinius. We talk about deployment as the “final boss” of software delivery. Not because teams do not know how to ship code, but because deployment is where everything collides: runtimes, registries, secrets, networking, cloud services, databases, rollbacks, and the internal platform nobody wants to touch anymore. A big theme is opinionated tooling. Engineers often say they want flexibility, but many teams are really asking for good defaults, a clear happy path, and fewer decisions to own. We also get into Terraform Enterprise, Terraform Cloud, Terragrunt, OpenTofu, Waypoint, Kubernetes, Heroku, ECS, container registries, and how AI changes the deployment conversation. AI can generate infrastructure code, but when that setup breaks, someone still has to understand it and be on call for it. Highlights • Why deployment is still painful after years of platforms and abstractions • What Evan learned from Terraform Enterprise and Waypoint • Why Terraform structure, state, modules, and repo layout remain hard • Why OpenTofu gained traction beyond the Terraform licensing change • Why Kubernetes can be too much surface area for some teams • What small teams actually need from deployment tooling • How AI changes infrastructure and deployment workflows • Why generated infrastructure still needs ownership and accountability Links • Miren: https://miren.dev • Miren on GitHub: https://github.com/mirendev • Evan Phoenix: https://evanphx.dev • Evan on Bluesky: https://bsky.app/profile/evanphx.dev • Evan on Linkedin: https://www.linkedin.com/in/evanphoenix/ Things mentioned • Terraform Enterprise: https://developer.hashicorp.com/terraform/enterprise • Terraform Cloud: https://developer.hashicorp.com/terraform/cloud-docs • Terragrunt: https://terragrunt.gruntwork.io • OpenTofu: https://opentofu.org • HashiCorp Waypoint: https://github.com/hashicorp/waypoint • Knative: https://knative.dev Our links More episodes + show notes: https://shipitweekly.fm On Call Brief: https://oncallbrief.com | |||
| Amazon Q CVEs, Hijacked npm and Go Packages, AWS WAF HTTP/2 Issues, Lambda MicroVMs, and Why Execution Is the Boundary Now | 03 Jul 2026 | 00:18:06 | |
This week on Ship It Weekly: Amazon Q Developer and the AWS language servers had a pair of trust-boundary CVEs, JFrog found hijacked npm and Go packages using hidden VS Code tasks to run malware when a workspace opens, AWS WAF had HTTP/2 request-body inspection issues, and AWS introduced Lambda MicroVMs for running user-generated and AI-generated code in isolated sandboxes. The bigger theme: execution is the boundary now. The repo, the IDE, the AI assistant, the WAF, and the sandbox all sit at the point where something gets to run, inspect, block, or decide. Before execution, trust is a policy. After execution, trust is a blast radius. In the lightning round, Brian covers GitHub’s record advisory volume, Git 2.55, Valkey 9.1 on Amazon ElastiCache, and a quick Fable 5 callback now that Anthropic’s Fable 5 is back online. Links AWS security bulletin: Amazon Q / AWS language server CVEs https://aws.amazon.com/security/security-bulletins/2026-047-aws/ JFrog: Hijacked npm packages using VS Code tasks https://research.jfrog.com/post/hijacked-npm-vscode-tasks-blockchain/ AWS security bulletin: AWS WAF HTTP/2 inspection issues https://aws.amazon.com/security/security-bulletins/2026-048-aws/ AWS Lambda MicroVMs https://aws.amazon.com/blogs/aws/run-isolated-sandboxes-with-full-lifecycle-control-aws-lambda-introduces-microvms/ GitHub Advisory Database record volume https://github.blog/security/supply-chain-security/inside-the-advisory-database-and-what-happens-when-vulnerability-volume-breaks-records/ Git 2.55 highlights https://github.blog/open-source/git/highlights-from-git-2-55/ Amazon ElastiCache Valkey 9.1 https://aws.amazon.com/blogs/database/announcing-valkey-9-1-for-amazon-elasticache/ Claude Fable 5 and Mythos 5 model docs https://platform.claude.com/docs/en/about-claude/models/introducing-claude-fable-5-and-claude-mythos-5 This week’s On Call Brief https://www.tellerstech.com/on-call-brief-news/2026-W27/ More episodes and full show notes https://shipitweekly.fm/ | |||
| Ship It Conversations: Kat Traxler of Vectra AI on AI Security, the Zero-Day Clock, IAM, and Cloud Risk | 28 Jun 2026 | 00:42:33 | |
This is a guest conversation episode of Ship It Weekly, separate from the weekly news recaps. In this Ship It: Conversations episode, I talk with Kat Traxler of Vectra AI about AI security, the zero-day clock, IAM, cloud risk, AI-assisted bug hunting, and why the scariest future security problems may still start with the boring fundamentals teams already struggle with today. Kat is a Principal Security Researcher at Vectra AI focused on abuse techniques and vulnerabilities in the public cloud, especially around the intersection of cloud security, AppSec, IAM, managed identities, and insecure-by-design flaws. We talk about the current AI security mood, from the excitement around faster research and bug hunting to the fear that AI could shrink the window between vulnerability disclosure and exploitation. Kat explains the “San Francisco Consensus,” why the zero-day clock is getting so much attention, and why she thinks the facts may be real while some of the conclusions are overextended. The bigger theme here is that AI is absolutely changing security work, but it does not erase the fundamentals. Attackers still take the lowest-friction path that works. For most teams, that still means credentials, IAM, misconfigurations, known vulnerabilities, and systems that were never threat-modeled as deeply as people assume. Highlights • Why AI security feels exciting and unsettling at the same time • What the “San Francisco Consensus” means and why people are talking about the zero-day clock • How AI may shrink the time between vulnerability disclosure and exploitation • Why Kat is skeptical of the full “zero-day apocalypse” narrative • Why credentials, IAM, misconfigurations, and known vulnerabilities still matter most for many teams • How AI helps narrow the search space in bug hunting and security research • Where AI is useful for code-level bugs, and where it still struggles with context and threat modeling • Why human expertise still matters when using AI for writing, research, and cloud security analysis • Why IAM remains hard because it sits at the intersection of people, access, and technology • What insecure-by-design flaws are, and why AI may not solve those anytime soon Kat / Vectra AI links • Kat Traxler at Vectra AI: https://www.vectra.ai/about/author/kat-traxler • Kat’s site: https://kattraxler.cloud/ • The San Francisco Consensus: https://kattraxler.cloud/the-san-francisco-consensus/ • Kat on X: https://x.com/NightmareJS • Vectra AI: https://www.vectra.ai/ Our links More episodes + show notes + links: https://shipitweekly.fm On Call Brief: https://oncallbrief.com | |||
| containerd CRI Vulnerabilities, Datadog PostgreSQL HA on Kubernetes, AWS DevOps Agent with Datadog MCP Server, EKS Control Plane Egress, and Why Users Feel the Wait | 26 Jun 2026 | 00:19:24 | |
This week on Ship It Weekly: containerd disclosed a batch of CRI plugin vulnerabilities, Datadog tested PostgreSQL high availability on Kubernetes and found that failover is not useful if it cannot happen safely, AWS DevOps Agent and Datadog MCP Server moved AI incident response closer to real production workflows, and Amazon EKS added customer-routed control-plane egress. The bigger theme: the control plane keeps getting wider. Runtimes, databases, incident agents, API-server egress, credentials, the cloud console, and object metadata are all becoming part of the production blast radius. And when something breaks, users do not experience your architecture diagram. They experience waiting. In the lightning round, Brian covers GitHub self-service credential revocation for incident response, AWS Management Console Private Access without internet connectivity, Vercel Connect and short-lived agent credentials, and Amazon S3 annotations. Links containerd CRI plugin vulnerabilities / AWS security bulletin https://aws.amazon.com/security/security-bulletins/2026-046-aws/ Datadog: PostgreSQL high availability on Kubernetes https://www.datadoghq.com/blog/engineering/postgresql-ha-kubernetes/ AWS DevOps Agent and Datadog MCP Server https://aws.amazon.com/blogs/devops/production-ready-autonomous-incident-resolution-with-aws-devops-agent-now-ga-and-datadog-mcp-server/ Amazon EKS customer-routed control-plane egress https://aws.amazon.com/blogs/containers/amazon-eks-now-supports-control-plane-egress-through-your-vpc/ GitHub self-service credential revocation for incident response https://github.blog/changelog/2026-06-24-self-service-credential-revocation-for-incident-response/ AWS Management Console Private Access https://aws.amazon.com/about-aws/whats-new/2026/06/aws-management-console-private/ Vercel Connect https://vercel.com/blog/introducing-vercel-connect Amazon S3 annotations https://aws.amazon.com/blogs/aws/amazon-s3-annotations-attach-rich-queryable-context-directly-to-your-objects/ Marc Brooker: Waiting, latency, MTTR, and the inspection paradox https://brooker.co.za/blog/2026/06/19/waiting.html This week’s On Call Brief https://www.tellerstech.com/on-call-brief-news/2026-W26/ More episodes and full show notes https://www.shipitweekly.fm | |||
| Ship It Conversations: Guardsquare’s Joel DeStefano on Mobile App Security, Runtime Protection, App Hardening, and Why Scanning Isn’t Enough | 21 Jun 2026 | 00:35:58 | |
This is a guest conversation episode of Ship It Weekly, separate from the weekly news recaps. In this Ship It: Conversations episode, I talk with Joel DeStefano from Guardsquare about mobile app security, why it is different from backend and cloud security, and why scanning alone is not enough once an app is shipped into the real world. We talk about the shift in trust model that happens with mobile apps. In backend and cloud systems, teams usually have more control over the runtime, infrastructure, policies, and monitoring. With mobile, the app becomes a public artifact running on someone else’s device, in an environment you do not fully control. The bigger theme here is that mobile security is not just “scan it before release.” Scanning matters, but teams also need to think about app hardening, obfuscation, runtime protection, monitoring, and whether the app connecting back to their APIs is genuine and uncompromised. Highlights • Why mobile changes the trust model compared to backend and cloud systems • What DevOps, SRE, and platform teams should understand about mobile app risk • Why scanning is useful, but not enough by itself • The danger of assuming app store approval means an app is secure • Why “we do not store sensitive data in the app” can be a misleading security argument • How attackers can reverse engineer apps, inspect workflows, and learn how the app talks to backend APIs • What code hardening and obfuscation actually help protect against • Why runtime checks matter for rooted devices, compromised environments, debuggers, hooking frameworks, overlays, and accessibility abuse • The difference between Android and iOS security assumptions • Why the OS is not responsible for protecting your app’s business logic • How mobile security should fit into CI/CD without destroying release velocity • What should block a release versus what should become tracked risk • Why testing, hardening, runtime protection, and monitoring should work together as one strategy • How AI may speed up attackers without fundamentally changing the need for strong security fundamentals • Joel’s advice for improving mobile security posture: start with the app’s critical workflows, backend interactions, and real business risk Joel / Guardsquare links • Guardsquare: https://hubs.ly/Q04fJgkJ0 • Guardsquare Blog: https://www.guardsquare.com/blog OWASP mobile security links • OWASP Mobile Application Security: https://owasp.org/www-project-mobile-app-security/ • OWASP MASVS: https://mas.owasp.org/MASVS/ Our links More episodes + show notes + links: https://shipitweekly.fm On Call Brief: https://oncallbrief.com | |||
| PeopleSoft Zero-Day Exploited, npm v12 Install Script Changes, GitHub Agentic Tokens, Anthropic Model Risk, and Default Trust Breaking | 19 Jun 2026 | 00:22:27 | |
This episode of Ship It Weekly is about default trust getting punished. Brian covers Oracle’s emergency PeopleSoft advisory for CVE-2026-35273, npm v12 changing install-script defaults, GitHub Agentic Workflows moving away from long-lived personal access tokens, and Anthropic disabling Fable 5 and Mythos 5 after a U.S. export-control directive. The common thread: legacy ERP systems, package installs, CI/CD agents, and AI models all become production risks when teams trust the default without checking what that trust can actually do. In the lightning round, Brian covers Tekton CloudEvents moving to a dedicated events controller, NVIDIA Triton Inference Server 26.04 changing inference defaults, AWS Nitro Isolation Engine bringing formal verification to Graviton5-based isolation, and Homebrew 6.0 adding explicit trust for third-party taps. The bigger theme: production does not care why you trusted the default. It only cares what that default was allowed to do. The bigger theme: production does not care why you trusted the default. It only cares what that default was allowed to do. Links Oracle PeopleSoft CVE-2026-35273 advisory https://www.oracle.com/security-alerts/alert-cve-2026-35273.html npm v12 breaking changes https://github.blog/changelog/2026-06-09-upcoming-breaking-changes-for-npm-v12/ GitHub Agentic Workflows no longer need PATs https://github.blog/changelog/2026-06-11-agentic-workflows-no-longer-need-a-personal-access-token/ Anthropic Fable 5 / Mythos 5 access statement https://www.anthropic.com/news/fable-mythos-access Tekton Pipelines releases https://github.com/tektoncd/pipeline/releases NVIDIA Triton Inference Server 26.04 release notes https://docs.nvidia.com/deeplearning/triton-inference-server/release-notes/rel-26-04.html AWS Nitro Isolation Engine https://aws.amazon.com/blogs/compute/aws-nitro-isolation-engine-formally-verifying-the-hypervisor-in-the-aws-nitro-system/ Homebrew 6.0.0 https://brew.sh/2026/06/11/homebrew-6.0.0/ This week’s On Call Brief https://www.tellerstech.com/on-call-brief-news/2026-W25/ More episodes and show notes https://shipitweekly.fm/ | |||
| Ship It Conversations: Meta’s Francois Richard on AI Incident Response, SLOs, and Reliability at Scale | 16 Jun 2026 | 00:42:56 | |
This is a guest conversation episode of Ship It Weekly, separate from the weekly news recaps. In this Ship It: Conversations episode, I talk with Francois Richard, Engineering Director at Meta, about reliability at scale, how AI is changing production risk, what teams actually learn from incidents, and why recovery practice matters just as much as prevention. We talk about the proactive and reactive sides of reliability, why SLOs should represent a promise to users instead of just another dashboard number, how incident reviews should drive real system improvements, and how teams can practice recovery before production forces the lesson on them. The bigger theme here is that reliability is not just about avoiding failure. It is about knowing what happens when prevention fails. That means practicing regional failure, understanding overload behavior, improving incident response, using AI carefully during investigation, and making reliability targets match the actual lifecycle and importance of the system. Highlights • Why reliability work starts with both prevention and recovery • The difference between reactive incident response and proactive reliability engineering • How Meta thinks about disaster recovery testing and regional failure practice • Why an SLO should be treated like a promise to users, not just a dashboard metric • How SLO trends help teams decide when to invest more in reliability or take more product risk • What engineers actually learn during the “pressure cooker” of an incident • Why incident reviews should produce follow-up work, not just a nicer explanation of what broke • The difference between finding the cause of an incident and improving the system • Where AI agents can help with incident investigation, telemetry, metrics, and query building • Why AI-generated code can increase change volume while reducing human context • How faster code generation changes the kinds of reliability problems teams should expect • Why recovery practice matters, especially for region loss, traffic spikes, overload, and restart behavior • What smaller DevOps and SRE teams can learn from Meta-scale reliability patterns • Why not every system needs six nines, especially early in a product lifecycle • How to think about reliability investment based on user promise, product maturity, and operational risk • Why At Scale Systems & Reliability is focused on the infrastructure behind AI and the use of AI to operate large-scale systems Francois’ links • LinkedIn: https://www.linkedin.com/in/francoisrichard/ At Scale links • Systems & Reliability 2026: https://bit.ly/4xd2FdG • At Scale Conferences: https://atscaleconference.com/ Our links More episodes + show notes + links: https://shipitweekly.fm On Call Brief: https://oncallbrief.com | |||
| Coinbase Outage, Meta AI Account Recovery, AWS AgentCore Code Injection, Apigee Tenant Isolation, and the Glue That Breaks Production | 12 Jun 2026 | 00:23:11 | |
This episode of Ship It Weekly is about the hidden glue holding production together. Brian covers Coinbase’s May 7 outage postmortem, where an AWS us-east-1 cooling failure exposed the difference between being “multi-AZ” on paper and actually being able to recover when stateful, low-latency systems are tied to a failed zone. Then he looks at Meta’s AI-assisted Instagram support issue and why account recovery is identity infrastructure, not just customer support. If AI can influence password resets, email changes, MFA resets, or account ownership flows, that workflow needs to be treated like a production control plane. The episode also covers AWS AgentCore CLI CVE-2026-11393, where collaborator metadata could break out into generated Python code during agent import, and an Apigee cross-tenant issue from Google’s Apigee security bulletins that shows why tenant isolation has to be tested beyond the obvious happy path. Links Coinbase May 7 outage postmortem https://www.coinbase.com/blog/a-postmortem-of-our-may-7-2026-outage Meta AI support / Instagram account recovery reporting https://www.theverge.com/tech/945658/meta-ai-support-chatbot-exploit-instagram-accounts AWS AgentCore CLI CVE-2026-11393 https://aws.amazon.com/security/security-bulletins/2026-040-aws/ AgentCore CLI GitHub advisory https://github.com/aws/agentcore-cli/security/advisories/GHSA-m4x6-gwgp-4pm7 Google Apigee security bulletins https://docs.cloud.google.com/apigee/docs/security-bulletins/security-bulletins Cloudflare real-time threat intel WAF rules https://blog.cloudflare.com/realtime-threat-intel-waf-rules/ AWS Lambda tenant isolation with event source mappings https://aws.amazon.com/blogs/compute/integrating-event-source-mappings-with-aws-lambda-tenant-isolation-mode/ Amazon OpenSearch Serverless next generation https://aws.amazon.com/about-aws/whats-new/2026/05/amazon-opensearch-serverless-next-generation-generally-available/ GitHub Enterprise Managed Users IP allow list coverage https://github.blog/changelog/2026-06-08-ip-allow-list-coverage-for-emu-namespaces-in-general-availability/ This week’s On Call Brief https://www.tellerstech.com/on-call-brief-news/2026-W24/ More episodes and show notes https://shipitweekly.fm/ | |||
| Kiro CLI Approval Bypass, Amazon Braket Pickle Risk, AWS Org Logging, KEDA Upgrades, and Automation’s Hidden Boundaries | 05 Jun 2026 | 00:20:27 | |
This episode of Ship It Weekly is about automation’s hidden boundaries. Brian covers Kiro CLI CVE-2026-9255, where piped stdin could act like user approval, Amazon Braket SDK CVE-2026-9291 and the very normal Python pickle risk hiding inside quantum job results, AWS Organizations finally emitting CloudTrail events when accounts join or leave an org, and KEDA updates that remind us autoscaling upgrades are production behavior changes. The bigger thread this week is that automation does not remove boundaries. It moves them. Approval paths, trusted data, account membership, scaling signals, platform access, and AI-generated output all need clear ownership and visibility. Brian also covers Kubernetes Dashboard being archived with Headlamp as the path forward, Google Cloud Remote MCP Server for AlloyDB, Apache Kafka 4.3.0, and Atlassian’s AI-native SDLC productivity claims. Sponsored by @Scale: Systems & Reliability, happening June 25 at the Meydenbauer Center in Bellevue, Washington. Register at https://bit.ly/4xd2FdG Links Kiro CLI CVE-2026-9255 https://aws.amazon.com/security/security-bulletins/2026-035-aws/ Amazon Braket SDK CVE-2026-9291 https://aws.amazon.com/security/security-bulletins/2026-036-aws/ AWS Organizations CloudTrail account events https://aws.amazon.com/about-aws/whats-new/2026/05/aws-organizations-cloudtrail/ KEDA v2.20.0 release https://github.com/kedacore/keda/releases/tag/v2.20.0 KEDA v2.19.0 release https://github.com/kedacore/keda/releases/tag/v2.19.0 Kubernetes Dashboard archived / Headlamp path forward https://kubernetes.io/blog/2026/06/04/dashboard-archived-what-now/ Google Cloud Remote MCP Server for AlloyDB https://cloud.google.com/blog/products/databases/alloydb-remote-mcp-server-now-ga Apache Kafka 4.3.0 https://www.confluent.io/blog/apache-kafka-4-3-release-announcement/ Atlassian AI-native SDLC productivity claims https://www.atlassian.com/blog/software-teams/ai-native-sdlc This week’s On Call Brief https://www.tellerstech.com/on-call-brief/2026-W23/ More episodes and show notes https://shipitweekly.fm/ | |||
| GitHub Supply Chain Attacks, Railway’s GCP Outage, Discord’s Voice Failure, AWS Retry Changes, and Trusted Tool Risk | 29 May 2026 | 00:23:47 | |
This episode of Ship It Weekly is about trusted tools becoming production dependencies. Brian covers a rough GitHub supply chain week, including the compromised Nx Console VS Code extension tied to exposed GitHub internal repositories and the Megalodon campaign abusing GitHub Actions workflows across thousands of public repos. The bigger thread this week is that the tools around production are increasingly part of production. Brian also covers Railway’s GCP account suspension outage, Discord’s voice outage during a Kubernetes migration, AWS changing SDK retry behavior, CVE-2026-9133 in the RabbitMQ AWS plugin, and a Reddit story about stolen AWS keys turning into a $14,000 Bedrock bill. Brian also touches on OpenTelemetry graduating from the CNCF, Claude Code security risk, GitLab Secrets Manager, Google Cloud AI spend caps, and a Redshift Python driver RCE. Full source list and extra links are available on this episode’s page at shipitweekly.fm. Links Nx Console compromise https://www.stepsecurity.io/blog/nx-console-vs-code-extension-compromised Megalodon GitHub Actions attack https://www.stepsecurity.io/blog/megalodon-mass-github-actions-secret-exfiltration-across-5-500-public-repositories Railway GCP outage https://blog.railway.com/p/incident-report-may-19-2026-gcp-account-outage Discord voice outage https://discord.com/blog/behind-the-scenes-of-the-3-25-26-voice-outage AWS SDK retry changes https://aws.amazon.com/blogs/developer/announcing-updated-retry-behavior-for-aws-sdks-and-tools/ RabbitMQ AWS plugin CVE-2026-9133 https://aws.amazon.com/security/security-bulletins/2026-034-aws/ AWS Bedrock cost spike Reddit thread https://www.reddit.com/r/aws/comments/1tm3ydo/aws_bedrock_cost_spike_14000_usd/ This week’s On Call Brief https://www.tellerstech.com/on-call-brief/2026-W22/ More episodes and show notes https://shipitweekly.fm/ | |||
| Ship It Conversations: Jake Warner on Cycle.io, Bare Metal’s Comeback, and Why Private Cloud Is Getting Interesting Again | 26 May 2026 | 00:36:06 | |
This is a guest conversation episode of Ship It Weekly, separate from the weekly news recaps. In this Ship It: Conversations episode, I talk with Jake Warner, founder and CEO of Cycle.io, about private cloud, bare metal, Kubernetes fatigue, and why some teams are rethinking how much infrastructure complexity they actually want to carry. We talk about why bare metal and private cloud are getting interesting again, especially around cost, performance, data sovereignty, compliance, and platform ownership. Jake explains how Cycle approaches infrastructure as a pool of resources, why he thinks in terms of “environments as code” instead of traditional infrastructure as code, and how teams can run containers and VMs together across bare metal, cloud, and hybrid environments. The bigger theme here is that this is not really a “cloud versus bare metal” conversation. It is about choosing the right level of abstraction. Sometimes Kubernetes is the right answer. Sometimes managed cloud services make sense. And sometimes teams just need a more opinionated platform that lets developers ship without requiring a large DevOps army to keep everything running. Highlights • Why some teams are moving back toward private cloud and bare metal • The role of cost, data sovereignty, compliance, and performance in infrastructure decisions • Why bare metal does not have to mean going back to old-school racking and stacking pain • How Cycle turns raw compute into a private cloud-style resource pool • Why Jake thinks about “environments as code” instead of only infrastructure as code • What “no DevOps army required” means in practice for engineering-heavy teams • Why some companies need VMs and containers running together on the same platform • Where Kubernetes still makes sense, especially for highly customized infrastructure needs • Why opinionated platforms can be valuable when teams want fewer knobs and better defaults • Active-active thinking, failover risk, and why application-level replication often matters more than platform-level storage magic • Why bandwidth, performance density, and predictable pricing can make bare metal attractive again • The weird continued gravity of AWS us-east-1, even for teams trying to move workloads elsewhere • How AI workloads, GPUs, and hype cycles fit into the private cloud and platform conversation • Jake’s advice for modernizing hybrid or on-prem infrastructure: containerize first, then look hard at your dependencies Jake’s links • Cycle Slack community: https://slack.cycle.io/ • Jake Warner on LinkedIn: https://www.linkedin.com/in/jakewarner/ Our links More episodes + show notes + links: https://shipitweekly.fm On Call Brief: https://oncallbrief.com | |||
| CISA’s GitHub Leak, AI Root Cause Analysis, Copilot Agents, Claude Code in CI/CD, and Kubernetes Seccomp Risk | 22 May 2026 | 00:22:23 | |
This episode of Ship It Weekly is about secrets, agents, risky defaults, and follow-up work that never gets done. Brian covers the CISA contractor GitHub leak involving AWS keys, internal docs, Terraform, Kubernetes, Argo CD, and CI/CD context, plus AWS DevOps Agent doing automated RCA across Datadog, Elasticsearch, CloudTrail, and EKS. Brian also covers MS Copilot Studio computer-using agents, Claude Code in Bitbucket Agentic Pipelines, CVE-2026-46333 and Kubernetes seccomp defaults, GitHub OIDC for Dependabot, Java pods getting OOMKilled, LLM-generated SQL that can be wrong but still run, and why postmortem action items die without ownership. Sponsored by Guardsquare https://hubs.ly/Q04fJgkJ0 Links CISA GitHub leak https://blog.gitguardian.com/how-we-got-a-cisa-github-leak-taken-down-in-26-hours/ AWS DevOps Agent RCA https://aws.amazon.com/blogs/devops/automate-root-cause-analysis-across-datadog-and-elasticsearch-with-aws-devops-agent/ Microsoft Copilot Studio computer-using agents https://techcommunity.microsoft.com/blog/copilot-studio-blog/computer-using-agents-in-microsoft-copilot-studio-are-now-generally-available/4519427 Atlassian Agentic Pipelines with Claude Code https://support.atlassian.com/bitbucket-cloud/docs/agentic-pipelines/ CVE-2026-46333 https://nvd.nist.gov/vuln/detail/CVE-2026-46333 Kubernetes seccomp https://kubernetes.io/docs/reference/node/seccomp/ GitHub OIDC for Dependabot and code scanning https://github.blog/changelog/2026-05-19-expanded-oidc-support-for-dependabot-and-code-scanning/ Java pods OOMKilled in Kubernetes https://dzone.com/articles/java-pod-oomkill-kubernetes LLM-generated SQL risks https://readyset.io/blog/why-llms-write-incorrect-sql-and-what-that-means-for-your-database Postmortem action items https://incident.io/blog/why-do-post-mortem-action-items-fail-how-to-make-incident-follow-ups-actually-get-done On Call Brief https://www.tellerstech.com/on-call-brief/2026-W21/ More episodes + show notes https://shipitweekly.fm/ | |||
| AI Agents Get API Access and Identity: GitHub Copilot Cloud Agents, MCP Auth, Ansible Automation, OpenAI Daybreak, and the New Production Risk | 14 May 2026 | 00:23:21 | |
This episode of Ship It Weekly is about AI agents moving from helpful coding assistants into real operational actors. Brian covers GitHub making Copilot cloud agent tasks available through a REST API, Auth0 bringing authentication and authorization to MCP servers, Red Hat positioning Ansible as a trusted execution layer for agentic IT operations, and OpenAI Daybreak pushing AI deeper into security research and remediation. The bigger thread this week is authority: what these agents can reach, what they can change, who approved the action, and who owns the outcome when something breaks. Brian also covers Discord’s ScyllaDB automation work, AWS GuardDuty crypto mining detection, queues and back pressure, and a Datadog PostgreSQL case where an index scan was still painfully slow. Sponsored by Guardsquare https://hubs.ly/Q04fJgkJ0 Links GitHub Copilot cloud agent tasks via REST API https://github.blog/changelog/2026-05-13-start-copilot-cloud-agent-tasks-via-the-rest-api/ GitHub REST API endpoints for agent tasks https://docs.github.com/en/rest/agent-tasks/agent-tasks Auth0 Auth for MCP is now generally available https://auth0.com/blog/auth0-auth-for-mcp-servers-generally-available/ Red Hat on Ansible as the execution layer for agentic IT https://www.redhat.com/en/about/press-releases/red-hat-establishes-ansible-automation-platform-trusted-execution-layer-it-operations-agentic-era OpenAI Daybreak https://openai.com/daybreak/ Discord automates ScyllaDB clusters at scale https://discord.com/blog/how-discord-automates-scylladb-clusters-at-scale AWS GuardDuty crypto mining detection and prevention https://aws.amazon.com/blogs/security/detecting-and-preventing-crypto-mining-in-your-aws-environment/ Queues do not absorb load, they delay failure https://dzone.com/articles/queues-dont-absorb-load-they-delay-bankruptcy Datadog on inefficient PostgreSQL index scans https://www.datadoghq.com/blog/detect-inefficient-index-scans-with-dbm/ This week’s On Call Brief https://www.tellerstech.com/on-call-brief/2026-W20/ More episodes and show notes https://shipitweekly.fm/ | |||
| Cursor Deletes PocketOS Prod DB, .de DNSSEC Outage, Bluesky Postmortem, Argo CD, and Copy Fail | 08 May 2026 | 00:21:57 | |
This episode of Ship It Weekly is about modern reliability getting squeezed from both directions. Old-school failures still hit hard, like broken DNSSEC, kernel privilege escalation bugs, and GitOps behavior changes. But newer automation layers add a second kind of risk, where AI agents, machine identity, and cloud control planes can do real damage fast when authority is too broad. Brian covers the Cursor and PocketOS production database wipe, the .de DNSSEC outage and Cloudflare’s response, Bluesky’s April outage postmortem, Argo CD v3.1.16 reaching end of life plus the v3.4.1 behavior change, Linux kernel CVE-2026-31431 under active exploitation, and why Google Cloud Agent Identity and AWS MCP Server GA both point to agents becoming first-class infrastructure actors. Sponsored by Guardsquare https://hubs.ly/Q04fJgkJ0 Links Cursor / PocketOS production database wipe https://www.tellerstech.com/on-call-brief/2026-W19/ Cloudflare on the .de DNSSEC outage https://blog.cloudflare.com/de-tld-outage-dnssec/ Bluesky April 2026 outage postmortem https://pckt.blog/b/jcalabro/april-2026-outage-post-mortem-219ebg2 Argo CD releases: v3.1.16 final release and v3.4.1 behavior change https://github.com/argoproj/argo-cd/releases Linux kernel CVE-2026-31431 https://nvd.nist.gov/vuln/detail/CVE-2026-31431 AWS bulletin for CVE-2026-31431 https://aws.amazon.com/security/security-bulletins/rss/2026-026-aws/ Google Cloud Agent Identity https://cloud.google.com/blog/products/identity-security/whats-new-in-iam-security-governance-and-runtime-defense AWS MCP Server is now generally available https://aws.amazon.com/blogs/aws/the-aws-mcp-server-is-now-generally-available/ Cross-region disaster recovery for Amazon EKS using AWS Backup https://aws.amazon.com/blogs/containers/cross-region-disaster-recovery-for-amazon-eks-using-aws-backup/ Google Ads new data retention policy starting June 1, 2026 https://ads-developers.googleblog.com/2026/05/new-data-retention-policy-for-google.html This week’s On Call Brief https://www.tellerstech.com/on-call-brief/2026-W19/ More episodes and show notes https://shipitweekly.fm/ | |||
| Ship It Conversations: Gareth Kersey on IaCConf 2026, AI, and Corey Quinn’s Terraform Keynote | 05 May 2026 | 00:31:54 | |
This is a guest conversation episode of Ship It Weekly, separate from the weekly news recaps. This episode is not sponsored. I wanted to cover IaCConf because the theme lines up closely with what Ship It Weekly focuses on: infrastructure, platform engineering, DevOps, SRE, and how teams are adapting to AI-driven change. In this Ship It: Conversations episode, I talk with Gareth Kersey about IaCConf 2026, a free virtual conference focused on infrastructure as code, platform engineering, DevOps, SRE, and infrastructure operations. The conference is May 14th 2026. The main theme is “keeping pace.” Not just keeping pace with new tools, but keeping pace with the speed of software delivery now that AI is changing how quickly application teams can write, ship, and change code. We talk about what that means for the infrastructure teams underneath it all: the people responsible for Terraform, Kubernetes, GitOps, policies, secrets, cost, security, rollback paths, and making sure faster delivery does not turn into faster chaos. Gareth walks through the IaCConf 2026 agenda, including Corey Quinn’s keynote, AI and Terraform sessions, platform engineering panels, Kubernetes and Argo CD talks, AI agents managing infrastructure as code, governance challenges, and the risk of 10x code velocity becoming 10x operational risk. The bigger theme here is that AI is not just changing how code gets written. It is changing the pressure on the systems around delivery. Infrastructure as code, platform engineering, policy, and operational guardrails matter even more when the pace of change goes up. Highlights • What “keeping pace” means for infrastructure, DevOps, SRE, and platform teams • Why faster application development can create more downstream operational pressure • Corey Quinn’s keynote, “AI Speaks Terraform Like a Tourist” • How AI-generated infrastructure changes create new governance and review challenges • Why infrastructure as code still matters as AI agents and automation become more common • Sessions covering Terraform, Kubernetes, Argo CD, GitOps, platform engineering, and AI-driven workflows • The risk of 10x code velocity turning into 10x operational risk • How platform teams can support faster developers without giving up safety or governance • Why IaCConf includes panels, demos, technical talks, and practitioner stories instead of only tool-specific content • How IaCConf has grown from its first event in 2025 into a broader infrastructure community • Why the event is trying to stay community-focused instead of becoming just another vendor marketing conference • The role of feedback, future spotlight events, in-person meetups, and possible community spaces around IaCConf • Why registering still makes sense even if you cannot attend live, since sessions are available afterward IaCConf links • IaCConf 2026 registration page - https://www.iacconf.com/iacconf-2026 • IaCConf LinkedIn page - https://www.linkedin.com/showcase/iac-conf/ • IaCConf: https://www.iacconf.com/ • IaCConf is supported by Spacelift: https://spacelift.com Our links More episodes + show notes + links: https://shipitweekly.fm On Call Brief: https://oncallbrief.com | |||
| GitHub RCE, AI Agent Prompt Injection, and the New Reality: Your Developer Toolchain Is Production Now | 01 May 2026 | 00:25:08 | |
This episode of Ship It Weekly is about the developer toolchain becoming part of production. Brian covers GitHub’s critical git push RCE, AI-assisted reverse engineering, prompt injection against AI agents in GitHub workflows, Elementary’s malicious CLI release, GitHub’s merge queue regression, Cal.com going closed source, and Copilot moving toward usage-based billing. Plus: MinIO’s repo archive, Ghostty leaving GitHub, Docker Hardened Images, and Azure DevOps security updates. Links GitHub git push RCE https://github.blog/security/securing-the-git-push-pipeline-responding-to-a-critical-remote-code-execution-vulnerability/ AI-assisted reverse engineering https://www.darkreading.com/application-security/reverse-engineering-ai-unearths-high-severity-github-bug AI agents + GitHub Actions prompt injection https://www.theregister.com/2026/04/15/claude_gemini_copilot_agents_hijacked/ Elementary malicious CLI release https://www.elementary-data.com/post/security-incident-report-malicious-release-of-elementary-oss-python-cli-v0-23-3 GitHub merge queue regression https://github.blog/news-insights/company-news/an-update-on-github-availability/ Cal.com going closed source https://cal.com/blog/cal-com-goes-closed-source-why GitHub Copilot billing https://github.blog/news-insights/company-news/github-copilot-is-moving-to-usage-based-billing/ MinIO archived repo https://github.com/minio/minio Ghostty leaving GitHub https://mitchellh.com/writing/ghostty-leaving-github Docker Hardened Images https://www.docker.com/blog/why-we-chose-the-harder-path-docker-hardened-images-one-year-later/ Azure DevOps security updates https://devblogs.microsoft.com/devops/one-click-security-scanning-and-org-wide-alert-triage-come-to-advanced-security/ On Call Brief https://oncallbrief.com/ More episodes https://shipitweekly.fm/ | |||
| Kubernetes 1.36, Gateway API v1.5, AWS Copilot End of Support, and Cloudflare Non-Human Identities | 24 Apr 2026 | 00:20:24 | |
This episode of Ship It Weekly is about platforms getting sharper about defaults, ownership, and the old paths they are no longer willing to quietly carry forever. Brian covers Kubernetes 1.36 and why it feels more like a cleanup-and-maturity release than a flashy feature dump, Gateway API v1.5 moving more networking behavior into the stable path, AWS Copilot CLI reaching end of support and what that means for teams still sitting on the older “easy” ECS workflow, Airbnb’s alert-development overhaul and why noisy or weak alerts are often a workflow problem long before they become an on-call problem, and Cloudflare’s push to treat scripts, agents, and third-party tools like real identities with real blast radius. He also hits the latest Azure DevOps Server patches and Google’s OTLP metrics support for Cloud Monitoring. Links Kubernetes v1.36 release https://kubernetes.io/blog/2026/04/22/kubernetes-v1-36-release/ Gateway API v1.5 https://kubernetes.io/blog/2026/04/21/gateway-api-v1-5/ AWS Copilot CLI end of support https://aws.amazon.com/blogs/containers/announcing-the-end-of-support-for-the-aws-copilot-cli/ Airbnb on alert development https://medium.com/airbnb-engineering/it-wasnt-a-culture-problem-upleveling-alert-development-at-airbnb-01e2290eb0f5 Cloudflare on non-human identities, OAuth visibility, and scoped permissions https://blog.cloudflare.com/improved-developer-security/ Azure DevOps Server April patches https://devblogs.microsoft.com/devops/april-patches-for-azure-devops-server/ OTLP metrics for Google Cloud Monitoring https://cloud.google.com/blog/products/management-tools/otlp-opentelemetry-protocol-for-google-cloud-monitoring-metrics Past episode where we talked about Cloudflare Mesh https://www.tellerstech.com/ship-it-weekly/aws-interconnect-ga-cloudflare-mesh-gitlab-19-eks-auto-mode-and-opentelemetry-config/ This week’s On Call Brief https://www.tellerstech.com/on-call-brief/2026-W16/ On Call Brief: https://oncallbrief.com/ More episodes and show notes https://shipitweekly.fm/ | |||