CoreDNS Resolution Failure — When Kubernetes Services Became Unreachable by Name Common Search Queries If you’re landing here from a search, this post covers: coredns resolution failure kubernetes kubernetes dns intermittent failure some pods work others don’t coredns resource limits not set throttled ndots:5 causing excessive dns queries search domain expansion coredns auto-scaling replicas insufficient for large cluster kubectl check coredns dns resolution troubleshooting The Incident Environment: K8S v1.28, Calico CNI, CoreDNS 1.10.1, 50 worker nodes, 500+ services, 2000+ pods. ...

MySQL Replication Lag — How a Long-Running Query on the Replica Broke Read-After-Write Consistency Common Search Queries English Chinese MySQL replication lag troubleshooting MySQL 主从延迟排查 Seconds_Behind_Master keeps increasing Seconds_Behind_Master 持续增长 Read-after-write consistency MySQL MySQL 读写一致性 MySQL slave lag cause MySQL 从库延迟原因 MySQL parallel replication setup MySQL 并行复制配置 pt-online-schema-change replication lag pt-osc 主从延迟 MySQL relay log space growing MySQL relay log 暴涨 Kill long running query on MySQL replica 杀掉 MySQL 从库慢查询 The Incident Environment Component Specification MySQL Version 8.0.32 (Community) Replication Mode Asynchronous (Async) Replication Format Row-Based (RBR) Topology 1 Master + 1 Replica App A Writes to Master App B Reads from Replica App C Analytics queries on Replica Table Size ~10 million rows Storage Engine InnoDB Timeline Time (UTC+8) Event 14:00 Peak business hours start 14:15 DBA runs ALTER TABLE orders ADD COLUMN on Master 14:16 Master write latency spikes; DDL blocks subsequent writes via MDL 14:18 Replica CPU reaches 100% 14:20 Customer support receives first reports of missing orders 14:22 On-call engineer is paged 14:25 SHOW REPLICA STATUS reveals Seconds_Behind_Master > 1800s 14:28 Root cause identified: analytical query on replica blocking SQL thread 14:30 Analytical query killed on replica 14:32 Replica starts catching up 14:45 Replication lag returns to 0; service fully recovered Symptoms Users reported that orders submitted successfully (via App A writing to Master) were invisible upon page refresh (App B reading from Replica). Replica server CPU was pegged at 100%. Replica disk IO wait was elevated. Application error logs showed no database connection errors — queries returned successfully but with stale data. The replica’s Relay_Log_Space had ballooned to several GB. Background MySQL Asynchronous Replication In MySQL’s default asynchronous replication, the Master writes events to its binary log (binlog), and the Replica pulls those events via the I/O thread into its relay log (relay-log). The SQL thread on the Replica then replays those events sequentially. ...

Prometheus OOM — When Your Monitoring System Became the Incident Common Search Queries Search Query Intent prometheus oom User’s Prometheus is crashing due to memory pressure prometheus high memory usage General troubleshooting for memory leaks or bloat prometheus tsdb memory Investigating TSDB memory consumption specifically prometheus label cardinality problem User suspects high-cardinality labels prometheus recording rules optimization Looking to reduce the cost of recording rules prometheus oom killed kubernetes Prometheus OOMing in a K8s container prometheus tsdb analyze cardinality Using promtool to diagnose series explosion prometheus wal replay memory WAL replay consuming too much memory after restart prometheus heap profile User wants to profile Prometheus memory prometheus /api/v1/status/tsdb Checking label series counts via the HTTP API The Incident Environment: ...

Common Search Queries PostgreSQL connection pool exhausted PostgreSQL max_connections reached PostgreSQL performance suddenly slow PostgreSQL pg_stat_activity all active How to find slow queries in PostgreSQL PostgreSQL missing index causing full table scan pg_stat_statements query performance analysis PostgreSQL connection storm thundering herd PostgreSQL kill long running queries PostgreSQL create index concurrently The Incident Environment: Component Spec PostgreSQL Version 15.4 RAM 16 GB vCPU 4 cores Storage 500 GB SSD max_connections 200 OS Ubuntu 22.04 LTS It was a typical Tuesday morning. At 10:00 AM sharp — the daily peak hour — the monitoring system began firing alerts in rapid succession. ...

Common Search Queries elasticsearch cluster red status fix es shard allocation failed elasticsearch disk watermark low high flood_stage elasticsearch unassigned shards troubleshooting es cluster red after node disk full elasticsearch replica shards not assigned why is my elasticsearch cluster red elk cluster health red fix The Incident Environment Item Detail Elasticsearch 7.17 (production cluster) Nodes 3 Indices 10 Primary Shards 50 Replica Shards 50 Total Shards 100 Role Full-text search for a production SaaS Timeline 14:23 (UTC+8) — The on-call engineer received a P0 alert: the product search API began returning HTTP 500 errors. Users reported that search results were incomplete or missing entirely in the web application. ...

Kafka Consumer Group Rebalance — How Frequent Rebalances Broke Our Real-Time Pipeline Common Search Queries If you arrived here searching for any of the following, you are in the right place: “kafka consumer group rebalance keeps happening” “kafka rebalance timeout too frequent” “kafka consumer lag spike rebalance” “kafka session.timeout.ms too low rebalance” “kafka max.poll.interval.ms exceeded rebalance” “kafka stop the world rebalance” “kafka consumer group not stable” “kafka rebalance rate high” “kafka consumer keeps leaving and joining group” The Incident Environment: Kafka 3.4, 12 consumers in a single consumer group, 24 partitions, real-time analytics pipeline processing user click events. Consumer application deployed on Kubernetes with 4 vCPUs and 8 GB RAM per pod, running Java 17 with Avro deserialization and enrichment logic. ...

Cloud Credential Leak — When Your AWS Key in a Public Repo Cost $50,000 Overnight Common Search Queries aws access key exposed in github public repository cloud credential leak incident response aws key compromised by automated scanner kubernetes service account token leaked secret scanning pre-commit hook prevention The Incident Environment: Kubernetes cluster running on AWS (EKS), 3 production clusters across us-east-1 and eu-west-1, 150+ microservices, Terraform-managed infrastructure in a public GitHub monorepo. ...

Audit Log Missing — When Your Kubernetes Cluster Was Breached and You Have No Forensics Common Search Queries kubernetes audit log not working k8s audit policy misconfiguration kubernetes forensics no audit logs audit log level metadata vs requestresponse kubernetes audit log shipping fluentd kube-apiserver audit logging best practices kubernetes incident response no logs The Incident Environment: K8S v1.28, kubeadm deployed, 3 control plane nodes (HA), 50 worker nodes, on-premise datacenter. Audit log storage configured on local control plane disk with no external shipping. ...

WAF Rule Bypass — When Your Web Application Firewall Missed the Obvious Common Search Queries WAF bypass techniques modsecurity OWASP CRS SQL injection bypass HTTP parameter pollution WAF evasion chunked transfer encoding WAF bypass modsecurity rule evasion real world example The Incident Environment: Kubernetes v1.28, Nginx Ingress Controller with ModSecurity enabled and OWASP CRS v3.3.5, MariaDB 10.11 on a dedicated node, single-region production cluster. Time: Tuesday 03:14. PagerDuty alerted on a sudden spike in database CPU usage (from 15% to 97% in under 2 minutes) and an unusually high number of error-level SQL queries in the application logs. ...

etcd Unencrypted Data Exposure — How a Backup File Leaked the Entire Cluster’s Secrets Common Search Queries etcd encryption at rest kubernetes etcd backup exposed secrets kubernetes etcd snapshot security k8s secret encryption etcd etcd data breach incident The Incident Environment: K8S v1.27, 3-node etcd cluster, single-region production, backup stored on a shared NAS. Time: Wednesday 14:00. Security team received an alert from an external security researcher: an etcd snapshot file was accessible on a publicly exposed NAS share. ...