Tag: Case Study

  • Case Study: The 13-Hour Delete — A Real SQL Server Performance Diagnosis Walkthrough

    Case Study: The 13-Hour Delete — A Real SQL Server Performance Diagnosis Walkthrough

    The problem: a nightly cleanup job deletes old records from a large partitioned table. It used to take 20 minutes. This week it took 13 hours, and blocked other processes the entire time. Let’s diagnose it exactly the way you would in production — using the Evidence-First workflow from the start of this course.

    The 13-Hour Delete, sketched out(evidence-first, applied to a real incident)1. Baselinejob normally takes20 minutes2. CaptureEvidenceXE: PAGEIOLATCH waits3. DiscoveryFK column hasNO index4. The FixONE nonclusteredindex, ONLINE=ON5. Validate22 minutes,Index Seek nowPrimary Keyautomatically indexedby SQL ServervsForeign KeyNOT automatically indexed— easy to overlook400M rows + full scan on the FK = 13 hours blocked

    Step 1 — Baseline

    -- What did "normal" look like? Check historical job duration logs first.
    -- Then measure the current run's I/O and duration directly:
    SET STATISTICS IO ON;
    SET STATISTICS TIME ON;
    -- (run a scoped-down version of the delete against a copy/test environment)

    Without a baseline, “13 hours” is just a scary number — you need “20 minutes normally” to even know how far off this is, and to confirm later that a fix actually worked.

    Step 2 — Capture Evidence

    CREATE EVENT SESSION DeleteDiagnosis ON SERVER
    ADD EVENT sqlserver.sql_statement_completed (ACTION (sqlserver.sql_text))
    ADD EVENT sqlos.wait_info (WHERE wait_type LIKE 'PAGEIOLATCH%' OR wait_type LIKE 'LCK%')
    ADD TARGET package0.event_file (SET filename = N'DeleteDiagnosis');
    GO
    ALTER EVENT SESSION DeleteDiagnosis ON SERVER STATE = START;
    -- Let the job run, then inspect captured wait types

    The captured evidence shows overwhelming PAGEIOLATCH_SH waits — the query is spending almost all its time waiting to read data pages from disk, not on CPU or locks.

    Step 3 — Discovery

    -- Capture the actual execution plan for the DELETE's WHERE clause
    SELECT * FROM dbo.LargeAuditTable
    WHERE customer_id IN (SELECT customer_id FROM dbo.DeactivatedCustomer);
    -- Plan shows: Clustered Index Scan on LargeAuditTable — no usable index on customer_id (a foreign key with no supporting index)

    The root cause: customer_id is a foreign key on a 400-million-row table with no index. Every batch of the delete performs a full clustered index scan to find matching rows — exactly the missing-index-on-a-foreign-key pattern that’s easy to overlook because foreign keys don’t automatically get indexed in SQL Server (unlike primary keys).

    Step 4 — The Fix

    CREATE NONCLUSTERED INDEX IX_LargeAuditTable_CustomerId
    ON dbo.LargeAuditTable (customer_id)
    WITH (ONLINE = ON, MAXDOP = 4);  -- ONLINE to avoid blocking production during creation

    One targeted index — not a rewrite of the whole delete process, not a hardware upgrade. This is the discipline from the Evidence-First loop: change one variable at a time.

    Step 5 — Validation

    -- Re-run the same scoped test, compare against baseline
    SET STATISTICS IO ON;
    -- Expect: Index Seek instead of Clustered Index Scan, dramatically fewer logical reads

    The rerun completes in 22 minutes — back in line with the historical baseline, with the execution plan now showing an Index Seek instead of a full scan.

    The Lesson, Beyond This One Incident

    Notice how this walkthrough used every earlier lesson in this module: the missing foreign-key index is what Ch.109’s evidence-based indexing lesson calls a hypothesis to verify; the clustered index scan is exactly the SARGability/plan-reading pattern from this module’s first lesson; and the whole five-step shape is the Evidence-First workflow from the very start of this course, applied for real.

    Foreign keys don’t get an index automatically in SQL Server — unlike the primary key side of the relationship. Any DELETE, UPDATE, or JOIN filtering on a foreign key column without a supporting index is a latent “13-hour delete” waiting for the table to grow large enough to matter.


    Enjoyed this?

    Subscribe to get every new lesson as soon as it’s published, and share it with a developer who’d find it useful.

    📡 Subscribe via RSS  | 
    Share on X  | 
    Share on LinkedIn  | 
    Share on Facebook

    Want the full structured course with quizzes and hands-on labs? Check out SQL Server Performance Tuning, coming soon on this site.

  • SQL Server Incident Report Template: How to Document a Performance Post-Mortem

    SQL Server Incident Report Template: How to Document a Performance Post-Mortem

    A fix without a written record teaches nothing to the next person who hits a similar symptom. This template mirrors the Evidence-First workflow directly, so writing the report is almost free once you’ve actually done the diagnosis.

    A post-mortem is just the workflow, written down(the report is free once you’ve done the work)BASELINEstep 2EVIDENCEstep 4ROOT CAUSEstep 5FIXstep 6VALIDATEstep 71. Summary — plain language, written for someone who wasn’t there3. Timeline — timestamped, start to finish8. Prevention — would monitoring (Module 6) have caught this sooner?A fix with no written record teaches nothing to thenext person who hits a similar symptom. 📌

    1. Summary

    One paragraph: what broke, for how long, and who/what was affected. Written for someone who wasn’t in the room.

    2. Baseline

    What did normal look like, with numbers? (duration, CPU, I/O, or whatever metric defines “working” for this system)

    3. Timeline

    Timestamped sequence: when was it first noticed, when did diagnosis start, when was the fix applied, when was it confirmed resolved.

    4. Evidence Captured

    Exactly what was captured and how (Extended Events session definition, DMV queries run, execution plan attached). Paste the actual queries — future-you will thank present-you.

    5. Root Cause

    The specific, technical cause — not “the database was slow,” but “missing nonclustered index on OrderLog.customer_id causing a clustered index scan on a 400M-row table.”

    6. Fix Applied

    The exact change made, plus why this specific fix (not a bigger rewrite, not a hardware upgrade) was the right scope.

    7. Validation

    Post-fix measurement against the Step 2 baseline, with numbers.

    8. Prevention

    Would monitoring (Module 6) have caught this earlier? Is this a pattern worth a standing alert or a schema review checklist item?

    Why the Structure Matters

    Following this template forces you to separate what happened from what you did about it from whether it actually worked — exactly the discipline the Evidence-First workflow teaches, just written down for the next person instead of held in your head.


    Enjoyed this?

    Subscribe to get every new lesson as soon as it’s published, and share it with a developer who’d find it useful.

    📡 Subscribe via RSS  | 
    Share on X  | 
    Share on LinkedIn  | 
    Share on Facebook

    Want the full structured course with quizzes and hands-on labs? Check out SQL Server Performance Tuning, coming soon on this site.