Summary
From September 24 to October 5, the nightly delivery of compiled HOS logs to Walmart did not complete, and your feed received little or no data on most of those days. This week we resent the affected days, resent the WK29–WK33 history you asked for, and changed our processing setup so a delivery that is in progress can no longer be stopped partway through.
Nightly delivery has completed on its own every night since October 6. We need your help to confirm the updated numbers match what your systems received, and we'd like to agree on alerting so both teams hear about a missed delivery the same morning.
What we did this week
All times Pacific.
| When | Action | Records sent |
|---|---|---|
| Mon Oct 5 | Started resending missed days in one-day batches, after the support case was opened. | — |
| Tue Oct 6, 2:30 AM | Nightly delivery completed on its own for the first time since September 23. | 22,270 |
| Tue Oct 6, 5 AM – 1:30 PM | After you reported data still missing, resent every log day from September 28 through October 6, including later edits to those logs. | 10 daily batches |
| Tue Oct 6 | Increased the minimum number of always-on processing workers for the HOS log delivery (details below). Applied immediately and made permanent in our deployment configuration the same day. | — |
| Wed Oct 7, 2:30 AM | Nightly delivery completed. | 23,931 |
| Thu Oct 8, 2:30 AM | Nightly delivery completed. | 37,370 |
| Thu Oct 8 – Fri Oct 9 | Rebroadcast WK29–WK33 (log days August 15 – September 18) at your request, one log day at a time. We also resent September 19–22. | 620,323 + 71,673 for Sep 19–22 |
| Fri Oct 9, 2:30 AM | Nightly delivery completed. | 36,220 |
The October 6 and 7 nightly deliveries were smaller because the resends that week had already covered part of their data.
Minimum worker change
The nightly delivery runs as one long job on a pool of processing workers that scales up and down automatically. Before the change, that pool could scale down to zero workers about an hour after a job started, because a job already in progress didn't count as work waiting to be done. On September 24 the delivery ran just past that point and was stopped before it finished. Each night after that had more data to cover, and the deliveries kept failing.
Minimum workers for each of the five HOS log delivery queues. An idle-looking queue could be scaled to zero while a delivery was still running.
At least one worker always stays running on each of the five queues, so an in-progress delivery is no longer shut down by automatic scale-down. Extra workers are still added during busy periods, and the pool returns to one, not zero.
Since the change, the workers have scaled up and back down each night without dropping to zero, and every nightly delivery has completed. This change prevents the specific failure that started the outage. The longer-term fixes to the delivery itself are listed under next steps.
Discussion points
Validate updated HOS log numbers
- Review the updated HOS log numbers after the WK29–WK33 data rebroadcast
- Confirm whether the data gaps have been resolved
Our response
Waiting on Walmart's comparisonThese are the records we sent in the rebroadcast, by Walmart week (Saturday to Friday). Each record is one driver's compiled HOS log for one day, in its latest version.
| Week | Log days | Records sent |
|---|---|---|
| WK29 | Aug 15 – Aug 21 | 123,210 |
| WK30 | Aug 22 – Aug 28 | 123,611 |
| WK31 | Aug 29 – Sep 4 | 123,971 |
| WK32 | Sep 5 – Sep 11 | 124,495 |
| WK33 | Sep 12 – Sep 18 | 125,036 |
| Total | 35 log days | 620,323 |
WK29
- Sat 8/1517,579
- Sun 8/1617,580
- Mon 8/1717,593
- Tue 8/1817,608
- Wed 8/1917,613
- Thu 8/2017,615
- Fri 8/2117,622
WK30
- Sat 8/2217,621
- Sun 8/2317,621
- Mon 8/2417,648
- Tue 8/2517,658
- Wed 8/2617,686
- Thu 8/2717,687
- Fri 8/2817,690
WK31
- Sat 8/2917,689
- Sun 8/3017,689
- Mon 8/3117,708
- Tue 9/117,713
- Wed 9/217,713
- Thu 9/317,726
- Fri 9/417,733
WK32
- Sat 9/517,735
- Sun 9/617,733
- Mon 9/717,748
- Tue 9/817,795
- Wed 9/917,819
- Thu 9/1017,829
- Fri 9/1117,836
WK33
- Sat 9/1217,836
- Sun 9/1317,837
- Mon 9/1417,848
- Tue 9/1517,868
- Wed 9/1617,871
- Thu 9/1717,879
- Fri 9/1817,897
Have the gaps been resolved?
- On our side, every log day from August 15 through October 8 has now been sent at least once. August 15 – September 22 came from the rebroadcast, September 23–27 from a resend on September 28, September 28 – October 6 from this week's resends, and October 7 onward from the nightly deliveries, which have completed every night since October 6.
- September 23–27 were resent on September 28, so edits drivers made to those days after September 28 may not be included. We plan to resend those five days with the same method as the rebroadcast to close that gap.
- The rebroadcast had no errors. All 39 daily batches completed, and the daily counts rise steadily from 17,579 to 17,897 with no sudden drops.
- Your side needs to confirm. We can confirm what we sent, not what your systems received and kept. Please compare the counts above with your counts per log day.
When comparing, please keep in mind
- Duplicates are expected. Some days were sent more than once during recovery. Please count unique driver + log day pairs and keep the most recent version of each.
- Active drivers only. The rebroadcast includes drivers who are active today. Drivers deactivated since August aren't included, so a small shortfall on older weeks may be expected if drivers have left.
- A small number of individual driver-days had log processing that didn't finish when the logs were edited, so the version sent may not include the latest edit. We're identifying and reprocessing these, and they'll be included in the nightly delivery automatically.
- If any day still looks short, please send two or three examples (driver ID and log date). We'll trace each one to find out whether it wasn't sent, wasn't received, or is one of the cases above.
Updated alerting: timeline and next steps
- Review the timeline for updated alerts from Platform Science and Walmart
- Align on alert criteria, ownership, and next steps
Our response
Proposed · to agree in this meetingThis time, the first failed night produced no error message, so our existing alert didn't fire until days later. The new alerts below watch for a delivery that didn't happen, not only for errors.
| Alert | Fires when | Owner | Timeline |
|---|---|---|---|
| Delivery not completed | The nightly delivery hasn't completed within 26 hours of the previous successful one. This would have fired on the first morning of this outage. | platformscience-oncall | Proposed first; date to confirm |
| Delivery running long | A nightly delivery is still running after 45 minutes, an early warning before it nears its limits. | platformscience-oncall | Proposed first; date to confirm |
| Unusual volume | A night's record count falls well below the recent daily average. | Platform Science SRE | To follow; date to confirm |
| Log processing failures | Individual driver logs fail to process after an edit, the cause of the small set of stale driver-days noted above. | Platform Science HOS engineering | To follow; date to confirm |
| Feed not received (suggested for Walmart) | Your feed hasn't received the nightly delivery by an agreed local time, for example 6:00 AM Central. | Walmart | For Walmart to decide |
To agree in this meeting
- Criteria: the 26-hour and 45-minute thresholds, what counts as "unusual volume", and the time by which your team expects the delivery.
- Ownership and contacts: who at Walmart we notify when one of our alerts fires, and who at Platform Science you contact when yours does.
- Notification path and timing: for example, notifying your named contact the same business morning with an estimated recovery time.
- Target dates for the alerts above.
Next steps
- Walmart: compare WK29–WK33 and recent days against your records, and send any driver/day examples that still look missing.
- Platform Science: resend log days September 23–27 with the same method as the WK29–WK33 rebroadcast, so later edits to those days are included.
- Platform Science: trace any examples you send and resend anything confirmed missing.
- Platform Science: reprocess the small set of driver-days whose log processing didn't finish. They'll go out in the nightly delivery automatically.
- Both teams: agree on alert criteria, ownership and target dates (Item 2). We'll confirm dates in writing after this meeting.
- Platform Science: make the nightly delivery more robust. Each run will cover a bounded amount of data, save progress as it goes, and read data more efficiently, so a single missed night can't snowball again. We'll share timing once it's scheduled.
- Platform Science: a root cause analysis document will follow once the longer-term fixes are in place.