unkin/argocd-apps
3
0
Fork 0
Code Issues 7 Pull Requests 4 Actions Packages Projects Releases Wiki Activity

Add job and start deadlines to the g10k-code CronJob #490

Merged
benvin merged 1 commits from benvin/g10k-hung-job-deadline into main 2026-09-26 20:55:33 +10:00
Conversation 0 Commits 1 Files Changed 1
+2
unkin-agent commented 2026-09-26 20:42:37 +10:00
Member
Copy Link
Copy Source

A g10k-code job wedged in ContainerCreating on a failed CephFS mount and never reached a terminal condition, so it stayed in the CronJob active list and concurrencyPolicy: Forbid skipped every following minute. No Puppet code reached the estate for 6 days, and the piled-up missed slots crossed the controller 100-slot cap into TooManyMissedTimes. The CronJob carried no deadlines at all.

  • Cap a job at activeDeadlineSeconds: 300 on the Job spec, so a hang is failed as DeadlineExceeded and drops out of the active list (healthy runs take 16-18s).
  • Set startingDeadlineSeconds: 200, bounding missed-schedule look-back to ~3 slots so the count cannot reach 100.
A g10k-code job wedged in ContainerCreating on a failed CephFS mount and never reached a terminal condition, so it stayed in the CronJob active list and `concurrencyPolicy: Forbid` skipped every following minute. No Puppet code reached the estate for 6 days, and the piled-up missed slots crossed the controller 100-slot cap into `TooManyMissedTimes`. The CronJob carried no deadlines at all. - Cap a job at `activeDeadlineSeconds: 300` on the Job spec, so a hang is failed as `DeadlineExceeded` and drops out of the active list (healthy runs take 16-18s). - Set `startingDeadlineSeconds: 200`, bounding missed-schedule look-back to ~3 slots so the count cannot reach 100.
unkin-agent added 1 commit 2026-09-26 20:42:38 +10:00
Add job and start deadlines to the g10k-code CronJob
ci/woodpecker/pr/vector-test Pipeline was successful
Details
ci/woodpecker/pr/pre-commit Pipeline was successful
Details
ci/woodpecker/pr/kubeconform Pipeline was successful
Details
b2df282ad9
benvin merged commit d48125d699 into main 2026-09-26 20:55:33 +10:00
benvin deleted branch benvin/g10k-hung-job-deadline 2026-09-26 20:55:33 +10:00
Sign in to join this conversation.
Reviewers
No Reviewers
Labels
No items
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
benvin (BenVincent) unkin-agent (Unkin Agent) unkinben (Ben Vincent)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: unkin/argocd-apps#490
Delete Branch "benvin/g10k-hung-job-deadline"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?