[00:04:04] Fundraising Dash, Fundraising-Backlog, Cleanup, MediaWiki-extensions-Other: Archive DjangoAnalytics extension - https://phabricator.wikimedia.org/T199850 (Krinkle) [00:04:27] Fundraising Dash, Fundraising-Backlog, Cleanup, MediaWiki-extensions-Other: Archive DjangoAnalytics extension - https://phabricator.wikimedia.org/T199850 (Krinkle) Open→Resolved a:Umherirrender [00:09:55] So the duplicate recurrings seems like the job somehow ran twice at once - digging more still [00:24:48] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Ingenico recurring failing due to lack of unique merchant id - https://phabricator.wikimedia.org/T212734 (Eileenmcnaughton) [01:02:58] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Ingenico recurring failing due to lack of unique merchant id - https://phabricator.wikimedia.org/T212734 (Eileenmcnaughton) Here is a list of failed from 29/11 ` 2018-12-29 00:23:05,750 INFO [failed] => Array 2018-12-29 00:23:05,750 INFO... [03:41:15] PROBLEM - check_disk on frbackup2001 is CRITICAL: DISK CRITICAL - free space: /dev 10 MB (100% inode=99%): /run 2872 MB (90% inode=99%): / 51231 MB (96% inode=98%): /dev/shm 7962 MB (100% inode=99%): /run/lock 5 MB (100% inode=99%): /sys/fs/cgroup 7962 MB (100% inode=99%): /boot 188 MB (75% inode=99%): /srv 839997 MB (5% inode=99%) [03:46:15] PROBLEM - check_disk on frbackup2001 is CRITICAL: DISK CRITICAL - free space: /dev 10 MB (100% inode=99%): /run 2872 MB (90% inode=99%): / 51231 MB (96% inode=98%): /dev/shm 7962 MB (100% inode=99%): /run/lock 5 MB (100% inode=99%): /sys/fs/cgroup 7962 MB (100% inode=99%): /boot 188 MB (75% inode=99%): /srv 819434 MB (5% inode=99%) [03:51:15] PROBLEM - check_disk on frbackup2001 is CRITICAL: DISK CRITICAL - free space: /dev 10 MB (100% inode=99%): /run 2872 MB (90% inode=99%): / 51231 MB (96% inode=98%): /dev/shm 7962 MB (100% inode=99%): /run/lock 5 MB (100% inode=99%): /sys/fs/cgroup 7962 MB (100% inode=99%): /boot 188 MB (75% inode=99%): /srv 798914 MB (5% inode=99%) [03:56:15] PROBLEM - check_disk on frbackup2001 is CRITICAL: DISK CRITICAL - free space: /dev 10 MB (100% inode=99%): /run 2872 MB (90% inode=99%): / 51231 MB (96% inode=98%): /dev/shm 7962 MB (100% inode=99%): /run/lock 5 MB (100% inode=99%): /sys/fs/cgroup 7962 MB (100% inode=99%): /boot 188 MB (75% inode=99%): /srv 778316 MB (5% inode=99%) [03:58:40] ACKNOWLEDGEMENT - check_disk on frbackup2001 is CRITICAL: DISK CRITICAL - free space: /dev 10 MB (100% inode=99%): /run 2872 MB (90% inode=99%): / 51231 MB (96% inode=98%): /dev/shm 7962 MB (100% inode=99%): /run/lock 5 MB (100% inode=99%): /sys/fs/cgroup 7962 MB (100% inode=99%): /boot 188 MB (75% inode=99%): /srv 778316 MB (5% inode=99%): Jeff_Green Ok for now, well address it on Jan 2. [04:34:41] Fundraising-Backlog, MediaWiki-extensions-CentralNotice, Readers-Web-Backlog (Tracking): Banner close button overlaps search controls on mobile - https://phabricator.wikimedia.org/T176092 (AndyRussG) [14:19:48] (PS4) Jforrester: Have wgNoticeProjects default to [] [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/471662 (https://phabricator.wikimedia.org/T208694) (owner: MacFan4000) [14:20:45] (CR) Jforrester: "Have added a dependency on the production config change, and CCed colleagues who know or use this system." [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/471662 (https://phabricator.wikimedia.org/T208694) (owner: MacFan4000) [14:22:45] (CR) jerkins-bot: [V: -1] Have wgNoticeProjects default to [] [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/471662 (https://phabricator.wikimedia.org/T208694) (owner: MacFan4000) [15:14:15] (CR) MacFan4000: "recheck" [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/471662 (https://phabricator.wikimedia.org/T208694) (owner: MacFan4000) [20:01:12] Fundraising-Backlog, fundraising-tech-ops: Create read-only user for civicrm devdb on staging - https://phabricator.wikimedia.org/T212745 (Eileenmcnaughton) [21:18:43] (PS1) Eileen: Disable job to process smashpig recurring. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/481692 (https://phabricator.wikimedia.org/T212727) [21:20:18] (PS2) Eileen: Disable job to process smashpig recurring. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/481692 (https://phabricator.wikimedia.org/T212727) [21:26:45] (PS3) Eileen: Disable job to process smashpig recurring. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/481692 (https://phabricator.wikimedia.org/T212727) [21:37:40] (CR) jenkins-bot: Localisation updates from https://translatewiki.net. [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/481707 (owner: L10n-bot) [22:23:14] Fundraising Sprint Window dressing is mostly olive oil, Fundraising Sprint XML ate my homework, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, Fr-Ingenico-integration_2017-18: Ingenico refunds in mismatched currency - https://phabricator.wikimedia.org/T210473 (Eileenmcnaughton) [22:23:16] Fundraising Sprint XML ate my homework, Fundraising-Backlog, Fr-Ingenico-integration_2017-18: currency mismatch between Ingenico Connect & Civi - https://phabricator.wikimedia.org/T211024 (Eileenmcnaughton) [22:25:07] Fundraising Sprint Sasquatches can't find us either, Fundraising Sprint They Live, Fundraising Sprint USB stands for underhanded socket bureaucracy, Fundraising Sprint Vestigial tails shoot from the hip, and 4 others: Fix performance issue on activities whe... - https://phabricator.wikimedia.org/T204908 [23:34:45] Fundraising Sprint Sasquatches can't find us either, Fundraising Sprint They Live, Fundraising Sprint USB stands for underhanded socket bureaucracy, Fundraising Sprint Vestigial tails shoot from the hip, and 4 others: Fix performance issue on activities whe... - https://phabricator.wikimedia.org/T204908