[00:31:17] PROBLEM - Long lived cherry-picks on puppetmaster on deployment-puppetmaster02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [05:51:33] 10Gerrit: Gerrit Change-Id should not be copied with "Change-Id: " prefix - https://phabricator.wikimedia.org/T195277#4221732 (10Dvorapa) [05:53:39] 10Gerrit: New Gerrit UI does not show parent commit anywhere - https://phabricator.wikimedia.org/T195278#4221742 (10Dvorapa) [05:56:47] 10Phabricator: Phabricator does not let me to move task on multiple workboards at once - https://phabricator.wikimedia.org/T195279#4221753 (10Dvorapa) [05:59:57] 10Gerrit: New Gerrit UI does not contain copyable Change-Id or parent commit id - https://phabricator.wikimedia.org/T195280#4221763 (10Dvorapa) [06:00:47] 10Gerrit: New Gerrit UI does not contain copyable Change-Id or parent commit id - https://phabricator.wikimedia.org/T195280#4221763 (10Dvorapa) [06:08:32] PROBLEM - Free space - all mounts on deployment-fluorine02 is CRITICAL: CRITICAL: deployment-prep.deployment-fluorine02.diskspace._srv.byte_percentfree (<10.00%) [06:48:34] RECOVERY - Free space - all mounts on deployment-fluorine02 is OK: OK: All targets OK [06:56:45] PROBLEM - Puppet errors on integration-slave-jessie-1003 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [07:16:55] 10Release-Engineering-Team (Kanban), 10Wikidata, 10Wikidata-Ministry-Of-Magic-Tech-Debt, 10Wikidata-Turtles-Tech-Debt, and 4 others: Run Wikibase daily browser tests on Jenkins - https://phabricator.wikimedia.org/T167432#4221838 (10WMDE-leszek) @zeljkofilipin: actually I went ahead at the hackathon and mad... [07:19:21] zeljkof: morning! following the very helpful discussion we had in barcelona, would you have some time to have a look at some patches I made there? In particular, the most helpful would be feedback on the most recent patches made for https://phabricator.wikimedia.org/T191537 and https://phabricator.wikimedia.org/T194252 [07:36:45] RECOVERY - Puppet errors on integration-slave-jessie-1003 is OK: OK: Less than 1.00% above the threshold [0.0] [07:48:03] RECOVERY - Host deployment-secureredirproto is UP: PING OK - Packet loss = 0%, RTA = 1.92 ms [08:00:12] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments: 1.32.0-wmf.5 deployment blockers - https://phabricator.wikimedia.org/T191051#4221889 (10mmodell) [08:05:47] 10Phabricator: iridium:/var/log/phd/daemons.log is growing too much (took 20% of filesystem space) - https://phabricator.wikimedia.org/T124651#4221892 (10mmodell) Yes, and best practice is to use logrotate to cycle your logs periodically in order to avoid running out of space. [08:06:18] 10Phabricator: iridium:/var/log/phd/daemons.log is growing too much (took 20% of filesystem space) - https://phabricator.wikimedia.org/T124651#4221893 (10mmodell) 05Open>03Resolved a:03mmodell AFAIK This is no longer an issue in WMF production. (and iridium doesn't exist anymore) [08:06:26] greg-g: you accidently removed my slot? =o https://wikitech.wikimedia.org/w/index.php?diff=1792258&oldid=1792256&title=Deployments [08:06:40] evil greg is evil :D [08:40:57] PROBLEM - Host deployment-secureredirproto is DOWN: PING CRITICAL - Packet loss = 100% [09:14:52] Project mwext-phpunit-coverage-publish build #4645: 04FAILURE in 3.6 sec: https://integration.wikimedia.org/ci/job/mwext-phpunit-coverage-publish/4645/ [09:15:07] Project mwext-phpunit-coverage-publish build #4646: 04STILL FAILING in 3.8 sec: https://integration.wikimedia.org/ci/job/mwext-phpunit-coverage-publish/4646/ [09:23:28] Yippee, build fixed! [09:23:29] Project mwext-phpunit-coverage-publish build #4647: 09FIXED in 7 min 29 sec: https://integration.wikimedia.org/ci/job/mwext-phpunit-coverage-publish/4647/ [09:58:17] PROBLEM - Free space - all mounts on deployment-tin is CRITICAL: CRITICAL: deployment-prep.deployment-tin.diskspace._mnt.byte_percentfree (No valid datapoints found)deployment-prep.deployment-tin.diskspace.root.byte_percentfree (<22.22%) WARN: deployment-prep.deployment-tin.diskspace._srv.byte_percentfree (<11.11%) [10:42:23] PROBLEM - Puppet errors on integration-quibble-demo is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [11:00:05] PROBLEM - Puppet errors on deployment-snapshot01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [11:17:23] RECOVERY - Puppet errors on integration-quibble-demo is OK: OK: Less than 1.00% above the threshold [0.0] [11:27:02] 10Gerrit: New Gerrit UI does not contain copyable Change-Id or parent commit id - https://phabricator.wikimedia.org/T195280#4221763 (10Paladox) "parent commit id" that will be fixed in gerrit 2.16 / 3.0. For "Change-Id" i doin't think they have done that yet. [11:40:09] 10Gerrit, 10Phabricator: Unify Gerrit merge behavior - https://phabricator.wikimedia.org/T195291#4222158 (10Dvorapa) [11:40:33] 10Gerrit, 10Phabricator: Unify Gerrit merge behavior - https://phabricator.wikimedia.org/T195291#4222169 (10Dvorapa) [11:40:43] 10Gerrit, 10Phabricator: Unify Gerrit merge behavior - https://phabricator.wikimedia.org/T195291#4222158 (10Dvorapa) [11:55:37] (03CR) 10Tobias Gritschacher: [C: 032] Temporarily increase the timeout of selenium jobs [integration/config] - 10https://gerrit.wikimedia.org/r/434037 (https://phabricator.wikimedia.org/T167432) (owner: 10WMDE-leszek) [11:56:52] (03CR) 10Tobias Gritschacher: [C: 031] "Sorry should just be +1" [integration/config] - 10https://gerrit.wikimedia.org/r/434037 (https://phabricator.wikimedia.org/T167432) (owner: 10WMDE-leszek) [12:28:16] RECOVERY - Free space - all mounts on deployment-tin is OK: OK: deployment-prep.deployment-tin.diskspace._mnt.byte_percentfree (No valid datapoints found) [12:39:17] PROBLEM - Free space - all mounts on deployment-tin is CRITICAL: CRITICAL: deployment-prep.deployment-tin.diskspace._mnt.byte_percentfree (No valid datapoints found)deployment-prep.deployment-tin.diskspace.root.byte_percentfree (<22.22%) [12:41:53] legoktm: curious if you know why https://gerrit.wikimedia.org/r/#/c/433774/ didn't fail given a test in Flow fails consistently with it. It looks like Flow is in the shared gate, right? https://gerrit.wikimedia.org/r/#/c/434483/ [12:45:46] leszek_wmde: sorry for the late reply, just noticed your question, will take a look probably tomorrow, I don't think I'll have the time today [13:00:43] leszek_wmde: sorry for the late reply, just noticed your question, will take a look probably tomorrow, I don't think I'll have the time today [13:10:42] zeljkof: sure thing, thanks! I should be available on IRC for most of the time tomorrow, so feel free to ping whenever it's not clear what I was up to [13:26:08] (03CR) 10WMDE-leszek: [C: 031] Whitelist Sjoerddebruin [integration/config] - 10https://gerrit.wikimedia.org/r/433960 (owner: 10Hoo man) [13:42:59] PROBLEM - Puppet errors on deployment-mediawiki-07 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [14:07:56] RECOVERY - Puppet errors on deployment-mediawiki-07 is OK: OK: Less than 1.00% above the threshold [0.0] [14:53:17] PROBLEM - Host deployment-puppetdb01 is DOWN: CRITICAL - Host Unreachable (10.68.23.76) [15:11:40] Shalom. [15:11:50] At which time today is the branch cut for the train? [15:55:24] PROBLEM - Puppet errors on deployment-ircd is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [16:09:17] RECOVERY - Free space - all mounts on deployment-tin is OK: OK: deployment-prep.deployment-tin.diskspace._mnt.byte_percentfree (No valid datapoints found) [16:15:17] PROBLEM - Free space - all mounts on deployment-tin is CRITICAL: CRITICAL: deployment-prep.deployment-tin.diskspace._mnt.byte_percentfree (No valid datapoints found)deployment-prep.deployment-tin.diskspace.root.byte_percentfree (<11.11%) [16:35:22] (03PS4) 10Zoranzoki21: Revert "Archive PushToWatch" because is extension now active. [integration/config] - 10https://gerrit.wikimedia.org/r/433197 (https://phabricator.wikimedia.org/T194785) [16:40:50] 10Beta-Cluster-Infrastructure, 10Performance-Team, 10Patch-For-Review: Make MediaWiki profiler in Beta match production - https://phabricator.wikimedia.org/T180766#3768748 (10Krinkle) [16:53:28] !log Created deployment-webperf instance (m1.small) - ref T195312 [16:53:31] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:53:31] T195312: Set up PHP flame graphs for Beta Cluster - https://phabricator.wikimedia.org/T195312 [16:59:01] 10Beta-Cluster-Infrastructure, 10Performance-Team: Set up webperf node in Beta Cluster - https://phabricator.wikimedia.org/T195314#4222943 (10Krinkle) [16:59:10] 10Beta-Cluster-Infrastructure, 10Performance-Team: Set up webperf node in Beta Cluster - https://phabricator.wikimedia.org/T195314#4222956 (10Krinkle) [17:05:06] 10Gerrit: New Gerrit UI does not show parent commit anywhere - https://phabricator.wikimedia.org/T195278#4222983 (10Lucas_Werkmeister_WMDE) >>! In T195280#4222126, @Paladox wrote: > "parent commit id" that will be fixed in gerrit 2.16 / 3.0. [17:06:13] 10Gerrit: New Gerrit UI does not show parent commit anywhere - https://phabricator.wikimedia.org/T195278#4222988 (10Paladox) New ui https://gerrit-review.googlesource.com/c/gerrit/+/142811 see the left sidebar, it shows "Parent" [17:08:03] 10Beta-Cluster-Infrastructure, 10Performance-Team: Set up webperf node in Beta Cluster - https://phabricator.wikimedia.org/T195314#4222991 (10Krinkle) >>! In T195312#4222916, @Stashbot wrote: > {nav icon=file, name=Mentioned in SAL (#wikimedia-releng), href=https://tools.wmflabs.org/sal/log/AWOIxRnQBEfgIt1jK9B... [17:30:52] 10Beta-Cluster-Infrastructure, 10Security-Team: Delete deployment-mediawiki06 - https://phabricator.wikimedia.org/T192996#4156778 (10Bawolff) I've never heard of this server before [17:45:55] 10Gerrit: New Gerrit UI does not show parent commit anywhere - https://phabricator.wikimedia.org/T195278#4223024 (10Dvorapa) I see, this can be closed then, sorry for noise [18:03:20] 10Gerrit: New Gerrit UI does not show parent commit anywhere - https://phabricator.wikimedia.org/T195278#4223047 (10Paladox) 05Open>03declined Task already fixed @ master :) [19:10:19] 10Beta-Cluster-Infrastructure: deployment-prep broken puppet, ldap-utils downgrade fail - https://phabricator.wikimedia.org/T195267#4223189 (10EddieGP) [19:10:22] 10Beta-Cluster-Infrastructure: puppet fails over libldap/ldap-utils downgrade on various deployment-prep hosts - https://phabricator.wikimedia.org/T194926#4223192 (10EddieGP) [19:15:07] 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10Release, 10Train Deployments: 1.32.0-wmf.4 deployment blockers - https://phabricator.wikimedia.org/T191050#4223210 (10mmodell) [19:36:19] 10Diffusion, 10Gerrit: Unify Gerrit merge behavior - https://phabricator.wikimedia.org/T195291#4223286 (10Legoktm) [19:39:12] 10Diffusion, 10Gerrit: Unify Gerrit merge behavior - https://phabricator.wikimedia.org/T195291#4222158 (10Paladox) This is because if the commit is not on the latest commit that was last merged, then when jenkins submits it, it follows which ever policy is defined for that project, most projects are merge if n... [19:56:21] 10Diffusion, 10Gerrit: Unify Gerrit merge behavior - https://phabricator.wikimedia.org/T195291#4223367 (10Dvorapa) So this is by design? Maybe jenkins should rebase the change before submitting (if mergeable of course). [20:02:18] 10Diffusion, 10Gerrit: Unify Gerrit merge behavior - https://phabricator.wikimedia.org/T195291#4223371 (10Paladox) Yep by desgn, there is a rebase option, but the project has to choose it. [20:02:46] 10Diffusion, 10Gerrit: Unify Gerrit merge behavior - https://phabricator.wikimedia.org/T195291#4222158 (10demon) The default for all new projects is "Merge if Necessary." That's not configurable as far as I know? Personally, I think the default for all projects should be "Rebase if Necessary." Jenkins/Zuul do... [20:10:02] 10Beta-Cluster-Infrastructure: puppet fails over libldap/ldap-utils downgrade on various deployment-prep hosts - https://phabricator.wikimedia.org/T194926#4213207 (10awight) Thanks for the pointer @EddieGP, I was able to fix ores-lb-02 with the following command: sudo apt-get -q -y --force-yes -o DPkg::Options... [20:12:13] PROBLEM - Free space - all mounts on integration-slave-jessie-1004 is CRITICAL: CRITICAL: integration.integration-slave-jessie-1004.diskspace._srv.byte_percentfree (<55.56%) [20:15:42] !log eddie@deployment-db03:~$ sudo apt-get -q -y --force-yes -o DPkg::Options::=--force-confold install ldap-utils [20:15:44] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [20:16:10] yay, puppet unblocked [20:18:15] 10Gerrit, 10Upstream: New Gerrit UI does not contain copyable Change-Id or parent commit id - https://phabricator.wikimedia.org/T195280#4223422 (10Peachey88) [20:18:25] 10Phabricator (Upstream), 10Upstream: Phabricator does not let me to move task on multiple workboards at once - https://phabricator.wikimedia.org/T195279#4223423 (10Peachey88) [20:19:01] 10Gerrit, 10Phabricator (Upstream), 10Upstream: Gerrit Change-Id should not be copied with "Change-Id: " prefix - https://phabricator.wikimedia.org/T195277#4223425 (10Peachey88) [20:19:20] 10Gerrit, 10Upstream: Gerrit Change-Id should not be copied with "Change-Id: " prefix - https://phabricator.wikimedia.org/T195277#4223427 (10Peachey88) [20:20:08] 10Gerrit, 10Upstream: New Gerrit UI does not contain copyable Change-Id or parent commit id - https://phabricator.wikimedia.org/T195280#4223431 (10Paladox) filled upstream at https://bugs.chromium.org/p/gerrit/issues/detail?id=9062 for "no copyable Change-Id" part [20:21:15] 10Gerrit, 10Upstream: Gerrit Change-Id should not be copied with "Change-Id: " prefix - https://phabricator.wikimedia.org/T195277#4221732 (10Paladox) is this for the old ui? If so, upstream are planning to remove it from *.googlesource.com on june 4 2018. [20:21:41] PROBLEM - Host deployment-secureredirexperiment is DOWN: CRITICAL - Host Unreachable (10.68.17.132) [20:29:03] RECOVERY - Puppet errors on deployment-db03 is OK: OK: Less than 1.00% above the threshold [0.0] [20:29:32] 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10Release, 10Train Deployments: 1.32.0-wmf.4 deployment blockers - https://phabricator.wikimedia.org/T191050#4223457 (10mmodell) 05Open>03Resolved [20:31:04] 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10Release, 10Train Deployments: 1.32.0-wmf.5 deployment blockers - https://phabricator.wikimedia.org/T191051#4223465 (10mmodell) [20:32:39] 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10Release, 10Train Deployments: 1.32.0-wmf.5 deployment blockers - https://phabricator.wikimedia.org/T191051#4223474 (10mmodell) [20:36:14] 10Gerrit, 10Upstream: Gerrit Change-Id should not be copied with "Change-Id: " prefix - https://phabricator.wikimedia.org/T195277#4223482 (10Dvorapa) Yes, it is. Probably feel free to close. I didn't know the new UI is planned to be replaced so soon. When could we await the change arrives to Wmf? [20:37:47] 10Gerrit, 10Upstream: Gerrit Change-Id should not be copied with "Change-Id: " prefix - https://phabricator.wikimedia.org/T195277#4223486 (10Paladox) PolyGerrit is @ the wmf now, we are planning to do T177201 soon. Which will bring alot of improvements to the new ui. (also adds alot of the missing features) [20:37:59] !log eddie@eddie-thinkpad:~$ for host in deployment-elastic07 deployment-ircd deployment-logstash2 deployment-mathoid deployment-ms-fe02 deployment-prometheus01 deployment-tin; do ssh $host.eqiad.wmflabs sudo apt-get -q -y --force-yes -o DPkg::Options::=--force-confold install ldap-utils; done T194926 [20:38:02] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [20:38:02] T194926: puppet fails over libldap/ldap-utils downgrade on various deployment-prep hosts - https://phabricator.wikimedia.org/T194926 [20:38:23] 10Gerrit, 10Upstream: Gerrit Change-Id should not be copied with "Change-Id: " prefix - https://phabricator.wikimedia.org/T195277#4223490 (10Paladox) You can access polygerrit by doing https://gerrit.wikimedia.org/r/?polygerrit=1 which stores a cookie so it remembers to use polygerrit. [20:38:26] At some point I should learn about cumin [20:40:57] 10Gerrit, 10Upstream: Gerrit Change-Id should not be copied with "Change-Id: " prefix - https://phabricator.wikimedia.org/T195277#4223493 (10Paladox) PolyGerrit is also going to get a new update to the change screen (might) https://groups.google.com/forum/#!searchin/repo-discuss/hackathon%7Csort:date/repo-dis... [20:41:07] 10Gerrit, 10Upstream: Gerrit Change-Id should not be copied with "Change-Id: " prefix - https://phabricator.wikimedia.org/T195277#4223495 (10Dvorapa) >>! In T195277#4223490, @Paladox wrote: > You can access polygerrit by doing https://gerrit.wikimedia.org/r/?polygerrit=1 which stores a cookie so it remembers t... [20:42:14] !log eddie@eddie-thinkpad:~$ for host in deployment-elastic07 deployment-ircd deployment-logstash2 deployment-mathoid deployment-ms-fe02 deployment-prometheus01 deployment-tin; do ssh $host.eqiad.wmflabs sudo puppet agent -tv; done (for T194926, finished without errors on any of the hosts) [20:42:16] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [20:42:23] 10Gerrit, 10Upstream: Gerrit Change-Id should not be copied with "Change-Id: " prefix - https://phabricator.wikimedia.org/T195277#4223507 (10Paladox) Yep, it should fix many of the bugs. [20:43:24] 10Gerrit, 10Upstream: Gerrit Change-Id should not be copied with "Change-Id: " prefix - https://phabricator.wikimedia.org/T195277#4223516 (10Dvorapa) >>! In T195277#4223493, @Paladox wrote: > PolyGerrit is also going to get a new update to the change screen (might) > > https://groups.google.com/forum/#!search... [20:43:51] PROBLEM - Puppet errors on integration-slave-jessie-android is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [20:46:41] o/ [20:46:52] greg-g: alive? :) [20:47:08] RECOVERY - Puppet errors on deployment-elastic07 is OK: OK: Less than 1.00% above the threshold [0.0] [20:48:16] no_justification it seems this logging change https://gerrit.wikimedia.org/r/#/c/423794/ does not add .gz at the end of the file [20:49:34] I guess that's because gerrit does it in /logs/ [20:50:23] RECOVERY - Puppet errors on deployment-ircd is OK: OK: Less than 1.00% above the threshold [0.0] [20:50:23] RECOVERY - Puppet errors on deployment-logstash2 is OK: OK: Less than 1.00% above the threshold [0.0] [20:51:00] yep, let me see if i can make a change to make it configurable :) [20:51:11] RECOVERY - Puppet errors on deployment-mathoid is OK: OK: Less than 1.00% above the threshold [0.0] [20:52:58] RECOVERY - Puppet errors on deployment-tin is OK: OK: Less than 1.00% above the threshold [0.0] [20:53:10] RECOVERY - Puppet errors on deployment-prometheus01 is OK: OK: Less than 1.00% above the threshold [0.0] [20:53:44] RECOVERY - Puppet errors on deployment-ms-fe02 is OK: OK: Less than 1.00% above the threshold [0.0] [20:57:50] 10Beta-Cluster-Infrastructure: puppet fails over libldap/ldap-utils downgrade on various deployment-prep hosts - https://phabricator.wikimedia.org/T194926#4223541 (10EddieGP) ``` 20:29 RECOVERY - Puppet errors on deployment-db03 is OK: OK: Less than 1.00% above the threshold [0.0] 20:47 ... [21:08:22] PROBLEM - Puppet errors on deployment-mx02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [21:12:12] hmm [21:12:39] though by using the log4j file it should have disabled gerrit's logging and log4j should take over the logging [21:21:24] 10MediaWiki-Codesniffer, 10MediaWiki-extensions-Variables: Allow configuring MediaWiki.NamingConventions.ValidGlobalName.wgPrefix to allow additional prefixes - https://phabricator.wikimedia.org/T191812#4223574 (10MGChecker) a:03MGChecker I can work on this (and will do it at some point), however it might ta... [21:27:11] * addshore was going to say something stupid like lets deploy WikibaseLexeme tonight, but lets not.... [21:34:05] (03PS1) 1020after4: Add CongressLookup extension to make-wmf-branch config [tools/release] - 10https://gerrit.wikimedia.org/r/434606 [21:34:21] (03CR) 1020after4: [C: 032] Add CongressLookup extension to make-wmf-branch config [tools/release] - 10https://gerrit.wikimedia.org/r/434606 (owner: 1020after4) [21:35:04] (03Merged) 10jenkins-bot: Add CongressLookup extension to make-wmf-branch config [tools/release] - 10https://gerrit.wikimedia.org/r/434606 (owner: 1020after4) [21:37:12] PROBLEM - Free space - all mounts on integration-slave-jessie-1004 is CRITICAL: CRITICAL: integration.integration-slave-jessie-1004.diskspace._srv.byte_percentfree (<22.22%) [21:40:08] folks, I'd like to get us on some sort of plan/schedule as far as getting terbium off of php5 so we can actually update MW php requirements [21:40:27] this is a continuation of Sunday's discussion, for those who were around, [21:40:40] the salient points are here in the last comment: [21:41:48] https://phabricator.wikimedia.org/T172165 [21:42:42] the first thing would be: which do you prefer, switch off php5 to hhvm, or wait and switch off terbium to mwmaint1001 (= php7.0 for those jobs)? [21:42:51] PROBLEM - SSH on integration-slave-docker-1015 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:46:18] PROBLEM - SSH on integration-slave-docker-1013 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:52:40] RECOVERY - SSH on integration-slave-docker-1015 is OK: SSH OK - OpenSSH_6.7p1 Debian-5+deb8u4 (protocol 2.0) [21:53:08] PROBLEM - SSH on integration-slave-docker-1014 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:02:57] RECOVERY - SSH on integration-slave-docker-1014 is OK: SSH OK - OpenSSH_6.7p1 Debian-5+deb8u4 (protocol 2.0) [22:06:09] RECOVERY - SSH on integration-slave-docker-1013 is OK: SSH OK - OpenSSH_6.7p1 Debian-5+deb8u4 (protocol 2.0) [22:31:52] and a fix for dark mode on safari https://gerrit-review.googlesource.com/c/gerrit/+/179911 :) [22:35:09] paladox: just curious [22:35:17] PROBLEM - Puppet errors on deployment-deploy1001 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [22:35:19] addshore yep? :) [22:35:22] but when you noticed the db errors earlier / privacy policy not loading [22:35:31] yep [22:35:33] why were you looking at the privacy policy ? :P [22:35:47] * addshore is trying to figure out how it is linked, if at all... [22:35:49] I was trying to find out about how the wmf will handle the gdpr [22:36:13] aah okay, it's funny, as the page was edited 2/3 mins before the db errors started [22:36:20] heh [22:36:30] and I just spotted https://phabricator.wikimedia.org/T195293#4222630 which also now mentions the privacy policy [22:37:34] heh [22:50:39] 10Beta-Cluster-Infrastructure: puppet fails over libldap/ldap-utils downgrade on various deployment-prep hosts - https://phabricator.wikimedia.org/T194926#4223738 (10Dzahn) Thanks for your work on this @EddieGP [22:58:18] 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10Release, 10Train Deployments: 1.32.0-wmf.5 deployment blockers - https://phabricator.wikimedia.org/T191051#4223754 (10mmodell)