[00:03:12] Project mwext-phpunit-coverage-publish build #4846: 04STILL FAILING in 36 sec: https://integration.wikimedia.org/ci/job/mwext-phpunit-coverage-publish/4846/ [00:04:06] 10Beta-Cluster-Infrastructure: Move puppetmaster to Stretch - https://phabricator.wikimedia.org/T195686#4234304 (10Krenair) [00:05:50] Project beta-scap-eqiad build #209217: 04STILL FAILING in 1 min 59 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/209217/ [00:06:52] legoktm question about planet, if we coulden't generate a atom ui instead only a rss file (which would be atom compatible) would that be ok? [00:15:37] Project beta-scap-eqiad build #209218: 04STILL FAILING in 1 min 54 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/209218/ [00:20:10] (03PS1) 10Legoktm: Generate coverage reports for Scribunto [integration/config] - 10https://gerrit.wikimedia.org/r/435673 [00:20:17] Project beta-update-databases-eqiad build #25741: 04STILL FAILING in 16 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/25741/ [00:20:33] paladox: I think so. tbh I don't really know what the actual difference between atom and rss are [00:20:40] Ok [00:20:43] my feed reader supports both so... :) [00:20:47] heh [00:20:51] i think atom is the ui one [00:20:59] prettifies the ui [00:21:06] so it's readable in the web browser [00:21:26] where as rss requires a feed reader (even though you can view it in the browser) it shows xml. [00:21:40] (03CR) 10Legoktm: [C: 032] Generate coverage reports for Scribunto [integration/config] - 10https://gerrit.wikimedia.org/r/435673 (owner: 10Legoktm) [00:22:47] PROBLEM - Puppet errors on deployment-certcentral is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [00:23:09] certcentral? [00:23:29] certral would be an OK name [00:23:45] (03Merged) 10jenkins-bot: Generate coverage reports for Scribunto [integration/config] - 10https://gerrit.wikimedia.org/r/435673 (owner: 10Legoktm) [00:24:33] !log deployed https://gerrit.wikimedia.org/r/435673 [00:24:36] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [00:24:41] PROBLEM - Puppet errors on deployment-puppetdb01 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [00:24:57] PROBLEM - Puppet errors on deployment-mediawiki-07 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [00:25:34] Yippee, build fixed! [00:25:35] Project mwext-phpunit-coverage-publish build #4847: 09FIXED in 57 sec: https://integration.wikimedia.org/ci/job/mwext-phpunit-coverage-publish/4847/ [00:25:50] Project beta-scap-eqiad build #209219: 04STILL FAILING in 2 min 1 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/209219/ [00:29:58] PROBLEM - Puppet errors on deployment-certcentral-testclient is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [00:35:40] Project beta-scap-eqiad build #209220: 04STILL FAILING in 1 min 55 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/209220/ [00:45:36] Project beta-scap-eqiad build #209221: 04STILL FAILING in 1 min 53 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/209221/ [00:47:58] PROBLEM - Puppet errors on deployment-tin is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [00:49:19] PROBLEM - Puppet errors on deployment-puppetmaster03 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [00:55:39] Project beta-scap-eqiad build #209222: 04STILL FAILING in 1 min 54 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/209222/ [01:05:45] Project beta-scap-eqiad build #209223: 04STILL FAILING in 1 min 58 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/209223/ [01:15:47] Project beta-scap-eqiad build #209224: 04STILL FAILING in 1 min 59 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/209224/ [01:20:15] Project beta-update-databases-eqiad build #25742: 04STILL FAILING in 14 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/25742/ [01:25:41] Project beta-scap-eqiad build #209225: 04STILL FAILING in 1 min 56 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/209225/ [01:35:41] Project beta-scap-eqiad build #209226: 04STILL FAILING in 1 min 55 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/209226/ [01:45:51] Project beta-scap-eqiad build #209227: 04STILL FAILING in 2 min 5 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/209227/ [01:55:37] Project beta-scap-eqiad build #209228: 04STILL FAILING in 1 min 53 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/209228/ [02:05:42] Project beta-scap-eqiad build #209229: 04STILL FAILING in 1 min 56 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/209229/ [02:15:38] Project beta-scap-eqiad build #209230: 04STILL FAILING in 1 min 53 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/209230/ [02:20:15] Project beta-update-databases-eqiad build #25743: 04STILL FAILING in 14 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/25743/ [02:25:31] Project beta-scap-eqiad build #209231: 04STILL FAILING in 1 min 48 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/209231/ [02:35:35] Project beta-scap-eqiad build #209232: 04STILL FAILING in 1 min 52 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/209232/ [02:45:34] Project beta-scap-eqiad build #209233: 04STILL FAILING in 1 min 50 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/209233/ [02:55:32] Project beta-scap-eqiad build #209234: 04STILL FAILING in 1 min 50 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/209234/ [02:57:04] PROBLEM - Puppet errors on deployment-puppetdb02 is CRITICAL: CRITICAL: 83.33% of data above the critical threshold [0.0] [02:57:57] 10Release-Engineering-Team, 10DNS, 10Operations, 10Traffic, and 2 others: Move Foundation Wiki to new URL when new Wikimedia Foundation website launches - https://phabricator.wikimedia.org/T188776#4234362 (10demon) a:05demon>03None Actually, I won't be able to get this done. Unlicking the cookie. [03:05:34] Project beta-scap-eqiad build #209235: 04STILL FAILING in 1 min 51 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/209235/ [03:07:47] RECOVERY - Puppet errors on deployment-certcentral is OK: OK: Less than 1.00% above the threshold [0.0] [03:09:19] RECOVERY - Puppet errors on deployment-puppetmaster03 is OK: OK: Less than 1.00% above the threshold [0.0] [03:09:57] RECOVERY - Puppet errors on deployment-mediawiki-07 is OK: OK: Less than 1.00% above the threshold [0.0] [03:09:59] RECOVERY - Puppet errors on deployment-certcentral-testclient is OK: OK: Less than 1.00% above the threshold [0.0] [03:15:34] Project beta-scap-eqiad build #209236: 04STILL FAILING in 1 min 51 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/209236/ [03:20:14] Project beta-update-databases-eqiad build #25744: 04STILL FAILING in 13 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/25744/ [03:20:21] PROBLEM - Puppet errors on deployment-puppetmaster03 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [03:25:32] Project beta-scap-eqiad build #209237: 04STILL FAILING in 1 min 49 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/209237/ [03:27:59] RECOVERY - Puppet errors on deployment-tin is OK: OK: Less than 1.00% above the threshold [0.0] [03:28:49] PROBLEM - Puppet errors on deployment-certcentral is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [03:30:57] PROBLEM - Puppet errors on deployment-mediawiki-07 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [03:30:59] PROBLEM - Puppet errors on deployment-certcentral-testclient is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [03:35:37] Project beta-scap-eqiad build #209238: 04STILL FAILING in 1 min 53 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/209238/ [03:45:33] Project beta-scap-eqiad build #209239: 04STILL FAILING in 1 min 51 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/209239/ [03:48:58] PROBLEM - Puppet errors on deployment-tin is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [03:55:46] Project beta-scap-eqiad build #209240: 04STILL FAILING in 2 min 0 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/209240/ [03:58:33] 03:25:31 Fatal error: Uncaught exception 'ConfigException' with message 'Failed to load configuration from etcd: (curl error: 60) Peer certificate cannot be authenticated with given CA certificates' in /srv/mediawiki-staging/php-master/includes/config/EtcdConfig.php:200 [03:58:45] Krenair: ^ any chance that's related to the stuff you were doing? [03:59:54] possibly [04:00:19] RECOVERY - Puppet errors on deployment-puppetmaster03 is OK: OK: Less than 1.00% above the threshold [0.0] [04:01:18] legoktm, is it using puppet certs? [04:02:06] RECOVERY - Puppet errors on deployment-puppetdb02 is OK: OK: Less than 1.00% above the threshold [0.0] [04:04:40] RECOVERY - Puppet errors on deployment-puppetdb01 is OK: OK: Less than 1.00% above the threshold [0.0] [04:05:43] Project beta-scap-eqiad build #209241: 04STILL FAILING in 1 min 57 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/209241/ [04:05:58] RECOVERY - Puppet errors on deployment-certcentral-testclient is OK: OK: Less than 1.00% above the threshold [0.0] [04:06:23] Krenair: No idea...it's configured in ops/mediawiki-config [04:07:03] well etcd.php doesn't say anything about certificates [04:08:47] RECOVERY - Puppet errors on deployment-certcentral is OK: OK: Less than 1.00% above the threshold [0.0] [04:10:57] RECOVERY - Puppet errors on deployment-mediawiki-07 is OK: OK: Less than 1.00% above the threshold [0.0] [04:11:28] Krenair: oh, it might be EtcdConfig in core then? :| [04:13:47] so if etcd is using puppet certs for HTTPS and MW is trying to validate that... [04:14:04] then yeah, since we have two different puppetmasters in use at the moment, that won't work [04:14:21] some servers have the jessie puppetmaster (02) and a couple have the stretch one (03) [04:14:45] this is on deployment-tin [04:14:48] could probably fix it by moving the etcd server to the stretch puppetmaster I guess [04:14:56] yeah that's using the stretch puppetmaster [04:15:45] Project beta-scap-eqiad build #209242: 04STILL FAILING in 1 min 58 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/209242/ [04:20:17] Project beta-update-databases-eqiad build #25745: 04STILL FAILING in 15 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/25745/ [04:23:58] RECOVERY - Puppet errors on deployment-tin is OK: OK: Less than 1.00% above the threshold [0.0] [04:25:49] Project beta-scap-eqiad build #209243: 04STILL FAILING in 2 min 2 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/209243/ [04:35:55] Project beta-scap-eqiad build #209244: 04STILL FAILING in 2 min 7 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/209244/ [04:45:51] Project beta-scap-eqiad build #209245: 04STILL FAILING in 2 min 5 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/209245/ [04:55:40] Project beta-scap-eqiad build #209246: 04STILL FAILING in 1 min 55 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/209246/ [05:05:49] Project beta-scap-eqiad build #209247: 04STILL FAILING in 2 min 2 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/209247/ [05:15:43] Project beta-scap-eqiad build #209248: 04STILL FAILING in 1 min 58 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/209248/ [05:18:05] PROBLEM - Puppet errors on deployment-puppetdb02 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [05:20:16] Project beta-update-databases-eqiad build #25746: 04STILL FAILING in 14 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/25746/ [05:21:20] PROBLEM - Puppet errors on deployment-puppetmaster03 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [05:25:40] PROBLEM - Puppet errors on deployment-puppetdb01 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [05:25:52] Project beta-scap-eqiad build #209249: 04STILL FAILING in 2 min 4 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/209249/ [05:29:47] PROBLEM - Puppet errors on deployment-certcentral is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [05:31:57] PROBLEM - Puppet errors on deployment-mediawiki-07 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [05:32:00] PROBLEM - Puppet errors on deployment-certcentral-testclient is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [05:35:41] Project beta-scap-eqiad build #209250: 04STILL FAILING in 1 min 54 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/209250/ [05:45:50] Project beta-scap-eqiad build #209251: 04STILL FAILING in 2 min 2 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/209251/ [05:49:58] PROBLEM - Puppet errors on deployment-tin is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [05:55:50] Project beta-scap-eqiad build #209252: 04STILL FAILING in 2 min 4 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/209252/ [06:00:33] (03PS1) 10Legoktm: Whitelist RazeSoldier [integration/config] - 10https://gerrit.wikimedia.org/r/435681 [06:00:51] (03CR) 10Legoktm: [C: 032] Whitelist RazeSoldier [integration/config] - 10https://gerrit.wikimedia.org/r/435681 (owner: 10Legoktm) [06:02:16] (03Merged) 10jenkins-bot: Whitelist RazeSoldier [integration/config] - 10https://gerrit.wikimedia.org/r/435681 (owner: 10Legoktm) [06:02:41] !log deployed https://gerrit.wikimedia.org/r/435681 [06:02:44] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [06:05:49] Project beta-scap-eqiad build #209253: 04STILL FAILING in 2 min 3 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/209253/ [06:15:46] Project beta-scap-eqiad build #209254: 04STILL FAILING in 1 min 59 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/209254/ [06:20:15] Project beta-update-databases-eqiad build #25747: 04STILL FAILING in 14 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/25747/ [06:25:39] Project beta-scap-eqiad build #209255: 04STILL FAILING in 1 min 55 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/209255/ [06:35:43] Project beta-scap-eqiad build #209256: 04STILL FAILING in 1 min 57 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/209256/ [06:46:02] Project beta-scap-eqiad build #209257: 04STILL FAILING in 2 min 11 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/209257/ [06:55:38] Project beta-scap-eqiad build #209258: 04STILL FAILING in 1 min 54 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/209258/ [07:05:47] Project beta-scap-eqiad build #209259: 04STILL FAILING in 2 min 1 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/209259/ [07:16:01] Project beta-scap-eqiad build #209260: 04STILL FAILING in 2 min 10 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/209260/ [07:20:15] Project beta-update-databases-eqiad build #25748: 04STILL FAILING in 14 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/25748/ [07:25:57] Project beta-scap-eqiad build #209261: 04STILL FAILING in 2 min 7 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/209261/ [07:35:40] Project beta-scap-eqiad build #209262: 04STILL FAILING in 1 min 55 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/209262/ [07:45:40] Project beta-scap-eqiad build #209263: 04STILL FAILING in 1 min 54 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/209263/ [07:55:38] Project beta-scap-eqiad build #209264: 04STILL FAILING in 1 min 53 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/209264/ [08:05:50] Project beta-scap-eqiad build #209265: 04STILL FAILING in 2 min 2 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/209265/ [08:15:39] Project beta-scap-eqiad build #209266: 04STILL FAILING in 1 min 54 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/209266/ [08:20:15] Project beta-update-databases-eqiad build #25749: 04STILL FAILING in 14 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/25749/ [08:23:29] 10Beta-Cluster-Infrastructure: Move puppetmaster to Stretch - https://phabricator.wikimedia.org/T195686#4234412 (10EddieGP) p:05Triage>03Normal [08:25:50] Project beta-scap-eqiad build #209267: 04STILL FAILING in 2 min 3 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/209267/ [08:35:48] Project beta-scap-eqiad build #209268: 04STILL FAILING in 2 min 1 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/209268/ [08:36:52] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team, 10Operations: Upgrade deployment-prep deployment servers to stretch - https://phabricator.wikimedia.org/T192561#4234419 (10EddieGP) It seems you used the same flavor for deploy1001 that tin had. This would've been a great time to switch to a differen... [08:45:50] Project beta-scap-eqiad build #209269: 04STILL FAILING in 2 min 3 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/209269/ [08:55:44] Project beta-scap-eqiad build #209270: 04STILL FAILING in 1 min 58 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/209270/ [09:05:48] Project beta-scap-eqiad build #209271: 04STILL FAILING in 2 min 1 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/209271/ [09:15:44] Project beta-scap-eqiad build #209272: 04STILL FAILING in 2 min 0 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/209272/ [09:20:17] Project beta-update-databases-eqiad build #25750: 04STILL FAILING in 16 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/25750/ [09:25:42] Project beta-scap-eqiad build #209273: 04STILL FAILING in 1 min 57 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/209273/ [09:35:43] Project beta-scap-eqiad build #209274: 04STILL FAILING in 1 min 58 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/209274/ [09:45:49] Project beta-scap-eqiad build #209275: 04STILL FAILING in 2 min 3 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/209275/ [09:49:43] 10Beta-Cluster-Infrastructure, 10Easy: en-rtl listed under other projects on beta.wmflabs.org - https://phabricator.wikimedia.org/T195675#4234443 (10EddieGP) `en_rtl` is in the table of projects. Under "other projects" there's `en-rtl`. Probably all that's needed is to swap the underscore with a dash in the la... [09:49:58] 10Beta-Cluster-Infrastructure, 10Easy: en-rtl listed under other projects on beta.wmflabs.org - https://phabricator.wikimedia.org/T195675#4234445 (10EddieGP) p:05Triage>03Low [09:55:52] Project beta-scap-eqiad build #209276: 04STILL FAILING in 2 min 5 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/209276/ [09:55:58] 10Beta-Cluster-Infrastructure: [betalabs] Uncaught TypeError: Cannot read property 'lat' of null - when 'Edit layer' option is used. - https://phabricator.wikimedia.org/T194440#4198958 (10EddieGP) Please provice a bit more information on what you were doing, which wiki in beta you specifically used, what part of... [10:05:58] Project beta-scap-eqiad build #209277: 04STILL FAILING in 2 min 9 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/209277/ [10:07:05] 10Beta-Cluster-Infrastructure, 10Operations, 10Puppet: Make deployment-prep puppetmaster more similar to Production puppetmaster - https://phabricator.wikimedia.org/T146627#4234462 (10EddieGP) 05Open>03Resolved a:03EddieGP According to openstack browser this uses ::standalone as of now. Seems the ony t... [10:15:49] Project beta-scap-eqiad build #209278: 04STILL FAILING in 2 min 3 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/209278/ [10:20:03] 10Beta-Cluster-Infrastructure, 10Continuous-Integration-Config, 10Release-Engineering-Team (Next): Use cron instead of Jenkins for beta deployments - https://phabricator.wikimedia.org/T188367#4004864 (10EddieGP) I wonder how this relates to T73305, which is about migrating away from a cron to jenkins for the... [10:20:15] Project beta-update-databases-eqiad build #25751: 04STILL FAILING in 14 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/25751/ [10:21:06] 10Beta-Cluster-Infrastructure, 10Continuous-Integration-Config: Convert puppetmaster sync cronjob to Jenkins job - https://phabricator.wikimedia.org/T73305#770430 (10EddieGP) See also T188367, which is about going the other way round (jenkins->cron) for mediawiki, and my comment there. [10:25:46] Project beta-scap-eqiad build #209279: 04STILL FAILING in 2 min 0 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/209279/ [10:35:51] Project beta-scap-eqiad build #209280: 04STILL FAILING in 2 min 5 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/209280/ [10:45:50] Project beta-scap-eqiad build #209281: 04STILL FAILING in 2 min 3 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/209281/ [10:55:42] Project beta-scap-eqiad build #209282: 04STILL FAILING in 1 min 57 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/209282/ [11:05:50] Project beta-scap-eqiad build #209283: 04STILL FAILING in 2 min 2 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/209283/ [11:15:56] Project beta-scap-eqiad build #209284: 04STILL FAILING in 2 min 8 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/209284/ [11:17:59] 10Phabricator (Upstream), 10Pywikibot-core, 10Upstream: Remove the use of inline flags in the middle of regular expression - https://phabricator.wikimedia.org/T195538#4234507 (10Xqt) 05Open>03stalled Stalled until an upstream solution is published, see http://bugs.python.org/issue33658 [11:20:16] Project beta-update-databases-eqiad build #25752: 04STILL FAILING in 14 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/25752/ [11:25:45] Project beta-scap-eqiad build #209285: 04STILL FAILING in 2 min 1 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/209285/ [11:35:46] Project beta-scap-eqiad build #209286: 04STILL FAILING in 2 min 0 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/209286/ [11:45:39] Project beta-scap-eqiad build #209287: 04STILL FAILING in 1 min 54 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/209287/ [11:55:41] Project beta-scap-eqiad build #209288: 04STILL FAILING in 1 min 56 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/209288/ [11:56:16] :/ [11:57:45] 11:55:41 [Sun May 27 11:55:41 2018] [hphp] [894:7f84c4d98200:0:000001] [] [11:57:45] 11:55:41 Fatal error: Uncaught exception 'ConfigException' with message 'Failed to load configuration from etcd: (curl error: 60) Peer certificate cannot be authenticated with given CA certificates' in /srv/mediawiki-staging/php-master/includes/config/EtcdConfig.php:200 [11:58:01] Krenair Reedy ^^ [12:05:48] Project beta-scap-eqiad build #209289: 04STILL FAILING in 2 min 1 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/209289/ [12:15:36] Project beta-scap-eqiad build #209290: 04STILL FAILING in 1 min 52 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/209290/ [12:20:15] Project beta-update-databases-eqiad build #25753: 04STILL FAILING in 14 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/25753/ [12:25:49] Project beta-scap-eqiad build #209291: 04STILL FAILING in 2 min 4 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/209291/ [12:35:43] Project beta-scap-eqiad build #209292: 04STILL FAILING in 1 min 57 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/209292/ [12:38:40] PROBLEM - Puppet errors on deployment-ms-be04 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [12:45:44] Project beta-scap-eqiad build #209293: 04STILL FAILING in 1 min 58 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/209293/ [12:54:28] PROBLEM - Puppet errors on deployment-ms-be03 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [12:55:51] Project beta-scap-eqiad build #209294: 04STILL FAILING in 2 min 3 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/209294/ [13:05:47] Project beta-scap-eqiad build #209295: 04STILL FAILING in 2 min 0 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/209295/ [13:15:36] Project beta-scap-eqiad build #209296: 04STILL FAILING in 1 min 52 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/209296/ [13:20:16] Project beta-update-databases-eqiad build #25754: 04STILL FAILING in 15 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/25754/ [13:25:45] Project beta-scap-eqiad build #209297: 04STILL FAILING in 1 min 59 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/209297/ [13:35:38] Project beta-scap-eqiad build #209298: 04STILL FAILING in 1 min 53 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/209298/ [13:36:33] PROBLEM - Puppet errors on deployment-eventlog05 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [13:45:45] Project beta-scap-eqiad build #209299: 04STILL FAILING in 1 min 58 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/209299/ [13:55:34] Project beta-scap-eqiad build #209300: 04STILL FAILING in 1 min 50 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/209300/ [14:03:35] 10Project-Admins, 10MediaWiki-Documentation, 10Documentation, 10Pywikibot-Documentation, and 2 others: Consider archiving #MediaWiki-Documentation & #Pywikibot-documentation in favour of single #Documentation - https://phabricator.wikimedia.org/T142345#4234624 (10Dvorapa) 05Open>03declined Unless we co... [14:05:36] Project beta-scap-eqiad build #209301: 04STILL FAILING in 1 min 51 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/209301/ [14:15:40] Project beta-scap-eqiad build #209302: 04STILL FAILING in 1 min 54 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/209302/ [14:20:15] Project beta-update-databases-eqiad build #25755: 04STILL FAILING in 14 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/25755/ [14:21:18] RECOVERY - Puppet errors on deployment-puppetmaster03 is OK: OK: Less than 1.00% above the threshold [0.0] [14:21:58] RECOVERY - Puppet errors on deployment-certcentral-testclient is OK: OK: Less than 1.00% above the threshold [0.0] [14:23:04] RECOVERY - Puppet errors on deployment-puppetdb02 is OK: OK: Less than 1.00% above the threshold [0.0] [14:24:47] RECOVERY - Puppet errors on deployment-certcentral is OK: OK: Less than 1.00% above the threshold [0.0] [14:24:59] RECOVERY - Puppet errors on deployment-tin is OK: OK: Less than 1.00% above the threshold [0.0] [14:25:41] Project beta-scap-eqiad build #209303: 04STILL FAILING in 1 min 57 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/209303/ [14:26:57] RECOVERY - Puppet errors on deployment-mediawiki-07 is OK: OK: Less than 1.00% above the threshold [0.0] [14:29:39] WHy has someone started naming hosts with a hyphen before the number? [14:35:33] PROBLEM - Puppet errors on deployment-etcd-01 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [14:35:34] Project beta-scap-eqiad build #209304: 04STILL FAILING in 1 min 52 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/209304/ [14:35:39] RECOVERY - Puppet errors on deployment-puppetdb01 is OK: OK: Less than 1.00% above the threshold [0.0] [14:38:38] 10Beta-Cluster-Infrastructure: Move puppetmaster to Stretch - https://phabricator.wikimedia.org/T195686#4234680 (10Krenair) p:05Normal>03Triage [14:38:40] 10Beta-Cluster-Infrastructure: Move puppetmaster to Stretch - https://phabricator.wikimedia.org/T195686#4234225 (10Krenair) p:05Triage>03Normal [14:41:57] legoktm, how's the cert problem looking now? [14:45:32] RECOVERY - Puppet errors on deployment-etcd-01 is OK: OK: Less than 1.00% above the threshold [0.0] [14:45:35] Project beta-scap-eqiad build #209305: 04STILL FAILING in 1 min 52 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/209305/ [14:48:45] hm [14:55:35] Project beta-scap-eqiad build #209306: 04STILL FAILING in 1 min 51 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/209306/ [15:02:53] alright I think I know what's up [15:02:55] plan of action [15:03:35] generate new cert for etcd, store it on puppetmaster as a secret and a cherry-picked files/ssl file [15:03:35] move other hosts that need to talk to etcd to the stretch puppetmaster [15:05:34] Project beta-scap-eqiad build #209307: 04STILL FAILING in 1 min 51 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/209307/ [15:15:32] Project beta-scap-eqiad build #209308: 04STILL FAILING in 1 min 50 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/209308/ [15:20:15] Project beta-update-databases-eqiad build #25756: 04STILL FAILING in 14 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/25756/ [15:25:48] Project beta-scap-eqiad build #209309: 04STILL FAILING in 2 min 2 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/209309/ [15:27:43] PROBLEM - Puppet errors on deployment-redis01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [15:33:42] PROBLEM - Puppet errors on deployment-redis02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [15:35:43] Project beta-scap-eqiad build #209310: 04STILL FAILING in 1 min 55 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/209310/ [15:45:37] Project beta-scap-eqiad build #209311: 04STILL FAILING in 1 min 54 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/209311/ [15:46:53] ok [15:55:34] Project beta-scap-eqiad build #209312: 04STILL FAILING in 1 min 49 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/209312/ [15:56:15] hmm why is it still upset [15:58:15] ah [16:00:45] hm it doesn't look like I can do much in jenkins while logged in anymore [16:06:41] alright it looks like it's running properly now [16:07:08] also, fml: [16:07:09] deployment-redis01 is a Jobqueue master (jobqueue_redis::master) [16:07:09] The last Puppet run was at Tue Nov 28 14:52:46 UTC 2017 (259266 minutes ago). [16:08:12] Heh [16:09:32] 10Beta-Cluster-Infrastructure: Move puppetmaster to Stretch - https://phabricator.wikimedia.org/T195686#4234857 (10Krenair) [16:09:40] Krenair: over a year [16:09:41] Wow [16:09:44] 4321 [16:09:46] no [16:09:58] Google says that [16:10:07] it's may now [16:10:15] last puppet run on the deployment-redis boxes - november [16:10:24] Google 259266 mins in hours [16:10:32] Oh hours [16:10:32] 6 months [16:10:43] 180 days [16:12:17] 10Beta-Cluster-Infrastructure: Move puppetmaster to Stretch - https://phabricator.wikimedia.org/T195686#4234864 (10Krenair) [16:16:49] Yippee, build fixed! [16:16:50] Project beta-scap-eqiad build #209313: 09FIXED in 13 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/209313/ [16:23:53] Yippee, build fixed! [16:23:54] Project beta-update-databases-eqiad build #25757: 09FIXED in 3 min 52 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/25757/ [16:28:37] PROBLEM - Puppet errors on deployment-jobrunner03 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [16:38:37] RECOVERY - Puppet errors on deployment-jobrunner03 is OK: OK: Less than 1.00% above the threshold [0.0] [16:45:28] 10Beta-Cluster-Infrastructure: Move puppetmaster to Stretch - https://phabricator.wikimedia.org/T195686#4234893 (10Krenair) [17:06:47] 10Beta-Cluster-Infrastructure: Move puppetmaster to Stretch - https://phabricator.wikimedia.org/T195686#4234897 (10Krenair) [17:27:31] 10Beta-Cluster-Infrastructure: Move puppetmaster to Stretch - https://phabricator.wikimedia.org/T195686#4234907 (10Krenair) [17:30:17] deployment-imagescaler02 still has salt stuff in hiera [17:33:22] 10Beta-Cluster-Infrastructure: Move puppetmaster to Stretch - https://phabricator.wikimedia.org/T195686#4234910 (10Krenair) [17:40:19] 10Beta-Cluster-Infrastructure: Move puppetmaster to Stretch - https://phabricator.wikimedia.org/T195686#4234923 (10Krenair) [17:41:26] 10Beta-Cluster-Infrastructure: Move puppetmaster to Stretch - https://phabricator.wikimedia.org/T195686#4234225 (10Krenair) [18:04:08] 10Beta-Cluster-Infrastructure, 10Continuous-Integration-Config, 10Release-Engineering-Team (Next): Use cron instead of Jenkins for beta deployments - https://phabricator.wikimedia.org/T188367#4234983 (10demon) I think we should decline T73305 then? [18:08:55] (03PS1) 10Umherirrender: Add tests for PrimarySources [integration/config] - 10https://gerrit.wikimedia.org/r/435706 [18:14:00] (03PS1) 10Umherirrender: Add tests for BlueSpiceEchoConnector [integration/config] - 10https://gerrit.wikimedia.org/r/435707 [18:14:01] PROBLEM - Puppet errors on deployment-webperf01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [18:21:23] 10Beta-Cluster-Infrastructure: Move puppetmaster to Stretch - https://phabricator.wikimedia.org/T195686#4234989 (10Krenair) [18:39:23] PROBLEM - Puppet errors on deployment-mx is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [18:43:45] 10Beta-Cluster-Infrastructure: Move puppetmaster to Stretch - https://phabricator.wikimedia.org/T195686#4234994 (10Krenair) [18:49:35] 10Project-Admins, 10MediaWiki-Documentation, 10Documentation, 10Pywikibot-Documentation, and 2 others: Consider archiving #MediaWiki-Documentation & #Pywikibot-documentation in favour of single #Documentation - https://phabricator.wikimedia.org/T142345#4234995 (10Aklapper) See T142345#3655080. (High-level:... [18:49:39] what the heck is wrong with deployment-restbase02? [18:52:55] 10Beta-Cluster-Infrastructure, 10Services (next), 10User-Eevans: Reset RESTBase deployment-prep environment - https://phabricator.wikimedia.org/T186750#4234996 (10Krenair) [18:53:43] 10Beta-Cluster-Infrastructure, 10Services: Something is wrong with deployment-restbase02.deployment-prep.eqiad.wmflabs - https://phabricator.wikimedia.org/T195709#4234997 (10Krenair) [18:54:27] 10Beta-Cluster-Infrastructure, 10Services: Something is wrong with deployment-restbase02.deployment-prep.eqiad.wmflabs - https://phabricator.wikimedia.org/T195709#4235007 (10Krenair) Where are these lines (example from 01): ```Linux deployment-restbase01 4.9.0-0.bpo.5-amd64 #1 SMP Debian 4.9.65-3+deb9u1~bpo8+2... [18:55:59] 10Beta-Cluster-Infrastructure: Move puppetmaster to Stretch - https://phabricator.wikimedia.org/T195686#4235008 (10Krenair) [19:00:21] 10Beta-Cluster-Infrastructure: Move puppetmaster to Stretch - https://phabricator.wikimedia.org/T195686#4235012 (10Krenair) [19:04:11] 10Gerrit, 10Release-Engineering-Team (Someday), 10Operations, 10Patch-For-Review: Gerrit shows HTTP 500 error when pasting extended unicode characters - https://phabricator.wikimedia.org/T145885#4235014 (10Paladox) The gerrit upgrade will be happening soon so as soon as the upgrade is done notedb will quic... [19:05:44] 10Project-Admins: Create a new phabricator project for Lingua Libre - https://phabricator.wikimedia.org/T195565#4235015 (10Aklapper) If we create a project in Wikimedia Phabricator, what would happen (by who) to the tickets in https://github.com/wikimedia-france/Lingua-Libre/issues ? If GitHub allows disabling "... [19:07:03] 10Beta-Cluster-Infrastructure, 10ORES, 10Scoring-platform-team: Reimage deployment-ores01 as Stretch - https://phabricator.wikimedia.org/T189790#4235019 (10Krenair) [19:19:45] 10Beta-Cluster-Infrastructure: Move puppetmaster to Stretch - https://phabricator.wikimedia.org/T195686#4235020 (10Krenair) [19:31:20] 10Beta-Cluster-Infrastructure: Move puppetmaster to Stretch - https://phabricator.wikimedia.org/T195686#4235022 (10Krenair) [19:36:54] 10Beta-Cluster-Infrastructure: Move puppetmaster to Stretch - https://phabricator.wikimedia.org/T195686#4235023 (10Krenair) [19:41:19] 10Beta-Cluster-Infrastructure: Move puppetmaster to Stretch - https://phabricator.wikimedia.org/T195686#4235025 (10Krenair) [19:53:47] 10Beta-Cluster-Infrastructure: Move puppetmaster to Stretch - https://phabricator.wikimedia.org/T195686#4235026 (10Krenair) [20:04:30] 10Beta-Cluster-Infrastructure: Move puppetmaster to Stretch - https://phabricator.wikimedia.org/T195686#4235034 (10Krenair) [20:09:01] RECOVERY - Puppet errors on deployment-webperf01 is OK: OK: Less than 1.00% above the threshold [0.0] [20:14:56] 10Beta-Cluster-Infrastructure: Move puppetmaster to Stretch - https://phabricator.wikimedia.org/T195686#4235035 (10Krenair) Have put in temporary overrides for the instances with broken puppet while I sort that out, new default: https://wikitech.wikimedia.org/w/index.php?title=Hiera:Deployment-prep&diff=next&old... [20:17:52] 10Beta-Cluster-Infrastructure: Move puppetmaster to Stretch - https://phabricator.wikimedia.org/T195686#4235036 (10Krenair) [20:33:44] Krenair: wooo, nice :))) [20:38:43] legoktm hi [20:38:56] hi paladox [20:39:08] legoktm wondering would you be able to review https://gerrit.wikimedia.org/r/#/c/435689/ please? :) [20:39:30] i've tested it [20:39:34] and confirmed it works [20:40:01] 10Beta-Cluster-Infrastructure, 10Operations, 10media-storage, 10Patch-For-Review, 10Puppet: Puppet broken on deployment-ms-be0[34] with evaluation error in swift module - https://phabricator.wikimedia.org/T184236#4235037 (10Krenair) Rebased the patches and re-cherry-picked [20:40:05] I'm about to leave for lunch, I'll look when I come back [20:40:08] ok [20:40:10] thanks! [20:42:55] legoktm, I'm not quite done yet :) [20:44:31] 10Beta-Cluster-Infrastructure: Move puppetmaster to Stretch - https://phabricator.wikimedia.org/T195686#4235038 (10Krenair) [20:46:02] RECOVERY - Puppet errors on deployment-ms-be03 is OK: OK: Less than 1.00% above the threshold [0.0] [20:51:54] RECOVERY - Puppet errors on deployment-ms-be04 is OK: OK: Less than 1.00% above the threshold [0.0] [20:55:10] 10Beta-Cluster-Infrastructure, 10Operations, 10Patch-For-Review, 10Prometheus-metrics-monitoring, 10User-fgiunchedi: Move deployment-prep redis instances to stretch - https://phabricator.wikimedia.org/T179371#4235039 (10Krenair) @fgiunchedi: So we basically need to find someone to review the puppet patch... [20:56:35] 10Beta-Cluster-Infrastructure: Move puppetmaster to Stretch - https://phabricator.wikimedia.org/T195686#4235050 (10Krenair) [21:24:43] RECOVERY - Puppet errors on deployment-deploy1001 is OK: OK: Less than 1.00% above the threshold [0.0] [21:26:36] 10Beta-Cluster-Infrastructure, 10Patch-For-Review: Move puppetmaster to Stretch - https://phabricator.wikimedia.org/T195686#4235156 (10Krenair) [21:30:42] PROBLEM - Puppet errors on deployment-deploy1001 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [21:34:01] 10Release-Engineering-Team (Kanban), 10User-greg, 10Wikimedia-extension-review-queue: Re-think/factor [[mw:Review queue]] and generally the process of getting new code into production - https://phabricator.wikimedia.org/T195244#4235175 (10Aklapper) > Why is the wiki page named "Review queue"? Not an answer,... [21:39:51] 10Project-Admins: setup a decommission project for server decommissions - https://phabricator.wikimedia.org/T193816#4235189 (10Aklapper) 05Open>03Resolved @RobH: Garr, looking at https://phabricator.wikimedia.org/project/manage/3364/ now it looks like a great project description (thanks!) so I'm not sure why... [21:48:28] 10Beta-Cluster-Infrastructure, 10Patch-For-Review: Move puppetmaster to Stretch - https://phabricator.wikimedia.org/T195686#4235212 (10Krenair) [22:08:02] 10Beta-Cluster-Infrastructure, 10Patch-For-Review: Move puppetmaster to Stretch - https://phabricator.wikimedia.org/T195686#4235248 (10Krenair) [22:13:08] 10Beta-Cluster-Infrastructure, 10Patch-For-Review: Move puppetmaster to Stretch - https://phabricator.wikimedia.org/T195686#4235253 (10Krenair) [22:13:32] 10Beta-Cluster-Infrastructure, 10Patch-For-Review: Move puppetmaster to Stretch - https://phabricator.wikimedia.org/T195686#4234225 (10Krenair) [22:20:33] 10Beta-Cluster-Infrastructure, 10Patch-For-Review: Move puppetmaster to Stretch - https://phabricator.wikimedia.org/T195686#4235262 (10Krenair) [22:25:13] 10Beta-Cluster-Infrastructure, 10Patch-For-Review: Move puppetmaster to Stretch - https://phabricator.wikimedia.org/T195686#4235263 (10Krenair) ```krenair@deployment-cumin:~$ sudo cumin '*' 'grep "server = " /etc/puppet/puppet.conf' 75 hosts will be targeted: deployment-apertium02.deployment-prep.eqiad.wmflabs... [22:29:14] 10Project-Admins: Create a new phabricator project for Lingua Libre - https://phabricator.wikimedia.org/T195565#4235264 (100x010C) Huuu, this is the repository of an old version of Lingua Libre, deprecated since one year. With the project grant linked in the description, when we moved to the current version, whi... [22:30:29] PROBLEM - Host deployment-puppetmaster02 is DOWN: CRITICAL - Host Unreachable (10.68.21.200) [22:33:28] 10Project-Admins: Create a new phabricator project for Lingua Libre - https://phabricator.wikimedia.org/T195565#4235279 (10Aklapper) >>! In T195565#4235264, @0x010C wrote: > Huuu, this is the repository of an old version of Lingua Libre, deprecated since one year. Heh. That makes me wonder what "LinguaLibre" on... [22:44:21] 10Beta-Cluster-Infrastructure, 10Patch-For-Review: Move puppetmaster to Stretch - https://phabricator.wikimedia.org/T195686#4235281 (10Krenair) 05Open>03Resolved Request ID Action Start Time User ID Message req-c1d7511f-d09e-45fd-aef8-a3748c4a3b66 Stop 27 May 2018, 10:33 p.m. krenair - Will delete the ol... [22:47:10] 10Project-Admins: Create a new phabricator project for Lingua Libre - https://phabricator.wikimedia.org/T195565#4235283 (100x010C) Huuuu (bis), I've completely forgotten this page... Lingua Libre is not ready (yet!) to welcome developers with little experience of MediaWiki, so I removed it from the page. [23:00:34] 10Gerrit, 10Developer-Relations, 10Documentation: Gerrit's test instance gerrit.git.wmflabs.org is not quite visible in the docs; no clear instructions how to use it - https://phabricator.wikimedia.org/T193788#4235287 (10Aklapper) I guess these docs should live on https://wikitech.wikimedia.org/wiki/Gerrit w... [23:12:52] 10Phabricator, 10Project-Admins, 10Security-Team: Create Tag for WMDE Fundraising Security issues - https://phabricator.wikimedia.org/T194286#4194474 (10Reedy) Yeah, we don't get any sort of high priority alerts or anything. And we don't have any reports...