[01:04:24] 06Release-Engineering-Team, 10Wikimedia-Developer-Summit, 15User-greg: Facilitate Wikidev'17 main topic "How to manage our technical debt" - https://phabricator.wikimedia.org/T147937#2715178 (10RobLa-WMF) Last year, @MrStradivarius suggested "Standardise procedures for deprecating public-facing code" (T11438... [01:13:03] PROBLEM - Puppet run on deployment-elastic08 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [01:53:03] RECOVERY - Puppet run on deployment-elastic08 is OK: OK: Less than 1.00% above the threshold [0.0] [02:23:33] 10Beta-Cluster-Infrastructure: DBQueryError when creating account on Beta - https://phabricator.wikimedia.org/T148111#2715246 (10Mattflaschen-WMF) [02:27:33] 10Beta-Cluster-Infrastructure: deployment-fluorine02 does not have logs - https://phabricator.wikimedia.org/T146723#2715259 (10Mattflaschen-WMF) [02:27:35] 10Beta-Cluster-Infrastructure: DBQueryError when creating account on Beta - https://phabricator.wikimedia.org/T148111#2715258 (10Mattflaschen-WMF) [02:32:54] 10Beta-Cluster-Infrastructure: deployment-fluorine02 does not have logs - https://phabricator.wikimedia.org/T146723#2715266 (10Mattflaschen-WMF) [02:32:56] 10Beta-Cluster-Infrastructure: DBQueryError when creating account on Beta - https://phabricator.wikimedia.org/T148111#2715265 (10Mattflaschen-WMF) [02:37:49] 10Beta-Cluster-Infrastructure: DBQueryError when creating account on Beta - https://phabricator.wikimedia.org/T148111#2715267 (10Mattflaschen-WMF) [02:38:22] 10Beta-Cluster-Infrastructure: DBQueryError when creating account on Beta - https://phabricator.wikimedia.org/T148111#2715246 (10Mattflaschen-WMF) @bd808, it doesn't seem like the actual DB error is available on logstash, just that there *was* a DB error. Am I missing something? [03:04:18] PROBLEM - Puppet run on deployment-eventlogging04 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [03:10:37] 10Beta-Cluster-Infrastructure: DBQueryError when creating account on Beta, or logging into newly-created accounts - https://phabricator.wikimedia.org/T148111#2715286 (10Mattflaschen-WMF) [03:21:15] 10Beta-Cluster-Infrastructure: DBQueryError when creating account on Beta, or logging into newly-created accounts - https://phabricator.wikimedia.org/T148111#2715287 (10Mattflaschen-WMF) The same happens when I try to log into newly created account on Beta (both created in the web UI, or created with createAndPr... [03:22:52] 10Beta-Cluster-Infrastructure: deployment-fluorine02 does not have logs - https://phabricator.wikimedia.org/T146723#2715289 (10Mattflaschen-WMF) [03:22:54] 10Beta-Cluster-Infrastructure: DBQueryError when creating account on Beta, or logging into newly-created accounts - https://phabricator.wikimedia.org/T148111#2715288 (10Mattflaschen-WMF) [03:44:19] RECOVERY - Puppet run on deployment-eventlogging04 is OK: OK: Less than 1.00% above the threshold [0.0] [03:46:41] PROBLEM - Puppet run on deployment-mathoid is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [04:21:43] RECOVERY - Puppet run on deployment-mathoid is OK: OK: Less than 1.00% above the threshold [0.0] [04:31:03] 10Beta-Cluster-Infrastructure: New wiki cluster wikipedia indonesian language - https://phabricator.wikimedia.org/T143557#2715346 (10Mbrt) Hi I want to request because for test about mediawiki script, gadget or else. [04:32:07] 10Beta-Cluster-Infrastructure: New wiki cluster wikipedia indonesian language - https://phabricator.wikimedia.org/T143557#2715347 (10Mbrt) @hashar @Aklapper Hi I want to request because for test about mediawiki script, gadget or else. [05:13:49] 03Scap3, 10ContentTranslation-CXserver, 10MediaWiki-extensions-ContentTranslation, 06Services, 15User-mobrovac: Enable Scap3 config deploys for CXServer - https://phabricator.wikimedia.org/T147634#2715364 (10KartikMistry) @mobrovac What's deadline for this? We can do this in the #language-q2-2016-17_spri... [05:28:43] 06Release-Engineering-Team, 10DBA, 10Phabricator, 13Patch-For-Review, 07Wikimedia-Incident: Contention on search phabricator database creating full phabricator outages - https://phabricator.wikimedia.org/T146673#2715369 (10MZMcBride) >>! In T146673#2704843, @jcrespo wrote: > * Does InnoDB FULLTEXT respec... [06:48:30] Yippee, build fixed! [06:48:30] Project selenium-Wikibase » chrome,beta,Linux,contintLabsSlave && UbuntuTrusty build #141: 09FIXED in 2 hr 8 min: https://integration.wikimedia.org/ci/job/selenium-Wikibase/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/141/ [06:58:44] PROBLEM - Puppet run on integration-slave-jessie-android is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [07:18:46] RECOVERY - Puppet run on integration-slave-jessie-android is OK: OK: Less than 1.00% above the threshold [0.0] [07:18:59] 06Release-Engineering-Team, 10DBA, 10Phabricator, 13Patch-For-Review, 07Wikimedia-Incident: Contention on search phabricator database creating full phabricator outages - https://phabricator.wikimedia.org/T146673#2715492 (10Paladox) That task is https://secure.phabricator.com/T10642 [08:14:16] 06Release-Engineering-Team, 10Phabricator: Phab Advanced Search no longer showing typical results - https://phabricator.wikimedia.org/T146789#2715538 (10jcrespo) [09:08:17] 10Beta-Cluster-Infrastructure: New wiki cluster wikipedia indonesian language - https://phabricator.wikimedia.org/T143557#2715604 (10Mbrt) 05declined>03Open p:05Normal>03Triage [09:08:54] 10Beta-Cluster-Infrastructure: New wiki cluster wikipedia indonesian language - https://phabricator.wikimedia.org/T143557#2571665 (10Mbrt) re-open [09:26:37] PROBLEM - Puppet run on deployment-kafka05 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [09:38:18] PROBLEM - Puppet run on deployment-restbase02 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [09:41:09] 03Scap3, 10Citoid, 10ContentTranslation-CXserver, 10ContentTranslation-Deployments, and 7 others: Depool and repool SCB services during deploys - https://phabricator.wikimedia.org/T144602#2715710 (10Joe) [09:42:30] PROBLEM - Puppet run on deployment-restbase01 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [09:42:40] PROBLEM - Puppet run on deployment-mathoid is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [09:49:41] (03PS1) 10Hashar: tox doc publishing jobs to Nodepool/Jessie [integration/config] - 10https://gerrit.wikimedia.org/r/315919 (https://phabricator.wikimedia.org/T119140) [09:51:33] PROBLEM - Puppet run on deployment-apertium01 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [09:52:39] PROBLEM - Puppet run on deployment-apertium02 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [09:53:26] (03CR) 10Hashar: [C: 032] tox doc publishing jobs to Nodepool/Jessie [integration/config] - 10https://gerrit.wikimedia.org/r/315919 (https://phabricator.wikimedia.org/T119140) (owner: 10Hashar) [09:53:46] PROBLEM - Puppet run on deployment-sca03 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [09:54:23] (03Merged) 10jenkins-bot: tox doc publishing jobs to Nodepool/Jessie [integration/config] - 10https://gerrit.wikimedia.org/r/315919 (https://phabricator.wikimedia.org/T119140) (owner: 10Hashar) [09:54:46] PROBLEM - Puppet run on deployment-mx is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [09:57:32] PROBLEM - Puppet run on deployment-sca02 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [09:58:10] (03PS3) 10Paladox: Update oojs-ui-coverage test to jessie [integration/config] - 10https://gerrit.wikimedia.org/r/296057 (https://phabricator.wikimedia.org/T138666) [09:59:10] (03CR) 10jenkins-bot: [V: 04-1] Update oojs-ui-coverage test to jessie [integration/config] - 10https://gerrit.wikimedia.org/r/296057 (https://phabricator.wikimedia.org/T138666) (owner: 10Paladox) [09:59:15] (03PS4) 10Paladox: Update oojs-ui-coverage test to jessie [integration/config] - 10https://gerrit.wikimedia.org/r/296057 (https://phabricator.wikimedia.org/T138666) [10:00:14] PROBLEM - Puppet run on deployment-sca01 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [10:00:43] (03CR) 10Hashar: "Tested and refreshed both repo :]" [integration/config] - 10https://gerrit.wikimedia.org/r/315919 (https://phabricator.wikimedia.org/T119140) (owner: 10Hashar) [10:01:36] RECOVERY - Puppet run on deployment-kafka05 is OK: OK: Less than 1.00% above the threshold [0.0] [10:06:06] (03PS5) 10Paladox: Update oojs-ui-coverage test to nodepool jessie [integration/config] - 10https://gerrit.wikimedia.org/r/296057 (https://phabricator.wikimedia.org/T138666) [10:08:21] 10Beta-Cluster-Infrastructure: New wiki cluster wikipedia indonesian language - https://phabricator.wikimedia.org/T143557#2715805 (10Luke081515) 05Open>03declined You can do that at existing wikis at the beta-Cluster. We don't Need a new wiki for that. [10:29:44] RECOVERY - Puppet run on deployment-mx is OK: OK: Less than 1.00% above the threshold [0.0] [10:45:59] PROBLEM - Puppet run on repository is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [11:14:13] any objections if I run update-ca-certificates --verbose --fresh on deployment-tin I run into the same cert issue (T145609) ? [11:20:58] RECOVERY - Puppet run on repository is OK: OK: Less than 1.00% above the threshold [0.0] [11:23:32] (03PS1) 10Hashar: mw-run-update-script allow unset ZUUL_PROJECT [integration/jenkins] - 10https://gerrit.wikimedia.org/r/315932 [11:24:11] (03CR) 10Hashar: [C: 032] mw-run-update-script allow unset ZUUL_PROJECT [integration/jenkins] - 10https://gerrit.wikimedia.org/r/315932 (owner: 10Hashar) [11:24:12] hashar: o/ [11:24:40] dcausse: o/ [11:24:44] (03Merged) 10jenkins-bot: mw-run-update-script allow unset ZUUL_PROJECT [integration/jenkins] - 10https://gerrit.wikimedia.org/r/315932 (owner: 10Hashar) [11:24:55] I have the same cert issue cert issue on deployment-tin (T145609) [11:25:07] lets hook in #wikimedia-operations [11:25:11] there are some ssl/cert expert there :] [11:25:17] ah [11:25:33] ah maybe related to the GlobalSign problem [11:30:05] !log deployment-prep running sudo update-ca-certificates --fresh on deployment-ton to fix curl error code 60 in cirrus maint script (T145609) [11:30:09] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [11:42:46] dcausse, hashar: I uploaded a patch to T145609, but I'm not entirely sure it is the correct solution in all cases... [11:44:04] gehel: thanks! [11:47:45] gehel: no idea really :( [11:47:55] gehel: I am probably good at finding out root causes [11:48:03] but less at proper fixing [11:48:08] I guess if we can manage to reproduce easily [11:48:20] hashar: well, the hard part is finding the root cause! [11:48:25] then apply your patch and confirm that the cert symlinks are properly refreshed, I guess it will be all good [11:48:32] yeah depend :] [11:48:39] the hard part for me is to find the right fix [11:48:55] hashar: I'm pretty sure that the patch fixes this specific issue, I'm not sure it will not break something else... [11:49:19] and that second part is hard to test, since I have no idea what it might break :) [11:55:16] !log ban deployment-elastic08 in preparation of decommission - T147777 [11:58:48] 10Continuous-Integration-Config: Migrate Jenkins job "operations-puppet-catalog-compiler" to Jenkins Job Builder - https://phabricator.wikimedia.org/T97513#1244534 (10Paladox) Should we also create a new pipeline and whitelist for this? Or do we just want a new pipeline with a command like check catalog or somet... [12:00:51] 10Continuous-Integration-Config, 10MediaWiki-extensions-Babel, 10MediaWiki-extensions-CLDR, 10MediaWiki-extensions-Translate, and 2 others: Add i18n related extensions to mediawiki-extensions-(php55|php70||hhvm) - https://phabricator.wikimedia.org/T86930#2715982 (10Paladox) [12:14:29] gehel: yeah that is why I said it is often harder to find the proper fix :( [12:14:56] hashar: I only blame my ignorance of SSL certs in this case :) [12:21:08] (03Draft1) 10Paladox: Add mw ext Translate to mediawiki-gate [integration/config] - 10https://gerrit.wikimedia.org/r/315941 (https://phabricator.wikimedia.org/T86930) [12:21:10] (03Draft2) 10Paladox: Add mw ext Translate to mediawiki-gate [integration/config] - 10https://gerrit.wikimedia.org/r/315941 (https://phabricator.wikimedia.org/T86930) [12:22:38] Project selenium-GettingStarted » firefox,beta,Linux,contintLabsSlave && UbuntuTrusty build #176: 04FAILURE in 38 sec: https://integration.wikimedia.org/ci/job/selenium-GettingStarted/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/176/ [12:25:18] 10Continuous-Integration-Config, 10BlueSky, 13Patch-For-Review: Fix structure tests for all skins - https://phabricator.wikimedia.org/T113860#2716141 (10matmarex) [12:30:14] (03PS1) 10Hashar: (WIP) MediaWiki tests with composer (WIP) [integration/config] - 10https://gerrit.wikimedia.org/r/315942 [12:52:44] 10Beta-Cluster-Infrastructure, 10CirrusSearch, 06Discovery, 06Operations, 13Patch-For-Review: Puppet sslcert::ca does not refresh the certificate symlinks when a .crt is updated - https://phabricator.wikimedia.org/T145609#2716391 (10Gehel) As pointed by @faidon, when running `update-ca-certificates` with... [12:54:22] 10Beta-Cluster-Infrastructure: New wiki cluster wikipedia indonesian language - https://phabricator.wikimedia.org/T143557#2716401 (10Mbrt) @Luke081515 can you change me to bureaucrat at deployment.? [12:54:25] (03CR) 10Nikerabbit: [C: 031] "If it works, why not." [integration/config] - 10https://gerrit.wikimedia.org/r/315941 (https://phabricator.wikimedia.org/T86930) (owner: 10Paladox) [12:55:17] (03PS9) 10Hashar: Create 'puppet-doc' macro and use it [integration/config] - 10https://gerrit.wikimedia.org/r/204983 (owner: 10Legoktm) [12:58:57] (03CR) 10Hashar: [C: 032] "Long overdue. Almost a noop, I have refreshed the couple jobs:" [integration/config] - 10https://gerrit.wikimedia.org/r/204983 (owner: 10Legoktm) [13:00:19] (03Merged) 10jenkins-bot: Create 'puppet-doc' macro and use it [integration/config] - 10https://gerrit.wikimedia.org/r/204983 (owner: 10Legoktm) [13:05:19] (03Abandoned) 10Hashar: [WIP] Need to lint some yaml [integration/config] - 10https://gerrit.wikimedia.org/r/295950 (owner: 10Awight) [13:06:41] (03PS5) 10Hashar: Add test to verify every extension+skin has an entry in zuul/layout.yaml [integration/config] - 10https://gerrit.wikimedia.org/r/198185 (owner: 10Legoktm) [13:07:44] (03CR) 10jenkins-bot: [V: 04-1] Add test to verify every extension+skin has an entry in zuul/layout.yaml [integration/config] - 10https://gerrit.wikimedia.org/r/198185 (owner: 10Legoktm) [13:23:11] Project beta-code-update-eqiad build #125725: 04FAILURE in 10 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/125725/ [13:23:24] :( [13:29:23] Project beta-code-update-eqiad build #125726: 04STILL FAILING in 10 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/125726/ [13:33:10] Project beta-code-update-eqiad build #125727: 04STILL FAILING in 10 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/125727/ [13:34:16] * hashar whistles [13:36:19] Project beta-code-update-eqiad build #125728: 04STILL FAILING in 23 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/125728/ [13:40:58] Yippee, build fixed! [13:40:59] Project beta-code-update-eqiad build #125729: 09FIXED in 2 min 4 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/125729/ [13:43:35] Project beta-scap-eqiad build #124356: 04FAILURE in 2 min 36 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/124356/ [13:43:58] stupid race [13:47:44] Yippee, build fixed! [13:47:45] Project beta-scap-eqiad build #124357: 09FIXED in 2 min 16 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/124357/ [13:58:05] 10Continuous-Integration-Infrastructure, 05Continuous-Integration-Scaling: Reduce CI permanent slaves - https://phabricator.wikimedia.org/T148183#2716800 (10hashar) [13:59:25] 10Continuous-Integration-Infrastructure, 05Continuous-Integration-Scaling: Reduce CI permanent slaves - https://phabricator.wikimedia.org/T148183#2716824 (10hashar) p:05Triage>03Low [14:03:53] PROBLEM - Puppet run on deployment-redis01 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [14:14:00] PROBLEM - Puppet run on deployment-elastic08 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [14:16:52] 10Gerrit, 06Release-Engineering-Team, 06Operations: Build warm slave for Gerrit in Dallas - https://phabricator.wikimedia.org/T148186#2716848 (10demon) [14:19:47] 10Gerrit, 06Release-Engineering-Team, 06Operations, 10hardware-requests: Requesting 1 spare misc box for Gerrit in codfw - https://phabricator.wikimedia.org/T148187#2716861 (10demon) [14:37:29] bah [14:37:33] Mediawiki is broken https://integration.wikimedia.org/ci/job/mediawiki-release-integration/lastCompletedBuild/testReport/ :( [14:39:10] PROBLEM - Puppet run on integration-slave-trusty-1011 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [14:43:55] RECOVERY - Puppet run on deployment-redis01 is OK: OK: Less than 1.00% above the threshold [0.0] [14:44:53] PROBLEM - Puppet run on integration-slave-jessie-1002 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [14:45:33] PROBLEM - Puppet run on integration-slave-trusty-1003 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [14:46:03] PROBLEM - Puppet run on integration-slave-jessie-1001 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [14:46:35] PROBLEM - Puppet run on zuul-dev-jessie is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [14:46:38] .... [14:46:46] Duplicate declaration: Class[Zuul] is already declared in file /etc/puppet/modules/role/manifests/ci/slave/labs.pp:43; cannot redeclare at /etc/puppet/modules/role/manifests/ci/slave/localbrowser.pp:11 [14:53:14] PROBLEM - Puppet run on integration-slave-trusty-1012 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [14:53:32] PROBLEM - Puppet run on integration-slave-trusty-1001 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [14:54:14] PROBLEM - Puppet run on integration-slave-trusty-1016 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [14:57:50] PROBLEM - Puppet run on integration-slave-trusty-1004 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [14:59:46] PROBLEM - Puppet run on integration-slave-jessie-1003 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [15:00:08] I never quite understood the that role::ci::slave::localbrowser [15:02:00] the only thing it really does is contint::browsers [15:02:01] bah [15:03:22] PROBLEM - Puppet run on integration-slave-trusty-1006 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [15:03:32] (03PS2) 10Hashar: (WIP) MediaWiki tests with composer (WIP) [integration/config] - 10https://gerrit.wikimedia.org/r/315942 [15:11:03] Error: Could not retrieve catalog from remote server: Error 400 on SERVER: Duplicate declaration: Class[Zuul] is already declared in file /etc/puppet/modules/role/manifests/ci/slave/labs.pp:43; cannot redeclare at /etc/puppet/modules/role/manifests/ci/slave/browsertests.pp:11 on node integration-slave-jessie-1001.integration.eqiad.wmflabs [15:11:05] yeah [15:11:07] that never ends [15:24:02] RECOVERY - Puppet run on deployment-elastic08 is OK: OK: Less than 1.00% above the threshold [0.0] [15:38:09] hashar i see one of your patches was finally merged for zuul [15:38:13] RECOVERY - Puppet run on integration-slave-trusty-1012 is OK: OK: Less than 1.00% above the threshold [0.0] [15:38:14] hiera integration [15:38:15] :) [15:38:21] RECOVERY - Puppet run on integration-slave-trusty-1006 is OK: OK: Less than 1.00% above the threshold [0.0] [15:38:33] RECOVERY - Puppet run on integration-slave-trusty-1001 is OK: OK: Less than 1.00% above the threshold [0.0] [15:38:56] Yippee, build fixed! [15:38:57] Project selenium-MobileFrontend » chrome,beta,Linux,contintLabsSlave && UbuntuTrusty build #193: 09FIXED in 16 min: https://integration.wikimedia.org/ci/job/selenium-MobileFrontend/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/193/ [15:39:09] RECOVERY - Puppet run on integration-slave-trusty-1011 is OK: OK: Less than 1.00% above the threshold [0.0] [15:39:13] RECOVERY - Puppet run on integration-slave-trusty-1016 is OK: OK: Less than 1.00% above the threshold [0.0] [15:39:45] RECOVERY - Puppet run on integration-slave-jessie-1003 is OK: OK: Less than 1.00% above the threshold [0.0] [15:39:49] PROBLEM - Puppet run on deployment-phab02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [15:39:53] RECOVERY - Puppet run on integration-slave-jessie-1002 is OK: OK: Less than 1.00% above the threshold [0.0] [15:40:33] RECOVERY - Puppet run on integration-slave-trusty-1003 is OK: OK: Less than 1.00% above the threshold [0.0] [15:41:01] RECOVERY - Puppet run on integration-slave-jessie-1001 is OK: OK: Less than 1.00% above the threshold [0.0] [15:42:49] RECOVERY - Puppet run on integration-slave-trusty-1004 is OK: OK: Less than 1.00% above the threshold [0.0] [15:45:41] Yippee, build fixed! [15:45:42] Project selenium-MobileFrontend » firefox,beta,Linux,contintLabsSlave && UbuntuTrusty build #193: 09FIXED in 23 min: https://integration.wikimedia.org/ci/job/selenium-MobileFrontend/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/193/ [15:51:04] PROBLEM - Puppet run on deployment-phab01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [15:53:34] yeahhh puppet fixed [15:54:14] PROBLEM - Puppet run on deployment-ores-redis is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [15:55:13] RECOVERY - Puppet run on deployment-lvs-realservertest2 is OK: OK: Less than 1.00% above the threshold [0.0] [16:01:46] (03PS1) 10Hashar: analytics/mediawiki-storage enable npm job [integration/config] - 10https://gerrit.wikimedia.org/r/315968 (https://phabricator.wikimedia.org/T148023) [16:02:06] (03CR) 10Hashar: [C: 032] analytics/mediawiki-storage enable npm job [integration/config] - 10https://gerrit.wikimedia.org/r/315968 (https://phabricator.wikimedia.org/T148023) (owner: 10Hashar) [16:03:33] (03Merged) 10jenkins-bot: analytics/mediawiki-storage enable npm job [integration/config] - 10https://gerrit.wikimedia.org/r/315968 (https://phabricator.wikimedia.org/T148023) (owner: 10Hashar) [16:05:27] 10Continuous-Integration-Config, 10Analytics-Dashiki, 13Patch-For-Review: Add CI job for analytics/mediawiki-storage - https://phabricator.wikimedia.org/T148023#2717037 (10hashar) 05Open>03Resolved a:03hashar With help of @mforns :] [16:08:42] 10Beta-Cluster-Infrastructure: DBQueryError when creating account on Beta, or logging into newly-created accounts - https://phabricator.wikimedia.org/T148111#2715246 (10Etonkovidova) One more thing to add: The error message on Special:CreateAccount page for attempting to create an account for an existing user w... [16:22:24] 10Continuous-Integration-Config: Migrate Jenkins job "operations-puppet-catalog-compiler" to Jenkins Job Builder - https://phabricator.wikimedia.org/T97513#2717160 (10hashar) I am not sure how doable it is really. Potentially we could use the parameter function in Zuul, it is passed the change queue item as firs... [16:25:08] 10Continuous-Integration-Infrastructure, 13Patch-For-Review, 07Puppet, 07Technical-Debt, 07Zuul: role::zuul::configuration should be replaced by hiera - https://phabricator.wikimedia.org/T139527#2717180 (10hashar) 05Open>03Resolved Solved with nice help from @joe and @Dzahn [16:25:50] 10Beta-Cluster-Infrastructure: DBQueryError when creating account on Beta, or logging into newly-created accounts - https://phabricator.wikimedia.org/T148111#2717186 (10bd808) >>! In T148111#2715267, @Mattflaschen-WMF wrote: > @bd808, it doesn't seem like the actual DB error is available on logstash, just that t... [16:26:57] PROBLEM - Puppet run on deployment-eventlogging03 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [16:34:14] RECOVERY - Puppet run on deployment-ores-redis is OK: OK: Less than 1.00% above the threshold [0.0] [16:48:26] 06Release-Engineering-Team, 10Wikimedia-Developer-Summit, 15User-greg: Facilitate Wikidev'17 main topic "How to manage our technical debt" - https://phabricator.wikimedia.org/T147937#2717272 (10greg) >>! In T147937#2715178, @RobLa-WMF wrote: > Perhaps we can spend some time before/during/after the summit com... [17:01:58] RECOVERY - Puppet run on deployment-eventlogging03 is OK: OK: Less than 1.00% above the threshold [0.0] [17:24:33] 10Continuous-Integration-Config, 10commit-message-validator, 10Pywikibot-core, 13Patch-For-Review: Check the style of the commit message - https://phabricator.wikimedia.org/T109119#2717361 (10Jdforrester-WMF) Two months later, with no issues, would it be possible to put it everywhere in non-voting mode and... [17:30:17] 10Continuous-Integration-Config, 10commit-message-validator, 10Pywikibot-core, 13Patch-For-Review: Check the style of the commit message - https://phabricator.wikimedia.org/T109119#2717389 (10greg) >>! In T109119#2717361, @Jdforrester-WMF wrote: > Two months later, with no issues, would it be possible to p... [17:37:35] PROBLEM - Puppet run on deployment-db04 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [17:47:55] PROBLEM - Puppet run on deployment-pdfrender is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [17:50:35] PROBLEM - Puppet run on deployment-mira is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [17:56:08] PROBLEM - Puppet run on deployment-tin is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [18:00:52] PROBLEM - Puppet run on deployment-db03 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [18:06:12] 10Beta-Cluster-Infrastructure: New wiki cluster wikipedia indonesian language - https://phabricator.wikimedia.org/T143557#2717471 (10Luke081515) Please read [[https://test.wikipedia.org/wiki/Wikipedia:What_Test_Wiki_is_not|this]] page first before making such requests. These rules apply for beta mostly too. Ther... [19:23:54] (03PS3) 10Hashar: (WIP) MediaWiki tests with composer (WIP) [integration/config] - 10https://gerrit.wikimedia.org/r/315942 [19:28:20] (03PS4) 10Hashar: (WIP) MediaWiki tests with composer (WIP) [integration/config] - 10https://gerrit.wikimedia.org/r/315942 [19:55:10] (03PS5) 10Hashar: (WIP) MediaWiki tests with composer (WIP) [integration/config] - 10https://gerrit.wikimedia.org/r/315942 [20:00:02] 10Beta-Cluster-Infrastructure: DBQueryError when creating account on Beta, or logging into newly-created accounts - https://phabricator.wikimedia.org/T148111#2718011 (10Mattflaschen-WMF) >>! In T148111#2717186, @bd808 wrote: >>>! In T148111#2715267, @Mattflaschen-WMF wrote: >> @bd808, it doesn't seem like the ac... [20:00:30] 10Beta-Cluster-Infrastructure: DBQueryError when creating account on Beta, or logging into newly-created accounts - https://phabricator.wikimedia.org/T148111#2718013 (10Mattflaschen-WMF) [20:00:42] 10Beta-Cluster-Infrastructure, 10MediaWiki-extensions-CentralAuth: DBQueryError when creating account on Beta, or logging into newly-created accounts - https://phabricator.wikimedia.org/T148111#2715246 (10Mattflaschen-WMF) a:03Mattflaschen-WMF [20:00:52] 10Beta-Cluster-Infrastructure, 10Android-app-goals-2016-Oct-Dec-Q2, 10MediaWiki-extensions-CentralAuth, 06Wikipedia-Android-App-Backlog: DBQueryError when creating account on Beta, or logging into newly-created accounts - https://phabricator.wikimedia.org/T148111#2718017 (10Mattflaschen-WMF) [20:04:23] 10Beta-Cluster-Infrastructure, 10MediaWiki-extensions-CentralAuth, 06Collaboration-Team-Triage (Collab-Team-Q2-Oct-Dec-2016): DBQueryError when creating account on Beta, or logging into newly-created accounts - https://phabricator.wikimedia.org/T148111#2718019 (10Mattflaschen-WMF) [20:06:03] 10Beta-Cluster-Infrastructure, 10MediaWiki-extensions-CentralAuth, 06Collaboration-Team-Triage (Collab-Team-Q2-Oct-Dec-2016): DBQueryError when creating account on Beta, or logging into newly-created accounts - https://phabricator.wikimedia.org/T148111#2718020 (10bd808) The `lu_local_id` stuff is new code Ce... [20:08:16] 10Beta-Cluster-Infrastructure, 10MediaWiki-extensions-CentralAuth, 06Collaboration-Team-Triage (Collab-Team-Q2-Oct-Dec-2016): DBQueryError when creating account on Beta, or logging into newly-created accounts - https://phabricator.wikimedia.org/T148111#2715246 (10Mattflaschen-WMF) >>! In T148111#2718020, @bd... [20:09:13] !log Applied CentralAuth's patch-lu_local_id.sql migration for T148111 [20:09:18] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [20:16:07] 10Beta-Cluster-Infrastructure, 10MediaWiki-extensions-CentralAuth: Add indices for local_user_id and global_user_id in Beta - https://phabricator.wikimedia.org/T148239#2718032 (10Mattflaschen-WMF) [20:17:58] 10Beta-Cluster-Infrastructure, 10MediaWiki-extensions-CentralAuth: Add indices for local_user_id and global_user_id in Beta - https://phabricator.wikimedia.org/T148239#2718071 (10Mattflaschen-WMF) [20:21:56] 10Beta-Cluster-Infrastructure, 10MediaWiki-extensions-CentralAuth, 06Collaboration-Team-Triage (Collab-Team-Q2-Oct-Dec-2016): DBQueryError when creating account on Beta, or logging into newly-created accounts - https://phabricator.wikimedia.org/T148111#2718120 (10Mattflaschen-WMF) [20:22:04] hashar Do you know what the scripts are for test operations-puppet-catalog-compiler? [20:22:09] For https://phabricator.wikimedia.org/T97513 [20:22:58] 10Beta-Cluster-Infrastructure, 10MediaWiki-extensions-CentralAuth, 06Collaboration-Team-Triage (Collab-Team-Q2-Oct-Dec-2016): DBQueryError when creating account on Beta, or logging into newly-created accounts - https://phabricator.wikimedia.org/T148111#2715246 (10Mattflaschen-WMF) [20:23:34] paladox: yeah let me find it [20:23:41] Ok, thanks [20:23:45] the thing is that it needs data from the production puppet [20:23:50] so the job runs on a different labs project [20:23:57] that is only accessible by a handful of people [20:23:58] 10Beta-Cluster-Infrastructure, 10MediaWiki-extensions-CentralAuth: Add indices for local_user_id and global_user_id in Beta - https://phabricator.wikimedia.org/T148239#2718133 (10Mattflaschen-WMF) [20:24:12] Oh [20:24:26] git clone ssh://gerrit.wikimedia.org:29418/operations/software/puppet-compiler.git [20:24:28] in python [20:24:37] I havent really looked at the code [20:24:41] oh [20:24:48] but the idea is that it needs metadata from the production puppet [20:24:55] oh [20:25:10] stuff that say for example that gallium.wikimedia.org is Ubuntu , Precise , 8GB ram in cluster misc etc [20:25:16] they are known as "facts" [20:25:21] provided by a tool named facter [20:25:24] Oh [20:25:27] + probably bunch of other things [20:25:43] 10Beta-Cluster-Infrastructure, 10MediaWiki-extensions-CentralAuth, 15User-bd808: DBQueryError when creating account on Beta, or logging into newly-created accounts - https://phabricator.wikimedia.org/T148111#2718148 (10Mattflaschen-WMF) a:05Mattflaschen-WMF>03bd808 [20:25:48] then the puppet-compiler create a puppet catalog for the node asked using tip of "production" branch [20:26:00] create a second using the patch [20:26:09] then does some difference between boths and report [20:26:31] yep, woulden zuul be able to reach that instance, or can it only work from users entering jenkins interface? [20:27:22] Oh if jenkins can do it carn't we use the variables in jenkins job builder?, or is it not that easy to migrate? [20:28:59] compiler02.puppet3-diffs.eqiad.wmflabs [20:30:20] the instance is connected to the Jenkins master [20:30:35] Yep [20:30:40] zuul trigger the job in the jenkins master which trigger the build on the compiler [20:30:44] so that part is all fine [20:30:48] ok [20:30:48] the thing [20:31:05] is finding out how whether Zuul knows about the comment entered by the user [20:31:07] well it does [20:31:10] eg "recheck" [20:31:14] Oh [20:31:43] but when it dispatch the job to jenkins, would it be able to parse the comment and dynamically craft the node to compile against [20:31:44] hashar we can setup a test test, so that we can test it [20:32:52] 10Beta-Cluster-Infrastructure, 10MediaWiki-extensions-CentralAuth: Add indices for local_user_id and global_user_id in Beta - https://phabricator.wikimedia.org/T148239#2718032 (10Krenair) Doesn't the update script take care of this? [20:33:05] (03PS1) 10Paladox: [operations-puppet-catalog-compiler] Adding it to jenkins job builder [integration/config] - 10https://gerrit.wikimedia.org/r/315994 [20:33:39] (03PS2) 10Paladox: [operations-puppet-catalog-compiler] Adding it to jenkins job builder [integration/config] - 10https://gerrit.wikimedia.org/r/315994 (https://phabricator.wikimedia.org/T97513) [20:34:40] 10Beta-Cluster-Infrastructure, 10MediaWiki-extensions-CentralAuth: Add indices for local_user_id and global_user_id in Beta - https://phabricator.wikimedia.org/T148239#2718177 (10Krenair) Ugh, right, CentralAuth. Never mind. [20:34:43] hashar ^^^ [20:34:48] That should not wipe anything [20:34:57] it will just add a new one, to test with [20:37:15] !log Applied CentralAuth's patch-lu_local_id.sql migration for T148111, to sql --write [20:37:20] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [20:40:00] hashar oh it exports NODES= which changes [20:40:06] Not sure how we will detect that [20:40:40] paladox: I thoguht of something like: puppet compile node.wikimedia.org [20:40:46] Oh [20:40:47] eg one comment that in Gerrit [20:40:47] :) [20:40:54] that comment-added is received by Zuul [20:40:58] Yep [20:41:16] to be recognized by a pipeline (maybe a dedicated one) [20:41:20] Yep [20:41:24] I was thinking that too [20:41:28] a dedicated pipeline [20:41:40] then zuul invokes the zuul parameter function and pass it the change in the queue + the job object [20:41:45] Project selenium-Echo » chrome,beta,Linux,contintLabsSlave && UbuntuTrusty build #179: 04FAILURE in 45 sec: https://integration.wikimedia.org/ci/job/selenium-Echo/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/179/ [20:41:52] Project selenium-Echo » firefox,beta,Linux,contintLabsSlave && UbuntuTrusty build #179: 04FAILURE in 52 sec: https://integration.wikimedia.org/ci/job/selenium-Echo/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/179/ [20:41:53] Yep. [20:42:05] but I dont think any of them have the event (and thus the comment) that triggered it [20:42:14] gotta dig what the ChangeItem has [20:42:26] whether it keeps the original event [20:42:32] So we could support mutiple hosts in puppet compile node.wikimedia.org for one test [20:42:34] oh [20:43:00] the thing is dont know whether the comment added is carried to the parameter function [20:43:05] gotta dig in Zuul [20:43:53] Ok [20:45:49] hashar we can always create a test on our test instance, and test it, ie test doing a comment and see if it reaches parameter_function? [20:48:34] 10Beta-Cluster-Infrastructure, 10MediaWiki-extensions-CentralAuth, 15User-bd808: DBQueryError when creating account on Beta, or logging into newly-created accounts - https://phabricator.wikimedia.org/T148111#2718235 (10Mattflaschen-WMF) >>! In T148111#2717050, @Etonkovidova wrote: > One more thing to add: >... [20:50:46] paladox: in integration/config there are a bunch of tests doing that [20:50:53] namely forging an event, entering it in a pipeline [20:50:55] Yep [20:50:57] and then doing some assertion [20:51:05] so maybe one can use the testsuite to confirm [20:51:06] the beta database is locked [20:51:11] "The database has been automatically locked while the replica database servers catch up to the master. " [20:51:17] can somebody take a look? [20:51:21] occurs since 10 minutes [20:51:48] Oh [20:53:06] Luke081515: no idea how to even diagnose that :( [20:53:47] !log Dropped lu_local_id, lu_global_id from replica db which were added improperly [20:53:52] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [20:55:31] bd808: db locked? [20:55:36] beat* [20:55:38] beta* [20:55:48] yeah so [20:55:55] that is all the difference between me having no clue [20:55:57] yeah. I think matt_flaschen and I caused it [20:56:10] and bd808 figuring it out all in less than 5 mins [20:56:29] not sure how to fix. I added columns to the slave db, matt properly added them to the master [20:56:46] I just dropped them from the slave but lock seems to persist [20:56:55] restart? [20:58:44] ori, let me check if either of the DBAs are around. They're on IRC, pinging them now. [21:01:33] hashar i've updated all the jobs now [21:01:38] how do i run the test script [21:05:33] paladox: which test suite? [21:05:48] and then doing some assertion [21:05:48] so maybe one can use the testsuite to confirm [21:05:48] ori, okay, apparently they're not around. I don't really know what effect restarting has, but if you think that's the best option, go for it, IMHO. [21:05:50] you said ^^ [21:06:03] so i presume the one in integration/config [21:06:04] ? [21:06:46] paladox: you need python2.7 and the python software "tox" [21:06:55] Oh [21:07:07] I think that is installed [21:07:11] tox -e py27 -- --help [21:07:23] that ask tox to setup/run in the "py27" environment [21:07:27] -- is a separator [21:07:32] then --help is passed to the test command [21:07:48] the test command is defined in tox.ini as: commands = nosetests {posargs} [21:08:00] oh [21:08:04] {posargs} is a magic token, that is replaced by whatever you pass to tox after the "--" [21:08:48] oh [21:09:08] tox -e py27 -- tests/test_zuul_scheduler.py:TestZuulScheduler.test_recheck_comment_trusted_user [21:09:30] nosetests (the test command) will look at test in the file tests/test_zuul_scheduler.py [21:09:54] and filter a test in the class "TestZuulScheduler" that is named "test_recheck_comment_trusted_user" [21:10:01] copy paste that somewhere for later reference :] [21:10:07] this way it runs a single test [21:10:34] ori, bd808, I'm going to try START SLAVE [21:10:47] I think it should work now that bd808 deleted the columns from the slave. [21:10:56] Last_Error: Error 'Duplicate column name 'lu_local_id'' on query. Default database: 'centralauth'. Query: 'ALTER TABLE localuser [21:11:01] Slave_IO_Running: Yes [21:11:05] Slave_SQL_Running: No [21:11:24] http://dev.mysql.com/doc/refman/5.7/en/replication-problems.html [21:12:25] matt_flaschen: which host is the slave? [21:12:44] oh [21:12:53] ori, don't know, I just ran START SLAVE, let's see if that works (I access replica/slave with sql centralauth' on deployment-tin) [21:12:59] sql centralauth [21:13:13] That worked [21:13:40] !log Ran START SLAVE to restart replication after columns created directly on replica were deleted. [21:13:44] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [21:20:26] hashar it says it requires tox 1.9, but i have 1.8 [21:23:10] 10Beta-Cluster-Infrastructure: Enable Quiz Extension on ca.wikipedia.beta.wmflabs.org for testing - https://phabricator.wikimedia.org/T142692#2718289 (10Krenair) [21:31:16] matt_flaschen: looks like that fixed it. Thanks for cleaning up my mess [21:32:57] hashar it seems im finding it hard to install tox [21:33:05] jessie one is out dated [21:34:49] bd808, no problem. I think I have done the same thing in the past (or something similar). And now I know how to restart replication. [21:35:00] bd808, it really shouldn't let you run updates on slaves. I think I filed a bug about that. [21:36:41] oh python -m tox [21:36:41] works [21:37:17] 10Beta-Cluster-Infrastructure, 10DBA, 06Operations: Possible to run writes (e.g. UPDATE) on slave - https://phabricator.wikimedia.org/T110115#2718307 (10Mattflaschen-WMF) This caused complications when trying to fix {T148111} too. @bd808 accidentally ran the ALTER on the slave, I then ran it on master, but... [21:37:29] 10Beta-Cluster-Infrastructure, 10DBA, 06Operations: Possible to run writes (e.g. UPDATE) on Beta Cluster replica - https://phabricator.wikimedia.org/T110115#2718311 (10Mattflaschen-WMF) [21:37:35] bd808: T110115 [21:42:31] hashar im getting this error now [21:42:31] MultipleInvalid: extra keys not allowed @ data['pipelines'][3]['failure']['oldgerrit'] [21:45:15] 03Scap3: Scap3 should announce all deploys - https://phabricator.wikimedia.org/T146386#2718345 (10demon) a:03demon [21:46:10] ahh [21:46:11] 10Beta-Cluster-Infrastructure, 13Patch-For-Review: Enable Quiz Extension on ca.wikipedia.beta.wmflabs.org for testing - https://phabricator.wikimedia.org/T142692#2718347 (10Krenair) 05Open>03Resolved a:03Krenair [21:46:19] matt_flaschen: thank you for https://gerrit.wikimedia.org/r/#/c/305691/1 that is exactly what I was looking for :) [21:46:35] paladox: not sure what happens [21:46:53] paladox: I was just polishing a patch and will head to bed :( [21:46:59] Ok [21:47:06] I am not around next week, but will still try to review/merger your patches as time allow [21:47:08] I think it may be my [21:47:11] try [21:47:19] at dual connections [21:47:19] dont obssess about the recheck for puppet compiler :] [21:47:22] in zuul [21:47:24] ok [21:47:43] hashar, no problem, glad it helps. [21:50:19] hashar i figured out the problem [21:50:27] im fixing it then will re run the test [21:51:21] hashar it passes [21:51:32] matt_flaschen: was hitting the same issue in Cite :] [21:51:48] anyway time to sleep [21:51:51] hashar https://phabricator.wikimedia.org/P4236 [21:52:08] neat :) [21:52:19] yep, so it passes [21:52:21] and dont run them as root ! [21:52:25] LOL [21:52:28] woops [21:53:00] note that the zuul version being used is defined in /test-requirements.txt [21:53:09] we have spotted that together a few months ago [21:53:28] basically installs integration/zuul.git @patch-queue/debian/precise-wikimedia [21:53:33] oh [21:53:43] so you might want to clone that if you want to look at the proper zuul code [21:53:52] zuul==2.0.1.dev857 [21:54:00] yeah [21:54:14] not sure how the version is forged [21:54:21] yep [21:54:27] Looks wrong, but im using [21:54:28] wmf4 [21:54:33] (03PS6) 10Hashar: (WIP) MediaWiki tests with composer (WIP) [integration/config] - 10https://gerrit.wikimedia.org/r/315942 [21:54:59] the current one were using on jessie [21:56:10] hashar: btw, beta DB works again now ;) [21:56:19] (03PS1) 10Paladox: Enable php56 tests in composer-test [integration/config] - 10https://gerrit.wikimedia.org/r/316012 [21:56:30] oh, I see it was already discussed :o [21:56:48] Luke081515: yeah the magic firefighter broke and fixed it up :] [21:57:00] :) [21:57:13] Luke081515: thanks for the notification! apparently that triggered the investigation/fix [21:57:41] I think they knew, but yeah, thanks for the ping :) [22:01:51] ha [22:01:57] it is officially my birthday now [22:01:59] and time to sleep :] [22:02:03] * hashar waves! [22:02:20] (03PS3) 10Paladox: [operations-puppet-catalog-compiler] Adding it to jenkins job builder [integration/config] - 10https://gerrit.wikimedia.org/r/315994 (https://phabricator.wikimedia.org/T97513) [22:02:30] hashar happy birthday [22:02:37] :D [22:02:43] good week-end! [22:02:54] :) [22:04:17] hashar: happy bday! [22:04:46] congratulations, hashar ! [22:05:32] hashar: happy bday, have a nice weekend! [22:05:58] ;] [22:46:56] 03Scap3: Scap3 should announce all deploys - https://phabricator.wikimedia.org/T146386#2718472 (10demon) 05Open>03Resolved [22:52:30] 06Release-Engineering-Team, 05WMF-NDA: Move mediawiki-config private files under /private/ - https://phabricator.wikimedia.org/T125542#2718481 (10demon) 05Open>03Resolved a:03demon There's nothing really private here, only most of this doesn't exist anymore. Only two files in private/ are PrivateSettings... [22:52:32] 06Release-Engineering-Team, 05WMF-NDA: Move mediawiki-config private files under /private/ - https://phabricator.wikimedia.org/T125542#2718484 (10demon) [22:55:35] +1, those are not private :P [23:16:20] Platonides, I think mwblocker.log used to contain private data, .svn may contain private history, I think 'xrumer' held a private anti-spam measure [23:16:27] no idea about x.php [23:17:13] 06Release-Engineering-Team: Move mediawiki-config private files under /private/ - https://phabricator.wikimedia.org/T125542#2718532 (10Krenair) [23:23:02] I don't think xrumer is used anymore, is it? [23:23:26] I heard it wasn't [23:37:10] Trying to remember where it was included