[03:44:03] Project mediawiki-core-code-coverage build #2314: 04STILL FAILING in 44 min: https://integration.wikimedia.org/ci/job/mediawiki-core-code-coverage/2314/ [04:03:29] 10Beta-Cluster-Infrastructure, 05Goal, 07Tracking: Consolidate, remove, and/or downsize Beta Cluster instances to help with [[wikitech:Purge_2016]] - https://phabricator.wikimedia.org/T142288#2702974 (10demon) >>! In T142288#2701724, @AlexMonk-WMF wrote: > @demon, @dcausse, @Gehel, what do you think about th... [04:17:33] Yippee, build fixed! [04:17:33] Project selenium-MultimediaViewer » firefox,beta,Linux,contintLabsSlave && UbuntuTrusty build #167: 09FIXED in 21 min: https://integration.wikimedia.org/ci/job/selenium-MultimediaViewer/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/167/ [08:02:42] 10Gerrit, 06Operations: cronspam from cobalt after the Gerrit migration - https://phabricator.wikimedia.org/T147776#2703125 (10hashar) I am pretty sure that is (**was**?) used for our community metrics tool at http://korma.wmflabs.org/browser/ with some cron fetching something like https://gerrit.wikimedia.or... [08:05:21] 06Release-Engineering-Team, 10Phabricator, 13Patch-For-Review, 07Wikimedia-Incident: Contention on search phabricator database creating full phabricator outages - https://phabricator.wikimedia.org/T146673#2703128 (10jcrespo) a:03jcrespo [08:08:31] 10Gerrit, 06Operations: cronspam from cobalt after the Gerrit migration - https://phabricator.wikimedia.org/T147776#2703133 (10hashar) On `lead.wikimedia.org`: ``` $ ls -l /var/www/reviewer-counts.json -rw-r--r-- 1 gerrit2 root 258 Oct 10 01:59 /var/www/reviewer-counts.json ``` [08:08:54] 10Gerrit, 06Operations: cronspam from cobalt after the Gerrit migration - https://phabricator.wikimedia.org/T147776#2703134 (10hashar) [08:13:52] 10Beta-Cluster-Infrastructure, 05Goal, 07Tracking: Consolidate, remove, and/or downsize Beta Cluster instances to help with [[wikitech:Purge_2016]] - https://phabricator.wikimedia.org/T142288#2703154 (10dcausse) @AlexMonk-WMF I think we can remove one, I'll start to update elastic config for this cluster (re... [08:21:24] 10Continuous-Integration-Config, 06Release-Engineering-Team: Switch MediaWiki coverage job from Trusty/Zend PHP 5.5 to Jessie/Zend PHP 7.0 - https://phabricator.wikimedia.org/T147778#2703156 (10hashar) [08:23:56] 10Continuous-Integration-Config, 06Release-Engineering-Team: Switch MediaWiki coverage job from Trusty/Zend PHP 5.5 to Jessie/Zend PHP 7.0 - https://phabricator.wikimedia.org/T147778#2703170 (10hashar) [08:24:59] 10Continuous-Integration-Config, 06Release-Engineering-Team, 10MediaWiki-Unit-tests: MediaWiki code coverage no more run parser tests - https://phabricator.wikimedia.org/T147779#2703173 (10hashar) [08:25:15] 10Continuous-Integration-Config, 06Release-Engineering-Team: Switch MediaWiki coverage job from Trusty/Zend PHP 5.5 to Jessie/Zend PHP 7.0 - https://phabricator.wikimedia.org/T147778#2703156 (10hashar) [08:26:28] (03PS3) 10Hashar: Try MediaWiki code coverage on Jessie/php7 [integration/config] - 10https://gerrit.wikimedia.org/r/314559 (https://phabricator.wikimedia.org/T147778) [08:27:15] (03CR) 10Hashar: [C: 04-2] "Attached to T147778" [integration/config] - 10https://gerrit.wikimedia.org/r/314559 (https://phabricator.wikimedia.org/T147778) (owner: 10Hashar) [08:38:49] (03PS1) 10Hashar: MediaWiki code coverage job now archive all logs [integration/config] - 10https://gerrit.wikimedia.org/r/315048 (https://phabricator.wikimedia.org/T147778) [08:39:54] (03CR) 10Hashar: [C: 032] MediaWiki code coverage job now archive all logs [integration/config] - 10https://gerrit.wikimedia.org/r/315048 (https://phabricator.wikimedia.org/T147778) (owner: 10Hashar) [08:40:30] (03PS2) 10Hashar: Kartographer depends on WikimediaMessages [integration/config] - 10https://gerrit.wikimedia.org/r/314825 (owner: 10Yurik) [08:40:42] (03CR) 10Hashar: [C: 032] "Rebased" [integration/config] - 10https://gerrit.wikimedia.org/r/314825 (owner: 10Yurik) [08:40:55] (03Merged) 10jenkins-bot: MediaWiki code coverage job now archive all logs [integration/config] - 10https://gerrit.wikimedia.org/r/315048 (https://phabricator.wikimedia.org/T147778) (owner: 10Hashar) [08:41:37] (03Merged) 10jenkins-bot: Kartographer depends on WikimediaMessages [integration/config] - 10https://gerrit.wikimedia.org/r/314825 (owner: 10Yurik) [08:42:46] 10Browser-Tests-Infrastructure, 10Gerrit, 10Wikidata, 13Patch-For-Review, 15User-Tobi_WMDE_SW: Retire wikidata/browsertests.git - https://phabricator.wikimedia.org/T144486#2601243 (10Tobi_WMDE_SW) 05Open>03Resolved a:03Tobi_WMDE_SW [08:44:17] (03PS4) 10Hashar: (WIP) Try MediaWiki code coverage on Jessie/php7 (WIP) [integration/config] - 10https://gerrit.wikimedia.org/r/314559 (https://phabricator.wikimedia.org/T147778) [08:53:07] 10Continuous-Integration-Config, 06Release-Engineering-Team, 10MediaWiki-Unit-tests: MediaWiki code coverage fails on Zend PHP 7.0 due to a database error - https://phabricator.wikimedia.org/T147781#2703230 (10hashar) [08:53:47] 10Continuous-Integration-Config, 06Release-Engineering-Team, 10MediaWiki-Unit-tests: MediaWiki code coverage fails on Zend PHP 7.0 due to a database error - https://phabricator.wikimedia.org/T147781#2703245 (10hashar) [08:53:49] 10Continuous-Integration-Infrastructure, 13Patch-For-Review: PHP7 support in CI (tracking) - https://phabricator.wikimedia.org/T144964#2703244 (10hashar) [08:55:07] PROBLEM - Puppet run on deployment-tin is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [08:55:47] (03CR) 10Hashar: "Rebased. Update the yaml to comment out the timed trigger and made the job to archive all files under /log/ for diagnostic purposes." [integration/config] - 10https://gerrit.wikimedia.org/r/314559 (https://phabricator.wikimedia.org/T147778) (owner: 10Hashar) [09:05:52] PROBLEM - Puppet run on deployment-redis01 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [09:18:31] PROBLEM - Puppet run on deployment-mira is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [09:45:54] RECOVERY - Puppet run on deployment-redis01 is OK: OK: Less than 1.00% above the threshold [0.0] [09:55:21] 10Beta-Cluster-Infrastructure, 05Goal, 07Tracking: Consolidate, remove, and/or downsize Beta Cluster instances to help with [[wikitech:Purge_2016]] - https://phabricator.wikimedia.org/T142288#2703380 (10Gehel) As @dcausse guessed, I'm not too keen on going below 3 nodes in an elasticsearch cluster. But I'll... [10:19:38] 06Release-Engineering-Team, 10Phabricator, 13Patch-For-Review, 07Wikimedia-Incident: Contention on search phabricator database creating full phabricator outages - https://phabricator.wikimedia.org/T146673#2703416 (10jcrespo) So I have applied the previous config settings and get rid of Aria on the slave. R... [10:35:31] 06Release-Engineering-Team, 10DBA, 10Phabricator, 13Patch-For-Review, 07Wikimedia-Incident: Contention on search phabricator database creating full phabricator outages - https://phabricator.wikimedia.org/T146673#2703452 (10jcrespo) a:05jcrespo>03None [10:40:24] 06Release-Engineering-Team, 10DBA, 10Phabricator, 13Patch-For-Review, 07Wikimedia-Incident: Contention on search phabricator database creating full phabricator outages - https://phabricator.wikimedia.org/T146673#2703471 (10jcrespo) To clarify, pending tasks: - let's schedule a downtime for the master fa... [11:14:08] 03Scap3, 06Services, 10service-runner, 10service-template-node, 15User-mobrovac: Enable config deploys for service::node services - https://phabricator.wikimedia.org/T144542#2703487 (10mobrovac) [11:51:15] Project selenium-Wikibase » firefox,test,Linux,contintLabsSlave && UbuntuTrusty build #135: 15ABORTED in 24 min: https://integration.wikimedia.org/ci/job/selenium-Wikibase/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=test,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/135/ [11:51:30] Project selenium-Wikibase » firefox,beta,Linux,contintLabsSlave && UbuntuTrusty build #136: 15ABORTED in 10 sec: https://integration.wikimedia.org/ci/job/selenium-Wikibase/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/136/ [11:53:57] 06Release-Engineering-Team, 10DBA, 10Phabricator, 13Patch-For-Review, 07Wikimedia-Incident: Contention on search phabricator database creating full phabricator outages - https://phabricator.wikimedia.org/T146673#2703551 (10Paladox) We would need to do the + changes to https://github.com/wikimedia/phabric... [12:03:09] 06Release-Engineering-Team, 10DBA, 10Phabricator, 13Patch-For-Review, 07Wikimedia-Incident: Contention on search phabricator database creating full phabricator outages - https://phabricator.wikimedia.org/T146673#2703554 (10Paladox) What we could do here https://github.com/wikimedia/phabricator/blob/wmf/... [12:54:37] PROBLEM - Puppet run on deployment-kafka05 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [13:20:17] 10Gerrit, 06Operations: cronspam from cobalt after the Gerrit migration - https://phabricator.wikimedia.org/T147776#2703638 (10Dzahn) root@cobalt:/var/www# touch reviewer-counts.json root@cobalt:/var/www# chown gerrit2 reviewer-counts.json puppetizing will follow tomorrow [13:34:37] RECOVERY - Puppet run on deployment-kafka05 is OK: OK: Less than 1.00% above the threshold [0.0] [13:52:10] Project selenium-Wikibase » firefox,test,Linux,contintLabsSlave && UbuntuTrusty build #136: 04STILL FAILING in 2 hr 0 min: https://integration.wikimedia.org/ci/job/selenium-Wikibase/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=test,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/136/ [14:14:33] PROBLEM - Puppet run on zuul-dev-jessie is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [14:15:26] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 13Patch-For-Review: Investigate again a central cache for package managers - https://phabricator.wikimedia.org/T147635#2703697 (10hashar) I crafted a [[ https://gerrit.wikimedia.org/r/314751 | very lame puppet manifest ]]. Sonatype provides... [14:23:52] 03Scap3, 10Citoid, 10ContentTranslation-CXserver, 10Graphoid, and 6 others: Depool and repool SCB services during deploys - https://phabricator.wikimedia.org/T144602#2703716 (10Mvolz) [14:43:28] 06Release-Engineering-Team, 10DBA, 10Phabricator, 13Patch-For-Review, 07Wikimedia-Incident: Contention on search phabricator database creating full phabricator outages - https://phabricator.wikimedia.org/T146673#2703759 (10Paladox) Or for https://github.com/wikimedia/phabricator/blob/wmf/stable/src/appli... [15:07:59] PROBLEM - Puppet run on repository is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [15:22:59] RECOVERY - Puppet run on repository is OK: OK: Less than 1.00% above the threshold [0.0] [15:39:47] PROBLEM - Puppet run on deployment-phab02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [15:40:17] (03PS2) 10Paladox: [DonationInterface] Remove test extension-unittests-non-voting [integration/config] - 10https://gerrit.wikimedia.org/r/313890 [15:40:25] (03Abandoned) 10Paladox: [DonationInterface] Remove test extension-unittests-non-voting [integration/config] - 10https://gerrit.wikimedia.org/r/313890 (owner: 10Paladox) [15:42:10] (03Abandoned) 10Paladox: Update squizlabs/php_codesniffer to 2.6.1 [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/293967 (owner: 10Paladox) [15:45:20] !log deployment-prep deployment-elastic0[5-8]: reduce the number of replicas to 1 max for all indices [15:45:24] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [15:47:15] Yippee, build fixed! [15:47:16] Project mediawiki-core-code-coverage build #2315: 09FIXED in 47 min: https://integration.wikimedia.org/ci/job/mediawiki-core-code-coverage/2315/ [15:51:04] PROBLEM - Puppet run on deployment-phab01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [16:16:43] 10Beta-Cluster-Infrastructure, 06Labs: Remove Labs uses of etcd and confd classes - https://phabricator.wikimedia.org/T147800#2703948 (10Andrew) [16:26:26] 10Beta-Cluster-Infrastructure, 06Labs, 13Patch-For-Review: Replace all class imports on Labs with role imports - https://phabricator.wikimedia.org/T147233#2703967 (10Andrew) [16:29:20] PROBLEM - Puppet run on deployment-restbase02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [16:31:56] 03Scap3, 10Citoid, 10ContentTranslation-CXserver, 10ContentTranslation-Deployments, and 7 others: Depool and repool SCB services during deploys - https://phabricator.wikimedia.org/T144602#2703971 (10Amire80) Mmmm, this is super-internal and technical, but it's on the #mediawiki-extensions-contenttranslatio... [16:43:32] PROBLEM - Puppet run on deployment-restbase01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [16:43:32] (03PS1) 10Paladox: Whitelist Pwirth [integration/config] - 10https://gerrit.wikimedia.org/r/315116 [17:04:42] 10Beta-Cluster-Infrastructure, 06Labs: Remove Labs uses of etcd and confd classes - https://phabricator.wikimedia.org/T147800#2703948 (10AlexMonk-WMF) deployment-conf03's includes match production's conf1001.eqiad.wmnet [17:08:01] (03PS1) 10Paladox: [BlueSpiceExtensions] Add dependance on extension BlueSpiceFoundation [integration/config] - 10https://gerrit.wikimedia.org/r/315123 [17:10:00] (03CR) 10Paladox: "We use composer for this, but it seems to be failing with https://integration.wikimedia.org/ci/job/mwext-testextension-php55-composer-non-" [integration/config] - 10https://gerrit.wikimedia.org/r/315123 (owner: 10Paladox) [17:25:46] 10Browser-Tests-Infrastructure, 06Reading-Web-Backlog, 07Browser-Tests, 13Patch-For-Review, and 3 others: Add helper to Selenium that allows you to query whether JavaScript module has loaded - https://phabricator.wikimedia.org/T146292#2656397 (10Jdlrobson) I've moved this to tracking as this is better plac... [18:15:21] (03PS1) 10Paladox: [PdfBook] Add jenkins tests [integration/config] - 10https://gerrit.wikimedia.org/r/315143 [18:16:19] (03PS2) 10Paladox: [PdfBook] Add jenkins tests [integration/config] - 10https://gerrit.wikimedia.org/r/315143 [18:34:03] (03CR) 10Robert Vogel: [C: 031] [BlueSpiceExtensions] Add dependance on extension BlueSpiceFoundation [integration/config] - 10https://gerrit.wikimedia.org/r/315123 (owner: 10Paladox) [18:42:51] 06Release-Engineering-Team, 10DBA, 10Phabricator, 13Patch-For-Review, 07Wikimedia-Incident: Contention on search phabricator database creating full phabricator outages - https://phabricator.wikimedia.org/T146673#2704167 (10mmodell) Upstream task about InnoDB support: https://secure.phabricator.com/T11741 [18:56:43] 10Beta-Cluster-Infrastructure, 06Labs: Remove Labs uses of etcd and confd classes - https://phabricator.wikimedia.org/T147800#2704189 (10hashar) The `etcd` labs project is most probably @joe sandbox area. [18:59:44] (03PS3) 10Hashar: [PdfBook] Add jenkins tests [integration/config] - 10https://gerrit.wikimedia.org/r/315143 (owner: 10Paladox) [18:59:56] (03CR) 10Hashar: [C: 032] [PdfBook] Add jenkins tests [integration/config] - 10https://gerrit.wikimedia.org/r/315143 (owner: 10Paladox) [19:00:34] (03Merged) 10jenkins-bot: [PdfBook] Add jenkins tests [integration/config] - 10https://gerrit.wikimedia.org/r/315143 (owner: 10Paladox) [19:10:42] hashar ^^ thanks [20:19:18] 10Continuous-Integration-Infrastructure: Favicon.ico on doc.wikimedia.org returns 500 - https://phabricator.wikimedia.org/T147814#2704362 (10Krinkle) [20:21:00] 10Continuous-Integration-Infrastructure: Favicon broken on doc.wikimedia.org and integration.wikimedia.org (HTTP 500) - https://phabricator.wikimedia.org/T147814#2704346 (10Krinkle) [20:25:52] (03PS2) 10Hashar: [BlueSpiceExtensions] Add dependance on extension BlueSpiceFoundation [integration/config] - 10https://gerrit.wikimedia.org/r/315123 (owner: 10Paladox) [20:25:56] (03CR) 10Hashar: [C: 032] [BlueSpiceExtensions] Add dependance on extension BlueSpiceFoundation [integration/config] - 10https://gerrit.wikimedia.org/r/315123 (owner: 10Paladox) [20:26:03] paladox: landing the bluespice thing :) [20:26:30] (03Merged) 10jenkins-bot: [BlueSpiceExtensions] Add dependance on extension BlueSpiceFoundation [integration/config] - 10https://gerrit.wikimedia.org/r/315123 (owner: 10Paladox) [20:27:53] paladox: not sure what it is supposed to fix but it looks harmless :] [20:34:19] (03CR) 10Hashar: "I did a recheck on https://gerrit.wikimedia.org/r/#/c/315120/ and what I noticed is that BlueSpiceExtensions bring in Foundation via compo" [integration/config] - 10https://gerrit.wikimedia.org/r/315123 (owner: 10Paladox) [20:42:25] (03PS2) 10Hashar: Whitelist Pwirth [integration/config] - 10https://gerrit.wikimedia.org/r/315116 (owner: 10Paladox) [20:42:39] (03CR) 10Hashar: [C: 032] Whitelist Pwirth [integration/config] - 10https://gerrit.wikimedia.org/r/315116 (owner: 10Paladox) [20:43:37] (03Merged) 10jenkins-bot: Whitelist Pwirth [integration/config] - 10https://gerrit.wikimedia.org/r/315116 (owner: 10Paladox) [20:47:26] 10Continuous-Integration-Infrastructure, 06Operations, 10Traffic, 07Regression: Favicon broken on doc.wikimedia.org and integration.wikimedia.org (HTTP 500) - https://phabricator.wikimedia.org/T147814#2704427 (10Krinkle) p:05Triage>03Normal Requesting from Varnish with `--compressed` includes it, reque... [20:49:39] hashar thanks [20:52:02] mobrovac, hey [21:08:32] RECOVERY - Puppet run on deployment-mira is OK: OK: Less than 1.00% above the threshold [0.0] [21:10:08] RECOVERY - Puppet run on deployment-tin is OK: OK: Less than 1.00% above the threshold [0.0] [21:19:19] RECOVERY - Puppet run on deployment-restbase02 is OK: OK: Less than 1.00% above the threshold [0.0] [21:23:29] RECOVERY - Puppet run on deployment-restbase01 is OK: OK: Less than 1.00% above the threshold [0.0] [21:41:55] (never mind, fixed everything myself) [21:42:06] just those deployment-phab machines left [21:43:38] RECOVERY - Keyholder status on deployment-tin is OK: OK: Less than 100.00% above the threshold [0.0] [21:45:04] 10Beta-Cluster-Infrastructure, 07Puppet: puppet failure on deployment-phab0[12] due to missing expected puppet:///modules/phabricator/sshd-phab.service - https://phabricator.wikimedia.org/T147818#2704482 (10AlexMonk-WMF) [22:50:55] 10Continuous-Integration-Config, 06Release-Engineering-Team, 13Patch-For-Review: Switch MediaWiki coverage job from Trusty/Zend PHP 5.5 to Jessie/Zend PHP 7.0 - https://phabricator.wikimedia.org/T147778#2704576 (10Krinkle) [22:51:41] 10Continuous-Integration-Config, 06Release-Engineering-Team, 10MediaWiki-Unit-tests: MediaWiki code coverage no longer runs parser tests - https://phabricator.wikimedia.org/T147779#2704580 (10Krinkle) [22:58:00] 10Continuous-Integration-Config, 06Release-Engineering-Team, 10MediaWiki-Unit-tests: MediaWiki code coverage no longer runs parser tests - https://phabricator.wikimedia.org/T147779#2704587 (10Krinkle) https://phabricator.wikimedia.org/diffusion/MW/history/master/;4a975b8099ee11b15421d03be02206935a8422f1 > 4...