[01:28:31] 10Continuous-Integration-Infrastructure, 06Operations, 10Traffic, 07Regression: Favicon broken on doc.wikimedia.org and integration.wikimedia.org (HTTP 500) - https://phabricator.wikimedia.org/T147814#2704346 (10BBlack) The response lacks `Content-Length` because it's sent with `Transfer-Encoding: chunked`... [03:28:38] (03CR) 10Legoktm: Publish code coverage post-merge in labs/tools/heritage (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/314171 (owner: 10Jean-Frédéric) [03:29:50] 10Continuous-Integration-Config, 06Release-Engineering-Team, 10MediaWiki-Unit-tests: MediaWiki code coverage no longer runs parser tests - https://phabricator.wikimedia.org/T147779#2703173 (10Legoktm) Also, do parser tests even have covers tags? [04:04:51] Project selenium-MultimediaViewer » safari,beta,OS X 10.9,contintLabsSlave && UbuntuTrusty build #168: 04FAILURE in 8 min 50 sec: https://integration.wikimedia.org/ci/job/selenium-MultimediaViewer/BROWSER=safari,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=OS%20X%2010.9,label=contintLabsSlave%20&&%20UbuntuTrusty/168/ [04:05:41] 10Beta-Cluster-Infrastructure: Request Admin at deployment or cluster - https://phabricator.wikimedia.org/T147828#2704779 (10Mbrt) [07:01:43] 10Beta-Cluster-Infrastructure, 07Puppet: puppet failure on deployment-phab0[12] due to missing expected puppet:///modules/phabricator/sshd-phab.service - https://phabricator.wikimedia.org/T147818#2704869 (10hashar) @mmodell what are those deployment-phab01 and deployment-phab02 instances? From the name that se... [07:04:22] Project selenium-Wikibase » firefox,beta,Linux,contintLabsSlave && UbuntuTrusty build #137: 04STILL FAILING in 2 hr 24 min: https://integration.wikimedia.org/ci/job/selenium-Wikibase/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/137/ [07:37:41] moritzm: sure! [07:39:20] so we would move the primary deployment server to mira, let people deploy from there for a few days, reimage tin in the mean time and once tin is all well, switch back? [07:40:44] given we are on offsite next week [07:41:05] switch to mira as primary today and reimage tomorrow? [07:41:10] though really it should just work :] [07:41:24] but I dont think we ever deployed in prod from mira/jessie [07:44:42] moritzm: will want to switch the dns entry deployment.eqiad.wmnet then a few hiera() values [07:45:03] I'm fine either way, we can also do it after the releng offsite [07:45:24] I think it is fine to switch at least today [07:45:27] errr [07:45:32] I am not awake yet [07:45:39] I think it is fine to switch the primary to mira / Jessie [07:45:49] so if we notice something too cumbersome we can revert at the end of the week [07:46:04] ok, sounds good. [07:46:06] and if all is good, indeed reimage tin to jessie in a couple weeks [07:47:07] this will also give people a bit time to save data from their home on tin (which is 12 GB in total ATM) [07:47:43] doh [07:48:58] zeljkof: sretan rodjendan! [07:52:53] mobrovac: hvala! :) [07:53:08] hashar: I currently need to finish something else, I can look into at in 2-3 hours, if you want to you can already prepare patches, otherwise I'll do it later on [07:54:14] moritzm: https://gerrit.wikimedia.org/r/315205 Switch primary deployment server from tin to mira [07:54:14] :D [07:54:23] it is a single hiera() value to change :] [07:54:46] ah, you're quick :-) will look into it later on [07:55:14] pretty sure trebuchet is going to be broken [08:26:38] (03PS2) 10Hashar: Filter out refs/meta/config from all pipelines [integration/config] - 10https://gerrit.wikimedia.org/r/313387 (https://phabricator.wikimedia.org/T52389) [08:28:02] (03CR) 10Hashar: [C: 032] Filter out refs/meta/config from all pipelines [integration/config] - 10https://gerrit.wikimedia.org/r/313387 (https://phabricator.wikimedia.org/T52389) (owner: 10Hashar) [08:29:00] (03Merged) 10jenkins-bot: Filter out refs/meta/config from all pipelines [integration/config] - 10https://gerrit.wikimedia.org/r/313387 (https://phabricator.wikimedia.org/T52389) (owner: 10Hashar) [08:29:33] !log Reloading Zuul to deploy https://gerrit.wikimedia.org/r/#/c/313387/ Filter out refs/meta/config from all pipelines T52389 [08:29:37] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [08:37:45] (03PS1) 10Hashar: Filter out refs/meta/config from check-only/check-voter [integration/config] - 10https://gerrit.wikimedia.org/r/315211 (https://phabricator.wikimedia.org/T52389) [08:39:29] (03CR) 10Hashar: [C: 032] Filter out refs/meta/config from check-only/check-voter [integration/config] - 10https://gerrit.wikimedia.org/r/315211 (https://phabricator.wikimedia.org/T52389) (owner: 10Hashar) [08:40:26] (03Merged) 10jenkins-bot: Filter out refs/meta/config from check-only/check-voter [integration/config] - 10https://gerrit.wikimedia.org/r/315211 (https://phabricator.wikimedia.org/T52389) (owner: 10Hashar) [08:43:27] (03PS1) 10Hashar: Filter out refs/meta/config from check-voter [integration/config] - 10https://gerrit.wikimedia.org/r/315213 (https://phabricator.wikimedia.org/T152389) [08:46:42] 10Continuous-Integration-Infrastructure, 13Patch-For-Review, 07Zuul: Zuul should not run jenkins-bot on changes for refs/meta/* - https://phabricator.wikimedia.org/T52389#2705012 (10hashar) 05Open>03Resolved a:03hashar Tested out on https://gerrit.wikimedia.org/r/#/c/315210/ . The Gerrit change is fetc... [08:47:53] 10Continuous-Integration-Infrastructure, 06Operations, 10puppet-compiler, 13Patch-For-Review: OSError: [Errno 28] No space left on device on compiler02.puppet3-diffs.eqiad.wmflabs - https://phabricator.wikimedia.org/T143671#2705015 (10hashar) 05Open>03Resolved [08:48:41] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 13Patch-For-Review: Investigate again a central cache for package managers - https://phabricator.wikimedia.org/T147635#2705017 (10hashar) p:05Triage>03Normal [08:49:41] 10Continuous-Integration-Infrastructure, 05Continuous-Integration-Scaling, 07Nodepool: Investigate why Nodepool instances are sometime slow to reach READY state - https://phabricator.wikimedia.org/T146813#2705020 (10hashar) 05Open>03Resolved a:03hashar Investigation is complete. Namely we have to lower... [08:59:20] 10Continuous-Integration-Infrastructure, 05Continuous-Integration-Scaling, 07Nodepool: Investigate why Nodepool instances are sometime slow to reach READY state - https://phabricator.wikimedia.org/T146813#2705023 (10hashar) Rate change is https://gerrit.wikimedia.org/r/#/c/315214/ [09:08:10] 10Continuous-Integration-Infrastructure, 10Wikidata: [Bug] github.com is 403ing downloads from Wikimedia CI during composer update - https://phabricator.wikimedia.org/T106519#2705026 (10hashar) That is quite an old task, I am not sure whether it is still happen of it got worked around somehow. A potential so... [09:13:29] PROBLEM - Puppet run on deployment-cache-text04 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [09:14:01] PROBLEM - Puppet run on deployment-elastic08 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [09:14:33] PROBLEM - Puppet run on integration-slave-trusty-1003 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [09:14:43] PROBLEM - Puppet run on deployment-mathoid is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [09:15:01] PROBLEM - Puppet run on integration-slave-jessie-1001 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [09:17:05] PROBLEM - Puppet run on deployment-conf03 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [09:17:35] PROBLEM - Puppet run on deployment-parsoid09 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [09:17:47] PROBLEM - Puppet run on deployment-fluorine02 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [09:17:47] PROBLEM - Puppet run on deployment-kafka04 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [09:18:27] PROBLEM - Puppet run on deployment-mediawiki05 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [09:18:35] PROBLEM - Puppet run on deployment-redis02 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [09:18:43] 10Continuous-Integration-Infrastructure, 10Wikidata: [Bug] github.com is 403ing downloads from Wikimedia CI during composer update - https://phabricator.wikimedia.org/T106519#2705041 (10hashar) [09:18:54] PROBLEM - Puppet run on integration-slave-jessie-1002 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [09:18:54] PROBLEM - Puppet run on deployment-logstash2 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [09:19:11] 10Continuous-Integration-Infrastructure, 10Wikidata: [Bug] github.com is 403ing downloads from Wikimedia CI during composer update - https://phabricator.wikimedia.org/T106519#1470716 (10hashar) **Reproduction steps** ``` git clone https://gerrit.wikimedia.org/r/p/mediawiki/extensions/WikibaseQualityConstraints... [09:19:14] PROBLEM - Puppet run on deployment-elastic06 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [09:19:32] PROBLEM - Puppet run on deployment-restbase01 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [09:19:32] PROBLEM - Puppet run on deployment-mira is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [09:19:36] PROBLEM - Puppet run on deployment-jobrunner02 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [09:19:50] PROBLEM - Puppet run on deployment-changeprop is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [09:20:11] PROBLEM - Puppet run on deployment-urldownloader is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [09:20:34] PROBLEM - Puppet run on deployment-apertium01 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [09:20:50] PROBLEM - Puppet run on deployment-zookeeper01 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [09:21:04] PROBLEM - Puppet run on deployment-zotero01 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [09:21:42] PROBLEM - Puppet run on deployment-apertium02 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [09:21:54] PROBLEM - Puppet run on deployment-pdfrender is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [09:22:18] PROBLEM - Puppet run on deployment-salt02 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [09:22:30] PROBLEM - Puppet run on integration-slave-trusty-1001 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [09:22:47] PROBLEM - Puppet run on deployment-sca03 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [09:23:13] PROBLEM - Puppet run on integration-slave-trusty-1016 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [09:24:15] PROBLEM - Puppet run on integration-puppetmaster01 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [09:24:21] PROBLEM - Puppet run on castor is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [09:25:13] PROBLEM - Puppet run on integration-slave-trusty-1012 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [09:25:37] PROBLEM - Puppet run on deployment-prometheus01 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [09:25:53] PROBLEM - Puppet run on integration-slave-trusty-1013 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [09:26:09] PROBLEM - Puppet run on integration-slave-jessie-1005 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [09:33:30] puppet errors are transient [09:35:03] !log reboot deployment-imagescaler01 to enable memory cgroup [09:35:08] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [09:36:04] RECOVERY - puppet last run on scandium is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [09:41:57] 10Beta-Cluster-Infrastructure, 07Puppet: puppet failure on deployment-phab0[12] due to missing expected puppet:///modules/phabricator/sshd-phab.service - https://phabricator.wikimedia.org/T147818#2705071 (10mmodell) @hashar: They are not for hosting phabricator per se, but rather for testing scap deployment of... [09:42:18] RECOVERY - Puppet run on deployment-salt02 is OK: OK: Less than 1.00% above the threshold [0.0] [09:44:21] RECOVERY - Puppet run on castor is OK: OK: Less than 1.00% above the threshold [0.0] [09:44:35] RECOVERY - Puppet run on deployment-jobrunner02 is OK: OK: Less than 1.00% above the threshold [0.0] [09:45:09] RECOVERY - Puppet run on deployment-urldownloader is OK: OK: Less than 1.00% above the threshold [0.0] [09:45:47] RECOVERY - Puppet run on deployment-zookeeper01 is OK: OK: Less than 1.00% above the threshold [0.0] [09:46:03] RECOVERY - Puppet run on deployment-zotero01 is OK: OK: Less than 1.00% above the threshold [0.0] [09:47:47] RECOVERY - Puppet run on deployment-sca03 is OK: OK: Less than 1.00% above the threshold [0.0] [09:47:49] RECOVERY - Puppet run on deployment-kafka04 is OK: OK: Less than 1.00% above the threshold [0.0] [09:48:37] RECOVERY - Puppet run on deployment-redis02 is OK: OK: Less than 1.00% above the threshold [0.0] [09:49:31] RECOVERY - Puppet run on deployment-restbase01 is OK: OK: Less than 1.00% above the threshold [0.0] [09:49:44] RECOVERY - Puppet run on deployment-mathoid is OK: OK: Less than 1.00% above the threshold [0.0] [09:49:48] RECOVERY - Puppet run on deployment-changeprop is OK: OK: Less than 1.00% above the threshold [0.0] [09:50:34] RECOVERY - Puppet run on deployment-apertium01 is OK: OK: Less than 1.00% above the threshold [0.0] [09:50:36] RECOVERY - Puppet run on deployment-prometheus01 is OK: OK: Less than 1.00% above the threshold [0.0] [09:51:56] RECOVERY - Puppet run on deployment-pdfrender is OK: OK: Less than 1.00% above the threshold [0.0] [09:52:06] RECOVERY - Puppet run on deployment-conf03 is OK: OK: Less than 1.00% above the threshold [0.0] [09:52:36] RECOVERY - Puppet run on deployment-parsoid09 is OK: OK: Less than 1.00% above the threshold [0.0] [09:53:28] RECOVERY - Puppet run on deployment-mediawiki05 is OK: OK: Less than 1.00% above the threshold [0.0] [09:53:30] RECOVERY - Puppet run on deployment-cache-text04 is OK: OK: Less than 1.00% above the threshold [0.0] [09:53:52] RECOVERY - Puppet run on integration-slave-jessie-1002 is OK: OK: Less than 1.00% above the threshold [0.0] [09:53:54] RECOVERY - Puppet run on deployment-logstash2 is OK: OK: Less than 1.00% above the threshold [0.0] [09:54:02] RECOVERY - Puppet run on deployment-elastic08 is OK: OK: Less than 1.00% above the threshold [0.0] [09:54:13] RECOVERY - Puppet run on deployment-elastic06 is OK: OK: Less than 1.00% above the threshold [0.0] [09:54:33] RECOVERY - Puppet run on deployment-mira is OK: OK: Less than 1.00% above the threshold [0.0] [09:54:33] RECOVERY - Puppet run on integration-slave-trusty-1003 is OK: OK: Less than 1.00% above the threshold [0.0] [09:55:01] RECOVERY - Puppet run on integration-slave-jessie-1001 is OK: OK: Less than 1.00% above the threshold [0.0] [09:56:41] RECOVERY - Puppet run on deployment-apertium02 is OK: OK: Less than 1.00% above the threshold [0.0] [09:57:31] RECOVERY - Puppet run on integration-slave-trusty-1001 is OK: OK: Less than 1.00% above the threshold [0.0] [09:57:49] RECOVERY - Puppet run on deployment-fluorine02 is OK: OK: Less than 1.00% above the threshold [0.0] [09:58:15] RECOVERY - Puppet run on integration-slave-trusty-1016 is OK: OK: Less than 1.00% above the threshold [0.0] [09:59:17] RECOVERY - Puppet run on integration-puppetmaster01 is OK: OK: Less than 1.00% above the threshold [0.0] [10:00:11] RECOVERY - Puppet run on integration-slave-trusty-1012 is OK: OK: Less than 1.00% above the threshold [0.0] [10:01:07] RECOVERY - Puppet run on integration-slave-jessie-1005 is OK: OK: Less than 1.00% above the threshold [0.0] [10:05:54] RECOVERY - Puppet run on integration-slave-trusty-1013 is OK: OK: Less than 1.00% above the threshold [0.0] [10:19:17] 10Beta-Cluster-Infrastructure: Request Admin at deployment or cluster - https://phabricator.wikimedia.org/T147828#2704779 (10Krenair) What do you plan to use it for? [10:23:19] 10Continuous-Integration-Config, 13Patch-For-Review: Run MediaWiki tests on PHP 7 - https://phabricator.wikimedia.org/T144962#2705176 (10hashar) p:05Triage>03Low [10:25:28] 10Continuous-Integration-Config, 10Fundraising-Backlog, 13Patch-For-Review: wikimedia/fundraising/tools should pass python lint checks (flake8) - https://phabricator.wikimedia.org/T114218#2705178 (10hashar) That is apparently also tracked in private task {T145012} https://gerrit.wikimedia.org/r/#/c/314481/... [10:27:29] 10Continuous-Integration-Config, 10Kartotherian: maps/kartotherian/deploy does not go via jenkins - https://phabricator.wikimedia.org/T142740#2705180 (10hashar) [10:28:58] 10Continuous-Integration-Config, 10Kartotherian: maps/kartotherian/deploy and maps/tilerator/deploy do not have Jenkins jobs - https://phabricator.wikimedia.org/T142740#2545192 (10hashar) [10:29:05] 10Continuous-Integration-Config, 10Kartotherian, 10Tilerator: maps/kartotherian/deploy and maps/tilerator/deploy do not have Jenkins jobs - https://phabricator.wikimedia.org/T142740#2545192 (10hashar) [10:31:34] 10Continuous-Integration-Config, 10Kartotherian, 10Tilerator: maps/kartotherian and maps/tilerator repos do not have Jenkins jobs - https://phabricator.wikimedia.org/T142740#2705191 (10hashar) [10:33:06] PROBLEM - Long lived cherry-picks on puppetmaster on deployment-puppetmaster is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [10:43:50] haha, what is this alert ^ [10:43:51] haha [10:53:38] (03PS1) 10Hashar: Experimental jobs for Tilerator and Kartotherian [integration/config] - 10https://gerrit.wikimedia.org/r/315230 (https://phabricator.wikimedia.org/T142740) [10:55:29] (03CR) 10Hashar: [C: 032] "On each of the four repos, you will be able to comment in Gerrit 'check experimental' which will trigger the job. The all fails due to th" [integration/config] - 10https://gerrit.wikimedia.org/r/315230 (https://phabricator.wikimedia.org/T142740) (owner: 10Hashar) [10:56:13] PROBLEM - Puppet run on integration-slave-trusty-1012 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [10:56:27] (03Merged) 10jenkins-bot: Experimental jobs for Tilerator and Kartotherian [integration/config] - 10https://gerrit.wikimedia.org/r/315230 (https://phabricator.wikimedia.org/T142740) (owner: 10Hashar) [11:00:43] 10Continuous-Integration-Config, 10Kartotherian, 10Tilerator, 13Patch-For-Review: maps/kartotherian and maps/tilerator repos do not have Jenkins jobs - https://phabricator.wikimedia.org/T142740#2705219 (10hashar) On each of the four repos, you will be able to comment in Gerrit 'check experimental' which w... [11:01:34] 10Continuous-Integration-Config, 10Kartotherian, 10Tilerator: maps/kartotherian and maps/tilerator repos do not have Jenkins jobs - https://phabricator.wikimedia.org/T142740#2705220 (10hashar) [11:31:13] RECOVERY - Puppet run on integration-slave-trusty-1012 is OK: OK: Less than 1.00% above the threshold [0.0] [11:38:03] 10Continuous-Integration-Config, 07Browser-Tests, 07Ruby: Cucumber linter should run for all repositories that contain Cucumber code - https://phabricator.wikimedia.org/T58251#2705282 (10hashar) p:05Normal>03Low [11:40:29] 10Continuous-Integration-Config, 06Operations, 06Operations-Software-Development: Flake8 for python files without extension in puppet repo - https://phabricator.wikimedia.org/T144169#2705286 (10hashar) @volans maybe drive this again? I have voted for my preference, but really one way or the other is all fine... [11:43:33] 10Beta-Cluster-Infrastructure: Request Admin at deployment or cluster - https://phabricator.wikimedia.org/T147828#2705303 (10Aklapper) @Mbrt: Currently this task does not explain well //why// this task was created. It describes a potential solution to an underlying root problem that still needs to get defined. K... [11:43:59] (03PS1) 10Hashar: [ArticleFeedbackv5] Add npm job [integration/config] - 10https://gerrit.wikimedia.org/r/315235 (https://phabricator.wikimedia.org/T63588) [11:45:39] (03PS2) 10Hashar: [ArticleFeedbackv5] Add composer/npm jobs [integration/config] - 10https://gerrit.wikimedia.org/r/315235 (https://phabricator.wikimedia.org/T63588) [11:47:46] (03CR) 10Hashar: [C: 032] [ArticleFeedbackv5] Add composer/npm jobs [integration/config] - 10https://gerrit.wikimedia.org/r/315235 (https://phabricator.wikimedia.org/T63588) (owner: 10Hashar) [11:48:46] (03Merged) 10jenkins-bot: [ArticleFeedbackv5] Add composer/npm jobs [integration/config] - 10https://gerrit.wikimedia.org/r/315235 (https://phabricator.wikimedia.org/T63588) (owner: 10Hashar) [12:06:42] 10Continuous-Integration-Config, 10ArticleFeedbackv5, 06Brickimedia, 13Patch-For-Review: ArticleFeedbackv5 should pass jshint - https://phabricator.wikimedia.org/T63588#2705345 (10hashar) CI now runs `npm install && npm test`. Still has to add jshint in it and fix all the issues :] [12:14:37] 10Continuous-Integration-Config, 10ArticleFeedbackv5, 06Brickimedia, 13Patch-For-Review: ArticleFeedbackv5 should pass jshint - https://phabricator.wikimedia.org/T63588#2705351 (10Paladox) Thankyou [12:25:49] thcipriani|afk: I'm ok with merging https://gerrit.wikimedia.org/r/#/c/310719 though it'll fail now I think since arcrc is missing, let me know when you are around [12:53:15] 10Continuous-Integration-Config, 10MediaWiki-extensions-ReplaceText, 13Patch-For-Review: Add phpcs to ReplaceText extension - https://phabricator.wikimedia.org/T136589#2705484 (10hashar) 05Open>03Resolved a:03Paladox [12:57:18] (03PS1) 10Hashar: [translatewiki] switch composer job to hhvm [integration/config] - 10https://gerrit.wikimedia.org/r/315251 (https://phabricator.wikimedia.org/T97889) [12:57:43] (03Abandoned) 10Hashar: Add new hhvm lint and use it [integration/config] - 10https://gerrit.wikimedia.org/r/270938 (https://phabricator.wikimedia.org/T97889) (owner: 10Paladox) [12:57:51] (03CR) 10Hashar: [C: 032] [translatewiki] switch composer job to hhvm [integration/config] - 10https://gerrit.wikimedia.org/r/315251 (https://phabricator.wikimedia.org/T97889) (owner: 10Hashar) [12:58:31] (03CR) 10jenkins-bot: [V: 04-1] [translatewiki] switch composer job to hhvm [integration/config] - 10https://gerrit.wikimedia.org/r/315251 (https://phabricator.wikimedia.org/T97889) (owner: 10Hashar) [13:04:50] (03PS2) 10Hashar: [translatewiki] switch composer job to hhvm [integration/config] - 10https://gerrit.wikimedia.org/r/315251 (https://phabricator.wikimedia.org/T97889) [13:05:21] (03CR) 10Hashar: [C: 04-1] "Need a more specific job such as 'translatewiki-composer-hhvm-jessie'" [integration/config] - 10https://gerrit.wikimedia.org/r/315251 (https://phabricator.wikimedia.org/T97889) (owner: 10Hashar) [13:18:46] 06Release-Engineering-Team, 06Operations, 07Beta-Cluster-reproducible, 13Patch-For-Review: mwscript on jessie mediawiki fails - https://phabricator.wikimedia.org/T146286#2705534 (10hashar) So we have switched today the primary deployment server to mira.codfw.wmnet which is running Jessie. The European SWA... [13:36:03] 06Release-Engineering-Team, 06Operations, 07Beta-Cluster-reproducible, 13Patch-For-Review: mwscript on jessie mediawiki fails - https://phabricator.wikimedia.org/T146286#2705567 (10hashar) Moritz has run puppet on mira.codfw.wmnet (Jessie) and that fixed the issue above. We will most probably want to a si... [13:40:36] Project selenium-Wikibase » chrome,test,Linux,contintLabsSlave && UbuntuTrusty build #138: 09SUCCESS in 1 hr 49 min: https://integration.wikimedia.org/ci/job/selenium-Wikibase/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=test,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/138/ [13:46:28] Yippee, build fixed! [13:46:28] Project selenium-VisualEditor » firefox,beta,Linux,contintLabsSlave && UbuntuTrusty build #176: 09FIXED in 2 min 27 sec: https://integration.wikimedia.org/ci/job/selenium-VisualEditor/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/176/ [13:52:13] PROBLEM - Puppet run on integration-slave-trusty-1012 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [13:58:19] Project selenium-Wikibase » chrome,beta,Linux,contintLabsSlave && UbuntuTrusty build #138: 09SUCCESS in 2 hr 6 min: https://integration.wikimedia.org/ci/job/selenium-Wikibase/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/138/ [13:58:20] 10Beta-Cluster-Infrastructure: Request Admin at deployment or cluster - https://phabricator.wikimedia.org/T147828#2705607 (10hashar) 05Open>03declined Might be related to previous task {T143557} ? Closing for now, please reopen this task with more details / explanation of what you want to do. [14:00:09] 10Continuous-Integration-Config, 10ArticleFeedbackv5, 06Brickimedia: ArticleFeedbackv5 should pass jshint - https://phabricator.wikimedia.org/T63588#2705611 (10hashar) [14:08:29] thcipriani|afk, hashar: the icinga failure you mentioned does not seem to be present on the current mira in production? [14:17:30] moritzm: hrm, the patch does appear to be unneeded in production https://phabricator.wikimedia.org/P4194 [14:18:23] but I'm not clear why, openssh client and server as well as libssh2 are all the same versions. [14:20:16] that is a mystery :] [14:20:28] hashar hi thanks for merging my patches today [14:20:34] hmm. me neither [14:20:40] hashar it seems mysql is down on precise instances [14:20:49] so either mira on prod has an issue [14:20:53] or that is beta misbehaving [14:20:57] hashar see https://integration.wikimedia.org/ci/job/mwext-testextension-php53/2816/console [14:21:04] paladox: will restart it [14:21:10] Ok thank you :) [14:21:39] there is a slight kernel version difference. ¯\_(ツ)_/¯ [14:23:05] paladox: yeah mysql refuses to start bah [14:23:35] hmm, really should not make a difference, I'd say let's wait until next week when tin gets reimaged [14:23:50] Oh [14:24:01] I'm guessing that is a bug in mysql on precise [14:24:12] Since it seems to not happen frequently on trusty [14:31:00] 10Continuous-Integration-Infrastructure, 07Zuul: Run zuul-clear-refs.py daily on all our repositories to reclaim Zuul references - https://phabricator.wikimedia.org/T103528#2705712 (10hashar) [14:31:03] 10Continuous-Integration-Infrastructure, 06Operations, 07Zuul: Upgrade Zuul on scandium to 2.5.0-8-gcbc7f62-wmf3jessie1 - https://phabricator.wikimedia.org/T147073#2705708 (10hashar) 05Open>03Resolved a:03hashar Solved by @elukey but that does not fix it :( I have screwed up the package and the shebang... [14:32:11] RECOVERY - Puppet run on integration-slave-trusty-1012 is OK: OK: Less than 1.00% above the threshold [0.0] [14:37:17] !log Mysql was down on Precise slaves. Apparently rebooted 17 days ago and I guess mysql does not spawn on boot. Restarted mysql on all Precise via: salt -v '*slave-precise*' cmd.run 'start mysql' [14:37:21] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [14:37:22] paladox_: should be good now [14:37:56] 10Continuous-Integration-Infrastructure, 07Zuul: Run zuul-clear-refs.py daily on all our repositories to reclaim Zuul references - https://phabricator.wikimedia.org/T103528#2705752 (10hashar) Gave the package a try but the shebang for zuul-clear-refs.py is not rewritten :( The /usr/bin/zuul* are symlinks to /... [14:48:31] godog: hi! I saw your message in the backscroll. How do we normally handle secrets? I have a vague memory of some kind of placeholder being needed... [14:49:23] hashar thanks [14:49:24] :) [14:56:39] hashar Microsoft fixed that bug in edge that made it not work correctly with gerrit :) [14:56:50] But downside is it is windows insider build 14342 [14:57:03] and it has a bug that prevents me from awaking my pc from sleep [14:58:55] 10Continuous-Integration-Infrastructure, 07Zuul: Upgrade Zuul on gallium to 2.5.0-8-gcbc7f62-wmf3precise1 - https://phabricator.wikimedia.org/T147070#2705790 (10hashar) 05Open>03Resolved a:03hashar ``` $ zuul --version Zuul version: 2.5.0-8-gcbc7f62-wmf3precise1 $ apt-cache policy zuul zuul: Installed... [15:04:48] 06Release-Engineering-Team, 06Operations, 07HHVM, 13Patch-For-Review: Migrate deployment servers (tin/mira) to jessie - https://phabricator.wikimedia.org/T144578#2705801 (10MoritzMuehlenhoff) mira is now the primary deployment server. tin will be reimaged to jessie on the 18th. After that we can switch bac... [15:07:54] 03Scap3, 15User-mobrovac: Scap deploy failed to sync git-fat artifacts - https://phabricator.wikimedia.org/T147856#2705818 (10Eevans) [15:22:48] 10Continuous-Integration-Infrastructure, 07Zuul: Run zuul-clear-refs.py daily on all our repositories to reclaim Zuul references - https://phabricator.wikimedia.org/T103528#2705866 (10hashar) [15:22:50] 10Continuous-Integration-Infrastructure, 10Packaging, 13Patch-For-Review, 07Zuul: Package / puppetize zuul-clear-refs.py - https://phabricator.wikimedia.org/T103529#2705864 (10hashar) 05Resolved>03Open Does not work. The script is not properly installed and the shebang is not adjusted to point to the v... [15:24:37] (03PS1) 10Hashar: WMF: install zuul-clear-refs [integration/zuul] (patch-queue/debian/precise-wikimedia) - 10https://gerrit.wikimedia.org/r/315281 [15:25:07] (03PS2) 10Hashar: WMF: install zuul-clear-refs [integration/zuul] (patch-queue/debian/precise-wikimedia) - 10https://gerrit.wikimedia.org/r/315281 (https://phabricator.wikimedia.org/T103529) [15:25:32] (03PS3) 10Hashar: WMF: install zuul-clear-refs [integration/zuul] (patch-queue/debian/precise-wikimedia) - 10https://gerrit.wikimedia.org/r/315281 (https://phabricator.wikimedia.org/T103529) [15:32:25] thcipriani: yeah in labs/private there are placeholders, not sure how secrets are handled in beta tho [15:32:31] some are in Hiera:Deployment-prep :( [15:32:37] oh good [15:33:12] my plan was to patch labs/private directly on the puppetmaster [15:33:28] since this is a *little* secret [15:33:53] didn't want to commit it into labs/private to live there forever [15:38:50] godog: Hiera:Deployment-prep I think we will end up phasing it out [15:38:55] to ease the transition to horizon [15:39:30] it is too confusing to have multiple sources between hiera:deployment-prep , puppet.git /hieradata , cherry pick on beta puppetmaster and now Horizon puppet stuff [15:40:58] yeah very confusing alright with multiple sources hashar [15:41:35] thcipriani: heh I don't know of a good solution for non-prod secrets [15:42:18] (03PS1) 10Hashar: Link zuul-launcher in /usr/bin [integration/zuul] (debian/precise-wikimedia) - 10https://gerrit.wikimedia.org/r/315289 [15:42:20] (03PS1) 10Hashar: Fix zuul-clear-refs install and shebang [integration/zuul] (debian/precise-wikimedia) - 10https://gerrit.wikimedia.org/r/315290 [15:42:46] fwiw, cherry-picking on beta-puppetmaster has, I think, been the pattern. With the foreknowledge that nothing there is very secret. [15:42:54] godog: the wikitech and horizon interfaces at least have an advantage, it is straightforward to edit :] [15:47:23] thcipriani: before I leave I have question for you about scap :] [15:47:38] do we have support for a standalone scap deployment server for a single repo ? [15:47:46] I would like to get some software material in a git repo [15:47:50] that would be deployed on a single instance [15:47:58] the workflow would be: git push whatever/software [15:48:02] ssh instance-software [15:48:12] cd /srv/deployment/whatever/software && git pull && scap deploy [15:48:19] which would install the package locally :D [15:48:29] oh [15:48:36] I don't know that we've tried that before [15:49:09] the use case is to deploy a Java application that comes with 80MBytes / 266 .jar files [15:49:24] so instead of a deb package, stuff all of that to a git [15:49:40] but then I would like to get the soft installed on integration labs project which does not have a deploy server :D [15:50:09] ah, yeah, I don't think that would work. Scap would want to fetch from and deploy to the same server which would get...confusing :\ [15:50:17] ;D [15:51:00] also, we currently have the dependency on apache for git repos, etc. :( [15:51:21] yeah I noticed that [15:51:31] and reading one of the puppet manifest ... that also install mediawiki / zend php etc [15:52:01] (03PS2) 10Hashar: Fix zuul-clear-refs install and shebang [integration/zuul] (debian/precise-wikimedia) - 10https://gerrit.wikimedia.org/r/315290 [15:52:12] maybe it will be easier to just .deb them afterall ;] [15:54:20] yeah, the puppet manifests are a bit in flux. _joe_ has been working on them recently. They should be better soon. But your usecase isn't really covered. I do have some ideas for something like this, but nothing that's coming very quickly. [15:57:16] hashar im thinking we can also install php 7.1 on jessie [15:57:26] since they can exist together so php 7 and php 7.1 [15:57:44] paladox: poke the QA or wikitech list about it ? [15:57:50] Ok [15:58:00] would probably just use php 7.1 and that is certainly straightforward to do [15:58:52] Why do we need to? :P [15:59:24] Reedy includes some fixes, but not sure [15:59:30] things can break in php releases [15:59:36] hashar done [15:59:42] We never tested individual PHP 5 releases [15:59:45] only versions of matter [16:00:09] yeah, we don't need to churn unless there's a good reason [16:00:49] heading out [16:00:51] be back later tonight [16:01:15] Reedy mcrypt is removed from php 7.1 is that going to be a problem for mw? [16:01:22] Since i see it is in use in mw [16:01:52] I don't think it's a compulsory usage [16:02:11] paladox: If it is being... Please file a task about it. We'll have to do some handling etc [16:02:33] Ok [16:02:58] 10Gerrit, 06Operations: cronspam from cobalt after the Gerrit migration - https://phabricator.wikimedia.org/T147776#2705957 (10Dzahn) [16:03:01] 10Gerrit, 06Operations, 13Patch-For-Review: setup/deploy cobalt as gerrit warm standby/replacement - https://phabricator.wikimedia.org/T147597#2705956 (10Dzahn) [16:04:16] Reedy already done https://phabricator.wikimedia.org/T143788 [16:04:33] here https://dotdev.co/upcoming-changes-in-php-7-1-76ebea53b820#.8pjnzzqfj shows it as being deprecated and removed [16:04:38] Ah, deprecated, not removed yet [16:04:53] It's on the radar, at least [16:04:54] Thanks [16:04:55] Conflicting since php 7.1 has not had a stable release [16:04:58] so not sure [16:07:02] Reedy it has been replaced by OpenSSL [16:07:13] paladox: We already can use openssl [16:07:13] But will mcrypt will be removed in php 7.2 [16:07:16] And probably do [16:07:17] Yes [16:07:39] Just needs some graceful degredation for those using versions where they can still use mcrypt etc [16:08:13] Yep [16:08:20] Reedy looks like only 4 usage here https://github.com/wikimedia/mediawiki/search?utf8=%E2%9C%93&q=mcrypt&type=Code [16:08:39] There's 14 in core alone [16:08:46] Oh [16:08:47] Granted, a few are comments [16:09:17] yep [16:10:41] (03PS5) 10Paladox: Move script out of assert-phpflavor macro [integration/config] - 10https://gerrit.wikimedia.org/r/296061 (https://phabricator.wikimedia.org/T124572) [16:11:07] (03PS4) 10Paladox: Add assert-phpflavor.sh shell script [integration/jenkins] - 10https://gerrit.wikimedia.org/r/296060 (https://phabricator.wikimedia.org/T124572) [16:11:13] (03CR) 10jenkins-bot: [V: 04-1] Move script out of assert-phpflavor macro [integration/config] - 10https://gerrit.wikimedia.org/r/296061 (https://phabricator.wikimedia.org/T124572) (owner: 10Paladox) [16:11:26] 10Gerrit, 06Operations: cronspam from cobalt after the Gerrit migration - https://phabricator.wikimedia.org/T147776#2705992 (10Dzahn) Oh, it turns out it's not just permissions on the file, it's also that the DB part must have changed. We now see on both, old and new server, that: ``` {"type":"error","mess... [16:12:30] (03PS5) 10Paladox: Add assert-phpflavor.sh shell script [integration/jenkins] - 10https://gerrit.wikimedia.org/r/296060 (https://phabricator.wikimedia.org/T124572) [16:14:34] (03CR) 10Paladox: "recheck" [integration/config] - 10https://gerrit.wikimedia.org/r/296061 (https://phabricator.wikimedia.org/T124572) (owner: 10Paladox) [16:18:52] 10Gerrit, 06Operations, 13Patch-For-Review: cronspam from cobalt after the Gerrit migration - https://phabricator.wikimedia.org/T147776#2706030 (10Dzahn) Why the syntax error? We just moved gerrit, we did not upgrade the version. If anything i'd expect a permissions issue, but syntax ?? [16:24:03] 10Beta-Cluster-Infrastructure, 06Labs: Remove Labs uses of etcd and confd classes - https://phabricator.wikimedia.org/T147800#2706048 (10Andrew) a:03Joe [16:29:27] 03Scap3, 15User-mobrovac: Scap deploy failed to sync git-fat artifacts - https://phabricator.wikimedia.org/T147856#2706062 (10Eevans) >>! In T147856#2705882, @thcipriani wrote: > Was this the deploy today? > > I looked at the log for `/srv/deployment/cassandra/twcs` (using `scap deploy-log -v`) and didn't see... [16:32:52] 10Beta-Cluster-Infrastructure, 06Labs, 15User-Joe: Remove Labs uses of etcd and confd classes - https://phabricator.wikimedia.org/T147800#2706069 (10Joe) p:05Triage>03Normal [16:44:39] 10Gerrit, 06Operations, 13Patch-For-Review: cronspam from cobalt after the Gerrit migration - https://phabricator.wikimedia.org/T147776#2706106 (10Paladox) >>! In T147776#2706030, @Dzahn wrote: > Why the syntax error? We just moved gerrit, we did not upgrade the version. If anything i'd expect a permissions... [16:49:45] 03Scap3, 15User-mobrovac: Scap deploy failed to sync git-fat artifacts - https://phabricator.wikimedia.org/T147856#2706128 (10thcipriani) p:05Triage>03Normal >>! In T147856#2706062, @Eevans wrote: > My understanding is that the deploy happened as the result of merging https://gerrit.wikimedia.org/r/313892;... [16:51:15] 06Release-Engineering-Team, 15User-greg: Reach out to WMDE re [[mw:dev/maint]] - https://phabricator.wikimedia.org/T147861#2706135 (10greg) [16:51:49] 06Release-Engineering-Team, 15User-greg: Once over of [[mw:dev/maint]] for obvious out of dateness - https://phabricator.wikimedia.org/T147862#2706149 (10greg) [17:12:10] 10Gerrit, 10MediaWiki-extensions-OAuth, 10MediaWiki-extensions-OAuthAuthentication: Support OAuth for login onto gerrit.wikimedia.org - https://phabricator.wikimedia.org/T147864#2706225 (10MarcoAurelio) [17:12:25] 10Gerrit: Support OAuth for login onto gerrit.wikimedia.org - https://phabricator.wikimedia.org/T147864#2706238 (10MarcoAurelio) [17:16:03] 10Gerrit: Support OAuth for login onto gerrit.wikimedia.org - https://phabricator.wikimedia.org/T147864#2706264 (10Aklapper) gerrit-oauth-provider exist in the wild, but I don't know LDAP implications and whether it makes sense to investigate with #Differential being the long-term goal... [17:19:13] 10Gerrit: Support OAuth for login onto gerrit.wikimedia.org - https://phabricator.wikimedia.org/T147864#2706287 (10MarcoAurelio) I thought about differential. However my impression is that a total migration will still take some time. In the meanwhile, maybe we can benefit from it. Thanks. [17:27:08] 10Gerrit, 06Operations, 13Patch-For-Review: cronspam from cobalt after the Gerrit migration - https://phabricator.wikimedia.org/T147776#2706315 (10Dzahn) Yes, but how did it ever work before and now suddenly fail on both, old and new server? Or was it broken all the time and only noticed now by coincidence? [17:27:24] 10Continuous-Integration-Infrastructure, 10Wikidata: [Bug] github.com is 403ing downloads from Wikimedia CI during composer update - https://phabricator.wikimedia.org/T106519#1470716 (10greg) >>! In T106519#2705041, @hashar wrote: > And since that is over HTTPS we would need some kind of SSL man in the middle... [17:28:03] 10Gerrit, 06Operations, 13Patch-For-Review: cronspam from cobalt after the Gerrit migration - https://phabricator.wikimedia.org/T147776#2706327 (10Paladox) @Dzahn probably it was broken all the time. Just wasn't noticed until now. [17:28:52] 10Gerrit: Support OAuth for login onto gerrit.wikimedia.org - https://phabricator.wikimedia.org/T147864#2706225 (10Krenair) I'm not sure it's worth the work, we don't really need non-technical users in gerrit do we? [17:34:53] 10Gerrit: Support OAuth for login onto gerrit.wikimedia.org - https://phabricator.wikimedia.org/T147864#2706363 (10MarcoAurelio) I'd say that we don't due to the nature of gerrit. I recognize that it's mainly because of lazyness that I'm asking for this, so I don't have to type so many times a password to log in... [17:38:26] (03PS1) 10Krinkle: favicon.php: Allow backend response without Content-Length header [integration/docroot] - 10https://gerrit.wikimedia.org/r/315312 (https://phabricator.wikimedia.org/T147814) [17:38:56] (03PS2) 10Krinkle: favicon.php: Allow backend response without Content-Length header [integration/docroot] - 10https://gerrit.wikimedia.org/r/315312 (https://phabricator.wikimedia.org/T147814) [17:39:07] hasharAway: ^ [17:46:43] 10Beta-Cluster-Infrastructure, 06Collaboration-Team-Triage, 10Flow, 07Technical-Debt: Use dedicated $wgFlowCluster and $wgFlowDefaultWikiDb on Beta Cluster - https://phabricator.wikimedia.org/T147523#2706464 (10jmatazzoni) [18:00:40] (03CR) 10Krinkle: [C: 032] favicon.php: Allow backend response without Content-Length header [integration/docroot] - 10https://gerrit.wikimedia.org/r/315312 (https://phabricator.wikimedia.org/T147814) (owner: 10Krinkle) [18:01:07] (03Merged) 10jenkins-bot: favicon.php: Allow backend response without Content-Length header [integration/docroot] - 10https://gerrit.wikimedia.org/r/315312 (https://phabricator.wikimedia.org/T147814) (owner: 10Krinkle) [18:03:27] 10Gerrit: Support OAuth for login onto gerrit.wikimedia.org - https://phabricator.wikimedia.org/T147864#2706550 (10Tgr) @bd808 has plans for eventually merging LDAP with SUL, I think. [18:16:02] 10Beta-Cluster-Infrastructure, 10Continuous-Integration-Infrastructure, 10DBA, 10MediaWiki-Database, 07WorkType-NewFunctionality: Enable MariaDB/MySQL's Strict Mode - https://phabricator.wikimedia.org/T108255#2706641 (10Huji) [18:17:43] hasharAway hi, operations-puppet-doc is failing again [18:17:44] se [18:17:52] see https://integration.wikimedia.org/ci/job/operations-puppet-doc/26984/console [18:20:20] 06Release-Engineering-Team (Deployment-Blockers), 05Release: MW-1.28.0-wmf.21 deployment blockers - https://phabricator.wikimedia.org/T145220#2706662 (10greg) 05Open>03Resolved T147520 is not blocking at this point, resolving. [18:20:44] 10Gerrit, 06Operations, 13Patch-For-Review: cronspam from cobalt after the Gerrit migration - https://phabricator.wikimedia.org/T147776#2706666 (10Dzahn) 05Open>03Resolved a:03Dzahn 11:18 < mutante> !log lead (old gerrit) manually remove reviewer-count cron, puppet is disabled 11:19 < mutante> !log co... [18:24:21] PROBLEM - Host integration-puppetmaster is DOWN: CRITICAL - Host Unreachable (10.68.16.42) [18:25:59] hey releng - just learned that my vacation coincides in time and place with your offsite :P [18:26:21] MaxSem: you're welcome! [18:26:53] might as weel meet for booze [18:30:51] oh, and place! yeah! [18:57:18] (03PS3) 10Hashar: Fix zuul-clear-refs install and shebang [integration/zuul] (debian/precise-wikimedia) - 10https://gerrit.wikimedia.org/r/315290 (https://phabricator.wikimedia.org/T103529) [18:57:20] (03PS2) 10Hashar: Link zuul-launcher in /usr/bin [integration/zuul] (debian/precise-wikimedia) - 10https://gerrit.wikimedia.org/r/315289 [18:57:22] (03PS1) 10Hashar: source ignore .pyc files and .tox directory [integration/zuul] (debian/precise-wikimedia) - 10https://gerrit.wikimedia.org/r/315323 [18:59:17] Krinkle: did your patches solved the favicon / content-length issue ? [19:10:23] hashar: Yes, confirmed. [19:10:36] 10Continuous-Integration-Infrastructure, 06Operations, 10Traffic, 13Patch-For-Review, 07Regression: Favicon broken on doc.wikimedia.org and integration.wikimedia.org (HTTP 500) - https://phabricator.wikimedia.org/T147814#2706944 (10Krinkle) 05Open>03Resolved a:03Krinkle [19:31:12] 10Beta-Cluster-Infrastructure, 10Continuous-Integration-Infrastructure, 10DBA, 10MediaWiki-Database, 07WorkType-NewFunctionality: Enable MariaDB/MySQL's Strict Mode - https://phabricator.wikimedia.org/T108255#2707037 (10RobLa-WMF) Thanks @Huji for noting the SecurePoll strict mode problems (T147875).... [19:35:49] MaxSem: I presume your contact info on the officewiki contact page is up to date? [19:39:30] yup! [19:43:58] PROBLEM - Puppet run on repository is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [19:49:44] hashar: Can we do something about code coverage failures resulting in an empty directory being published? [19:50:05] I'm unsure what to do. It seems we ignore the error intentionally, but I forgot why. [19:50:25] Krinkle: ah poor code coverage [19:50:32] yeah it tends to segfault or get some random errors [19:50:41] so the result is ignored and the output rsynced away [19:50:52] hashar: when it segfaults, is there output? [19:52:16] maybe after the error is ignored we could add an assert to make sure there is a clover.xml generated [19:52:20] might be sufficent [19:52:33] (given a clover.xml implies some html tree got generated in 'cover' dir) [19:52:53] would have to inspect a build that caused the html report to be wiped out [19:55:45] 06Release-Engineering-Team (Deployment-Blockers), 05Release: MW-1.28.0-wmf.21 deployment blockers - https://phabricator.wikimedia.org/T145220#2707073 (10greg) [20:13:19] (03CR) 10Hashar: [C: 032] Fix zuul-clear-refs install and shebang [integration/zuul] (debian/precise-wikimedia) - 10https://gerrit.wikimedia.org/r/315290 (https://phabricator.wikimedia.org/T103529) (owner: 10Hashar) [20:13:21] (03CR) 10Hashar: [C: 032] Link zuul-launcher in /usr/bin [integration/zuul] (debian/precise-wikimedia) - 10https://gerrit.wikimedia.org/r/315289 (owner: 10Hashar) [20:13:23] (03CR) 10Hashar: [C: 032] source ignore .pyc files and .tox directory [integration/zuul] (debian/precise-wikimedia) - 10https://gerrit.wikimedia.org/r/315323 (owner: 10Hashar) [20:19:52] 10Continuous-Integration-Infrastructure, 07Zuul: Fix Zuul package "postinst called with unknown argument `triggered' - https://phabricator.wikimedia.org/T146084#2707135 (10hashar) [20:24:00] RECOVERY - Puppet run on repository is OK: OK: Less than 1.00% above the threshold [0.0] [20:36:00] (03PS1) 10Hashar: 2.5.0-8-gcbc7f62-wmf4precise1 [integration/zuul] (debian/precise-wikimedia) - 10https://gerrit.wikimedia.org/r/315352 (https://phabricator.wikimedia.org/T103529) [20:46:48] 10Beta-Cluster-Infrastructure, 10MediaWiki-extensions-CentralAuth, 10MobileFrontend, 06Reading-Web-Backlog: beta cluster: Notice: Undefined variable: wmgMFUseCentralAuthToken in /srv/mediawiki-staging/wmf-config/mobile-labs.php on line 20 - https://phabricator.wikimedia.org/T146945#2707235 (10Jdlrobson) [20:48:48] PROBLEM - Puppet run on deployment-fluorine02 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [20:54:35] RECOVERY - Puppet run on zuul-dev-jessie is OK: OK: Less than 1.00% above the threshold [0.0] [21:08:32] 06Release-Engineering-Team (Deployment-Blockers), 05Release: MW-1.28.0-wmf.22 deployment blockers - https://phabricator.wikimedia.org/T146998#2707328 (10Jdforrester-WMF) [21:10:20] PROBLEM - Puppet run on deployment-restbase02 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [21:10:30] PROBLEM - Puppet run on deployment-restbase01 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [21:15:34] PROBLEM - Puppet run on zuul-dev-jessie is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [21:30:45] 06Release-Engineering-Team, 10Wikimedia-Developer-Summit, 06Developer-Relations (Oct-Dec-2016): Developer Summit 2017: Work with TPG and RelEng on solution to event documenting - https://phabricator.wikimedia.org/T132400#2707475 (10Rfarrand) p:05Lowest>03High [21:30:51] 10Continuous-Integration-Infrastructure, 07Zuul: Fix Zuul package "postinst called with unknown argument `triggered' - https://phabricator.wikimedia.org/T146084#2707476 (10hashar) 05Open>03Resolved a:03hashar self note: I wanted to look at the actual postinst content. The way to do it is by using dpkg-d... [21:35:41] !log Force pushed Zuul patchqueue 5628f95...fc6a118 HEAD -> patch-queue/debian/precise-wikimedia [21:35:45] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [21:36:52] PROBLEM - Puppet run on deployment-redis01 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [21:41:46] (03PS1) 10Hashar: Merge branch 'debian/precise-wikimedia' into debian/jessie-wikimedia [integration/zuul] (debian/jessie-wikimedia) - 10https://gerrit.wikimedia.org/r/315429 [21:41:48] (03PS1) 10Hashar: 2.5.0-8-gcbc7f62-wmf4jessie1 [integration/zuul] (debian/jessie-wikimedia) - 10https://gerrit.wikimedia.org/r/315430 (https://phabricator.wikimedia.org/T103529) [21:50:19] RECOVERY - Puppet run on deployment-restbase02 is OK: OK: Less than 1.00% above the threshold [0.0] [21:50:31] RECOVERY - Puppet run on deployment-restbase01 is OK: OK: Less than 1.00% above the threshold [0.0] [22:16:52] RECOVERY - Puppet run on deployment-redis01 is OK: OK: Less than 1.00% above the threshold [0.0] [22:18:32] 10Beta-Cluster-Infrastructure, 10Flow, 06Collaboration-Team-Triage (Collab-Team-Q2-Oct-Dec-2016), 07Technical-Debt: Use dedicated $wgFlowCluster and $wgFlowDefaultWikiDb on Beta Cluster - https://phabricator.wikimedia.org/T147523#2707604 (10jmatazzoni) [22:55:19] 06Release-Engineering-Team, 15User-greg: Create RelEng FY1617Q1 QR slides (due 10/14) - https://phabricator.wikimedia.org/T146515#2707685 (10greg) [22:55:21] 06Release-Engineering-Team, 15User-greg: Compile time spent graphs for FY1617Q1 - https://phabricator.wikimedia.org/T146517#2707684 (10greg) 05Open>03Resolved [23:09:58] 10Gerrit, 06Operations, 13Patch-For-Review: setup/deploy cobalt as gerrit warm standby/replacement - https://phabricator.wikimedia.org/T147597#2707712 (10Dzahn) [23:10:06] 10Gerrit, 06Operations, 10hardware-requests: Allocate spare misc box in eqiad for gerrit replacement - https://phabricator.wikimedia.org/T147596#2707714 (10Dzahn) [23:10:09] 10Gerrit, 06Operations, 13Patch-For-Review: setup/deploy cobalt as gerrit warm standby/replacement - https://phabricator.wikimedia.org/T147597#2697753 (10Dzahn) 05Open>03Resolved [23:17:42] 10scap, 06Operations, 06Performance-Team, 07Epic: During deployment old servers may populate new cache URIs - https://phabricator.wikimedia.org/T47877#2707719 (10Krinkle) [23:57:45] 06Release-Engineering-Team (Deployment-Blockers), 05Release: MW-1.28.0-wmf.22 deployment blockers - https://phabricator.wikimedia.org/T146998#2707818 (10AlexMonk-WMF)