[00:04:42] PROBLEM - Puppet run on deployment-ms-fe01 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [00:44:43] RECOVERY - Puppet run on deployment-ms-fe01 is OK: OK: Less than 1.00% above the threshold [0.0] [00:55:36] PROBLEM - Puppet run on deployment-kafka05 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [01:17:18] 06Release-Engineering-Team: Remove .gitreview from MediaWiki and Extensions - https://phabricator.wikimedia.org/T146293#2658034 (10mmodell) git-review can figure out the right branch to use if you set `--track` [01:21:47] 06Release-Engineering-Team: Remove .gitreview from MediaWiki and Extensions - https://phabricator.wikimedia.org/T146293#2658036 (10mmodell) We could just remove the defaultbranch setting from the .gitreview file and require people to use --track [01:30:36] RECOVERY - Puppet run on deployment-kafka05 is OK: OK: Less than 1.00% above the threshold [0.0] [01:50:08] (03PS1) 10Legoktm: Whitelist Sam Wilson [integration/config] - 10https://gerrit.wikimedia.org/r/312158 [01:51:33] (03CR) 10Legoktm: [C: 032] Whitelist Sam Wilson [integration/config] - 10https://gerrit.wikimedia.org/r/312158 (owner: 10Legoktm) [01:52:10] (03Merged) 10jenkins-bot: Whitelist Sam Wilson [integration/config] - 10https://gerrit.wikimedia.org/r/312158 (owner: 10Legoktm) [01:52:17] !log deploying https://gerrit.wikimedia.org/r/312158 [01:52:21] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [02:46:37] PROBLEM - Puppet run on deployment-elastic08 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [03:06:18] PROBLEM - Puppet run on deployment-eventlogging04 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [03:11:22] Project mediawiki-core-code-coverage build #2277: 04STILL FAILING in 11 min: https://integration.wikimedia.org/ci/job/mediawiki-core-code-coverage/2277/ [03:13:41] PROBLEM - Puppet run on deployment-mathoid is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [03:21:38] RECOVERY - Puppet run on deployment-elastic08 is OK: OK: Less than 1.00% above the threshold [0.0] [03:41:24] RECOVERY - Puppet run on deployment-eventlogging04 is OK: OK: Less than 1.00% above the threshold [0.0] [03:46:10] (03PS1) 10Legoktm: Release 0.8.0-alpha.1 [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/312169 [03:46:15] James_F: ^ [03:47:14] Have we done an alpha release before [03:47:15] ? [03:47:39] If we like it, we'll re-label it as 0.8.0 in the HISTORY.md file? [03:50:18] James_F: no and yes [03:50:26] Kk. [03:50:37] (03CR) 10Jforrester: [C: 032] Release 0.8.0-alpha.1 [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/312169 (owner: 10Legoktm) [03:51:12] legoktm: Hmm. Both jobs queued. Not a great sign. [03:51:37] nodepool is probably building more [03:51:42] yeah, there they go [03:51:47] Ah, neat. [03:52:11] (03Merged) 10jenkins-bot: Release 0.8.0-alpha.1 [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/312169 (owner: 10Legoktm) [03:53:42] RECOVERY - Puppet run on deployment-mathoid is OK: OK: Less than 1.00% above the threshold [0.0] [03:53:53] tag pushed, I'll send an email out after dinner [03:53:58] Cool. [04:19:03] Yippee, build fixed! [04:19:04] Project selenium-MultimediaViewer » firefox,beta,Linux,contintLabsSlave && UbuntuTrusty build #149: 09FIXED in 23 min: https://integration.wikimedia.org/ci/job/selenium-MultimediaViewer/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/149/ [05:04:30] PROBLEM - Puppet run on deployment-tmh01 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [05:26:40] PROBLEM - Puppet staleness on deployment-aqs01 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [43200.0] [05:39:29] RECOVERY - Puppet run on deployment-tmh01 is OK: OK: Less than 1.00% above the threshold [0.0] [06:26:19] allah is doing [06:26:24] sun is not doing allah is doing [06:26:30] moon is not doing allah is doing [06:26:36] stars are not doing allah is doing [06:26:47] planets are not doing allah is doing [06:26:54] galaxies are not doing allah is doing [06:27:01] oceans are not doing allah is doing [06:27:11] mountains are not doing allah is doing [06:27:19] trees are not doing allah is doing [06:27:27] mom is not doing allah is doing [06:27:34] dad is not doing allah is doing [06:27:44] boss is not doing allah is doing [06:27:50] job is not doing allah is doing [06:27:57] dollar is not doing allah is doing [06:28:01] shut up [06:28:28] degree is not doing allah is doing [06:28:35] medicine is not doing allah is doing [06:28:44] customers are not doing allah is doing [06:29:02] you can not get a job without the permission of allah [06:29:14] you can not get married without the permission of allah [06:29:29] nobody can get angry at you without the permission of allah [06:29:43] light is not doing allah is doing [06:29:49] fan is not doing allah is doing [06:29:57] businessess are not doing allah is doing [06:30:03] america is not doing allah is doing [06:30:39] fire can not burn without the permission of allah [06:30:50] knife can not cut without the permission of allah [06:30:56] rulers are not doing allah is doing [06:31:18] governments are not doing allah is doing [06:31:33] sleep is not doing allah is doing [06:31:58] hunger is not doing allah is doing [06:32:29] food does not take away the hunger allah takes away the hunger [06:32:42] water does not take away the thirst allah takes away the thirst [06:32:51] seeing is not doing allah is doing [06:32:57] hearing is not doing allah is doing [06:33:03] seasons are not doing allah is doing [06:33:10] weather is not doing allah is doing [06:33:22] humans are not doing allah is doing [06:33:31] animals are not doing allah is doing [06:33:56] the best amongst you are those who learn and then teach quran [06:34:21] one letter read from book of allah amounts to one good deed and allah multiplies one good deed ten times [06:35:03] hearts get rusted as does iron with water to remove rust from heart recitation of quran and rememberance of death [06:35:13] heart is likened to a mirror [06:35:26] when a person commits one sin a black dot sustains the heart [06:36:27] to accept islam say that i bear witness that there is no deity worthy of worship except allah and muhammad peace be upon him is his slave and messenger [06:39:36] read book http://www.fazaileamaal.com [06:39:53] read book http://www.muntakhabahadith.com [06:41:16] need spiritual teacher visit http://www.alhaadi.org.za [06:57:52] 06Release-Engineering-Team: Remove .gitreview from MediaWiki and Extensions - https://phabricator.wikimedia.org/T146293#2658247 (10Paladox) But that will be a breaking change, unless we write too all our forums and make sure ops know about this change. [07:54:17] 10Gerrit: Update gerrit to 2.13 - https://phabricator.wikimedia.org/T146350#2658303 (10Paladox) [08:44:53] 06Release-Engineering-Team, 10MediaWiki-JobRunner, 06Operations, 10Trebuchet: Some Trebuchet minions are not responding to salt call when deploying jobrunner - https://phabricator.wikimedia.org/T146352#2658377 (10hashar) [08:46:07] I've imported the prometheus node stats in labs grafana, looks like beta prometheus is working \o/ [08:46:13] https://grafana-labs-admin.wikimedia.org/dashboard/db/prometheus-machine-stats?from=1474523125125&to=1474533925125&var-server=deployment-puppetmaster%3A9100&var-datasource=Beta%20Prometheus [08:46:29] err, the public version https://grafana-labs.wikimedia.org/dashboard/db/prometheus-machine-stats?from=1474523125125&to=1474533925125&var-server=deployment-puppetmaster%3A9100&var-datasource=Beta%20Prometheus [09:01:00] 06Release-Engineering-Team, 10MediaWiki-JobRunner, 06Operations, 10Trebuchet: Some Trebuchet minions are not responding to salt call when deploying jobrunner - https://phabricator.wikimedia.org/T146352#2658407 (10hashar) 05Open>03declined I am giving up trying to deploy jobrunner update. The whole Tre... [09:03:49] 10Deployment-Systems, 03Scap3 (Scap3-Adoption-Phase1), 10scap, 10MediaWiki-JobRunner: Deploy jobrunner with scap3 - https://phabricator.wikimedia.org/T129148#2096570 (10hashar) I tried to deploy the jobrunner service via Trebuchet and it is completely broken: * 7 of 41 deployment target are mysteriously n... [09:07:07] godog: I am all confused by that board :] [09:07:35] godog: is prometheus collecting it is own metric or does it reuse the Diamond metrics that are in Graphite servers.* ? [09:07:57] the former, via prometheus-node-exporter [09:08:47] so we collect twice right ? :)] [09:09:37] the board looks nice for sure :] [09:10:24] heheh indeed, but yeah collected twice [09:40:55] 06Release-Engineering-Team, 10MediaWiki-JobRunner, 06Operations, 10Trebuchet: Some Trebuchet minions are not responding to salt call when deploying jobrunner - https://phabricator.wikimedia.org/T146352#2658493 (10hashar) 05declined>03Open So @volans found: NameError: global name '__pillar__' is n... [09:43:12] 06Release-Engineering-Team, 10MediaWiki-JobRunner, 06Operations, 10Trebuchet: Some Trebuchet minions are not responding to salt call when deploying jobrunner - https://phabricator.wikimedia.org/T146352#2658510 (10Volans) >>! In T146352#2658493, @hashar wrote: > So @volans found: > > NameError: global... [09:44:07] 06Release-Engineering-Team, 10MediaWiki-JobRunner, 06Operations, 10Trebuchet: Some Trebuchet minions are not responding to salt call when deploying jobrunner - https://phabricator.wikimedia.org/T146352#2658511 (10hashar) 05Open>03Resolved a:03Joe Giuseppe has done all the magic restart of salt minion... [10:11:14] 06Release-Engineering-Team (Deployment-Blockers), 05Release: MW-1.28.0-wmf.20 deployment blockers - https://phabricator.wikimedia.org/T144644#2606121 (10Addshore) So what will be getting deployed today? [10:16:10] (03PS1) 10Hashar: tox: archive tox logs [integration/config] - 10https://gerrit.wikimedia.org/r/312221 [10:18:28] (03CR) 10Hashar: [C: 032] tox: archive tox logs [integration/config] - 10https://gerrit.wikimedia.org/r/312221 (owner: 10Hashar) [10:19:25] (03Merged) 10jenkins-bot: tox: archive tox logs [integration/config] - 10https://gerrit.wikimedia.org/r/312221 (owner: 10Hashar) [11:03:51] 10Gerrit: Update gerrit to 2.13 - https://phabricator.wikimedia.org/T146350#2658622 (10Aklapper) p:05Triage>03Low [11:25:14] PROBLEM - Puppet run on deployment-ores-redis is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [11:26:32] 06Release-Engineering-Team (Deployment-Blockers), 05Release: MW-1.28.0-wmf.20 deployment blockers - https://phabricator.wikimedia.org/T144644#2658643 (10Addshore) Scrap that, I have just seen wikitech-l! > Group0 wikis (mediawikiwiki, test2wiki, testwiki, testwikidatawiki, and > zerowiki) are running versio... [11:53:23] 10Gerrit: In changeset view, clicking the "Owner" name does not display list of their changesets but of unrelated other user - https://phabricator.wikimedia.org/T146361#2658656 (10Aklapper) [12:00:13] RECOVERY - Puppet run on deployment-ores-redis is OK: OK: Less than 1.00% above the threshold [0.0] [12:42:14] 10Gerrit, 06Operations: Update gerrit to 2.13 - https://phabricator.wikimedia.org/T146350#2658694 (10Peachey88) [12:43:42] 10Gerrit, 07Upstream: In changeset view, clicking the "Owner" name does not display list of their changesets but of unrelated other user - https://phabricator.wikimedia.org/T146361#2658695 (10Peachey88) [13:05:27] 06Release-Engineering-Team, 10TimedMediaHandler-Player: ogv.js player fails due to extension files being from newer version of MediaWiki than is deployed - https://phabricator.wikimedia.org/T146363#2658738 (10brion) [13:05:43] PROBLEM - Puppet run on deployment-ms-fe01 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [13:08:30] I'm pretty sure $wgExtensionAssetsPath used to point to a versioned directory [13:08:44] but now it seems to load a version that may, or may not, match what's deployed? [13:08:45] * brion confused [13:11:58] specifically in $wgExtensionAssetsPath I'm seeing file versions from master that aren't in 1.28.0wmf-.18 [13:12:07] which results in a version mismatch [13:12:10] which breaks my code :D [13:12:51] should I assume that $wgExtensionAssetsPath must never be used anymore? [13:13:02] or can we make them match again? [13:14:30] PROBLEM - Puppet run on deployment-cache-text04 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [13:19:15] 06Release-Engineering-Team: Remove .gitreview from MediaWiki and Extensions - https://phabricator.wikimedia.org/T146293#2658764 (10hashar) TLDR: `git-review` is now smart enough to push to the tracked branch if instructed to do so with the `track` option. Given a local checkout having a branch `REL1_27` set up... [13:22:12] 06Release-Engineering-Team: Remove .gitreview from MediaWiki and Extensions - https://phabricator.wikimedia.org/T146293#2658766 (10hashar) [13:23:10] 06Release-Engineering-Team, 10MediaWiki-ResourceLoader, 10TimedMediaHandler-Player: ogv.js player fails due to extension files being from newer version of MediaWiki than is deployed - https://phabricator.wikimedia.org/T146363#2658769 (10hashar) [13:23:42] brion: I cant remember the details sorry :( [13:23:58] brion: I remember one of i18n extension (maybe ULS) suffered from a similar issue [13:24:59] fun :D [13:25:52] brion: I am trying to find some doc [13:26:00] tx [13:26:21] oh [13:26:35] and also /srv/mediawiki/php points to php-1.28.0-wmf.19 [13:26:49] thcipriani|afk: oddity /php points to wmf.19 :) [13:28:12] so the files i'm seeing in commons.wikimedia.org/w/extensions are in master and in 1.28.0-wmf.20 but not in 1.28-wmf.18 (deployed MW on commons) or 1.28-wmf.19 [13:28:20] (file versions) [13:29:18] RL correctly serves out files from 1.28.0-wmf.18, as i expect [13:29:29] but the secondary files loaded over https are the master ones [13:29:57] which I have hard time figuring out [13:30:06] since master is not on the servers! [13:30:09] heh [13:30:18] hmm [13:31:30] hashar: are you talking about the fonts? We changed ULS to append the file hashes which are handled by this magic wrapper around static files [13:31:57] I cant remember [13:32:11] but I think ULS had an issue loading to the hardcoded paths instead of RL [13:32:28] there is also the new static.php I havent caught up with [13:32:37] yeah that thing exactly [13:32:59] it's supposed to know by hash and domain which file to serve [13:33:17] static stuff states: "Apache configuration rewrites "/w/skins/*", "/w/resources/*", and "/w/extension/*" to static.php [13:33:23] static.php streams the file from the appropiate MediaWiki branch directory. [13:33:37] hmmmm, sounds like it _should_ work then [13:34:37] unless there's bugs ;) [13:35:41] brion: the commit * ca15328 - Update ogv.js to 1.2.0 [13:35:49] that is in 1.28.0-wmf.20 [13:35:56] right [13:36:02] and commons is running 1.28.0-wmf.18 [13:36:07] so not just master :] [13:36:13] that part was confusing me [13:36:15] yeah, master and .20 :) [13:36:41] 06Release-Engineering-Team, 10MediaWiki-ResourceLoader, 10TimedMediaHandler-Player: ogv.js player fails due to extension files being from newer version of MediaWiki than is deployed - https://phabricator.wikimedia.org/T146363#2658793 (10hashar) [13:36:44] edited [13:39:09] 06Release-Engineering-Team, 10MediaWiki-ResourceLoader, 10TimedMediaHandler-Player: ogv.js player fails due to extension files being from newer version of MediaWiki than is deployed - https://phabricator.wikimedia.org/T146363#2658800 (10brion) per IRC discussion: the /w/extensions/* files are rewritten via A... [13:39:36] the script is in operations/mediawiki-config.git under [13:39:37] just explaining to myself for later reference that it should work :D [13:39:39] w/static.php [13:39:44] yeah I do that as well :] [13:40:01] that is how I end up with bugs that are several pages long [13:41:59] so somehow static.php fails to lookup commons [13:42:11] or think it uses wmf.20 [13:42:17] or fallback to it [13:42:40] brion: if you feed adventurous, you could live edit it on mw1099.eqiad.wmnet :] [13:43:11] it is probably easier than trying to figure out the code pass traversing several different stacks [13:43:18] oh hey: // - Requests without a validation hash will get the latest version. [13:43:40] haha [13:43:41] i'm guessing the hash format i'm using in ogv.js isn't getting matched here :D [13:45:02] if ( strlen( $urlHash ) !== 5 ) { // Garbage query string. Give same response as for requests with [13:45:07] o_O [13:45:43] RECOVERY - Puppet run on deployment-ms-fe01 is OK: OK: Less than 1.00% above the threshold [0.0] [13:46:38] Yippee, build fixed! [13:46:39] Project selenium-VisualEditor » firefox,beta,Linux,contintLabsSlave && UbuntuTrusty build #153: 09FIXED in 2 min 37 sec: https://integration.wikimedia.org/ci/job/selenium-VisualEditor/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/153/ [13:47:29] ok so this will basically strike any third-party library that has resources to load via url that might change from version to version unless there's a very specific way to hack in a specific md5 hash 5-char prefix as a query string suffix on all the urls :D [13:47:33] let me ponder on this [13:48:57] Timo most probably wrote a spec / doc about its behavior [13:48:59] hashar: where are we with tests of the jessie deployment servers, can we upgrade mira in production tomorrow? [13:49:34] moritzm: yes! [13:50:00] moritzm: I think we validated it somewhere, but the comment must have been lost in the spam of various comments [13:50:11] moritzm: I event got deployment-tin02 running Jessie [13:50:42] ok, let's do this tomorrow morning. the releng can run some more tests during the week of the ops offsite and we can pick a date for reimaging tin [13:50:49] ok, let's do this tomorrow morning. then releng can run some more tests during the week of the ops offsite and we can pick a date for reimaging tin [13:52:44] it'd be nice if it used the deployed branch instead of the most recent of all available branches, that would fix my problem [13:53:59] i'll see if i can rig up a hackaround for now [13:54:01] brion: I guess you are up to fill a feature request for Timo :D [13:54:30] RECOVERY - Puppet run on deployment-cache-text04 is OK: OK: Less than 1.00% above the threshold [0.0] [13:55:03] :D [13:55:41] hashar: well per spec it's consolidating across domains, so using deployed version is a no-go [13:56:06] i can just change my hashing to use static.php's format. not scalable but it'll do ;) [13:58:10] (03Abandoned) 10Hashar: Hardcode wiki page header [integration/dashboard] - 10https://gerrit.wikimedia.org/r/244723 (owner: 10Hashar) [13:58:13] (03Abandoned) 10Hashar: (WIP) wrapp update_composer in a class [integration/dashboard] - 10https://gerrit.wikimedia.org/r/245870 (owner: 10Hashar) [13:58:37] (03Abandoned) 10Hashar: make-wmf-branch: forge Zuul cloner parameters [tools/release] - 10https://gerrit.wikimedia.org/r/161225 (owner: 10Hashar) [14:03:31] moritzm: elukey: we did test the master switch and had the Jessie dpeloy server acting as the deploy system. So it is all green for us [14:03:39] moritzm: elukey: we could use https://gerrit.wikimedia.org/r/#/c/311947/ to land :] [14:03:56] hashar: ack, will look into it 15 mins [14:05:07] (03Abandoned) 10Hashar: debian-glue: fix zuul-cloner detached branch [integration/config] - 10https://gerrit.wikimedia.org/r/301763 (https://phabricator.wikimedia.org/T141607) (owner: 10Hashar) [14:06:47] hmmm no comment at top says it should be current ver by host name [14:08:21] AHA [14:08:22] ok i see [14:08:32] because i have a hash, it's not putting the *current* version on top of the stack [14:08:56] but because i have a bogus hash, it's using the *latest* [14:08:59] ok now it makes sense [14:09:17] i think i just need to make it reject the non-compliant hash earlier? [14:09:30] I have no idea really :( [14:11:21] oh i'm just thinking out loud :D [14:12:56] 06Release-Engineering-Team, 10MediaWiki-ResourceLoader, 10TimedMediaHandler-Player: ogv.js player fails due to extension files being from newer version of MediaWiki than is deployed - https://phabricator.wikimedia.org/T146363#2658884 (10brion) Looking at static.php, looks like it's not pulling from the curre... [14:18:03] hashar: ok to merge https://gerrit.wikimedia.org/r/#/c/311947/ now? [14:18:15] moritzm: yeah [14:18:40] moritzm: while at it you can land https://gerrit.wikimedia.org/r/#/c/311954/ which is some cleanup of legacy code [14:20:35] hashar: merged, will have a look at the second patch later [14:21:01] I have stomped on some old code yesterday while making deployment-mira a jenkins slave [14:21:10] so I thought it was a good time to drop the legacy code :D [14:26:22] ok i have a patch for static.php in https://gerrit.wikimedia.org/r/#/c/312254/ if anyone dares look it over :D i think it should resolve my bug (might or might not need to clear cache entries too though) [14:29:00] hmm, probably applying that fix and then SWATing the ogv.js 1.2.0 update into the deployment branch would fix it completely [14:44:44] RECOVERY - Puppet run on integration-slave-precise-1011 is OK: OK: Less than 1.00% above the threshold [0.0] [14:49:18] 10Continuous-Integration-Infrastructure, 07WorkType-Maintenance: phplint fails on paths containing a space - https://phabricator.wikimedia.org/T89380#1034975 (10Nemo_bis) The bug is back on https://gerrit.wikimedia.org/r/#/c/262087/2 / https://integration.wikimedia.org/ci/job/php55lint/25375/console , AFAICT. [14:51:21] 10Continuous-Integration-Infrastructure, 07WorkType-Maintenance: phplint fails on paths containing a space - https://phabricator.wikimedia.org/T89380#2659151 (10Nemo_bis) >>! In T89380#1035312, @Krinkle wrote: > Use the `-z` option. This seems a trivial and appropriate solution. >>! In T89380#1761715, @hasha... [15:02:00] PROBLEM - Puppet run on deployment-tin02 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [15:02:10] ostriches hi, i think we should wait for gerrit 2.12.5 since there is a bug in gerrit 2.12.4 that prevents you from allowing any new emails [15:06:44] 10Continuous-Integration-Infrastructure: Migrate CI labs slaves to use /srv instead of /mnt - https://phabricator.wikimedia.org/T146381#2659219 (10hashar) [15:10:37] Project mediawiki-core-code-coverage build #2278: 04STILL FAILING in 10 min: https://integration.wikimedia.org/ci/job/mediawiki-core-code-coverage/2278/ [15:11:59] RECOVERY - Puppet run on deployment-tin02 is OK: OK: Less than 1.00% above the threshold [0.0] [15:14:25] PROBLEM - Puppet run on integration-slave-precise-1002 is CRITICAL: CRITICAL: 11.11% of data above the critical threshold [0.0] [15:17:01] 10Gerrit: Update gerrit to 2.12.4 - https://phabricator.wikimedia.org/T143089#2659244 (10Paladox) I think we should not go with gerrit 2.12.4 but instead go with 2.12.5 when released. Since there Is a very bad bug that prevents any new emails from being processed which is fixed in gerrit 2.12.5. [15:21:11] hashar hi, i made this https://github.com/paladox/EdgeWikimediaDebug last night, not actually just converted from chrome to edge with a few minor ajustments [15:21:22] using an chrome to edge convert from microsoft :) [15:23:13] 10Continuous-Integration-Infrastructure: Jenkins: /srv/deployment/integration/slave-scripts/bin/mw-install-mysql.sh: No such file or directory - https://phabricator.wikimedia.org/T146382#2659255 (10Mattflaschen-WMF) [15:24:24] RECOVERY - Puppet run on integration-slave-precise-1002 is OK: OK: Less than 1.00% above the threshold [0.0] [15:28:32] We don't lint sql? https://gerrit.wikimedia.org/r/#/c/121853/4/Console/databaseTemplate.sql [15:28:57] maybe I have broken CI again [15:29:44] Nemo_bis: we run update.php [15:32:26] maybe that step was not reached because php55lint failed [15:33:42] hashar did you switch from mnt to srv? [15:33:49] Nemo_bis: for that one, it is directory having spaces in them [15:34:05] Nemo_bis: that is not supported by the lame phplint command [15:34:21] paladox: yeah a few [15:34:29] will do more tonigh [15:34:30] t [15:34:32] Oh, i guess jenkins repo needs updating [15:34:38] bbl [15:34:41] Ok [15:37:25] hashar oh i know why, we wipe [15:37:28] the drive [15:37:32] after we test [15:37:39] So that will delete mw-* [15:37:42] shell scripts [15:38:06] Yippee, build fixed! [15:38:06] Project selenium-MobileFrontend » chrome,beta,Linux,contintLabsSlave && UbuntuTrusty build #167: 09FIXED in 16 min: https://integration.wikimedia.org/ci/job/selenium-MobileFrontend/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/167/ [15:38:42] 10Continuous-Integration-Infrastructure: Jenkins: /srv/deployment/integration/slave-scripts/bin/mw-install-mysql.sh: No such file or directory - https://phabricator.wikimedia.org/T146382#2659255 (10Paladox) This might be because we switched from /mnt to /srv I belive I know the problem. The problem is that it... [15:45:24] Yippee, build fixed! [15:45:25] Project selenium-MobileFrontend » firefox,beta,Linux,contintLabsSlave && UbuntuTrusty build #167: 09FIXED in 23 min: https://integration.wikimedia.org/ci/job/selenium-MobileFrontend/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/167/ [16:15:48] 03Scap3: Scap3 should announce all deploys - https://phabricator.wikimedia.org/T146386#2659389 (10thcipriani) [16:16:19] 03Scap3: Scap3 should announce all deploys - https://phabricator.wikimedia.org/T146386#2659401 (10thcipriani) p:05Triage>03Normal [16:18:26] 10Continuous-Integration-Infrastructure, 13Patch-For-Review, 07Technical-Debt: Migrate CI labs slaves to use /srv instead of /mnt - https://phabricator.wikimedia.org/T146381#2659407 (10hashar) [16:24:44] brion: in production wgResourcePath (/w for skins, resources, extensions) is rewritten to /w/static.php. It used to be a symlink to /php which is unrelated to the hostname and the same accross all domains. But earlier this year I wrote static.php which makes it multi-version aware. This means that it will default to the current version for that wiki. [16:25:21] brion: However if a version query is given, it will try to satisfy that requirement which helps avoid race conditions during deployment, by trying other script paths on disk as well (php-1.XX) [16:25:33] The same if there is no query parameter and the file is not found. [16:25:40] *nod* [16:25:42] It'll return the newest version that exists in that case. [16:25:55] brion: Cache-control maxage is short unless a version hash is given. [16:26:27] brion: To bypass version verification and still have long caching, use /static/current/, though ideally we'd have version hashes where possible (ULS was migrated as such). [16:26:51] brion: using wgExtensionAssetsPath should be fine. [16:27:55] Krinkle: so the problem I was seeing is that in the case where there is a query string that isn't a 5-char md5 hash prefix, it pulls the latest known branch instead of the deployed branch [16:28:04] i have a patch in to tweak it to use current branch for that case too [16:28:08] is that an ok change to make? [16:29:03] -> https://gerrit.wikimedia.org/r/#/c/312254/ [16:29:46] at some point in future i can migrate to use appropriate hashes but i'll hvae to retool my dynamic module loading [16:29:56] for the library [16:30:28] brion: Ah, interesting. [16:30:58] 10Gerrit, 06Developer-Relations (Jul-Sep-2016), 07Documentation: Update Gerrit docs on mw.org after 2.12 upgrade - https://phabricator.wikimedia.org/T140272#2659439 (10Aklapper) a:03Aklapper [16:31:49] brion: Which responseType and maxage is this case supposed to trigger? [16:32:45] It'll get long caching right now [16:32:47] Krinkle: a long maxage would be ideal, since there's a version+hash in the query string [16:33:26] brion: Right, normally the assumption with a gabarge hash is that this hash will likely not change when the contents change since if it does, it would presumably be the actual content hash. [16:33:38] Which is why nohash defaults to short caching. [16:34:23] in this case it's specifically supposed to change when contents change :D [16:34:24] For garbage hash we default to long caching to avoid performance overhead for invalid requests so it's kind of lucky that it triggers that. [16:34:30] heh [16:34:59] There was a lot of third party hits to images with random global query parameters from social networks and such. [16:35:17] brion: I plan to make it a bit more flexible by using a named query parameter instead of the whole query string. [16:35:27] But not right now :) [16:35:29] ah nice [16:35:41] if that happens i'll try to make my hashes match in future :D [16:36:01] though i'm trying not to be too wmf-speific in the base library [16:37:25] Krinkle: is there a wikipage documenting static.php somewhere? if not I'd like to start one by copying this convorsation to wiki [16:37:58] PROBLEM - Keyholder status on deployment-tin02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [16:38:16] brion: That's alright. the idea is that by using a named parameter, intent is more explicit so it's easier to distinguish internal traffic from generic traffic. [16:38:46] *nod* [16:38:55] Right now we're forced to make the same decision for unknown hashes (e.g. a very old but once verified url), plain garbage, and things like your use case. [16:39:27] 10Gerrit, 06Developer-Relations (Jul-Sep-2016), 07Documentation: Update Gerrit docs on mw.org after 2.12 upgrade - https://phabricator.wikimedia.org/T140272#2659446 (10Aklapper) [16:39:29] And any extra query parameter added by middleware invalidates it. That strictness isn't too bad, but perhaps not as useful here. [16:40:35] twentyafterfour: Would be good to document, probably part of multiversion in some way or separately (general name I gave the program is 'wmfstatic'). It may be better to start with the comments in the code though. I think those are in better shape than what I generalised here. [16:40:38] But feel free eithier way. [16:40:53] Krinkle: thanks [16:41:25] Woot! [16:41:50] 10Gerrit, 06Developer-Relations (Jul-Sep-2016), 07Documentation: Update Gerrit docs on mw.org after 2.12 upgrade - https://phabricator.wikimedia.org/T140272#2659447 (10Aklapper) [16:41:50] https://grafana.wikimedia.org/dashboard/db/mediawiki-static [16:43:11] twentyafterfour: Also, there is an aspect to this that is part of Varnish VCL. Similar to /static/, /w/(skins,resources,extensions)/.+ with a hex query string is cached in Varnish in hostname agnostic way. [16:43:18] brion: This may complicate your patch, now that I think about it. [16:43:29] 10MediaWiki-Codesniffer: Should we require documentation for constructors? - https://phabricator.wikimedia.org/T146388#2659448 (10Jdforrester-WMF) [16:43:38] Not sure how precise that regex is. Can you double check? [16:44:28] All I'm doing is applying the existing strlen check earlier :) [16:44:56] brion: Yeah, but the assumption is that in case of a query string, the response is agnostic of the hostname, which you're violating by prepending 'current wiki' [16:45:09] Ah [16:45:19] currently it may cause cache populating in ways that is racey [16:45:25] Then yes let's confirm that doesn't break [16:46:48] brion: Also, if you reach version 12345 or ?v=12, it's strlen=5. We may wanna make the current check stricter to better distinguish between 'tolerate random query' from 'v hash wasn't found in any directory, defaulting to latest' [16:47:11] brb [16:47:17] so in theory for this case, as long as deployments don't break, it should maintain consistency across domains for a given query string. but could break in case of a version rollback [16:56:03] 10Gerrit, 06Developer-Relations (Jul-Sep-2016), 07Documentation: Update Gerrit docs on mw.org after 2.12 upgrade - https://phabricator.wikimedia.org/T140272#2659483 (10Aklapper) [16:56:09] 10Gerrit, 06Developer-Relations (Jul-Sep-2016), 07Documentation: Update Gerrit docs on mw.org after 2.12 upgrade - https://phabricator.wikimedia.org/T140272#2458793 (10Aklapper) 05Open>03Resolved * Updated screenshot https://www.mediawiki.org/wiki/File:Chrome_diff_9332.png * Updated screenshot https://w... [16:58:08] hasharAway: https://doc.wikimedia.org/cover/mediawiki-core/master/php/ is empty again :/ [17:03:51] ok i think i found the right regex, shouldn't be a problem for query strings with key-value pairs, as mine is [17:04:08] if (req.url ~ "^/w/(skins|resources|extensions)/.+\?[a-fA-F0-9]+$" ) { [17:04:36] my query string won't match so won't get sent to static-host [17:06:03] yeah i see how that would break everything if it did match prematurely :D [17:29:22] Krinkle: :-( [17:29:45] hasharAway: https://gerrit.wikimedia.org/r/#/c/312282/ [17:29:47] Krinkle: it has been failling for a few days now [17:31:19] Krinkle: you are fast [17:43:00] hasharAway hi, i think we fixed all the puppet lint errors now, the only one left to fix is autoload now https://phabricator.wikimedia.org/P4048 :) [17:43:17] Final fix https://gerrit.wikimedia.org/r/312284 [17:49:11] 10Beta-Cluster-Infrastructure, 10Wikimedia-Extension-setup, 07Category, 10FileAnnotations (Beta Cluster Release): Release FileAnnotations on the Beta Cluster - https://phabricator.wikimedia.org/T144302#2659647 (10matmarex) [17:54:09] 10Gerrit: Update gerrit to 2.12.4 - https://phabricator.wikimedia.org/T143089#2659675 (10Aklapper) When making statements please link to resources, always. Thanks. [17:55:27] 10Gerrit: Update gerrit to 2.12.4 - https://phabricator.wikimedia.org/T143089#2659681 (10Paladox) @Aklapper sorry, here https://bugs.chromium.org/p/gerrit/issues/detail?id=4521 [18:13:15] greg-g: What's the status of wmf.20 and SWAT? Is it OK to SWAT not-super-urgent things again or are we still in lockdown mode? [18:14:24] SWATs appear to be going forward atm [18:24:15] PROBLEM - Puppet run on integration-puppetmaster01 is CRITICAL: CRITICAL: 11.11% of data above the critical threshold [0.0] [18:29:14] RECOVERY - Puppet run on integration-puppetmaster01 is OK: OK: Less than 1.00% above the threshold [0.0] [18:36:27] 06Release-Engineering-Team, 10MediaWiki-ResourceLoader, 10TimedMediaHandler-Player, 13Patch-For-Review: ogv.js player fails due to extension files being from newer version of MediaWiki than is deployed - https://phabricator.wikimedia.org/T146363#2659900 (10brion) Symptoms temporarily resolved by backportin... [19:18:44] PROBLEM - Puppet run on integration-slave-jessie-1003 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [19:19:11] PROBLEM - Puppet run on integration-slave-jessie-1005 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [19:19:39] PROBLEM - Puppet run on integration-slave-precise-1012 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [19:19:43] PROBLEM - Puppet run on integration-slave-jessie-1004 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [19:19:53] PROBLEM - Puppet run on integration-slave-jessie-1002 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [19:20:27] PROBLEM - Puppet run on integration-slave-precise-1002 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [19:20:31] PROBLEM - Puppet run on integration-slave-trusty-1001 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [19:20:43] PROBLEM - Puppet run on integration-slave-precise-1011 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [19:21:03] PROBLEM - Puppet run on integration-slave-jessie-1001 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [19:21:42] I messed up ^^ [19:22:21] PROBLEM - Puppet run on integration-slave-trusty-1006 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [19:22:51] PROBLEM - Puppet run on integration-slave-trusty-1014 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [19:22:51] PROBLEM - Puppet run on integration-slave-trusty-1004 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [19:22:55] PROBLEM - Puppet run on integration-slave-trusty-1017 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [19:23:07] PROBLEM - Puppet run on integration-slave-trusty-1018 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [19:23:13] PROBLEM - Puppet run on integration-slave-trusty-1016 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [19:23:13] PROBLEM - Puppet run on integration-slave-trusty-1012 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [19:23:31] PROBLEM - Puppet run on integration-slave-trusty-1003 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [19:23:55] PROBLEM - Puppet run on integration-slave-trusty-1013 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [19:24:09] PROBLEM - Puppet run on integration-slave-trusty-1011 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [19:26:15] PROBLEM - Puppet run on deployment-ores-redis is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [19:29:03] !log switching Jenkins slaves workspace from /mnt/jenkins-workspace to /srv/jenkins-workspace (actually the same dir/inode on the filesystem) [19:29:09] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [19:30:04] (03CR) 10Niedzielski: "@krinkle, would it be ok to merge this change then?" [integration/config] - 10https://gerrit.wikimedia.org/r/308579 (owner: 10Niedzielski) [19:32:50] RECOVERY - Puppet run on integration-slave-trusty-1004 is OK: OK: Less than 1.00% above the threshold [0.0] [19:33:54] RECOVERY - Puppet run on integration-slave-trusty-1013 is OK: OK: Less than 1.00% above the threshold [0.0] [19:34:38] RECOVERY - Puppet run on integration-slave-precise-1012 is OK: OK: Less than 1.00% above the threshold [0.0] [19:34:52] RECOVERY - Puppet run on integration-slave-jessie-1002 is OK: OK: Less than 1.00% above the threshold [0.0] [19:35:24] RECOVERY - Puppet run on integration-slave-precise-1002 is OK: OK: Less than 1.00% above the threshold [0.0] [19:35:32] RECOVERY - Puppet run on integration-slave-trusty-1001 is OK: OK: Less than 1.00% above the threshold [0.0] [19:35:42] RECOVERY - Puppet run on integration-slave-precise-1011 is OK: OK: Less than 1.00% above the threshold [0.0] [19:36:02] RECOVERY - Puppet run on integration-slave-jessie-1001 is OK: OK: Less than 1.00% above the threshold [0.0] [19:37:21] RECOVERY - Puppet run on integration-slave-trusty-1006 is OK: OK: Less than 1.00% above the threshold [0.0] [19:37:49] RECOVERY - Puppet run on integration-slave-trusty-1014 is OK: OK: Less than 1.00% above the threshold [0.0] [19:37:57] RECOVERY - Puppet run on integration-slave-trusty-1017 is OK: OK: Less than 1.00% above the threshold [0.0] [19:38:09] RECOVERY - Puppet run on integration-slave-trusty-1018 is OK: OK: Less than 1.00% above the threshold [0.0] [19:38:13] RECOVERY - Puppet run on integration-slave-trusty-1012 is OK: OK: Less than 1.00% above the threshold [0.0] [19:38:15] RECOVERY - Puppet run on integration-slave-trusty-1016 is OK: OK: Less than 1.00% above the threshold [0.0] [19:38:33] RECOVERY - Puppet run on integration-slave-trusty-1003 is OK: OK: Less than 1.00% above the threshold [0.0] [19:38:45] RECOVERY - Puppet run on integration-slave-jessie-1003 is OK: OK: Less than 1.00% above the threshold [0.0] [19:39:09] RECOVERY - Puppet run on integration-slave-trusty-1011 is OK: OK: Less than 1.00% above the threshold [0.0] [19:39:09] RECOVERY - Puppet run on integration-slave-jessie-1005 is OK: OK: Less than 1.00% above the threshold [0.0] [19:39:23] Yippee, build fixed! [19:39:23] Project mediawiki-core-code-coverage build #2279: 09FIXED in 1 hr 57 min: https://integration.wikimedia.org/ci/job/mediawiki-core-code-coverage/2279/ [19:39:43] RECOVERY - Puppet run on integration-slave-jessie-1004 is OK: OK: Less than 1.00% above the threshold [0.0] [19:39:53] Krinkle: mw coverage fixed bravo! [19:51:54] (03PS1) 10Hashar: Migrate castor-save to use /srv instead of /mnt [integration/config] - 10https://gerrit.wikimedia.org/r/312323 (https://phabricator.wikimedia.org/T146381) [19:54:31] (03CR) 10Hashar: [C: 032] Migrate castor-save to use /srv instead of /mnt [integration/config] - 10https://gerrit.wikimedia.org/r/312323 (https://phabricator.wikimedia.org/T146381) (owner: 10Hashar) [19:56:16] (03Merged) 10jenkins-bot: Migrate castor-save to use /srv instead of /mnt [integration/config] - 10https://gerrit.wikimedia.org/r/312323 (https://phabricator.wikimedia.org/T146381) (owner: 10Hashar) [20:01:16] RECOVERY - Puppet run on deployment-ores-redis is OK: OK: Less than 1.00% above the threshold [0.0] [20:22:14] PROBLEM - Puppet run on deployment-ores-redis is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [20:25:23] (03PS1) 10Hashar: Change tmpfs to /srv instead of /mnt [integration/jenkins] - 10https://gerrit.wikimedia.org/r/312330 (https://phabricator.wikimedia.org/T146381) [20:35:52] (03CR) 10Hashar: [C: 04-1] "Will do in the morning." [integration/jenkins] - 10https://gerrit.wikimedia.org/r/312330 (https://phabricator.wikimedia.org/T146381) (owner: 10Hashar) [20:39:30] 10Continuous-Integration-Infrastructure, 13Patch-For-Review, 07Technical-Debt: Migrate CI labs slaves to use /srv instead of /mnt - https://phabricator.wikimedia.org/T146381#2660344 (10hashar) I am 80% done with it. * have to point jobs to use the tmpfs under /srv https://gerrit.wikimedia.org/r/#/c/312330/... [20:39:38] 10Continuous-Integration-Infrastructure, 13Patch-For-Review, 07Technical-Debt: Migrate CI labs slaves to use /srv instead of /mnt - https://phabricator.wikimedia.org/T146381#2660345 (10hashar) a:03hashar [20:46:22] 06Release-Engineering-Team: Remove .gitreview from MediaWiki and Extensions - https://phabricator.wikimedia.org/T146293#2660356 (10mmodell) git-review version 1.25.0 is currently available as a .deb for Ubuntu 16.04 but debian jessie has only [[ https://packages.debian.org/jessie/git-review | 1.24-2 ]] [20:57:26] is there a known issue with jenkins? There is a bunch of puppet jobs stuck for ~20' waiting on operations-puppet-tox-jessie... [20:57:38] maybe that's part of the "intermittent CI issues"... [20:57:48] oh, that topic is old [20:57:50] is there anything I can do? [20:58:07] greg-g: not so old it seems :P [20:58:16] everything old is new again [20:58:29] i don't see them here: https://integration.wikimedia.org/zuul/ [20:58:55] yeah, I just had to ask the question here to have them all fixed... [20:59:04] you're welcome [20:59:16] have a nice day. thank you for flying WMF RelEng. [20:59:17] greg-g: do you have a bot in here automatically fixing things when people complain? releng rules! [20:59:30] we're a complaint driven team [20:59:30] thanks! [21:00:05] (It was probably just nodepool waiting for instances to be ready to run the tests) [21:01:12] as long as it works... [21:02:16] RECOVERY - Puppet run on deployment-ores-redis is OK: OK: Less than 1.00% above the threshold [0.0] [21:03:21] 10Continuous-Integration-Infrastructure, 06Labs, 06Operations, 07Nodepool: Upgrade Nodepool to 0.1.1-wmf5 to reduce requests made to OpenStack API - https://phabricator.wikimedia.org/T145142#2621216 (10Dzahn) ``` root@carbon:~/nodepool# diff -ru 4/ 5/ diff -ru 4/usr/lib/python2.7/dist-packages/nodepool/pro... [21:05:00] PROBLEM - Puppet run on deployment-elastic07 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [21:10:05] 10Continuous-Integration-Infrastructure, 07Nodepool, 13Patch-For-Review: Investigate use of Nodepool ListFloatingIPsTask - https://phabricator.wikimedia.org/T143943#2660499 (10hashar) [21:10:07] 10Continuous-Integration-Infrastructure, 06Labs, 06Operations, 07Nodepool: Upgrade Nodepool to 0.1.1-wmf5 to reduce requests made to OpenStack API - https://phabricator.wikimedia.org/T145142#2660496 (10hashar) 05Open>03Resolved a:05hashar>03Dzahn I stopped Nodepool. Daniel apt-get installed. Done... [21:11:42] 10Continuous-Integration-Infrastructure, 07Nodepool, 13Patch-For-Review: Investigate use of Nodepool ListFloatingIPsTask - https://phabricator.wikimedia.org/T143943#2584195 (10hashar) Nodepool has been upgraded. It should no more query OpenStack for a list of floating IP. Can be check via https://grafana.w... [21:12:09] 10Continuous-Integration-Infrastructure, 07Nodepool: 2016-08-10 CI incident follow-ups - https://phabricator.wikimedia.org/T142952#2660506 (10hashar) [21:12:11] 10Continuous-Integration-Infrastructure, 07Nodepool, 13Patch-For-Review: Investigate use of Nodepool ListFloatingIPsTask - https://phabricator.wikimedia.org/T143943#2660505 (10hashar) 05stalled>03Open [21:17:26] 10Continuous-Integration-Infrastructure, 07Nodepool: 2016-08-10 CI incident follow-ups - https://phabricator.wikimedia.org/T142952#2660515 (10hashar) [21:17:28] 10Continuous-Integration-Infrastructure, 07Nodepool, 13Patch-For-Review: Investigate use of Nodepool ListFloatingIPsTask - https://phabricator.wikimedia.org/T143943#2660513 (10hashar) 05Open>03Resolved I have checked on labnet1001 and there are definitely no more requests for `/v2/contintcloud/os-floatin... [21:27:46] 06Release-Engineering-Team: Remove .gitreview from MediaWiki and Extensions - https://phabricator.wikimedia.org/T146293#2660559 (10hashar) On Debian Jessie one would want to use the backports repo: deb http://ftp.debian.org/debian/ jessie-backports main contrib non-free sudo apt-get install -t jessie-ba... [21:30:32] 10Beta-Cluster-Infrastructure, 06Release-Engineering-Team, 10DBA, 13Patch-For-Review, 07WorkType-Maintenance: Upgrade mariadb in deployment-prep from Precise/MariaDB 5.5 to Jessie/MariaDB 5.10 - https://phabricator.wikimedia.org/T138778#2660564 (10greg) I went ahead and [[ https://www.mediawiki.org/w/ind... [21:40:59] 10Continuous-Integration-Infrastructure: Jenkins: /srv/deployment/integration/slave-scripts/bin/mw-install-mysql.sh: No such file or directory - https://phabricator.wikimedia.org/T146382#2659255 (10hashar) @Mattflaschen-WMF thank you to have filled a bug! For the record the issue was: line 2: /srv/deploy... [21:41:02] 10Continuous-Integration-Infrastructure: Jenkins: /srv/deployment/integration/slave-scripts/bin/mw-install-mysql.sh: No such file or directory - https://phabricator.wikimedia.org/T146382#2660644 (10hashar) 05Open>03Resolved a:03hashar [21:41:23] 10Continuous-Integration-Infrastructure: Jenkins: /srv/deployment/integration/slave-scripts/bin/mw-install-mysql.sh: No such file or directory - https://phabricator.wikimedia.org/T146382#2659255 (10hashar) [21:41:23] 10Continuous-Integration-Infrastructure, 13Patch-For-Review, 07Technical-Debt: Migrate CI labs slaves to use /srv instead of /mnt - https://phabricator.wikimedia.org/T146381#2659219 (10hashar) [21:42:47] 10Continuous-Integration-Infrastructure, 07WorkType-Maintenance: phplint fails on paths containing a space - https://phabricator.wikimedia.org/T89380#2660655 (10hashar) Yeah the bug has always been there and is not fixed. I dont want to fix it either in favor of having repositories to transition to use compo... [21:45:01] RECOVERY - Puppet run on deployment-elastic07 is OK: OK: Less than 1.00% above the threshold [0.0] [21:45:33] (03PS3) 10Krinkle: Limit performance jobs IRC notification channels [integration/config] - 10https://gerrit.wikimedia.org/r/308579 (owner: 10Niedzielski) [21:52:49] (03CR) 10Krinkle: [C: 032] "INFO:jenkins_jobs.builder:Reconfiguring jenkins job performance-webpagetest-wmf" [integration/config] - 10https://gerrit.wikimedia.org/r/308579 (owner: 10Niedzielski) [21:53:51] (03Merged) 10jenkins-bot: Limit performance jobs IRC notification channels [integration/config] - 10https://gerrit.wikimedia.org/r/308579 (owner: 10Niedzielski) [22:11:08] 06Release-Engineering-Team (Deployment-Blockers), 05Release: MW-1.28.0-wmf.20 deployment blockers - https://phabricator.wikimedia.org/T144644#2660740 (10Krenair) [22:11:19] greg-g, ^ fyi [22:11:41] I have a fix for that new issue [22:11:41] I need to find a reviewer and hopefully it'll be done ready for the last swat [22:11:57] otherwise I'll have to do it later outside a window [22:12:18] that's fine (doing outside window) [22:12:34] thanks for responding/fixing [22:18:16] 10Beta-Cluster-Infrastructure, 10CirrusSearch, 06Discovery: On Beta cluster, CirrusSearch yields: unknown: Unknown error:60 - https://phabricator.wikimedia.org/T145609#2660773 (10debt) Removing the Search tag from this ticket - please let us know if there is anything else for us to investigate. [22:28:53] greg-g, alright, thanks to Aaron, it's going into master now. [22:29:40] rock [22:48:53] 10Deployment-Systems, 06Release-Engineering-Team, 15User-greg: List all SWAT deployers for each window (no longer segment) - https://phabricator.wikimedia.org/T144297#2660850 (10greg) a:03greg [22:54:09] 03Scap3: Local config deploys should use the target's current version - https://phabricator.wikimedia.org/T145373#2660877 (10thcipriani) 05Open>03Resolved [22:56:48] 10Deployment-Systems, 06Release-Engineering-Team, 15User-greg: List all SWAT deployers for each window (no longer segment) - https://phabricator.wikimedia.org/T144297#2660897 (10greg) 05Open>03Resolved https://wikitech.wikimedia.org/w/index.php?title=Deployments&diff=859186&oldid=858991