[00:01:01] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [10.0] [00:11:02] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [10.0] [00:58:46] PROBLEM - Free space - all mounts on deployment-fluorine02 is CRITICAL: CRITICAL: deployment-prep.deployment-fluorine02.diskspace._srv.byte_percentfree (<40.00%) [01:01:01] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [10.0] [02:11:03] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [10.0] [04:11:02] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [10.0] [04:54:44] 10Release-Engineering-Team, 10Recommendation-API, 10Research, 10Google-Summer-of-Code (2019): Merge the patch for GSoC - https://phabricator.wikimedia.org/T230859 (10leila) [06:11:01] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [10.0] [07:03:47] RECOVERY - Free space - all mounts on deployment-fluorine02 is OK: OK: All targets OK [07:22:57] 10Continuous-Integration-Infrastructure, 10Zuul: Stop/Restart tests for zuul - https://phabricator.wikimedia.org/T230019 (10hashar) I guess Matthias wants to restart a build of a job before all jobs for the change have completed. Typically my flow is: * send patch to Gerrit * head to https://integration.wiki... [07:40:40] 10Continuous-Integration-Infrastructure: mwext-Math-testextensions-master should build texvc - https://phabricator.wikimedia.org/T51884 (10hashar) 05Open→03Declined Eventually texvc has been removed entirely (thank you @Physikerwelt !). Hence CI no more has to build texvc nor support objective caml. **Refer... [07:44:01] 10Continuous-Integration-Infrastructure, 10Jenkins: Jenkins: Consistently getting 503 Varnish in response to succesful login - https://phabricator.wikimedia.org/T63710 (10hashar) At the time I think I encountered the issue a couple time at most. Since 2015 lot of things have changed (Jenkins upgrades, Varnish/... [07:46:25] 10Beta-Cluster-Infrastructure, 10CirrusSearch, 10Wikidata-Termbox-Hike, 10Discovery-Search (Current work), and 2 others: PHP Warning: Invalid argument supplied for foreach() - https://phabricator.wikimedia.org/T226969 (10WMDE-leszek) [07:58:36] (03CR) 10Phedenskog: [C: 03+1] "Yes please go ahead!" [integration/config] - 10https://gerrit.wikimedia.org/r/531286 (https://phabricator.wikimedia.org/T225416) (owner: 10Jforrester) [08:01:06] (03CR) 10Hashar: [C: 03+2] "Checked with Peter." [integration/config] - 10https://gerrit.wikimedia.org/r/529858 (https://phabricator.wikimedia.org/T225416) (owner: 10Phedenskog) [08:03:21] (03Merged) 10jenkins-bot: jjb: Drop cron triggered WebPageTest run [integration/config] - 10https://gerrit.wikimedia.org/r/529858 (https://phabricator.wikimedia.org/T225416) (owner: 10Phedenskog) [08:09:24] PROBLEM - App Server Main HTTP Response on deployment-mediawiki-09 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:11:01] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [10.0] [08:14:14] RECOVERY - App Server Main HTTP Response on deployment-mediawiki-09 is OK: HTTP OK: HTTP/1.1 200 OK - 48223 bytes in 2.516 second response time [08:15:41] PROBLEM - Host webperformance is DOWN: CRITICAL - Host Unreachable (172.16.3.26) [08:20:53] (03CR) 10Hashar: [C: 03+2] layout: [performance/WebPageTest] Archive [integration/config] - 10https://gerrit.wikimedia.org/r/531286 (https://phabricator.wikimedia.org/T225416) (owner: 10Jforrester) [08:27:35] (03CR) 10Hashar: [C: 04-1] [quibble-coverage] Generate coverage with unit tests in codehealth (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/530866 (https://phabricator.wikimedia.org/T230423) (owner: 10Kosta Harlan) [08:27:54] (03CR) 10Hashar: [C: 03+2] [quibble-coverage] Generate coverage with unit tests in codehealth [integration/config] - 10https://gerrit.wikimedia.org/r/530866 (https://phabricator.wikimedia.org/T230423) (owner: 10Kosta Harlan) [08:28:13] (03Merged) 10jenkins-bot: layout: [performance/WebPageTest] Archive [integration/config] - 10https://gerrit.wikimedia.org/r/531286 (https://phabricator.wikimedia.org/T225416) (owner: 10Jforrester) [08:36:29] (03Merged) 10jenkins-bot: [quibble-coverage] Generate coverage with unit tests in codehealth [integration/config] - 10https://gerrit.wikimedia.org/r/530866 (https://phabricator.wikimedia.org/T230423) (owner: 10Kosta Harlan) [08:37:15] <_joe_> hi, I've been waiting for what is now 5 minutes for a gate-and-submit job where the tests don't last 1 minute. And this is one of the less busy times in the day. Can something be done about this constant slowness? [08:37:44] <_joe_> (it's a genuine question, I'm not sure what's the cause at all :P) [08:38:24] <_joe_> it took my patch 8 minutes to be processed, of which 1.47 of processing [08:38:40] <_joe_> (that is very slow too, but that's another issue) [08:39:35] (03CR) 10Hashar: jjb: [mwext-codehealth*] Use quibble-coverage 0.0.34-4 (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/530867 (https://phabricator.wikimedia.org/T230423) (owner: 10Kosta Harlan) [08:40:09] kostajh: I have +2 ed your fix to the coverage generator script and the container is being rebuild. For the Jenkins job ( https://gerrit.wikimedia.org/r/#/c/integration/config/+/530867/2/jjb/mediawiki.yaml@573 ) you pass --skip-zuul which would hmmm NOT clone any repository :] [08:40:21] kostajh: I guess it is a left over from a local hack? [08:41:08] !log Build docker-registry.discovery.wmnet/releng/quibble-coverage:0.0.34-4 # T230423 [08:41:10] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [08:41:11] T230423: Generate PHP code coverage via unit tests only, not integration tests - https://phabricator.wikimedia.org/T230423 [08:41:14] hashar: ah, I intended to avoid cloning all dependencies, but I do need the extension to get cloned :) [08:41:33] I'll remove the --skip-zuul bit [08:42:00] well that depends on what you need [08:42:17] if it is just mediawiki/core and the extension, we might well just not use quibble at all [08:43:20] (03PS3) 10Kosta Harlan: jjb: [mwext-codehealth*] Use quibble-coverage 0.0.34-4 [integration/config] - 10https://gerrit.wikimedia.org/r/530867 (https://phabricator.wikimedia.org/T230423) [08:43:59] hashar: yeah, but I still need composer install and npm install, and I'd rather not reinvent everything since quibble already does a lot of the tedious stuff :) [08:44:30] kostajh: +1 :) [08:45:42] _joe_: I . Yeah there is somewhere in Jenkins a race condition to acquire a lock for an instance to run build on. And somehow when some kind of job acquire the lock, no other jobs can run on that instance [08:46:45] something like that [08:47:02] but I have never been able to even acquire a meaningful log of the issue :-\ [08:47:09] <_joe_> uh, we can't be the only ones with such a problem [09:06:53] hashar: I'm around for the next ~20 minutes if you want me to verify the new Jenkins job when it's deployed, otherwise I'll be back around 230 CEST [09:07:11] good [09:09:29] kostajh: deploying them [09:10:27] (03CR) 10Hashar: [C: 03+2] jjb: [mwext-codehealth*] Use quibble-coverage 0.0.34-4 [integration/config] - 10https://gerrit.wikimedia.org/r/530867 (https://phabricator.wikimedia.org/T230423) (owner: 10Kosta Harlan) [09:10:33] kostajh: deployed all four jobs [09:11:06] * kostajh looks [09:13:17] (03Merged) 10jenkins-bot: jjb: [mwext-codehealth*] Use quibble-coverage 0.0.34-4 [integration/config] - 10https://gerrit.wikimedia.org/r/530867 (https://phabricator.wikimedia.org/T230423) (owner: 10Kosta Harlan) [09:15:02] Hm, my "check code health" comments seem to have done nothing [09:15:10] `check codehealth` rather [09:15:51] maybe the change is already in the pipeline? [09:15:53] Ah, they're queued [09:16:22] unfortunately my battery probably won't last that long but I'll check back a bit later. Thanks for deploying hashar! [09:16:30] :)) [09:18:50] hashar: oops `11:18:22 /usr/local/bin/mwext-phpunit-coverage: line 55: --testsuite=extensions:unit: command not found` [09:18:58] ):(( [09:19:14] <_joe_> hashar: can you remind me how do I make a repo sync to github? [09:19:30] I think it's because of the code comment :( [09:19:38] kostajh: haven't tested the container :-( [09:20:52] (03PS1) 10Hashar: Revert "jjb: [mwext-codehealth*] Use quibble-coverage 0.0.34-4" [integration/config] - 10https://gerrit.wikimedia.org/r/531444 [09:20:56] kostajh: jobs reverted ^^^ [09:21:09] I have a patch coming [09:21:14] _joe_: so yeah easy. git clone --mirror github.com/foo/bar [09:21:30] _joe_: in Gerrit create a new empty project and grant you owner rights (which should gives everything) [09:21:53] _joe_: then git remote add gerrit ssh://gerrit.wikimedia.org/p/foo/bar.git [09:22:06] and finally push the whole git repo to gerrit: git push --mirror gerrit [09:22:29] (you might be able to push directly to the ssh://gerrit.wikimedia.org/p/foo/bar.git url without having to add the remote. Not sure [09:22:48] with --mirror, all references are cloned so you also get the history of github pull requests for example [09:22:58] (03CR) 10Hashar: [C: 03+2] Revert "jjb: [mwext-codehealth*] Use quibble-coverage 0.0.34-4" [integration/config] - 10https://gerrit.wikimedia.org/r/531444 (owner: 10Hashar) [09:23:33] <_joe_> hashar: no the other way around [09:23:38] <_joe_> I have the repo in gerrit [09:23:38] kostajh: probably we should move those entry points to be in a composer package instead. Will make it easier to bump and collaborate on the code [09:23:51] <_joe_> I want to activate the sync to gh [09:23:58] <_joe_> the way we do for puppet et al [09:24:01] oh [09:24:04] the mirroring [09:24:08] from gerrit to github [09:24:08] <_joe_> yep [09:24:17] <_joe_> sorry I wasn't very clear :P [09:24:33] so given a Gerrit repo named foo/bar-baz , Gerrit autoamtically attempt to replicate it to github.com/wikimedia/foo-bar-baz [09:24:49] <_joe_> ok so I just have to create the empty repo in gh? [09:24:55] it just replaces slashes with dashes ( s%/%-%g ) [09:24:57] so yeah [09:25:00] just create it in github [09:25:02] (03PS1) 10Kosta Harlan: [quibble-coverage] Fix mwext-phpunit-coverage command [integration/config] - 10https://gerrit.wikimedia.org/r/531445 (https://phabricator.wikimedia.org/T230423) [09:25:04] (03PS1) 10Kosta Harlan: jjb [mwext-codehealth*] Bump jobs to use quibble-coverage:0.0.34-5 [integration/config] - 10https://gerrit.wikimedia.org/r/531446 (https://phabricator.wikimedia.org/T230423) [09:25:07] and eventually it will be replicated [09:25:18] kostajh: awesome. [09:25:36] (03CR) 10Hashar: [C: 03+2] [quibble-coverage] Fix mwext-phpunit-coverage command [integration/config] - 10https://gerrit.wikimedia.org/r/531445 (https://phabricator.wikimedia.org/T230423) (owner: 10Kosta Harlan) [09:26:03] (03Merged) 10jenkins-bot: Revert "jjb: [mwext-codehealth*] Use quibble-coverage 0.0.34-4" [integration/config] - 10https://gerrit.wikimedia.org/r/531444 (owner: 10Hashar) [09:27:28] (03Merged) 10jenkins-bot: [quibble-coverage] Fix mwext-phpunit-coverage command [integration/config] - 10https://gerrit.wikimedia.org/r/531445 (https://phabricator.wikimedia.org/T230423) (owner: 10Kosta Harlan) [09:29:41] (03PS1) 10Pwirth: parameter_functions: Add missing dependencies for BlueSpiceSocial* extensions [integration/config] - 10https://gerrit.wikimedia.org/r/531449 [09:30:39] (03CR) 10Hashar: [C: 03+2] jjb [mwext-codehealth*] Bump jobs to use quibble-coverage:0.0.34-5 [integration/config] - 10https://gerrit.wikimedia.org/r/531446 (https://phabricator.wikimedia.org/T230423) (owner: 10Kosta Harlan) [09:30:42] kostajh: deployed! [09:30:48] (03CR) 10jerkins-bot: [V: 04-1] jjb [mwext-codehealth*] Bump jobs to use quibble-coverage:0.0.34-5 [integration/config] - 10https://gerrit.wikimedia.org/r/531446 (https://phabricator.wikimedia.org/T230423) (owner: 10Kosta Harlan) [09:30:51] bah [09:32:18] (03PS2) 10Hashar: jjb [mwext-codehealth*] Bump jobs to use quibble-coverage:0.0.34-5 [integration/config] - 10https://gerrit.wikimedia.org/r/531446 (https://phabricator.wikimedia.org/T230423) (owner: 10Kosta Harlan) [09:33:48] (03PS3) 10Hashar: jjb [mwext-codehealth*] Bump jobs to use quibble-coverage:0.0.34-5 [integration/config] - 10https://gerrit.wikimedia.org/r/531446 (https://phabricator.wikimedia.org/T230423) (owner: 10Kosta Harlan) [09:33:49] stupid rebase [09:35:08] (03CR) 10jerkins-bot: [V: 04-1] parameter_functions: Add missing dependencies for BlueSpiceSocial* extensions [integration/config] - 10https://gerrit.wikimedia.org/r/531449 (owner: 10Pwirth) [09:37:20] hashar: ah, sorry. Are they deployed now? [09:37:24] yes [09:41:38] hashar: seems to work [09:44:37] hashar: so for https://gerrit.wikimedia.org/r/c/mediawiki/extensions/GrowthExperiments/+/530837#message-ce06240fed4b98f5eaf98ec94a5e914698d4a3bb, 4 minutes instead of 7 and 12 minutes seen pre-deployment of those jobs [09:44:50] :))))))) [09:46:24] This could be sped up further (1 or 2 minutes?) if we pass just the extension we are looking at as an argument to quibble and also if we could figure out how to cache the sonar plugins [09:48:00] (03PS2) 10Pwirth: parameter_functions: Add missing dependencies for BlueSpiceSocial* extensions [integration/config] - 10https://gerrit.wikimedia.org/r/531449 [09:49:29] (03CR) 10jerkins-bot: [V: 04-1] parameter_functions: Add missing dependencies for BlueSpiceSocial* extensions [integration/config] - 10https://gerrit.wikimedia.org/r/531449 (owner: 10Pwirth) [09:51:47] (03PS3) 10Pwirth: parameter_functions: Add missing dependencies for BlueSpiceSocial* extensions [integration/config] - 10https://gerrit.wikimedia.org/r/531449 [10:04:57] (03CR) 10Hashar: [C: 03+2] parameter_functions: Add missing dependencies for BlueSpiceSocial* extensions [integration/config] - 10https://gerrit.wikimedia.org/r/531449 (owner: 10Pwirth) [10:06:42] (03Merged) 10jenkins-bot: parameter_functions: Add missing dependencies for BlueSpiceSocial* extensions [integration/config] - 10https://gerrit.wikimedia.org/r/531449 (owner: 10Pwirth) [10:08:53] (03CR) 10Hashar: [C: 03+2] "deployed" [integration/config] - 10https://gerrit.wikimedia.org/r/531449 (owner: 10Pwirth) [10:11:03] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [10.0] [10:11:55] (03CR) 10Pwirth: "thank you!" [integration/config] - 10https://gerrit.wikimedia.org/r/531449 (owner: 10Pwirth) [10:48:19] (03CR) 10TheDJ: [C: 03+1] [TimedMediaHandler] Add phan dependency [integration/config] - 10https://gerrit.wikimedia.org/r/528592 (https://phabricator.wikimedia.org/T224766) (owner: 10Umherirrender) [10:55:56] (03CR) 10Hashar: [C: 03+2] [LiquidThreads] Add phan dependency [integration/config] - 10https://gerrit.wikimedia.org/r/528937 (https://phabricator.wikimedia.org/T224757) (owner: 10Umherirrender) [10:56:48] (03CR) 10Hashar: [C: 03+2] [TimedMediaHandler] Add phan dependency [integration/config] - 10https://gerrit.wikimedia.org/r/528592 (https://phabricator.wikimedia.org/T224766) (owner: 10Umherirrender) [10:57:31] (03Merged) 10jenkins-bot: [LiquidThreads] Add phan dependency [integration/config] - 10https://gerrit.wikimedia.org/r/528937 (https://phabricator.wikimedia.org/T224757) (owner: 10Umherirrender) [10:58:23] (03Merged) 10jenkins-bot: [TimedMediaHandler] Add phan dependency [integration/config] - 10https://gerrit.wikimedia.org/r/528592 (https://phabricator.wikimedia.org/T224766) (owner: 10Umherirrender) [11:01:42] (03CR) 10Hashar: [C: 03+2] Restore tests for SemanticSifter and SemanticGenealogy [integration/config] - 10https://gerrit.wikimedia.org/r/515615 (https://phabricator.wikimedia.org/T199423) (owner: 10Umherirrender) [11:03:16] (03Merged) 10jenkins-bot: Restore tests for SemanticSifter and SemanticGenealogy [integration/config] - 10https://gerrit.wikimedia.org/r/515615 (https://phabricator.wikimedia.org/T199423) (owner: 10Umherirrender) [11:12:53] (03CR) 10Hashar: "> Perhaps for cases like libup and maybe others we can force a 'queue-name' to something that enforces no shared dependencies in the gate," [integration/config] - 10https://gerrit.wikimedia.org/r/526749 (owner: 10Jforrester) [11:20:36] (03CR) 10Hashar: [C: 03+2] "recheck" [integration/config] - 10https://gerrit.wikimedia.org/r/531446 (https://phabricator.wikimedia.org/T230423) (owner: 10Kosta Harlan) [11:22:48] (03Merged) 10jenkins-bot: jjb [mwext-codehealth*] Bump jobs to use quibble-coverage:0.0.34-5 [integration/config] - 10https://gerrit.wikimedia.org/r/531446 (https://phabricator.wikimedia.org/T230423) (owner: 10Kosta Harlan) [11:24:40] (03CR) 10Gergő Tisza: "Nitpicks aside, this seems good. Not sure if it's worth the complexity as opposed to just getting rid of this sniff altogether, though." (033 comments) [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/530910 (https://phabricator.wikimedia.org/T222853) (owner: 10Daimona Eaytoy) [11:29:22] (03CR) 10Daimona Eaytoy: ">" (033 comments) [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/530910 (https://phabricator.wikimedia.org/T222853) (owner: 10Daimona Eaytoy) [11:31:45] (03PS8) 10Daimona Eaytoy: Allow consecutive single-line comments not to start with a single space [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/530910 (https://phabricator.wikimedia.org/T222853) [11:33:58] (03CR) 10Hashar: [C: 03+2] Port tests to pytest (031 comment) [integration/commit-message-validator] - 10https://gerrit.wikimedia.org/r/525202 (owner: 10Legoktm) [11:35:18] (03Merged) 10jenkins-bot: Port tests to pytest [integration/commit-message-validator] - 10https://gerrit.wikimedia.org/r/525202 (owner: 10Legoktm) [11:47:09] (03CR) 10Gergő Tisza: Allow consecutive single-line comments not to start with a single space (031 comment) [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/530910 (https://phabricator.wikimedia.org/T222853) (owner: 10Daimona Eaytoy) [11:48:04] (03CR) 10Gergő Tisza: [C: 03+2] Allow consecutive single-line comments not to start with a single space [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/530910 (https://phabricator.wikimedia.org/T222853) (owner: 10Daimona Eaytoy) [11:49:20] (03Merged) 10jenkins-bot: Allow consecutive single-line comments not to start with a single space [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/530910 (https://phabricator.wikimedia.org/T222853) (owner: 10Daimona Eaytoy) [11:49:49] (03CR) 10jenkins-bot: Allow consecutive single-line comments not to start with a single space [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/530910 (https://phabricator.wikimedia.org/T222853) (owner: 10Daimona Eaytoy) [11:50:06] 10MediaWiki-Codesniffer: Improve or disable SingleSpaceBeforeSingleLineComment - https://phabricator.wikimedia.org/T222853 (10Daimona) 05Open→03Resolved Thanks for the review! [11:57:22] 10Release-Engineering-Team-TODO (201908): Need permission to push to core repo - https://phabricator.wikimedia.org/T230916 (10Jpita) [11:59:44] 10Release-Engineering-Team-TODO (201908): Need permission to push to core repo - https://phabricator.wikimedia.org/T230916 (10zeljkofilipin) He's trying to amend my commit in core: https://gerrit.wikimedia.org/r/c/mediawiki/core/+/530914 [12:02:26] (03PS2) 10Hashar: tests: Use yaml.safe_load() [integration/config] - 10https://gerrit.wikimedia.org/r/525215 (owner: 10Legoktm) [12:02:54] (03CR) 10Hashar: [C: 03+2] "cherry picked against tip of master :)" [integration/config] - 10https://gerrit.wikimedia.org/r/525215 (owner: 10Legoktm) [12:04:31] (03Merged) 10jenkins-bot: tests: Use yaml.safe_load() [integration/config] - 10https://gerrit.wikimedia.org/r/525215 (owner: 10Legoktm) [12:06:17] 10Phabricator, 10Project-Admins: Add GrowthExperiments to tasks tagged as GrowthExperiments-* - https://phabricator.wikimedia.org/T230831 (10Catrope) a:03JTannerWMF I think subprojects is the best way to go here. @JTannerWMF agreed to do this. [12:10:43] 10Phabricator, 10Project-Admins, 10Growth-Team (Current Sprint): Add GrowthExperiments to tasks tagged as GrowthExperiments-* - https://phabricator.wikimedia.org/T230831 (10JTannerWMF) I created the above tags that should be subprojects of #growthexperiments instead of stand alone projects. I will add this t... [12:11:02] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [10.0] [12:13:23] 10MediaWiki-Codesniffer: Improve or disable SingleSpaceBeforeSingleLineComment - https://phabricator.wikimedia.org/T222853 (10Tgr) Thanks for actually doing it :) [13:23:50] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO (201908), 10Wikidata, 10Release, and 3 others: Moved Wikidata Item link to Other Projects might break gadgets - https://phabricator.wikimedia.org/T230926 (10alaa_wmde) [13:24:08] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO (201908), 10Wikidata, 10Release, and 3 others: Moved Wikidata Item link to Other Projects might break gadgets - https://phabricator.wikimedia.org/T230926 (10alaa_wmde) p:05Triage→03High [13:29:12] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO (201908), 10Wikidata, 10Release, and 3 others: Moved Wikidata Item link to Other Projects might break gadgets - https://phabricator.wikimedia.org/T230926 (10alaa_wmde) [13:29:31] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO (201908), 10Wikidata, 10Release, and 3 others: Moved Wikidata Item link to Other Projects might break gadgets - https://phabricator.wikimedia.org/T230926 (10RhinosF1) @alaa_wmde - if you're blocking the train then this should... [13:36:20] thcipriani: what are the scap steps that usually failed with permissions problems? branch cut? cleaning up old branches? I'm trying to update https://wikitech.wikimedia.org/wiki/How_to_deploy_code with a reference to the fix permissions kludge but I'm not sure where [13:37:08] the step that usually fails is syncing files out https://wikitech.wikimedia.org/wiki/How_to_deploy_code#Step_4:_synchronize_the_changes_to_the_cluster [13:37:30] scap tries to update all the git hashes on https://en.wikipedia.org/wiki/Special:Version [13:37:35] and that's what's failing [13:38:47] hrm, css looks a little screwed up for that page... [13:43:36] haha wow [13:45:43] it's really scap sync that usually fails? does that touch the local git repo on deploy1001? I thought it was operations against the repo like cutting a branch or cleaning up old branches [13:47:13] sync-file was what was failing yesterday. Another frequent failure is https://wikitech.wikimedia.org/wiki/How_to_deploy_code#Step_2:_get_the_code_on_the_deployment_host [13:47:31] the git operations for getting code to deployment hosts [13:47:55] hm [13:48:14] actually I think I'll add a new "Problem" section to the end of the page [13:48:15] I guess there were two permission failures yesterday. The first was for cleanup, the second was for a sync-file that failed due to permission changes to fix cleanup IIRC [13:49:08] "cleanup" is something only train folks do, so it's documented on https://wikitech.wikimedia.org/wiki/Heterogeneous_deployment/Train_deploys#Clean_up_old_stuff [13:49:14] aha [13:49:58] thanks! [13:50:21] (03CR) 10Hashar: Begin migration to pytest, use it as test runner (035 comments) [integration/config] - 10https://gerrit.wikimedia.org/r/525209 (owner: 10Legoktm) [13:54:40] pytest [13:54:47] how to ruin my afternoon :-] [13:56:27] (03CR) 10Hashar: [C: 03+2] Whitelist Viztor [integration/config] - 10https://gerrit.wikimedia.org/r/529584 (owner: 10Urbanecm) [13:58:04] (03CR) 10Hashar: [C: 03+2] "Désolé oui ce n'est pas bien simple. Bon hack sur WikiCommons!" [integration/config] - 10https://gerrit.wikimedia.org/r/530778 (owner: 10Don-vip) [13:59:26] (03Merged) 10jenkins-bot: Whitelist Viztor [integration/config] - 10https://gerrit.wikimedia.org/r/529584 (owner: 10Urbanecm) [13:59:28] (03CR) 10jerkins-bot: [V: 04-1] Whitelist Don-vip [integration/config] - 10https://gerrit.wikimedia.org/r/530778 (owner: 10Don-vip) [14:01:19] (03PS2) 10Hashar: Whitelist Don-vip [integration/config] - 10https://gerrit.wikimedia.org/r/530778 (owner: 10Don-vip) [14:01:39] (03CR) 10Hashar: [C: 03+2] "Trivial conflict with another change :)" [integration/config] - 10https://gerrit.wikimedia.org/r/530778 (owner: 10Don-vip) [14:03:47] (03Merged) 10jenkins-bot: Whitelist Don-vip [integration/config] - 10https://gerrit.wikimedia.org/r/530778 (owner: 10Don-vip) [14:09:56] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO (201908), 10Release, 10Train Deployments, 10User-zeljkofilipin: 1.34.0-wmf.19 deployment blockers - https://phabricator.wikimedia.org/T220744 (10zeljkofilipin) [14:11:01] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [10.0] [14:51:46] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO (201908), 10Release, 10Train Deployments, 10User-zeljkofilipin: 1.34.0-wmf.19 deployment blockers - https://phabricator.wikimedia.org/T220744 (10zeljkofilipin) [14:57:05] 10Continuous-Integration-Infrastructure, 10Jenkins: Jenkins: Consistently getting 503 Varnish in response to succesful login - https://phabricator.wikimedia.org/T63710 (10Krinkle) 05Open→03Resolved Nope. [15:15:27] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO (201908), 10Release, 10Train Deployments, 10User-zeljkofilipin: 1.34.0-wmf.20 deployment blockers - https://phabricator.wikimedia.org/T220745 (10zeljkofilipin) [15:17:55] (03Abandoned) 10Hashar: Import zuul-migrate [integration/config] - 10https://gerrit.wikimedia.org/r/517545 (owner: 10Hashar) [15:17:57] (03Abandoned) 10Hashar: Adapt zuul-migrate [integration/config] - 10https://gerrit.wikimedia.org/r/517546 (owner: 10Hashar) [15:45:05] (03PS2) 10Jforrester: [LiquidThreads] Run phan job [integration/config] - 10https://gerrit.wikimedia.org/r/528941 (owner: 10Umherirrender) [15:45:18] (03PS3) 10Jforrester: layout: [LiquidThreads] Run phan job [integration/config] - 10https://gerrit.wikimedia.org/r/528941 (owner: 10Umherirrender) [15:45:23] (03CR) 10Jforrester: [C: 03+2] layout: [LiquidThreads] Run phan job [integration/config] - 10https://gerrit.wikimedia.org/r/528941 (owner: 10Umherirrender) [15:47:09] (03Merged) 10jenkins-bot: layout: [LiquidThreads] Run phan job [integration/config] - 10https://gerrit.wikimedia.org/r/528941 (owner: 10Umherirrender) [15:49:23] !log Zuul: Add phan for LiquidThreads [15:49:24] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:11:03] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [10.0] [16:24:40] zeljkof: we believe we have cooked up a solution to T230937 [16:24:40] T230937: TermboxView.php: Call to a member function getSerialization() on a non-object (null) - https://phabricator.wikimedia.org/T230937 [16:26:13] we're all planning on leaving the office soon; should/can we backport it and deploy it ourselves now? Or should we leave it to you? [16:26:35] tarrow: please backport/deploy [16:26:52] is it ok to go now? [16:27:06] tarrow: you can also do it during eu swat tomorrow, if that's better for you [16:27:19] let me check the deployments page... [16:28:07] tarrow: looks like morning swat is in progress, but there's nothing scheduled, so you should be able to deploy now https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20190821T1600 [16:28:43] ok cool! we'll do it now [16:32:13] zeljkof: actually we just realised that we'd need to wait for quite some time for it to make it's way through jenkins so we will aim for eu morning SWAT tomorrow [16:41:36] 10Release-Engineering-Team-TODO, 10Performance-Team, 10serviceops: Create warmup procedure for MediaWiki app servers - https://phabricator.wikimedia.org/T230037 (10Jdforrester-WMF) [16:41:51] 10Release-Engineering-Team, 10Performance-Team, 10serviceops: Create warmup procedure for MediaWiki app servers - https://phabricator.wikimedia.org/T230037 (10Jdforrester-WMF) [18:11:02] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [10.0] [18:17:32] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO (201908), 10Release, 10Train Deployments, 10User-zeljkofilipin: 1.34.0-wmf.19 deployment blockers - https://phabricator.wikimedia.org/T220744 (10WMDE-leszek) [18:20:31] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO (201908), 10Release, 10Train Deployments, 10User-zeljkofilipin: 1.34.0-wmf.19 deployment blockers - https://phabricator.wikimedia.org/T220744 (10Jdforrester-WMF) [18:44:14] tarrow: thanks! (sorry for the late reply, I was in meetings, then away) [18:45:24] cool, we will do the merge tomorrow; since we wrote the solution in a rush tonight I'm happy to sleep on it [18:45:38] we did the other WMDE train blocker tonight though [20:11:03] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [10.0] [20:42:04] Fun when your going accross the border of Ireland and Northern Ireland and you suddenly start roaming :P [20:44:07] * paladox lucky that he had a period in Northern Ireland that he was not roaming [20:45:43] You don't even need to cross the border sometimes for your phone to roam [20:46:02] Heh [20:46:10] The fun one is leaving your mobile out of airplane mode on long haul flights and seeing what "welcome to" texts you get [20:46:15] My mobile company is in three yet still makes me roam :( [20:46:20] *ireland [20:46:29] Lol [20:46:42] I got a text telling me I was roaming when I landed :P [20:47:02] But you also don't need to use three if it exists when abroad. So you can take advantage of whoever has the best signal etc [20:47:29] Oh, three appears to be the best here [20:47:48] At least I’ve been on three in most of Ireland [20:49:48] Signal seems to be ok, patchy in some places. Wasent going to try the speed app to see how fast it was going. [20:55:25] Reedy: fun seeing coke on offer :P [21:10:48] 10Phabricator, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO (201908), 10Operations, and 2 others: Prepare Phame to support heavy traffic for a Tech Department blog - https://phabricator.wikimedia.org/T226044 (10JAufrecht) > title, subtitle and description I think what... [21:30:16] 10Phabricator, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO (201908), 10Operations, and 2 others: Prepare Phame to support heavy traffic for a Tech Department blog - https://phabricator.wikimedia.org/T226044 (10Aklapper) General reminder about [naming things](https://w... [22:11:02] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [10.0] [22:29:38] 10Release-Engineering-Team, 10Recommendation-API, 10Research, 10Google-Summer-of-Code (2019): Merge the patch for GSoC - https://phabricator.wikimedia.org/T230859 (10Aklapper) The Gerrit owners for the group [`mediawiki-services-recommendation-api`](https://gerrit.wikimedia.org/r/#/admin/groups/1335,member... [23:58:12] 10Project-Admins, 10Phlogiston: Rename #Category to #Phlogiston-Category - https://phabricator.wikimedia.org/T224450 (10Aklapper) [23:58:52] 10Project-Admins, 10Phlogiston: Rename #Category to #Phlogiston-Category - https://phabricator.wikimedia.org/T224450 (10Aklapper) 05Open→03Resolved a:03Aklapper Thanks everyone! Renamed to `#Phlogiston-category`.