[00:07:20] 10Continuous-Integration-Config, 10Composer: Composer\Downloader\TransportException in mwext-testextension-hhvm-jessie - https://phabricator.wikimedia.org/T182266#3818591 (10Tgr) p:05Triage>03Unbreak! Seems to be breaking all MediaWiki extension tests. [00:08:10] greg-g: upped that to unbreak now as gerrit says all MW extension commits have failed with this error in the last half hour [00:09:27] ugh [00:10:28] huh, just ran into this for mwconfig for swat too [00:11:26] Its affecting mw-testskin [00:12:02] (03PS1) 10Umherirrender: Archive mediawiki/extensions/Wikidata [integration/config] - 10https://gerrit.wikimedia.org/r/395895 (https://phabricator.wikimedia.org/T181838) [00:13:21] (03CR) 10jerkins-bot: [V: 04-1] Archive mediawiki/extensions/Wikidata [integration/config] - 10https://gerrit.wikimedia.org/r/395895 (https://phabricator.wikimedia.org/T181838) (owner: 10Umherirrender) [00:15:08] RECOVERY - Puppet errors on deployment-cache-upload04 is OK: OK: Less than 1.00% above the threshold [0.0] [00:17:39] (03PS2) 10Umherirrender: Archive mediawiki/extensions/Wikidata [integration/config] - 10https://gerrit.wikimedia.org/r/395895 (https://phabricator.wikimedia.org/T181838) [00:18:35] tgr: our composer isn't that out of date, is it? [00:18:56] (03CR) 10jerkins-bot: [V: 04-1] Archive mediawiki/extensions/Wikidata [integration/config] - 10https://gerrit.wikimedia.org/r/395895 (https://phabricator.wikimedia.org/T181838) (owner: 10Umherirrender) [00:19:00] no idea [00:19:06] we use 1.2 [00:19:12] whereas there's 1.5 [00:19:13] I'm trying to find out where even to check that [00:19:42] actually 1.1 [00:19:52] tgr integration/composer [00:20:35] in the dib folder [00:20:43] then we call out to a lot of puppet in operations/puppet [00:20:45] from there [00:21:02] (03PS1) 10Paladox: Update composer to 1.4.3 [integration/composer] - 10https://gerrit.wikimedia.org/r/395898 (https://phabricator.wikimedia.org/T125343) [00:21:14] ah, thanks, didn't realise we had our own fork [00:22:21] https://github.com/wikimedia/integration-config/blob/376657f64e789df4b85af896362fc2527d5d1065/dockerfiles/composer/prebuild.sh#L9 [00:22:25] (03CR) 10Umherirrender: parameter_functions switch extensions to use Wikibase (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/395704 (owner: 10Addshore) [00:22:51] oh, we use: https://github.com/wikimedia/integration-composer [00:25:45] following the git trail: https://github.com/wikimedia/integration-config/blob/master/dib/puppet/ciimage.pp#L25 -> https://github.com/wikimedia/puppet/blob/production/modules/contint/manifests/composer.pp#L10 -> https://github.com/wikimedia/integration-composer [00:27:07] doesn't look like there's been much change in that composer repo recently. I'm going to try regenerating the nodepool snapshot to see if that changes anything... [00:29:12] 10Continuous-Integration-Config, 10Composer: Composer\Downloader\TransportException in mwext-testextension-hhvm-jessie - https://phabricator.wikimedia.org/T182266#3818615 (10Tgr) https://github.com/wikimedia/integration-composer is at 1.1.3. Can't reproduce the error with that version locally; installation see... [00:36:53] (03CR) 10Reedy: [C: 031] "Should try again sometime soon..." [integration/composer] - 10https://gerrit.wikimedia.org/r/395898 (https://phabricator.wikimedia.org/T125343) (owner: 10Paladox) [00:43:17] 10Continuous-Integration-Config, 10Composer: Composer\Downloader\TransportException in mwext-testextension-hhvm-jessie - https://phabricator.wikimedia.org/T182266#3818521 (10Reedy) https://gerrit.wikimedia.org/r/#/c/395898/1 is long overdue happening [00:43:39] so...bizarrely...doing something like: curl -I https://packagist.org/p/provider-2017-04%2447568496b5ec7309a3b1a6a172620bc4729dda2acb59c85648bd6d25b6a99ffc.json inside a nodepool instance is a 404 but outside is a 200... [00:43:57] thats a 404 for me [00:44:01] outside [00:44:17] "Oh noes, 404!" [00:45:22] Same for me, with curl -I [00:49:09] grabbed that url from: https://integration.wikimedia.org/ci/job/operations-mw-config-composer-hhvm-jessie/9076/console [00:49:54] http://packagist.org/p/provider-2017-07%2478aeddd89a6af3af1d1501c1bfb25e1b61a4dfac12001c4c0520b106a21053f1.json works for me [00:51:48] https://github.com/composer/composer/blob/6a08755c254d50384e48a36f75dd56e56e942692/src/Composer/Repository/ComposerRepository.php#L300-L316 [01:03:52] Reedy: what am I looking at? [01:04:02] thcipriani: php [01:04:03] ;) [01:04:08] ohhhhhh :) [01:04:25] It's where it seems to build the strings of those provider-foobar urls [01:04:31] 10Continuous-Integration-Config, 10Composer: Composer\Downloader\TransportException in mwext-testextension-hhvm-jessie - https://phabricator.wikimedia.org/T182266#3818665 (10Tgr) Looking at the Composer code it seems like these URLs come from https://packagist.org/packages.json by replacing `%hash%` with the s... [01:04:50] gotcha [01:04:53] I wonder how cached $cacheKey is [01:05:42] Based on Gergos link above.. [01:05:55] Does it use those depending on the composer version? [01:07:04] "p\/provider-2014$%hash%.json":{"sha256":"67ac94409dcd4da75fc57158057f09f00619a11a17511cac4b7b3bd3ceff49d4"} [01:07:43] $this->cache = new Cache($io, $config->get('cache-repo-dir').'/'.preg_replace('{[^a-z0-9.]}i', '-', $this->url), 'a-z0-9.$'); [01:07:54] thcipriani: Can we just nuke the composer cache dirs? [01:07:56] 10Continuous-Integration-Config, 10Composer: Composer\Downloader\TransportException in mwext-testextension-hhvm-jessie - https://phabricator.wikimedia.org/T182266#3818521 (10thcipriani) I've regenerated the nodepool jessie snapshot using the instructions from: https://wikitech.wikimedia.org/wiki/Nodepool#Manua... [01:09:12] Reedy: maybe...there is some rsync thing that keeps the cache around IIRC lemme see if I can find the instance [01:09:50] castor02.integration.eqiad.wmflabs [01:12:27] Reedy: the .composer/cache there? [01:13:11] 'cache-dir' => '{$home}/cache', [01:13:17] 'cache-files-dir' => '{$cache-dir}/files', [01:13:17] 'cache-repo-dir' => '{$cache-dir}/repo', [01:13:17] 'cache-vcs-dir' => '{$cache-dir}/vcs', [01:13:19] Looks like it.. [01:14:58] k, done for operations-mw-config-composer-hhvm-jessie [01:15:14] https://integration.wikimedia.org/ci/job/operations-mw-config-composer-hhvm-jessie/9079/console [01:15:23] same fail [01:15:26] :( [01:16:15] hrm there are no more files in the cache there afaict [01:18:14] computers suck [01:24:15] Completely wrong hash again [01:25:05] and latest changes regularly... [01:27:34] sometimes I get: file could not be downloaded (HTTP/1.1 302 Found) [01:28:22] * Reedy waits for the hash to change again [01:30:01] thcipriani: curl... seems to get 404'd every time [01:36:07] 10Continuous-Integration-Config, 10Composer: Composer\Downloader\TransportException in mwext-testextension-hhvm-jessie - https://phabricator.wikimedia.org/T182266#3818688 (10thcipriani) Composer diag on a nodepool instance shows with mw-config ``` jenkins@ci-jessie-wikimedia-910534:~/space$ composer diag... [01:39:01] hrm seems setting HTTPS_PROXY_REQUEST_FULLURI=false works... [01:39:45] I don't know what that setting does [01:39:57] found via: https://github.com/composer/composer/issues/1839 [01:40:09] https://github.com/composer/composer/blob/4072d6bdb8f8c625a316aa0305c195f543ecc3d8/doc/03-cli.md#L834 [01:40:23] "If you use a proxy but it does not support the request_fulluri flag for HTTPS requests, then you should set this env var to false or 0 to prevent Composer from setting the request_fulluri option." [01:40:49] ok... [01:41:01] Seems a weird thing to suddenly break [01:41:42] yeah [01:42:31] What does it use as a proxy? [01:43:10] the networking for the nodepool vms is routed through whatever openstack magic there is [01:44:40] we might be looking at two different errors [01:44:47] https://integration.wikimedia.org/ci/job/mediawiki-extensions-hhvm-jessie/29657/console is 302 and the file does exist [01:44:58] most of the errors are 404s though [01:45:03] I did see that some of the time [01:45:04] and point at wrong URLs [01:45:13] the 302 sometimes, 404 others. [01:46:03] I got the envvar to work in ssh. I want to see if it works on a job... [01:47:37] fwiw the cache file for the hashes is .composer/cache/repo/https---packagist.org/packages.json , if something got stuck I think it must have gotten stuck there [01:52:14] hrm, I got rid of the .composer/cache directory entirely [01:52:30] https://integration.wikimedia.org/ci/job/operations-mw-config-composer-hhvm-jessie/9083/console [01:52:45] I have no idea why that worked. [01:52:50] well, kinda [01:52:55] I set HTTPS_PROXY_REQUEST_FULLURI=false [01:53:29] but I did: export HTTPS_PROXY_REQUEST_FULLURI=false and that didn't work, only when I put it in front of the composer command itself did it work [01:55:04] clearer: https://gist.github.com/thcipriani/a600286ec8fe076999b3ce169b9d6cd9 [01:55:27] https://gerrit.wikimedia.org/r/#/c/395817 seems to be failing jenkins during composer install for now appearent reason I can make sense of [01:56:07] oh wait, its everybody [01:56:07] bawolff: probably related to https://phabricator.wikimedia.org/T182266 [01:58:17] 10Continuous-Integration-Config, 10Composer: Composer\Downloader\TransportException in mwext-testextension-hhvm-jessie - https://phabricator.wikimedia.org/T182266#3818728 (10thcipriani) so for some reason adding `HTTPS_PROXY_REQUEST_FULLURI=false` seems to fix this https://integration.wikimedia.org/ci/job/oper... [02:07:26] well [02:07:32] I guess my workaround is working: https://integration.wikimedia.org/ci/job/mwext-testextension-hhvm-jessie/25195/console [02:07:34] ish [02:19:22] (03PS1) 10Thcipriani: HTTPS_PROXY_REQUEST_FULLURI=false for composer [integration/config] - 10https://gerrit.wikimedia.org/r/395913 (https://phabricator.wikimedia.org/T182266) [02:27:20] (03CR) 10Gergő Tisza: "Is the set -x intentional?" [integration/config] - 10https://gerrit.wikimedia.org/r/395913 (https://phabricator.wikimedia.org/T182266) (owner: 10Thcipriani) [02:29:17] (03CR) 10Thcipriani: "> Is the set -x intentional?" [integration/config] - 10https://gerrit.wikimedia.org/r/395913 (https://phabricator.wikimedia.org/T182266) (owner: 10Thcipriani) [02:29:30] the pessimistic explanation is that changing the request URI broke whatever proxy cache packages.json got stuck in, and the issue will resurface eventually [02:30:06] (03CR) 10Gergő Tisza: [C: 031] HTTPS_PROXY_REQUEST_FULLURI=false for composer [integration/config] - 10https://gerrit.wikimedia.org/r/395913 (https://phabricator.wikimedia.org/T182266) (owner: 10Thcipriani) [02:30:09] Running newer composer is only going to help in various ways too [02:31:17] alright, going to update the 41 jobs ^ effects now :) [02:32:36] thcipriani: <3 thank you (and the rest of ya'll) [02:33:34] Yay [02:33:38] rechecking my patch now [02:34:11] If the CI pipeline works, should I SWAT my patch 2h late, or should I defer it to 11am tomorrow? [02:36:20] (03CR) 10Thcipriani: [C: 032] "Going live now" [integration/config] - 10https://gerrit.wikimedia.org/r/395913 (https://phabricator.wikimedia.org/T182266) (owner: 10Thcipriani) [02:37:29] (03Merged) 10jenkins-bot: HTTPS_PROXY_REQUEST_FULLURI=false for composer [integration/config] - 10https://gerrit.wikimedia.org/r/395913 (https://phabricator.wikimedia.org/T182266) (owner: 10Thcipriani) [02:37:40] RoanKattouw: SWATing 2h late always sounds like fun to me :) I'm still deploying these fixes to all the jobs (in a big loop) - I can ping you when done. [02:37:49] OK cool thanks [02:38:00] * greg-g sends thcipriani a cold drink [02:38:04] RoanKattouw: ok, just finished [02:38:05] The jobs I need seem to be playing ball so far [02:39:13] cool, I'll be around watching for a bit [02:52:28] OK, my thing merged, going to deploy now [02:52:35] nice \o/ [02:53:12] 10Continuous-Integration-Config, 10Composer, 10Patch-For-Review: Composer\Downloader\TransportException in mwext-testextension-hhvm-jessie - https://phabricator.wikimedia.org/T182266#3818774 (10thcipriani) 05Open>03Resolved a:03thcipriani Alright, I've deployed the fix for 41 jobs and saw a few jobs th... [02:55:06] Ugh, scap needs better progress reporting [02:55:19] I'm running scap pull for what is possibly the first time after the train [02:55:41] yeah, IIRC there's a task [02:55:43] And it got suck without telling me what it's stuck on. I used ps and found out it was rebuilding CDBs which obviously does take a while [02:55:54] rebuilding cdb files, yeah [02:56:03] 31947 31950 31937 31930 pts/1 31937 D+ 998 1:31 | \_ /usr/bin/python /usr/bin/scap cdb-rebuild --no-progress [02:56:06] RoanKattouw: we could change it back to the real-time dump of dsh output ;) [02:56:13] So something somewhere it's passing in --no-progress [02:56:14] haha [02:56:21] I remember that [02:56:23] -vvvvvvvv [02:56:29] https://phabricator.wikimedia.org/T162207 [02:56:34] As bad as that was, there was something satisfying about it [02:56:44] The new progress indicators are much better though [02:56:49] the first time I ran scap I thought I must have done something horribly wrong [02:56:51] Watching the # of synced hosts count up is also satisfying [02:57:04] haha yeah usually when your screen is flooded like that it's with errors [02:57:05] how that was the UX for years escapes my understanding [02:57:33] you can thank reedy for the progress bar. ori thought it was a waste of energy to implement [02:57:37] Well we didn't have a releng team [02:57:42] That was probably one factor [02:57:46] Nobody maintained those tools [02:57:50] I did :) [02:58:02] for 3 months anyway [02:58:29] oh right, you mean why it was left so sketchy [02:58:39] wrapping dsh wasn't very easy [02:58:50] we didn't get better until we threw it out [03:01:16] thcipriani: thanks for fixing! is there a tool for forcing recheck on all the failing changesets? [03:02:01] I can go through them by hand if that's the only way, just asking in case somebody wrote a script for that [03:02:06] not that I'm aware of :( [03:11:10] OK I'm done [03:13:46] tgr: thanks for all the "recheck"s! :) [03:16:52] (03PS2) 10KartikMistry: Add VE dependency for ContentTranslation [integration/config] - 10https://gerrit.wikimedia.org/r/395734 [03:21:13] thcipriani: ty for the fix, [03:24:07] (03CR) 10KartikMistry: ">" (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/395734 (owner: 10KartikMistry) [03:28:28] alright, I am now shuffling off this digital coil for one evening, have a good evening all! [03:41:25] Project mediawiki-core-code-coverage build #3177: 04STILL FAILING in 41 min: https://integration.wikimedia.org/ci/job/mediawiki-core-code-coverage/3177/ [04:01:00] (03CR) 10Santhosh: [C: 031] Add VE dependency for ContentTranslation [integration/config] - 10https://gerrit.wikimedia.org/r/395734 (owner: 10KartikMistry) [04:03:19] Project selenium-MultimediaViewer » firefox,beta,Linux,BrowserTests build #600: 04FAILURE in 7 min 19 sec: https://integration.wikimedia.org/ci/job/selenium-MultimediaViewer/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/600/ [04:07:51] 10Continuous-Integration-Config, 10Composer, 10Patch-For-Review: Composer\Downloader\TransportException in mwext-testextension-hhvm-jessie - https://phabricator.wikimedia.org/T182266#3818521 (10Legoktm) Do we know why this suddenly just happened? [05:09:01] 10Release-Engineering-Team, 10Deployments, 10HHVM, 10Wikimedia-Incident: Figure out why HHVM kept running stale code for hours - https://phabricator.wikimedia.org/T181833#3803747 (10mmodell) This (or similar) seems to happen pretty regularly. Is there even any real justification for the mess that is Initia... [05:56:01] 10Release-Engineering-Team (Watching / External), 10Wikidata, 10Patch-For-Review, 10User-Addshore: Undeploy the Wikidata extension - https://phabricator.wikimedia.org/T181708#3818948 (10greg) [05:56:19] 10Release-Engineering-Team (Someday), 10MediaWiki-Release-Tools: make-release doesn't list mediawiki-core - https://phabricator.wikimedia.org/T180540#3818950 (10greg) [05:56:22] 10MediaWiki-Releasing, 10Release-Engineering-Team (Someday), 10MediaWiki-Release-Tools: make-release vendor diffing broken for REL1_27/REL1_28 - https://phabricator.wikimedia.org/T180532#3818952 (10greg) [05:56:27] 10MediaWiki-Releasing, 10Release-Engineering-Team (Someday), 10MediaWiki-Release-Tools: Make make-release patch structure like we do for production security patches et al - https://phabricator.wikimedia.org/T180526#3818960 (10greg) [05:56:31] 10MediaWiki-Releasing, 10Release-Engineering-Team (Someday), 10MediaWiki-Release-Tools: Make make-release not need to clone MW for every branch - https://phabricator.wikimedia.org/T180522#3818962 (10greg) [05:56:36] 10MediaWiki-Releasing, 10Release-Engineering-Team (Someday): Removal of EOL release branches broke SemanticMediaWiki CI - https://phabricator.wikimedia.org/T179754#3818967 (10greg) [05:56:40] 10Release-Engineering-Team (Watching / External), 10Operations, 10User-Joe: Create jenkins job for creating deployment artifacts for `docker-pkg-deploy` - https://phabricator.wikimedia.org/T179562#3818969 (10greg) [05:57:34] Yippee, build fixed! [05:57:35] Project mediawiki-core-code-coverage build #3178: 09FIXED in 1 hr 26 min: https://integration.wikimedia.org/ci/job/mediawiki-core-code-coverage/3178/ [06:30:51] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [10.0] [06:37:28] (03CR) 10Thiemo Mättig (WMDE): [C: 031] parameter_functions switch extensions to use Wikibase (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/395704 (owner: 10Addshore) [06:48:09] (03CR) 10Legoktm: "gated is for the mediawiki-extensions- jobs, it should also be updated." [integration/config] - 10https://gerrit.wikimedia.org/r/395704 (owner: 10Addshore) [06:53:52] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Watching / External), 10Performance-Team, 10Availability (Multiple-active-datacenters): Performance Q2 2017/18 goal: Install and use mcrouter in deployment-prep - https://phabricator.wikimedia.org/T151466#3819095 (10aaron) [06:54:02] RECOVERY - Free space - all mounts on deployment-fluorine02 is OK: OK: All targets OK [07:21:38] (03CR) 10Nikerabbit: [C: 031] Add VE dependency for ContentTranslation [integration/config] - 10https://gerrit.wikimedia.org/r/395734 (owner: 10KartikMistry) [07:27:26] Can anyone +2: https://gerrit.wikimedia.org/r/#/c/395734 [07:27:43] (or in better wording: review :)) [07:30:31] (03CR) 10Legoktm: [C: 031] Add VE dependency for ContentTranslation [integration/config] - 10https://gerrit.wikimedia.org/r/395734 (owner: 10KartikMistry) [07:30:57] kart_: there you go, it's +3 now [07:31:02] ;) [07:40:10] Project selenium-Wikibase » chrome,beta,Linux,BrowserTests build #567: 15ABORTED in 3 hr 0 min: https://integration.wikimedia.org/ci/job/selenium-Wikibase/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/567/ [08:00:49] RECOVERY - Mediawiki Error Rate on graphite-labs is OK: OK: Less than 1.00% above the threshold [1.0] [08:08:58] legoktm: lol [08:27:13] 10Release-Engineering-Team (Kanban), 10Discovery, 10Discovery-Search (Current work), 10Patch-For-Review, 10User-zeljkofilipin: Run selenium-EXTENSION-jessie Jenkins job for CirrusSearch - https://phabricator.wikimedia.org/T175179#3819224 (10dcausse) @zeljkofilipin we might be ready to port our selenium-C... [08:54:11] PROBLEM - Long lived cherry-picks on puppetmaster on deployment-puppetmaster02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [09:03:36] hashar: hi [09:04:15] kart_: good morning :) [09:04:22] hashar: https://gerrit.wikimedia.org/r/#/c/395734 - when you're free. [09:04:31] hashar: Good Morning! [09:04:56] kart_: ah yeah ;] [09:06:07] (03CR) 10Hashar: [C: 032] Add VE dependency for ContentTranslation [integration/config] - 10https://gerrit.wikimedia.org/r/395734 (owner: 10KartikMistry) [09:06:20] kart_: I should have adjusted the commit myself but I lacked time to do so yeasterday [09:06:43] hashar: no worries! [09:07:12] (03Merged) 10jenkins-bot: Add VE dependency for ContentTranslation [integration/config] - 10https://gerrit.wikimedia.org/r/395734 (owner: 10KartikMistry) [09:07:20] kart_: has some change been force merged that caused the job to fail ? [09:07:29] kart_: or is that just that new patches require VE ? [09:07:46] hashar: It started failing yesterday only. [09:07:53] strange [09:08:06] However, VE is hard dependency for CX. So, I should have added it earlier. [09:08:18] hashar: did anything change in jenkins setup? [09:09:33] https://gerrit.wikimedia.org/r/#/c/371133/ is from August 12th and added the requirement to extension.json [09:09:46] most probably, that is mediawiki/core now enforcing the requirement [09:09:58] * hashar rechecks https://gerrit.wikimedia.org/r/#/c/383570/ [09:11:03] kart_: I am not sure what happened really [09:12:32] hmm. strange. [09:14:05] some weird failure now: [09:14:08] Fatal error: Call to undefined method Mock_AnalysisConfigBuilder_f4ad6bbf::buildLanguageConfigs() in /home/jenkins/workspace/mwext-testextension-hhvm-jessie/src/extensions/Wikidata/extensions/Wikibase/repo/includes/Search/Elastic/ConfigBuilder.php on line 48 [09:14:51] kart_: yeah that is better [09:14:57] :) [09:15:03] the extensions/wikidata is being removed though [09:15:10] there are a few patches to switch it to the new repo Wikibase [09:15:40] I see. [09:15:52] I have no clue what AnalysisConfigBuilder can be though [09:15:54] hmm [09:16:10] eek sorry, I closed the wrong window [09:16:51] this is cirrus <> wikibase integration, does it fail on all patches? [09:16:56] hashar: Should we move from Wikidata->Wikibase? [09:17:28] dcausse: currently failed on last 2 patches atleast. [09:17:44] dcausse: eg https://integration.wikimedia.org/ci/job/mwext-testextension-hhvm-jessie/25211/console [09:18:28] perhaps it fails because cirrus is not there, but I think we added some guards to not run the tests in that case [09:18:36] looking [09:19:01] (03CR) 10Hashar: [C: 031] parameter_functions switch extensions to use Wikibase (032 comments) [integration/config] - 10https://gerrit.wikimedia.org/r/395704 (owner: 10Addshore) [09:24:51] addshore: I am gonna polish up the CI changes for the Wikidata -> Wikibase transition [09:24:55] kart_: that will take a bit of time [09:25:38] yes the guards are here https://github.com/wikimedia/mediawiki-extensions-Wikibase/blob/master/repo/tests/phpunit/includes/Search/Elastic/AnalysisConfigBuilderTest.php#L15 [09:25:46] so not sure what's causing the failure :( [09:30:20] (03Abandoned) 10Hashar: Load extensions of wikibase for wikibase CI [integration/config] - 10https://gerrit.wikimedia.org/r/395705 (owner: 10Addshore) [09:30:39] (03Abandoned) 10Hashar: Remove Wikidata.git from mirror-gerrit-repos script [integration/config] - 10https://gerrit.wikimedia.org/r/395707 (owner: 10Addshore) [09:30:40] (03Abandoned) 10Hashar: Archive mediawiki/extensions/Wikidata [integration/config] - 10https://gerrit.wikimedia.org/r/395895 (https://phabricator.wikimedia.org/T181838) (owner: 10Umherirrender) [09:31:12] (03PS2) 10Hashar: Migrate from Wikidata to Wikibase [integration/config] - 10https://gerrit.wikimedia.org/r/395704 (https://phabricator.wikimedia.org/T181838) (owner: 10Addshore) [09:32:26] (03CR) 10jerkins-bot: [V: 04-1] Migrate from Wikidata to Wikibase [integration/config] - 10https://gerrit.wikimedia.org/r/395704 (https://phabricator.wikimedia.org/T181838) (owner: 10Addshore) [09:32:28] (03CR) 10Hashar: "I have squashed all the other changes in there, else the CI test suite would fail for sure." [integration/config] - 10https://gerrit.wikimedia.org/r/395704 (https://phabricator.wikimedia.org/T181838) (owner: 10Addshore) [09:32:56] dcausse: kart_ yeah the build is going to stay broken for a little bit till I Fix it up [09:33:47] hi hashar! [09:34:08] yes, legoktm also noticed yesterday core is still running against the wikidata build instead of wikibase and extensions! [09:34:17] I dont know which list to change for that though [09:34:49] hashar: Okay! [09:35:08] addshore: I squashed all the changes in a single on [09:35:09] e [09:35:14] ack! [09:35:15] and I am trying to polish up something [09:35:27] all the changes you guys made are nice [09:35:41] but they need to be done in the same commit else the integration/config would fail for arndom reasons [09:37:07] hahaa, okay [09:37:12] addshore:I am not sure what will happen to all the jobs such as mwext-Wikibase-client-tests-mysql-php55-jessie [09:37:36] (03PS3) 10Hashar: Migrate from Wikidata to Wikibase [integration/config] - 10https://gerrit.wikimedia.org/r/395704 (https://phabricator.wikimedia.org/T181838) (owner: 10Addshore) [09:38:41] and most probably the build will be broken for wmf/ and REL branches but hey... cant fix everything in one go :D [09:38:55] yupo [09:39:07] ah tests passed [09:39:09] that is a start [09:43:19] addshore: I am gonna give https://gerrit.wikimedia.org/r/#/c/395704/ a try [09:43:35] it regroups all the different changes that have been made to get rid of "Wikidata" [09:43:49] okay! [09:43:59] but the wmf/* branches would be broken most probably [09:44:10] the REL ones as well, but that needs a bit more work [09:44:21] namely the gatedextension should only run for master and wmf branches [09:44:28] but yeah hmm bah [09:45:05] (03CR) 10Hashar: [C: 032] "Lets give it a try. That might fix a few of the breakage currently occurring." [integration/config] - 10https://gerrit.wikimedia.org/r/395704 (https://phabricator.wikimedia.org/T181838) (owner: 10Addshore) [09:45:13] addshore: hopefully mediawiki/vendor has all the dependencies required [09:45:31] it does :) [09:45:38] \o/ [09:46:53] (03Merged) 10jenkins-bot: Migrate from Wikidata to Wikibase [integration/config] - 10https://gerrit.wikimedia.org/r/395704 (https://phabricator.wikimedia.org/T181838) (owner: 10Addshore) [09:48:02] !log CI: removed Wikidata from configuration, replaced by Wikibase. wmf/* and REL branches are going to be broken though | https://gerrit.wikimedia.org/r/395704 | T181838 [09:48:14] so, it needs more than just wikibase i guess [09:48:16] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [09:48:16] T181838: Mark extension-Wikidata & wikidata-build-resources on Gerrit as ARCHIVED - https://phabricator.wikimedia.org/T181838 [09:48:28] it should have wikibase and all extensions to wikibase [09:48:36] yeah [09:48:46] so there are a lot more adjustements to be made I guess [09:49:16] though I think the testsuite validates that all extensions in the 'gatedextensions' list actually trigger the mediawiki-extensions- jobs [09:49:35] so I think the dependencies added to Wikibase are already using those jobs. I forgot to check though [09:50:25] hmm no [09:50:33] Scribunto / Capiunto are not [09:50:53] neither are PropertySuggested / WikibaseQuality / WikibaseQualityConstraints / WikimediaBadges [09:50:54] bah [09:51:10] or Wikidata.Org! [09:51:14] EEEK [09:52:05] I am afraid everything is going to be broken [09:52:34] i oudbt it will be broken, it just means less tests will run [09:52:35] but things could break [09:52:44] as the tesst wont all be run together [09:52:48] yeah [09:53:16] we need a way to differentiate between tests that are solely for an extension, and integration tests that rely on other extensions [09:53:30] like if do a tiny change to CiontentTranslation, surely I dont need all the Wikibase tests to run [09:53:42] (and we still have xdebug enabled which slow it down grr) [09:55:48] OH [09:56:02] kart_: so apparently ContentTranslation got fixed [09:56:08] kart_: at least on https://gerrit.wikimedia.org/r/#/c/383570/10 [09:57:08] for Wikibase, I am watching a recheck on https://gerrit.wikimedia.org/r/#/c/70395/ [10:06:05] !log integration: unbroke puppet on some permanent slaves. Add been broken since Nov 29th ~ 19:50UTC | https://gerrit.wikimedia.org/r/#/c/395961/ [10:06:10] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [10:06:50] dcausse: kart_: addshore: I think CI is fixed for the extensions "master" branch [10:06:59] at least Wikibase and ContentTranslation pass now [10:07:27] (03CR) 10Hashar: "ContentTranslation and Wikibase ( https://gerrit.wikimedia.org/r/#/c/70395/ ) pass now!" [integration/config] - 10https://gerrit.wikimedia.org/r/395704 (https://phabricator.wikimedia.org/T181838) (owner: 10Addshore) [10:09:52] !log integration: sudo cumin --force 'name:integration-slave-jessie-100*' /usr/local/sbin/run-puppet-agent | https://gerrit.wikimedia.org/r/395961 [10:09:56] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [10:13:16] Thanks hashar! [10:17:26] RECOVERY - Puppet errors on integration-slave-jessie-1001 is OK: OK: Less than 1.00% above the threshold [0.0] [10:21:44] RECOVERY - Puppet errors on integration-slave-jessie-1003 is OK: OK: Less than 1.00% above the threshold [0.0] [10:22:10] RECOVERY - Puppet errors on integration-slave-jessie-1004 is OK: OK: Less than 1.00% above the threshold [0.0] [10:22:22] ah puppet fixed [10:25:23] RECOVERY - Puppet errors on integration-slave-jessie-1002 is OK: OK: Less than 1.00% above the threshold [0.0] [10:40:46] :> [11:24:09] (03CR) 10Zfilipin: "Thanks! Sorry, I have forgot to deploy it after merging. :|" [integration/config] - 10https://gerrit.wikimedia.org/r/394111 (https://phabricator.wikimedia.org/T174591) (owner: 10Zfilipin) [11:27:16] 10Release-Engineering-Team (Kanban), 10Phabricator, 10monitoring, 10Browser-Tests, 10User-zeljkofilipin: Develop tests for phabricator search to detect regressions / search quality issues - https://phabricator.wikimedia.org/T182160#3819627 (10zeljkofilipin) @mmodell in which repository the tests should l... [12:04:18] (03CR) 10Zfilipin: [C: 032] Update RuboCop Ruby gem [ruby/api] - 10https://gerrit.wikimedia.org/r/395511 (https://phabricator.wikimedia.org/T180878) (owner: 10Zfilipin) [12:04:36] (03CR) 10Zfilipin: [C: 032] Update RuboCop Ruby gem [selenium] - 10https://gerrit.wikimedia.org/r/395513 (https://phabricator.wikimedia.org/T180878) (owner: 10Zfilipin) [12:06:27] (03Merged) 10jenkins-bot: Update RuboCop Ruby gem [selenium] - 10https://gerrit.wikimedia.org/r/395513 (https://phabricator.wikimedia.org/T180878) (owner: 10Zfilipin) [12:07:40] (03CR) 10jenkins-bot: Update RuboCop Ruby gem [selenium] - 10https://gerrit.wikimedia.org/r/395513 (https://phabricator.wikimedia.org/T180878) (owner: 10Zfilipin) [12:12:33] 10Continuous-Integration-Infrastructure (shipyard), 10PAWS, 10monitoring, 10Graphite: Purge labs graphite metrics of Docker ephemeral partitions - https://phabricator.wikimedia.org/T181476#3819762 (10hashar) [12:12:39] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10Cloud-Services, 10Operations, and 4 others: Grafana reports ALL docker mounts in a spammy way - https://phabricator.wikimedia.org/T177052#3819760 (10hashar) 05Resolved>03Open [12:12:40] 10Continuous-Integration-Infrastructure (shipyard), 10PAWS, 10monitoring, 10Graphite: Purge labs graphite metrics of Docker ephemeral partitions - https://phabricator.wikimedia.org/T181476#3791555 (10hashar) [12:12:48] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10Cloud-Services, 10Operations, and 4 others: Grafana reports ALL docker mounts in a spammy way - https://phabricator.wikimedia.org/T177052#3645705 (10hashar) 05Open>03Resolved [12:39:14] (03PS1) 10Hashar: Enable castor on npm-node-6-docker jobs [integration/config] - 10https://gerrit.wikimedia.org/r/395992 [12:39:37] (03CR) 10Hashar: [C: 032] "Tested on wikibase-javascript-api-npm-browser-node-6-docker" [integration/config] - 10https://gerrit.wikimedia.org/r/395992 (owner: 10Hashar) [12:41:03] (03Merged) 10jenkins-bot: Enable castor on npm-node-6-docker jobs [integration/config] - 10https://gerrit.wikimedia.org/r/395992 (owner: 10Hashar) [13:34:17] (03PS1) 10Hashar: rake job: gemspec is a file suffix [integration/config] - 10https://gerrit.wikimedia.org/r/396004 [13:35:28] (03PS2) 10Hashar: rake job: gemspec is a file suffix [integration/config] - 10https://gerrit.wikimedia.org/r/396004 [13:37:33] (03CR) 10Zfilipin: [C: 032] rake job: gemspec is a file suffix [integration/config] - 10https://gerrit.wikimedia.org/r/396004 (owner: 10Hashar) [13:38:37] (03Merged) 10jenkins-bot: rake job: gemspec is a file suffix [integration/config] - 10https://gerrit.wikimedia.org/r/396004 (owner: 10Hashar) [13:39:59] (03CR) 10Hashar: "recheck" [ruby/api] - 10https://gerrit.wikimedia.org/r/395511 (https://phabricator.wikimedia.org/T180878) (owner: 10Zfilipin) [13:41:48] (03CR) 10Zfilipin: [C: 032] Update RuboCop Ruby gem [ruby/api] - 10https://gerrit.wikimedia.org/r/395511 (https://phabricator.wikimedia.org/T180878) (owner: 10Zfilipin) [13:42:10] (03Merged) 10jenkins-bot: Update RuboCop Ruby gem [ruby/api] - 10https://gerrit.wikimedia.org/r/395511 (https://phabricator.wikimedia.org/T180878) (owner: 10Zfilipin) [13:42:38] (03CR) 10jenkins-bot: Update RuboCop Ruby gem [ruby/api] - 10https://gerrit.wikimedia.org/r/395511 (https://phabricator.wikimedia.org/T180878) (owner: 10Zfilipin) [13:42:39] PROBLEM - Puppet errors on deployment-sentry01 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [13:43:16] addshore: twist for ya. Instead of: [13:43:19] rm -fr src [13:43:41] and having src with mode 2777 + a umask [13:43:52] we can use a container to delete the files owned by nobody: [13:43:54] docker run --rm -v /srv/jenkins-workspace/workspace/apps-android-wikipedia-npm-node-6-docker:/workspace --entrypoint=/bin/rm docker-registry.wikimedia.org/wikimedia-stretch:latest -fR /workspace/src [13:43:55] :] [13:44:54] hah, indeed [13:45:25] eg npm install would end up creating a node_modules file belonging to nobody:nogroup [13:45:34] hapilly ignoring the setgid flag on the parent directory bah [13:45:44] so on next build, rm -fR fails to delete the node_modules dir bah [13:46:21] I had the trouble with tox / pip as well which create files such as cache/.pip/http with mode 600 (eg not readable by group) [13:46:22] bah [13:46:23] yeh, so you can just use a base image as user nobody to control the files [13:46:30] yeah [13:46:40] or I make ci-src-setup-simple to discard the content of src [13:46:47] yeh [13:46:53] oooh, thats actually even nicer [13:47:00] ORR [13:47:03] OORRRR [13:47:20] I stop using ci-src-setup-simple and let the npm-test image to do the clonining ;] [13:47:23] too many options hehe [13:47:30] naaah, i dont like the last one [13:47:41] right now i can still use all of these tests to test my code locally [13:47:42] :D [13:47:47] well I am surely going to do it so we rely on a single container [13:47:50] but not right now [13:48:10] if you do, all I ask is that you have a big wrapper around the clonding /fetching of the code, so i can pas NOFETH=true or something :D [13:48:19] yeah that was the idea [13:48:20] *NOFETCH [13:49:21] I will just [ "$USER" != "addshore" ] && rm -fR src [13:49:50] I am just shooting non sense [13:49:58] gotta find how I fixed the perm issue for tox [13:50:50] mv hashar /mnt/field/tomatoes [13:51:12] ;D [14:17:38] RECOVERY - Puppet errors on deployment-sentry01 is OK: OK: Less than 1.00% above the threshold [0.0] [14:23:26] 10Release-Engineering-Team, 10Wikidata: Make one group1 wiki a client of testwikidata (preferably a test wiki) - https://phabricator.wikimedia.org/T182326#3820084 (10Ladsgroup) [14:27:53] (03PS1) 10Hashar: Delete log/src as nobody via a docker container [integration/config] - 10https://gerrit.wikimedia.org/r/396009 [14:34:47] 10Release-Engineering-Team, 10Wikidata: Make one group1 wiki a client of testwikidata (preferably a test wiki) - https://phabricator.wikimedia.org/T182326#3820108 (10thiemowmde) p:05Triage>03High [15:16:19] 10Continuous-Integration-Config, 10Easy, 10Need-volunteer: adjust recheck regex to also act on comment with additional inline comments - https://phabricator.wikimedia.org/T130640#3820233 (10thiemowmde) p:05Triage>03Low [15:24:45] (03PS2) 10Hashar: Delete log/src as nobody via a docker container [integration/config] - 10https://gerrit.wikimedia.org/r/396009 [15:25:21] 10Release-Engineering-Team, 10Wikidata: Make one group1 wiki a client of testwikidata (preferably a test wiki) - https://phabricator.wikimedia.org/T182326#3820259 (10thiemowmde) [15:29:08] 10Release-Engineering-Team, 10Wikidata: Make one group1 wiki a client of testwikidata (preferably a test wiki) - https://phabricator.wikimedia.org/T182326#3820084 (10thiemowmde) [15:30:24] 10Release-Engineering-Team, 10Wikidata: Make one group1 wiki a client of testwikidata (preferably a test wiki) - https://phabricator.wikimedia.org/T182326#3820084 (10Addshore) +1 we should do this. It is an easy way to spot possible issues with wikidata related compat deployments. These cases will never be ca... [15:33:41] (03CR) 10Addshore: [C: 031] Delete log/src as nobody via a docker container [integration/config] - 10https://gerrit.wikimedia.org/r/396009 (owner: 10Hashar) [15:35:13] (03CR) 10Hashar: "Went a find -mindepth 1 to prevent deleting the actual log or src directory (which belong to jenkins-deploy)." [integration/config] - 10https://gerrit.wikimedia.org/r/396009 (owner: 10Hashar) [15:35:48] (03CR) 10Hashar: [C: 032] Delete log/src as nobody via a docker container [integration/config] - 10https://gerrit.wikimedia.org/r/396009 (owner: 10Hashar) [15:35:52] addshore: that is terrible :D [15:36:03] what is? [15:36:07] the testwiki idea? :P [15:36:26] 10Release-Engineering-Team, 10Wikidata: Make one group1 wiki a client of testwikidata (preferably a test wiki) - https://phabricator.wikimedia.org/T182326#3820318 (10Ladsgroup) Making test2wiki a group1 wiki won't be super hard but @greg needs to sign it off first. [15:36:47] addshore: na using a docker container to delete files :D [15:37:10] haha, i like it [15:37:10] (03Merged) 10jenkins-bot: Delete log/src as nobody via a docker container [integration/config] - 10https://gerrit.wikimedia.org/r/396009 (owner: 10Hashar) [15:37:40] 10Release-Engineering-Team, 10Wikidata: Make one group1 wiki a client of testwikidata (preferably a test wiki) - https://phabricator.wikimedia.org/T182326#3820320 (10Addshore) I think it would be less confusing to simply make a new wiki for this specific purpose. [15:41:04] (03PS1) 10Hashar: Migrate oojs/core npm job to Docker [integration/config] - 10https://gerrit.wikimedia.org/r/396028 [15:44:17] (03PS2) 10Hashar: Migrate oojs/core npm job to Docker [integration/config] - 10https://gerrit.wikimedia.org/r/396028 [15:44:27] (03CR) 10Hashar: [C: 032] Migrate oojs/core npm job to Docker [integration/config] - 10https://gerrit.wikimedia.org/r/396028 (owner: 10Hashar) [15:45:57] (03Merged) 10jenkins-bot: Migrate oojs/core npm job to Docker [integration/config] - 10https://gerrit.wikimedia.org/r/396028 (owner: 10Hashar) [15:46:25] 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10Release Pipeline (Blubber): Establish blubber config in mathoid - https://phabricator.wikimedia.org/T173127#3820361 (10thcipriani) 05stalled>03Open [15:46:32] (03PS1) 10Hashar: Migrate some npm jobs to Docker [integration/config] - 10https://gerrit.wikimedia.org/r/396029 [15:47:07] 10Continuous-Integration-Config, 10Technical-Debt: Move setting of DOC_* variables from Zuul function to inside the build scripts - https://phabricator.wikimedia.org/T113250#3820366 (10thiemowmde) 05Open>03declined [15:47:09] 10Continuous-Integration-Config, 10Patch-For-Review: Broken coverage report published to literal dir name "$DOC_PROJECT" - https://phabricator.wikimedia.org/T113195#3820367 (10thiemowmde) [15:49:22] 10Continuous-Integration-Config, 10Technical-Debt: Move setting of DOC_* variables from Zuul function to inside the build scripts - https://phabricator.wikimedia.org/T113250#3820374 (10Legoktm) @thiemowmde why did you decline this? [15:51:56] 10Continuous-Integration-Config, 10Incident-20160126-WikimediaDomainRedirection, 10Operations, 10Regression, 10Wikimedia-Incident: operations-apache-config-lint replacement doesn't check syntax - https://phabricator.wikimedia.org/T114801#3820379 (10thiemowmde) 05Open>03Invalid [15:51:57] 10Continuous-Integration-Infrastructure, 10Operations, 10Patch-For-Review: Jenkins: Re-enable lint checks for Apache config in operations-puppet - https://phabricator.wikimedia.org/T72068#3820380 (10thiemowmde) [15:53:59] eh [15:54:25] (03PS2) 10Hashar: Migrate some npm jobs to Docker [integration/config] - 10https://gerrit.wikimedia.org/r/396029 [15:54:34] eh? [15:54:45] ah [15:54:51] (03CR) 10Hashar: [C: 032] Migrate some npm jobs to Docker [integration/config] - 10https://gerrit.wikimedia.org/r/396029 (owner: 10Hashar) [15:55:24] legoktm: tldr of today: do not rm -fR cache log src [15:55:24] thiemo is closing legit looking tasks without comments - I pinged him in a different channel [15:56:08] legoktm: instead use something like: docker run --user=nobody --entrypoint=/usr/bin/find stretch -mindepth 1 cache log src [15:56:08] bah [15:56:21] (03Merged) 10jenkins-bot: Migrate some npm jobs to Docker [integration/config] - 10https://gerrit.wikimedia.org/r/396029 (owner: 10Hashar) [15:58:16] 10Continuous-Integration-Infrastructure, 10Operations, 10Patch-For-Review: Jenkins: Re-enable lint checks for Apache config in operations-puppet - https://phabricator.wikimedia.org/T72068#3820401 (10Legoktm) [15:58:18] 10Continuous-Integration-Config, 10Incident-20160126-WikimediaDomainRedirection, 10Operations, 10Regression, 10Wikimedia-Incident: operations-apache-config-lint replacement doesn't check syntax - https://phabricator.wikimedia.org/T114801#3820399 (10Legoktm) 05Invalid>03Open This hasn't been fixed yet. [15:59:35] * addshore is sleepy [16:00:59] 10Continuous-Integration-Config, 10Technical-Debt: Move setting of DOC_* variables from Zuul function to inside the build scripts - https://phabricator.wikimedia.org/T113250#3820413 (10Legoktm) 05declined>03Open [16:00:59] 10Continuous-Integration-Config, 10Patch-For-Review: Broken coverage report published to literal dir name "$DOC_PROJECT" - https://phabricator.wikimedia.org/T113195#3820414 (10Legoktm) [16:09:24] 10Continuous-Integration-Infrastructure, 10Wikidata: [Bug] github.com is 403ing downloads from Wikimedia CI during composer update - https://phabricator.wikimedia.org/T106519#3820508 (10thiemowmde) [16:22:32] 10Continuous-Integration-Config, 10Technical-Debt: Move setting of DOC_* variables from Zuul function to inside the build scripts - https://phabricator.wikimedia.org/T113250#3820559 (10thiemowmde) I reviewed the backlog of tickets our former employee @JanZerebecki created. This appears to be a request for some... [16:54:53] 10Continuous-Integration-Config, 10MediaWiki-Core-Tests: mediawiki/core tests failing for REL1_30 due to Wikibase - https://phabricator.wikimedia.org/T182336#3820620 (10Umherirrender) [16:56:46] 10Continuous-Integration-Config, 10MediaWiki-Core-Tests, 10MW-1.30-release: mediawiki/core tests failing for REL1_30 due to Wikibase - https://phabricator.wikimedia.org/T182336#3820634 (10Legoktm) p:05Triage>03High +cc @Addshore we probably need to use the build for everything pre-1.31 to make tests stil... [16:59:18] 10Continuous-Integration-Config, 10MediaWiki-Core-Tests, 10MW-1.30-release: mediawiki/core tests failing for REL1_30 due to Wikibase - https://phabricator.wikimedia.org/T182336#3820649 (10Umherirrender) Or Wikibase is removed from gated extensions for the release branches, but I did not know if that is wante... [17:02:56] 10Continuous-Integration-Config, 10MediaWiki-Core-Tests, 10MW-1.30-release: mediawiki/core tests failing for REL1_30 due to Wikibase - https://phabricator.wikimedia.org/T182336#3820655 (10Addshore) I'm not sure how (In ci) we can make different branches load a different set of extensions :/ @hashar has been... [17:03:27] no_justification i've almost completed the log4j2 update :). I am now uncertain which class to use for the daily rolling as there is CronTriggeringPolicy and TimeBasedTriggeringPolicy [17:03:37] * paladox wishes log4j kept the daily rolling class [17:03:41] * no_justification shrugs [17:04:54] anyways we can set a custom var for logstash to identify it. I also want to get the gelf lib into gerrit too. [17:05:24] Do we have an exact date for 1.30 release? [17:07:14] 10Continuous-Integration-Config, 10MediaWiki-Core-Tests, 10MW-1.30-release: mediawiki/core tests failing for REL1_30 due to Wikibase - https://phabricator.wikimedia.org/T182336#3820667 (10Legoktm) The change was made in parameter_functions.py right? We can add a check for the branch in that file, it's just p... [17:11:37] 10Continuous-Integration-Config, 10MediaWiki-Core-Tests, 10MW-1.30-release: mediawiki/core tests failing for REL1_30 due to Wikibase - https://phabricator.wikimedia.org/T182336#3820671 (10Addshore) Yup, parameter_functions.py! [17:54:17] it's now complete (log4j2) :) [17:56:07] !log deployed mobileapps@71f581c to beta cluster [17:56:11] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [17:56:47] When are we releasing REL1_30 [18:05:38] no_justification actually great news looking at log4j2 docs looks like it supports gelf internally. As socket appender supports udp now and also includes a gelf layout we can use. [18:05:44] * paladox wants to test this udp :) [18:35:18] 10Continuous-Integration-Config, 10Incident-20160126-WikimediaDomainRedirection, 10Operations, 10Regression, 10Wikimedia-Incident: operations-apache-config-lint replacement doesn't check syntax - https://phabricator.wikimedia.org/T114801#3820885 (10Dzahn) This is correct, i believe it's still a valid ToD... [19:25:01] (03PS1) 10Reedy: Make GraphViz depend on ImageMap [integration/config] - 10https://gerrit.wikimedia.org/r/396093 [19:25:39] 10Scap, 10Packaging: SCAP: Upload debian package version 3.7.4-1 - https://phabricator.wikimedia.org/T182347#3821018 (10mmodell) [19:45:46] 10Release-Engineering-Team, 10Wikidata: Make one group1 wiki a client of testwikidata (preferably a test wiki) - https://phabricator.wikimedia.org/T182326#3821068 (10greg) Which makes sense? Give me the pros/cons (not looking for an RFC :P ). [19:51:50] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [10.0] [19:56:24] no_justification lol i just found a bug in gerrit [19:56:33] some how i managed to create a change without a subject [19:56:33] just one? [19:56:37] https://gerrit.wikimedia.org/r/#/q/owner:thomasmulhall410%2540yahoo.com+status:open [19:56:41] I got 99 bugs but a..... [19:56:46] it's a deaft [19:56:59] Looks like leading newline wasn't stripped [19:57:02] draft, and now i carn't delete it and doint know the patch number for it [19:57:11] https://gerrit.wikimedia.org/r/#/c/395983/ that one? [19:57:24] ah yep [19:57:26] thanks [20:06:08] 10Release-Engineering-Team, 10Wikidata: Make one group1 wiki a client of testwikidata (preferably a test wiki) - https://phabricator.wikimedia.org/T182326#3821139 (10daniel) @addshore in which way would it be confusing? Who would be confused? What is test2wiki used for, specifically? Adding a wiki is not as e... [20:09:36] 10Release-Engineering-Team, 10Wikidata: Make one group1 wiki a client of testwikidata (preferably a test wiki) - https://phabricator.wikimedia.org/T182326#3821144 (10Addshore) >>! In T182326#3821139, @daniel wrote: > @addshore in which way would it be confusing? Who would be confused? What is test2wiki used fo... [20:12:24] 10Release-Engineering-Team, 10Wikidata: Make one group1 wiki a client of testwikidata (preferably a test wiki) - https://phabricator.wikimedia.org/T182326#3821149 (10daniel) > Test2 is primarily used to trial and debug global and cross-wiki features in conjunction with https://test.wikipedia.org and https://te... [20:18:07] beta cluster recent changes looks disgusting [20:18:07] * Hauskatze cleans [20:18:56] !log deployment-prep maurelio@deployment-tin:~$ mwscript cleanupSpam.php --wiki=deploymentwiki --delete *.loginidol.org [20:19:01] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [20:20:55] PROBLEM - Puppet errors on deployment-ms-be04 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [20:20:56] 10Continuous-Integration-Config, 10MediaWiki-Core-Tests, 10MW-1.30-release: mediawiki/core tests failing for REL1_30 due to Wikibase - https://phabricator.wikimedia.org/T182336#3821184 (10hashar) The `mediawiki-extensions-*` jobs are shared by a fixed set of extensions and the list of dependencies is `gatede... [20:21:35] 10Continuous-Integration-Config, 10Release-Engineering-Team (Kanban), 10MediaWiki-Core-Tests, 10MW-1.30-release: mediawiki/core tests failing for REL1_30 due to Wikibase - https://phabricator.wikimedia.org/T182336#3821186 (10hashar) [20:24:32] 10Continuous-Integration-Config, 10Technical-Debt: Move setting of DOC_* variables from Zuul function to inside the build scripts - https://phabricator.wikimedia.org/T113250#3821195 (10hashar) [20:25:23] !log maurelio@deployment-tin:~$ mwscript namespaceDupes.php --wiki=deploymentwiki --fix --add-prefix=Broken/ [20:25:27] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [20:27:26] 10Continuous-Integration-Config, 10Technical-Debt: Move setting of DOC_* variables from Zuul function to inside the build scripts - https://phabricator.wikimedia.org/T113250#3821200 (10hashar) @thiemowmde thank you for the backlog clean up. That is indeed a valid request to reduce confusion. It is probably no... [20:28:10] Yippee, build fixed! [20:28:10] Project selenium-Wikibase-chrome » chrome,beta,Linux,DebianJessie && contintLabsSlave build #35: 09FIXED in 41 min: https://integration.wikimedia.org/ci/job/selenium-Wikibase-chrome/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=DebianJessie%20&&%20contintLabsSlave/35/ [20:28:13] legoktm: so eventually I have found out extension.json supports defining dependencies/requirements :]]]]] [20:28:18] https://phabricator.wikimedia.org/T117277 ! [20:28:55] lol [20:29:26] hmm I think --fix and --add-prefix ain't compatible each other? [20:29:53] I fixed the dupes but the "Broken/" thing was not added [20:30:23] and eventually we can drop https://github.com/wikimedia/integration-config/blob/66920a4/zuul/parameter_functions.py#L133-L150 Reedy :] [20:30:23] in any case, I'll delete them all :P [20:30:36] lol [20:30:53] I don't think it tries to load the extensions does it? Just checks if it's loaded? [20:31:14] yeah that is enough for CI purpose [20:31:44] we can then just blindly wfLoadExtensions( glob( "$wgExtensionDir/*" ) ); or something like that [20:32:35] Having random extensions loaded... Has some use... Maybe [20:34:27] (03CR) 10Hashar: "Tyler, would you be able to regenerate the operations-puppet image please? I am afraid I haven't dig in how to regenerate the docker-pkg " [integration/config] - 10https://gerrit.wikimedia.org/r/394551 (https://phabricator.wikimedia.org/T181794) (owner: 10Filippo Giunchedi) [20:34:51] thcipriani: I am lamely pretending I have no idea how CI/docker-pkg works ^^ :D [20:35:18] oh boy :) [20:36:00] sure: I'll push it up to wmfreleng :P [20:38:11] danke ! [20:38:23] and guess who is running commands and let them fail happilly ??? npm! [20:39:19] id=98 ns=0 dbk=Commons:Picture_of_the_day -> Commons-Picture_of_the_day (no conflict) [20:39:29] but on Special:Seach it is still there [20:39:33] :| [20:41:47] RECOVERY - Mediawiki Error Rate on graphite-labs is OK: OK: Less than 1.00% above the threshold [1.0] [20:46:30] PROBLEM - Puppet errors on deployment-ms-be03 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [20:50:07] the 'page' table isn't showing it so I guess it's gone and it's just a caching issue [20:50:21] btw does beta supports 'foreachwikiindblist' ? [20:51:43] (03PS2) 10Hashar: Migrate a VisualEditor npm job to Docker [integration/config] - 10https://gerrit.wikimedia.org/r/395553 [20:53:02] (03CR) 10Hashar: [C: 032] "CCing James since I guess he still has some interest in VisualEditor:" [integration/config] - 10https://gerrit.wikimedia.org/r/395553 (owner: 10Hashar) [20:53:44] !log maurelio@deployment-tin:~$ mwscript namespaceDupes.php --wiki=dewiki --fix [20:53:49] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [20:54:32] (03Merged) 10jenkins-bot: Migrate a VisualEditor npm job to Docker [integration/config] - 10https://gerrit.wikimedia.org/r/395553 (owner: 10Hashar) [20:55:16] 10Beta-Cluster-Infrastructure: de.wiktionary.beta is using Commons Beta logo - https://phabricator.wikimedia.org/T182355#3821297 (10MarcoAurelio) [20:55:43] 10Beta-Cluster-Infrastructure: de.wiktionary.beta is using Commons Beta logo - https://phabricator.wikimedia.org/T182355#3821307 (10MarcoAurelio) [20:58:21] 10Beta-Cluster-Infrastructure, 10User-MarcoAurelio: Broken namespaces on enwiki Beta Cluster - https://phabricator.wikimedia.org/T182356#3821327 (10MarcoAurelio) [21:00:34] 10Beta-Cluster-Infrastructure, 10User-MarcoAurelio: Broken namespaces / pagelinks on enwiki Beta Cluster - https://phabricator.wikimedia.org/T182356#3821356 (10MarcoAurelio) [21:01:25] 10Beta-Cluster-Infrastructure, 10User-MarcoAurelio: Broken namespaces / pagelinks on enwiki Beta Cluster - https://phabricator.wikimedia.org/T182356#3821327 (10MarcoAurelio) My idea was to run `mwscript namespaceDupes.php --wiki=enwiki --fix` but will that fix **pagelinks**? Thanks. [21:01:46] (03CR) 10Thcipriani: "Needs to bump the changelog version in dockerfiles/operations-puppet/changelog as well so that docker-pkg will build a new version." [integration/config] - 10https://gerrit.wikimedia.org/r/394551 (https://phabricator.wikimedia.org/T181794) (owner: 10Filippo Giunchedi) [21:01:48] hashar: https://hub.docker.com/r/wmfreleng/docker-pkg-operations-puppet/ [21:02:21] =) [21:02:26] if it looks good you can +2 to merge and use the fab task https://gerrit.wikimedia.org/r/#/c/391592/ to deploy :) [21:02:45] isn't it missing tags ? [21:02:50] oh no that is a transient image [21:04:35] thcipriani: how you wrote all of that in less than half an hour AND rebuild the image is ... above my grade [21:04:59] 8-] [21:05:36] 10Release-Engineering-Team, 10Wikidata: Make one group1 wiki a client of testwikidata (preferably a test wiki) - https://phabricator.wikimedia.org/T182326#3821362 (10Addshore) >>! In T182326#3821149, @daniel wrote: >> Test2 is primarily used to trial and debug global and cross-wiki features in conjunction with... [21:06:24] heh, I wrote that thing a while ago :P [21:06:46] guess I will have a look at it now :] [21:07:02] I need some hot water with some kind of herbs in it [21:07:05] (not tea, "verveine" in french) [21:07:20] (03CR) 10Thcipriani: [C: 031] "-1 for changelog change needed." [integration/config] - 10https://gerrit.wikimedia.org/r/394551 (https://phabricator.wikimedia.org/T181794) (owner: 10Filippo Giunchedi) [21:07:26] (03CR) 10Thcipriani: [C: 04-1] operations-puppet: install mtail [integration/config] - 10https://gerrit.wikimedia.org/r/394551 (https://phabricator.wikimedia.org/T181794) (owner: 10Filippo Giunchedi) [21:07:29] damnit [21:07:32] :) [21:08:31] I think I need some hot water with extraction of coffee bean in it. [21:08:45] *reads up* [21:12:44] hashar: yes, my eventual plan is to have CI use extension.json dependencies :) but I don't think we're there yet [21:13:01] hashar: mostly because there's no distinction between real dependency and dev dependency yet [21:14:22] legoktm: im about to do it for the phan version loaded in CI [21:14:23] ;) [21:14:48] oh wait, no, i was going to use composer.json for that for now.. [21:14:50] meh [21:15:37] yes, that should go in composer [21:15:45] hashar: thcipriani if all the patches get merged moving the system for docker building to the pkg system, then dockerizing CI will be one of my christmas projects ;) [21:16:28] addshore: also I was thinking of having a mediawiki/phan-config composer package so we can ship the default config in one place instead of putting it everywhere [21:16:55] sounds good [21:16:55] I think it'll be useful especially with the new plugin coming soon [21:17:14] i look forward to this plugin [21:17:25] although bawolf said it takes a while to run [21:17:32] thcipriani: for your fab task. I think I am too tired to properly review it. If it works for you do self merge it, we can always enhance it later [21:18:09] hashar: ok, will do, seems to work fine [21:18:12] addshore: I'm sure people will figure out how to speed it up later :) [21:18:14] legoktm: addshore: if you wanna make that some kind of epic, we can probably created a dedicated Phabricator project for it. That might be helpful [21:18:19] hehe, yup [21:18:25] addshore: yep, I know I'm blocking you, have started down the docker-pkg php path a bit. [21:18:36] hashar: for phan or extension dependencies? [21:18:43] (03CR) 10Hashar: [C: 031] "I remember we talked about it a few weeks ago. if it works I guess you can merge it :)" [integration/config] - 10https://gerrit.wikimedia.org/r/391592 (owner: 10Thcipriani) [21:18:45] took me an afternoon to catchup with all the work you've done :) [21:18:49] legoktm: both ? :] [21:18:57] phan I was just going to do it [21:19:03] thcipriani: im super busy right now anyway :) but i saw that you already documented the pkg docker stuff on wiki which is excellet :D [21:19:31] legoktm: that sounds good to me as well [21:19:32] hashar: for extension dependencies I think we need more work in MediaWiki related to extension management (that I plan on working on soon!) but CI is definitely one of my usecases that a lot of this was designed around [21:19:39] legoktm: my point being if it takes more than a few days and there is a lot of things to do, a subproject in Phabricator might make it easier to track the advance. But a giant task would do as well :] [21:20:02] yeap, got it documented, it's pretty straight-forward really, just needs some careful testing to move stuff to using it and I ran out of afternoon the day I made real progress on it. [21:20:19] (03CR) 10Thcipriani: [C: 032] "wfm!" [integration/config] - 10https://gerrit.wikimedia.org/r/391592 (owner: 10Thcipriani) [21:20:53] thcipriani: oh and python-debian might help to drop the dch requirement :] [21:22:06] (03Merged) 10jenkins-bot: New fab task for rebuilding docker-pkg images [integration/config] - 10https://gerrit.wikimedia.org/r/391592 (owner: 10Thcipriani) [21:22:07] hashar: that would be nice :) [21:22:12] legoktm: if we can drop the list of dependencies from integration/config zuul/parameters_functions.py , that would be a HUGE plus [21:22:24] I did spend some time just now discovering the gbp dch has the path debian/changelog hard-coded all over afaict :) [21:25:45] (03PS10) 10Hashar: (WIP) Cache node_modules between runs [integration/config] - 10https://gerrit.wikimedia.org/r/395610 (https://phabricator.wikimedia.org/T159591) [21:27:39] (03Abandoned) 10Hashar: castor: cache binaries on a per job basis [integration/config] - 10https://gerrit.wikimedia.org/r/346152 (https://phabricator.wikimedia.org/T159591) (owner: 10Hashar) [21:28:34] 10Continuous-Integration-Config, 10Patch-For-Review: Castor: mediawiki-core-qunit-jessie node_modules cache ineffective - https://phabricator.wikimedia.org/T159591#3821390 (10hashar) I am migrating the npm jobs to Docker containers. https://gerrit.wikimedia.org/r/#/c/395610/ would potentially save and restore... [21:29:15] thcipriani: https://pypi.python.org/pypi/python-debian desperately lacks any doc pointer :( [21:29:37] thcipriani: but it has a debian.changelog module, and maybe it is pure python [21:29:57] at least it only depends on python modules [21:30:47] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10Patch-For-Review: Castor: mediawiki-core-qunit-jessie node_modules cache ineffective - https://phabricator.wikimedia.org/T159591#3821403 (10hashar) [21:56:33] 10Continuous-Integration-Infrastructure (shipyard): Create docker capacity dashboard similar to nodepool one - https://phabricator.wikimedia.org/T177686#3821460 (10hashar) I have added a row for `*Docker` labels on https://grafana-admin.wikimedia.org/dashboard/db/zuul?orgId=1 [21:59:58] 10Continuous-Integration-Infrastructure (shipyard): Create docker capacity dashboard similar to nodepool one - https://phabricator.wikimedia.org/T177686#3821463 (10hashar) And @addshore created https://grafana-admin.wikimedia.org/dashboard/db/ci-docker-jobs to show the break down by permanent/nodepool/docker exe... [22:02:50] 10Beta-Cluster-Infrastructure, 10User-MarcoAurelio: de.wiktionary.beta is using Commons Beta logo - https://phabricator.wikimedia.org/T182355#3821464 (10MarcoAurelio) [23:47:48] 10Beta-Cluster-Infrastructure, 10User-MarcoAurelio: de.wiktionary.beta is using Commons Beta logo - https://phabricator.wikimedia.org/T182355#3821297 (10TerraCodes) Might just be due to it having no wiki there