[01:57:28] 10Project-Admins: Tags for WCAG levels - https://phabricator.wikimedia.org/T189558#4048548 (10tstarling) 05Open>03Resolved a:03tstarling [02:37:41] PROBLEM - Free space - all mounts on deployment-tin is CRITICAL: CRITICAL: deployment-prep.deployment-tin.diskspace._mnt.byte_percentfree (No valid datapoints found)deployment-prep.deployment-tin.diskspace._srv.byte_percentfree (<20.00%) [05:06:52] Krinkle: we might need to force it with --ansi or something like we do for composer [05:08:38] Krinkle: it uses symfony/console for coloring, so at least locally it's colored for me [05:08:46] but getting it on jenkins would be pretty nice [07:49:14] 10Continuous-Integration-Config, 10Wikidata: integration-slave-jessie-1002.integration.eqiad.wmflabs editting under 10.68.16.199 on test.wikidata - https://phabricator.wikimedia.org/T189047#4048860 (10hashar) p:05Triage>03Low This are for the Wikibase end to end tests. The tests do have to make edit / crea... [07:52:51] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10User-greg: integration-slave-jessie-1004 is out of disk space - https://phabricator.wikimedia.org/T189365#4048863 (10hashar) 05Resolved>03Open [07:53:09] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10User-greg: integration-slave-jessie-1004 is out of disk space - https://phabricator.wikimedia.org/T189365#4040013 (10hashar) [07:53:12] 10Continuous-Integration-Infrastructure: integration-slave-jessie-1001 and integration-slave-jessie-1002 out of disk space - https://phabricator.wikimedia.org/T189587#4048868 (10hashar) [07:55:36] (03PS3) 10Hashar: parsoidsvc-hhvm-parsertests to Docker [integration/config] - 10https://gerrit.wikimedia.org/r/418984 [08:02:31] (03PS1) 10Hashar: Migrate phpunit-coverage-patch jobs to Nodepool [integration/config] - 10https://gerrit.wikimedia.org/r/419361 (https://phabricator.wikimedia.org/T189365) [08:05:07] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10User-greg: integration-slave-jessie-1004 is out of disk space - https://phabricator.wikimedia.org/T189365#4048896 (10hashar) That keeps happening, those slaves are kept between builds and the build workspace... [08:05:27] !log Move Jenkins jobs mwext-phpunit-coverage-patch mwext-phpunit-coverage-publish mediawiki-phpunit-coverage-patch to Nodepool (they use too much disk space) | T189365 [08:05:30] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [08:05:30] T189365: integration-slave-jessie-1004 is out of disk space - https://phabricator.wikimedia.org/T189365 [08:06:45] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10User-greg: integration-slave-jessie-XXXX is out of disk space - https://phabricator.wikimedia.org/T189365#4048900 (10hashar) [08:09:14] 10Continuous-Integration-Infrastructure (shipyard): mwgate-npm-node-6-docker failed on a patch with an error saying "No space left on device" - https://phabricator.wikimedia.org/T189616#4048904 (10hashar) 05duplicate>03Open T189587 is about permanent slaves This one is about the Docker slaves. [08:10:35] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10Technical-Debt: Phaseout CI mediawiki config / extensions_load.txt to load extensions - https://phabricator.wikimedia.org/T189567#4048912 (10hashar) >>! In T189567#4047054, @Legoktm wrote: >> For... [08:13:19] (03CR) 10Hashar: [C: 032] Migrate phpunit-coverage-patch jobs to Nodepool [integration/config] - 10https://gerrit.wikimedia.org/r/419361 (https://phabricator.wikimedia.org/T189365) (owner: 10Hashar) [08:14:32] (03Merged) 10jenkins-bot: Migrate phpunit-coverage-patch jobs to Nodepool [integration/config] - 10https://gerrit.wikimedia.org/r/419361 (https://phabricator.wikimedia.org/T189365) (owner: 10Hashar) [08:14:41] (03PS2) 10Hashar: Use shallow clone for phplint jobs [integration/config] - 10https://gerrit.wikimedia.org/r/405722 (https://phabricator.wikimedia.org/T179963) [08:39:51] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Someday): Install php-yaml for Translate - https://phabricator.wikimedia.org/T189244#4048939 (10Nikerabbit) Translate on WMF production does not even use any YAML features. It is primarily used in translatewiki.net to read 1) group configurat... [08:50:09] PROBLEM - Puppet errors on integration-slave-jessie-1004 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [08:50:59] PROBLEM - Puppet errors on integration-slave-jessie-1001 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [08:54:13] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10Technical-Debt: Phaseout CI mediawiki config / extensions_load.txt to load extensions - https://phabricator.wikimedia.org/T189567#4048944 (10hashar) [09:14:04] PROBLEM - Puppet errors on deployment-mira is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [09:18:36] PROBLEM - Puppet errors on deployment-tin is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [09:33:56] (03PS1) 10Hashar: jjb: remove unused macro [integration/config] - 10https://gerrit.wikimedia.org/r/419373 [09:34:44] (03CR) 10Hashar: "That comes from Ie080af4a6717c5df6c284118dc85408ac6f42af1 and the job got deleted via I35c117a57c00ff6a6c0a6b2925cb31e99e77da79 :]" [integration/config] - 10https://gerrit.wikimedia.org/r/419373 (owner: 10Hashar) [09:40:19] (03PS1) 10Hashar: mw-apply-settings: get LocalSettings.php archived [integration/jenkins] - 10https://gerrit.wikimedia.org/r/419375 [09:40:36] (03CR) 10Hashar: [C: 032] mw-apply-settings: get LocalSettings.php archived [integration/jenkins] - 10https://gerrit.wikimedia.org/r/419375 (owner: 10Hashar) [09:41:17] (03Merged) 10jenkins-bot: mw-apply-settings: get LocalSettings.php archived [integration/jenkins] - 10https://gerrit.wikimedia.org/r/419375 (owner: 10Hashar) [09:52:50] !log Refreshing Nodepool snapshot for "mw-apply-settings: get LocalSettings.php archived" https://gerrit.wikimedia.org/r/#/c/419375/ [09:52:52] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [10:04:10] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10User-greg: integration-slave-jessie-XXXX is out of disk space - https://phabricator.wikimedia.org/T189365#4049128 (10Gilles) Something's still wrong with integration-slave-jessie-1001: https://integration.w... [10:05:56] 10Phabricator, 10Release-Engineering-Team (Kanban), 10Phlogiston: Phlogiston reports don't have new data since mid-February - https://phabricator.wikimedia.org/T188149#4049142 (10Aklapper) See T188149#4010091 - clear steps that allow someone else to see the problem, to start with. [10:07:27] !log nodepool: Image snapshot-ci-jessie-1521021138 in wmflabs-eqiad is ready [10:07:29] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [10:18:18] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10User-greg: integration-slave-jessie-XXXX is out of disk space - https://phabricator.wikimedia.org/T189365#4049175 (10hashar) @Gilles Indeed though that error is unrelated to disk space. That specific slave h... [10:22:01] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban): phabricator-jessie-diffs fails with "echo : No such file or directory" - https://phabricator.wikimedia.org/T189660#4049184 (10Gilles) [11:21:07] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10MW-1.31-release-notes (WMF-deploy-2018-03-20 (1.31.0-wmf.26)), 10Patch-For-Review, 10Technical-Debt: Phaseout CI mediawiki config / extensions_load.txt to load exte... - https://phabricator.wikimedia.org/T189567#4049327 [11:22:44] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban): phabricator-jessie-diffs fails with "echo : No such file or directory" - https://phabricator.wikimedia.org/T189660#4049333 (10hashar) [11:22:56] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban): phabricator-jessie-diffs fails with "echo : No such file or directory" - https://phabricator.wikimedia.org/T189660#4049184 (10hashar) [11:22:59] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10User-greg: integration-slave-jessie-XXXX is out of disk space - https://phabricator.wikimedia.org/T189365#4049335 (10hashar) [11:25:29] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban): phabricator-jessie-diffs fails with "echo : No such file or directory" - https://phabricator.wikimedia.org/T189660#4049336 (10hashar) From the Jenkins job: ``` lang=shell set +e if [ -f 'tox.ini' ]; then echo 'Found tox.ini, runn... [12:14:58] 10Release-Engineering-Team, 10DNS, 10Operations, 10Traffic, and 2 others: Move Foundation Wiki to new URL when new Wikimedia Foundation website launches - https://phabricator.wikimedia.org/T188776#4049434 (10Pcoombe) The current plan is to move the thank you pages and some other fundraising specific pages... [12:16:17] !log update cxserver to c355d0c [12:16:18] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [13:14:53] kart_: whii [13:40:23] (03PS2) 10Thiemo Kreuz (WMDE): Add missing newlines to multi-line if() [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/419126 [14:18:26] Project beta-scap-eqiad build #199537: 04FAILURE in 4 min 14 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/199537/ [14:18:52] more full [14:21:56] Reedy: can I rm -Rf /srv/mediawiki-staging/php-master/cache/l10n/* ? [14:22:17] I've just done it on /srv/mediawiki [14:22:24] doing some git gc to see if I can free up some space [14:22:33] awesome :) [14:23:01] just disabled beta-scap-eqiad for the moment [14:23:17] !log beta-scap-eqiad disabled while freeing up space on deployment-tin [14:23:19] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [14:27:30] srv/jenkins and srv/jenkins-workspace are clear as well [14:37:42] PROBLEM - Free space - all mounts on deployment-tin is CRITICAL: CRITICAL: deployment-prep.deployment-tin.diskspace._mnt.byte_percentfree (No valid datapoints found)deployment-prep.deployment-tin.diskspace._srv.byte_percentfree (<40.00%) [14:47:39] RECOVERY - Free space - all mounts on deployment-tin is OK: OK: deployment-prep.deployment-tin.diskspace._mnt.byte_percentfree (No valid datapoints found) [15:27:45] PROBLEM - Puppet errors on deployment-redis01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [15:28:11] is something amiss with mediawiki CI? Puppet changes are getting checked as I'd expect but https://gerrit.wikimedia.org/r/#/c/417974/ has been sitting there for 10 minutes [15:30:13] https://integration.wikimedia.org/zuul/ [15:30:28] queuing for something.. [15:31:47] I don't fully understand how to read that page [15:32:03] two of the three tasks for your patch are "queued" [15:32:20] There's a lot of idle slaves though [15:33:18] yeah, and nodepool looks fine (for once) [15:33:42] PROBLEM - Puppet errors on deployment-redis02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [15:38:03] hmmmmmm 55lint job is stuck? [15:39:33] wait, that runs on nodepool now? D: [15:40:02] oh no, just on a labs slave... [15:40:46] and they are currently all backlogged with some selenium tests and code coverage.... [15:41:43] bah, the code coverage job will block those 2 slaves for agggessss [15:42:18] they normally take around an hour or so [15:42:20] hashar: :( [15:42:29] another integration slave would be sweeeeet [15:42:50] andrewbogott: ^^ [15:43:15] did you enabled again beta-scap-equiad after the cleanup Reedy ? :) [15:43:19] Not yet [15:43:38] Because it's not going to last for long [15:44:13] Project mediawiki-core-code-coverage build #3382: 15ABORTED in 44 min: https://integration.wikimedia.org/ci/job/mediawiki-core-code-coverage/3382/ [15:44:15] Project mediawiki-core-code-coverage-php7 build #143: 15ABORTED in 44 min: https://integration.wikimedia.org/ci/job/mediawiki-core-code-coverage-php7/143/ [15:44:31] !log killed https://integration.wikimedia.org/ci/job/mediawiki-core-code-coverage/3382/ and https://integration.wikimedia.org/ci/job/mediawiki-core-code-coverage-php7/143/ to unblock integration slaves [15:44:33] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [15:44:58] addshore: thanks [15:45:13] I guess I should file a ticket or something [15:45:14] :D [15:50:04] Project selenium-Wikibase-chrome » chrome,beta,Linux,DebianJessie && contintLabsSlave build #145: 04FAILURE in 45 min: https://integration.wikimedia.org/ci/job/selenium-Wikibase-chrome/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=DebianJessie%20&&%20contintLabsSlave/145/ [15:50:20] 10Release-Engineering-Team: mediawiki-core-code-coverage & mediawiki-core-code-coverage-php7 jobs lock up labs integration slaves - https://phabricator.wikimedia.org/T189693#4050252 (10Addshore) [15:50:21] andrewbogott: https://phabricator.wikimedia.org/T189693 [15:51:23] 10Release-Engineering-Team (Kanban): mediawiki-core-code-coverage & mediawiki-core-code-coverage-php7 jobs lock up labs integration slaves - https://phabricator.wikimedia.org/T189693#4050263 (10Addshore) a:03hashar [15:58:19] Yippee, build fixed! [15:58:19] Project beta-scap-eqiad build #199538: 09FIXED in 13 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/199538/ [15:59:00] Project beta-code-update-eqiad build #197693: 04FAILURE in 40 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/197693/ [15:59:13] fucking things [15:59:50] yeah always the fucking things... [16:04:45] rip [16:05:29] Project beta-code-update-eqiad build #197694: 04STILL FAILING in 36 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/197694/ [16:05:30] 10Release-Engineering-Team (Kanban): mediawiki-core-code-coverage & mediawiki-core-code-coverage-php7 jobs lock up labs integration slaves - https://phabricator.wikimedia.org/T189693#4050286 (10Legoktm) I didn't get a chance to finish investigating everything, but I think we can drop the php56 job, and rename th... [16:12:52] 10Release-Engineering-Team (Watching / External), 10Operations, 10ops-eqiad, 10Patch-For-Review: setup/install/deploy deploy1001 as deployment server - https://phabricator.wikimedia.org/T175288#4050320 (10RobH) Ok, IRC conversation update: Next Steps: * boot deploy1001 off live CD and check SMART attribu... [16:13:38] Project beta-code-update-eqiad build #197695: 04STILL FAILING in 37 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/197695/ [16:20:03] error: cannot update the ref 'refs/remotes/origin/master': unable to append to '/srv/mediawiki-staging/php-master/extensions/.git/modules/WikimediaMaintenance/logs/refs/remotes/origin/master': Permission denied [16:23:08] looks like some chmod permission error? [16:24:10] reedy@deployment-tin:~$ ls -al /srv/mediawiki-staging/php-master/extensions/.git/modules/WikimediaMaintenance/logs/refs/remotes/origin/master [16:24:10] -rw-r--r-- 1 root root 2870 Mar 14 15:09 /srv/mediawiki-staging/php-master/extensions/.git/modules/WikimediaMaintenance/logs/refs/remotes/origin/master [16:25:09] drwxr-xr-x 860 jenkins-deploy wikidev 36864 Mar 8 21:33 extensions [16:25:42] maurelio@deployment-tin:/srv/mediawiki-staging/php-master$ ls -la [16:25:50] the above is for the extensions folder [16:26:28] fixed to jenkins-deploy:wikidev [16:26:34] drwxr-xr-x 7 jenkins-deploy wikidev 4096 Mar 10 09:13 WikimediaMaintenance [16:27:00] lets see if the insecte says anything :) [16:27:35] https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/197696/console is building [16:27:42] I've got a strange issue, I've received an email with "[Maniphest] [Created] T189698: Wikidata API occasionally returns HTML instead of JSON" [16:27:42] T189698: Wikidata API occasionally returns HTML instead of JSON - https://phabricator.wikimedia.org/T189698 [16:27:49] though I'm not subscribed to that task, nor ever was [16:27:50] Yippee, build fixed! [16:27:50] Project beta-code-update-eqiad build #197696: 09FIXED in 46 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/197696/ [16:27:57] :D [16:27:58] so why would this task be sent to me? [16:28:26] Reception|away: maybe you're 'watching' the project attached to the task? [16:28:34] or you got added as subscriber? [16:28:40] or have a Herald rule? [16:28:58] Reedy: for future reference, how did you fix that? [16:29:10] Don't have a herald rule, and I checked and I was not added as a subscriber [16:29:18] sudo chown -R jenkins-deploy:wikidev /srv/mediawiki-staging/php-master [16:29:20] and there are no projects for said task [16:29:34] ah, I don't think I can sudo there [16:29:37] 10Scap, 10Operations, 10Packaging, 10Patch-For-Review: Install git-lfs client (at least on scap targets & masters) - https://phabricator.wikimedia.org/T180628#4050430 (10demon) I can't think of any compelling reason why it would be *required* on the masters...but I could see it as being //useful// if you w... [16:30:04] cloud-services [16:30:31] Reedy: Oh, how come that's listed under "subscribers" and not "tags"? [16:30:36] Because magnus filed it wrong [16:30:56] ah, ok [16:30:59] Reception|away: because they added 'cloud-services' as subscriber and you're a member of that project [16:31:05] that's why you got the mail [16:31:18] As did 100+ other people [16:31:22] heh [16:31:32] unless they disabled mail from the project [16:31:38] few people do that though [16:32:19] Project beta-scap-eqiad build #199541: 04FAILURE in 4 min 28 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/199541/ [16:32:22] ok, thanks for the help. I was looking at the tags section and didn't think about the project being in "subscribers" instead [16:33:00] * Hauskatze tests his (lack of) sudo permissions [16:36:48] !log maurelio@deployment-tin:~$ sudo chown -R jenkins-deploy:wikidev /srv/mediawiki-staging/php-master [16:36:50] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:37:36] Project beta-scap-eqiad build #199542: 04STILL FAILING in 3 min 56 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/199542/ [16:37:58] 10Deployments, 10Release-Engineering-Team (Someday), 10Operations, 10Beta-Cluster-reproducible, and 2 others: Switch mwscript from Zend PHP5 to default php alternative (e.g. HHVM or PHP7) - https://phabricator.wikimedia.org/T146285#4050464 (10demon) 05Open>03Resolved a:03demon [16:38:19] the .cdb files are the ones giving issues [16:38:43] 10Release-Engineering-Team (Watching / External), 10Operations, 10ops-eqiad, 10Patch-For-Review: setup/install/deploy deploy1001 as deployment server - https://phabricator.wikimedia.org/T175288#4050484 (10RobH) p:05Normal>03High [16:38:58] 10Release-Engineering-Team (Watching / External), 10Operations, 10ops-eqiad, 10Patch-For-Review: setup/install/deploy deploy1001 as deployment server - https://phabricator.wikimedia.org/T175288#3588945 (10RobH) a:05RobH>03Cmjohnson [16:39:22] added g+w to the cdb... [16:39:24] -rw-r--r-- 1 jenkins-deploy wikidev 3953364 Mar 14 16:25 l10n_cache-zu.cdb [16:39:24] -rwxrwxrwx 1 jenkins-deploy wikidev 6657605 Feb 27 07:20 lad.l10n.php [16:39:35] different sets of permissions [16:44:31] new build in progress, let's see if it passes this time :) [16:46:36] 10Phabricator, 10Release-Engineering-Team (Kanban), 10Phlogiston: Phlogiston reports don't have new data since mid-February - https://phabricator.wikimedia.org/T188149#4050526 (10jmatazzoni) In T188149#4049142, @Aklapper wrote: > See T188149#4010091 - clear steps that allow someone else to see the problem, t... [16:47:45] Project beta-scap-eqiad build #199543: 04STILL FAILING in 4 min 1 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/199543/ [16:48:03] cp: cannot create regular file ‘/srv/mediawiki-staging/php-master/cache/l10n/l10n_cache-ab.cdb’: Permission denied [16:48:19] chmod 777 ? [16:48:39] PROBLEM - Puppet errors on deployment-jobrunner02 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [16:49:14] Hauskatze: where is this ? [16:49:41] doesn't look like tin [16:49:43] beta-scap-equiad mutante [16:50:23] those .cdb files are -rw-rw-r-- for jenkins-deploy:wikidev [16:50:33] maybe scap is using none of those? [16:50:39] I admit I'm a bit lost here [16:51:04] and of course that jobrunner failure I'd not know how to start to debug it [16:51:10] (unrelated though) [16:51:36] Hauskatze: can you give me the full hostname (hostname -f )? [16:52:35] I think Sam has just chmod the whole lot of the files [16:52:53] deployment-tin.deployment-prep.eqiad.wmflabs [16:55:43] Hauskatze: confirmed. somebody already changed it. [16:55:52] they are writable by _anyone_ now [16:56:12] anyone with access to the project [16:56:46] well, if group and user wasn't enough [16:56:59] so mutante maybe we should check who triggers https://integration.wikimedia.org/ci/job/beta-scap-eqiad/199544/console and add it either to jenkins-deploy or wikidev [16:57:19] as for the puppet errors on -jobrunner02, no idea [16:57:36] i have another comment [16:58:03] it's passing now [16:58:04] on deployment-prep you have /srv/mediawiki-staging/php-master [16:58:10] on prod this does not exist [16:58:22] because beta runs master [16:58:23] :P [16:58:25] just saying because "prep not like prod" and stuff [16:58:51] Yippee, build fixed! [16:58:52] Project beta-scap-eqiad build #199544: 09FIXED in 5 min 12 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/199544/ [16:59:00] i never know if this is for testing new stuff or for being like prod [16:59:02] \o/ [17:03:06] (03PS1) 10WMDE-leszek: Run node selenium tests as part of WikibaseLexeme CI build [integration/config] - 10https://gerrit.wikimedia.org/r/419479 (https://phabricator.wikimedia.org/T189677) [17:05:07] (03CR) 10WMDE-leszek: Run node selenium tests as part of WikibaseLexeme CI build (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/419479 (https://phabricator.wikimedia.org/T189677) (owner: 10WMDE-leszek) [17:06:33] (03PS2) 10WMDE-leszek: Run node selenium tests as part of WikibaseLexeme CI build [integration/config] - 10https://gerrit.wikimedia.org/r/419479 (https://phabricator.wikimedia.org/T189677) [17:07:05] (03PS3) 10WMDE-leszek: Run node selenium tests as part of WikibaseLexeme CI build [integration/config] - 10https://gerrit.wikimedia.org/r/419479 (https://phabricator.wikimedia.org/T189677) [17:08:08] (03Restored) 10Hashar: Skip php 5.5 for mediawiki wmf branches [integration/config] - 10https://gerrit.wikimedia.org/r/344642 (https://phabricator.wikimedia.org/T94149) (owner: 10Hashar) [17:08:17] (03PS3) 10Hashar: Skip php 5.5 for mediawiki wmf branches [integration/config] - 10https://gerrit.wikimedia.org/r/344642 (https://phabricator.wikimedia.org/T94149) [17:08:19] (03CR) 10jerkins-bot: [V: 04-1] Run node selenium tests as part of WikibaseLexeme CI build [integration/config] - 10https://gerrit.wikimedia.org/r/419479 (https://phabricator.wikimedia.org/T189677) (owner: 10WMDE-leszek) [17:28:41] RECOVERY - Puppet errors on deployment-jobrunner02 is OK: OK: Less than 1.00% above the threshold [0.0] [17:29:18] (03CR) 10Hashar: [C: 032] jjb: remove unused macro [integration/config] - 10https://gerrit.wikimedia.org/r/419373 (owner: 10Hashar) [17:29:24] (03CR) 10Hashar: [C: 032] "Noop in JJB" [integration/config] - 10https://gerrit.wikimedia.org/r/419373 (owner: 10Hashar) [17:30:28] (03Merged) 10jenkins-bot: jjb: remove unused macro [integration/config] - 10https://gerrit.wikimedia.org/r/419373 (owner: 10Hashar) [17:40:44] no_justification: Ping re my email about maps deployments [17:44:37] In my inbox :) [17:44:39] * no_justification digs it up [17:45:20] Hmmm. Does it need a dedicated window, or could you piggyback on the existing service deploy window? [18:05:28] Project selenium-Wikibase-chrome » chrome,test,Linux,DebianJessie && contintLabsSlave build #145: 15ABORTED in 3 hr 0 min: https://integration.wikimedia.org/ci/job/selenium-Wikibase-chrome/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=test,PLATFORM=Linux,label=DebianJessie%20&&%20contintLabsSlave/145/ [18:06:40] no_justification: I probably could piggyback in the long term, but for the first time, maybe not [18:06:46] This service hasn't had a deploy in >1y [18:08:41] Fair enough. You were saying Wednesday 9am our time? [18:19:07] 10Release-Engineering-Team (Kanban), 10Vector, 10MW-1.31-release-notes (WMF-deploy-2018-03-06 (1.31.0-wmf.24)), 10Patch-For-Review, 10User-zeljkofilipin: Move one Selenium tests from mediawiki/core to mediawiki/skins/Vector - https://phabricator.wikimedia.org/T187859#4051006 (10zeljkofilipin) [[ https://... [18:22:17] (03PS3) 10Thcipriani: Add totals to deploy-notes [tools/release] - 10https://gerrit.wikimedia.org/r/416873 [18:27:19] thcipriani: I guess next step is to rig ^ up to a jenkins job that polls wmf/* branches and when a new one is found, do the changelog from it .. old one? [18:28:26] no_justification: that sounds like a good plan :) [18:30:08] 10Release-Engineering-Team, 10DNS, 10Operations, 10Traffic, and 2 others: Move Foundation Wiki to new URL when new Wikimedia Foundation website launches - https://phabricator.wikimedia.org/T188776#4051066 (10Varnent) >>! In T188776#4046204, @jhsoby wrote: >>>! In T188776#4021634, @Varnent wrote: >>>>! In T... [18:39:21] PROBLEM - Puppet errors on deployment-mx is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [18:51:08] 10Release-Engineering-Team (Watching / External), 10Operations, 10ops-eqiad, 10Patch-For-Review: setup/install/deploy deploy1001 as deployment server - https://phabricator.wikimedia.org/T175288#4051147 (10Cmjohnson) The errors are disk related, same as bast1002. In bast1002 we replaced dev/sda and the I/... [18:59:06] no_justification lol upstream did not catch the courrupt cache bug that was introduced today heh [18:59:10] only took me to find it [18:59:14] with a simple reindex [18:59:16] https://gerrit-review.googlesource.com/c/gerrit/+/165952 [19:23:42] Project beta-scap-eqiad build #199557: 04FAILURE in 30 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/199557/ [19:28:25] Project beta-scap-eqiad build #199558: 04STILL FAILING in 4 min 1 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/199558/ [19:35:02] no_justification: Does anything need to happen regarding https://phabricator.wikimedia.org/T187761 ? [19:35:39] Documentation changes? [19:35:51] Otherwise, I say just announce it as Official Policy [19:37:43] Project beta-scap-eqiad build #199559: 04STILL FAILING in 4 min 5 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/199559/ [19:38:43] lol @ https://github.com/wikimedia/operations-mediawiki-config/commit/64fb7dddc6a44154ad (by: no_justification ) [19:39:48] Hehe [19:43:42] !log deployment-prep: fixed permissions on i18n cache to unbreak scap job [19:43:44] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [19:47:44] Project beta-scap-eqiad build #199560: 04STILL FAILING in 4 min 2 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/199560/ [19:52:11] Krinkle, is sync-file of directories really fine? [19:53:30] yup [19:53:37] it's the same as sync-dir now [19:57:47] Project beta-scap-eqiad build #199561: 04STILL FAILING in 4 min 7 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/199561/ [20:00:48] Reedy, as I recall, sync-dir wasn't fine? [20:07:47] 11:08:41 <+no_justification> Fair enough. You were saying Wednesday 9am our time? [20:07:57] Sorry, got distracted by meetings + going to the office [20:08:01] Yes Wednesday 9am [20:08:05] Project beta-scap-eqiad build #199562: 04STILL FAILING in 4 min 22 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/199562/ [20:08:08] Krenair: The code between sync-file and sync-dir was nearly identical. [20:08:10] I merged them [20:08:14] sync-dir is just a back-compat alias [20:08:22] ok [20:08:24] (that i'll likely drop in a release or two) [20:08:43] We think that's a good time timezone-wise, because Stephane (east coast) is usually going to do it, but we also want me (SF) and Guillaume (Europe) to be able to do it [20:09:15] but wouldn't sync-dir would update files in some arcane order and leave PHP to read in some updated and some non-updated files in the middle? [20:23:27] !log deployed to BC: [mobileapps/deploy@0f9625a]: Update mobileapps to 9f4a80c [20:23:29] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [20:29:34] Yippee, build fixed! [20:29:35] Project selenium-Wikibase-chrome » chrome,beta,Linux,DebianJessie && contintLabsSlave build #146: 09FIXED in 42 min: https://integration.wikimedia.org/ci/job/selenium-Wikibase-chrome/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=DebianJessie%20&&%20contintLabsSlave/146/ [20:39:02] PROBLEM - Puppet errors on deployment-mediawiki07 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [20:48:21] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10MW-1.31-release-notes (WMF-deploy-2018-03-06 (1.31.0-wmf.24)), 10Patch-For-Review, 10Technical-Debt: Phaseout CI mediawiki config / extensions_load.txt to load exte... - https://phabricator.wikimedia.org/T189567#4051592 [20:50:28] Project beta-scap-eqiad build #199563: 04STILL FAILING in 36 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/199563/ [20:52:53] (03CR) 10Legoktm: Add totals to deploy-notes (031 comment) [tools/release] - 10https://gerrit.wikimedia.org/r/416873 (owner: 10Thcipriani) [20:58:37] Yippee, build fixed! [20:58:37] Project beta-scap-eqiad build #199564: 09FIXED in 7 min 24 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/199564/ [21:19:57] no_justification i like the new 404 / 403 error screen with my change https://gerrit-review.googlesource.com/c/gerrit/+/160132 :) [22:11:38] paladox: A-ha! I think I figured out how to configure URLs and other things /inside/ gitiles [22:11:47] no_justification oh :) [22:11:49] how? [22:11:49] All-Projects.git -> refs/meta/config -> gitiles.config [22:11:54] ah [22:12:00] * paladox tests [22:12:04] what do i put in gitiles.config ? [22:12:05] no_justification ^^ [22:12:21] Well, any config we want there, I'm looking [22:12:24] archive.format is one [22:13:24] logFormat.*? [22:13:37] * paladox wonders what logFormat.* is . [22:13:41] log.follow? [22:13:45] These are all undocumented [22:13:57] lol they have no docs [22:13:58] yeh [22:14:50] gitiles.fixedTimeZone? [22:15:08] gitiles.fixedTimeZone is that to fix it so it wont convert it in that users timezone [22:15:17] I guess? [22:16:14] no_justification seems the config was introduced in https://github.com/GerritCodeReview/gitiles/commit/2b2f34bba3c2be7e2506ce6b1f040949da350cf9 [22:17:08] I *think*..... [22:17:17] We can set gitiles.basePath in All-Projects.... [22:18:27] no_justification https://gerrit.git.wmflabs.org/r/#/c/59/ [22:18:31] * paladox submits that [22:18:58] (03Draft2) 10Chad: Edit Project Config [All-Projects] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/419606 [22:19:04] hmm nope https://gerrit.git.wmflabs.org/g/All-Projects/+/17283008b3a45016bac687a72d4232a1cad3d830 [22:19:32] Bleh [22:20:21] I was looking at getConfig() in gitiles {the plugin} which is what got me thinking this [22:21:07] (03PS4) 10Hashar: parsoidsvc-hhvm-parsertests to Docker [integration/config] - 10https://gerrit.wikimedia.org/r/418984 [22:21:15] ah [22:21:23] (03PS16) 10Zoranzoki21: Add few extensions in zuul/layout.yaml to Jenkins can run builds and remove mediawiki/extensions/Collection/OfflineContentGenerator/node_modules [integration/config] - 10https://gerrit.wikimedia.org/r/406524 (https://phabricator.wikimedia.org/T183674) [22:22:28] (03CR) 10jerkins-bot: [V: 04-1] Add few extensions in zuul/layout.yaml to Jenkins can run builds and remove mediawiki/extensions/Collection/OfflineContentGenerator/node_modules [integration/config] - 10https://gerrit.wikimedia.org/r/406524 (https://phabricator.wikimedia.org/T183674) (owner: 10Zoranzoki21) [22:23:13] no_justification i doin't think any gitiles config will work without trying to pass it through the gitiles gerrit plugin [22:23:47] maybe this https://github.com/GerritCodeReview/plugins_gitiles/blob/master/src/main/java/com/googlesource/gerrit/plugins/gitiles/HttpModule.java [22:23:52] is generating the url [22:23:52] If we didn't have any private repos, we could just use gitiles outta the box [22:23:55] As its own daemon [22:24:33] yep [22:24:42] Ahhhh, so the plugin is basically rewriting any urls it gets from gitiles. [22:24:46] Gotcha [22:25:02] So what we need is a change to use the baseUrl/basePath stuff we did already for the Weblinks part [22:25:20] i think so [22:25:32] as it has got to be getting plugins/* from some where [22:25:45] that file looks likely the culprit [22:26:18] It's whatever HttpModule does as default for plugins [22:26:57] yeh [22:28:29] no_justification aha [22:28:31] https://github.com/GerritCodeReview/gitiles/blob/master/java/com/google/gitiles/GitilesUrls.java#L66 [22:29:36] Well that's the base gerrit url, which you still need [22:29:40] For login, links back, etc [22:31:57] oh [22:32:08] what about getBaseGitUrl no_justification ? [22:32:27] That's for clone links and such, right? [22:33:52] ah right i guess so [22:33:54] getHostName [22:35:11] https://github.com/GerritCodeReview/gitiles/blob/HEAD/java/com/google/gitiles/GitilesFilter.java#L341 is where I'm looking right now [22:35:19] getHostName() is just the host [22:35:32] 10Scap: Support git-lfs - https://phabricator.wikimedia.org/T180627#3764014 (10awight) I'm doing an experiment with git-lfs today. I'm going to enable a git-lfs submodule on a branch of the ORES deployment repo, and try it on deployment-prep. [22:36:20] (03PS1) 10Awight: Automatically merge on a new repo [integration/config] - 10https://gerrit.wikimedia.org/r/419610 (https://phabricator.wikimedia.org/T180627) [22:38:06] hasharAway: would you have a look into https://gerrit.wikimedia.org/r/#/c/407166/ & https://integration.wikimedia.org/ci/job/oojs-ui-npm-run-jenkins-node-6-docker/126/console – it's plaguing us for some days now [22:39:22] no_justification i wonder how google do it with gitiles [22:39:31] ie have it through the deamon [22:39:37] but have the same access controls [22:39:38] They don't run it as a gerrit plugin I bet [22:39:42] so private repos are hidden [22:39:43] yeh [22:39:46] They also get secret sauce we don't ;-) [22:39:52] oh [22:40:02] They could very easily do it based on a replication destination [22:40:07] Replicate all public repos to $some_dir [22:40:12] oh i see [22:40:12] And then gitiles runs standalone from there? [22:40:14] * no_justification shrugs [22:41:15] no_justification i've asked here https://groups.google.com/forum/#!topic/repo-discuss/tttrqlBXZ_w :) [22:41:21] * paladox goes through the docs [22:42:15] no_justification ah [22:42:17] https://github.com/GerritCodeReview/gitiles/blob/master/Documentation/markdown.md#access-controls [22:48:22] no_justification i wonder how do i start this gitiles thingy [22:48:33] The daemon? I guess you build it and you end up with a gitiles.war? [22:49:05] right [22:49:10] bazel build gitiles then :) [22:49:22] Yep, bazel build gitiles [22:49:28] or [22:49:28] Results in bazel-bin/gitiles.war [22:49:29] bazel build //... [22:49:33] yeh [22:50:16] Oh well, we'll figure it out later [22:50:20] [22:52:57] yep gitiles.war is created :) [22:55:26] (03PS1) 10QChris: Allow “Project and Group creators” to import history [extensions/TinyMCE] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/419612 [22:55:28] (03CR) 10QChris: [V: 031 C: 031] Allow “Project and Group creators” to import history [extensions/TinyMCE] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/419612 (owner: 10QChris) [22:57:32] (03CR) 10MarcoAurelio: [C: 04-1] "I'm not sure we want to "Tweeterize" Gerrit. What 'hashtags' are for in Gerrit and which advantages would they bring? I can see this being" [All-Projects] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/416357 (owner: 10Paladox) [22:58:46] (03CR) 10Paladox: "> Patch Set 3: Code-Review-1" [All-Projects] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/416357 (owner: 10Paladox) [22:59:44] no_justification you can merge https://gerrit-review.googlesource.com/c/gitiles/+/166091 if you want [22:59:49] (need to set v+1) [22:59:50] :) [23:02:37] (03CR) 10MarcoAurelio: ""Project and Group creators" no longer exists. It is my understanding that this is now called "Gerrit Managers"." [extensions/TinyMCE] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/419612 (owner: 10QChris) [23:06:02] I don't understand what happened ^^ [23:06:17] Hauskatze what do you mean? [23:06:36] refs/meta/config do not reflect the change [23:06:44] and was uploaded directly merged?! [23:07:00] the project name do not even exist no more [23:07:03] Hauskatze yes he has been doing that since we started using gerrit [23:07:14] also, some groups are hidden [23:07:23] and cannot be accessed even by an admin [23:08:21] But didn't no_justification renamed "Project and Group creators" to "Gerrit managers" ? [23:08:47] "and cannot be accessed even by an admin" - if that's true, that'd be utterly ridiculous [23:09:14] nope [23:09:18] Hauskatze he renamed this one [23:09:23] https://gerrit.wikimedia.org/r/plugins/gitiles/All-Projects/+/10107cf5056eb6ef3903f49d59cd27387831c5b5 [23:10:33] and before https://gerrit.wikimedia.org/r/plugins/gitiles/All-Projects/+/b3ea82221b5d46b1411abbac5e2ef5b444eaef40 [23:10:56] Project and Group Creators -> Project and Group Managers -> Gerrit Managers [23:12:21] [ which can't manage much as to handle requests to create new repositories you need +admin in order to edit the newly created project later ] [23:13:38] (03PS1) 10MarcoAurelio: Revert "Allow “Project and Group creators” to import history" [extensions/TinyMCE] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/419621 [23:13:57] (03PS2) 10MarcoAurelio: Revert "Allow “Project and Group creators” to import history" [extensions/TinyMCE] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/419621 [23:14:36] (03Abandoned) 10MarcoAurelio: Revert "Allow “Project and Group creators” to import history" [extensions/TinyMCE] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/419621 (owner: 10MarcoAurelio) [23:15:21] paladox: it looks he reverted that later https://gerrit.wikimedia.org/r/#/c/419621/1..2//COMMIT_MSG [23:15:28] but that commit is not anywhere [23:15:42] or I am not correctly looking for it [23:16:39] oh [23:16:41] 10Phabricator, 10Release-Engineering-Team (Kanban), 10Phlogiston: Phlogiston reports don't have new data since mid-February - https://phabricator.wikimedia.org/T188149#4052000 (10JAufrecht) **Steps to Reproduce:** 1) Download a current dump from https://docs.google.com/spreadsheets/d/1yVx3Fmmxccu2noU18IBztcK... [23:17:20] https://gerrit.wikimedia.org/r/plugins/gitiles/mediawiki/extensions/TinyMCE/+log/refs/meta/config [23:17:33] sigh, browsing that branch is always a pain :) [23:21:58] (03CR) 10MarcoAurelio: Grant Edit hashtag right to everyone. [All-Projects] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/416357 (owner: 10Paladox) [23:22:40] (03PS4) 10Paladox: Grant Edit hashtag right to everyone. [All-Projects] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/416357 (https://phabricator.wikimedia.org/T37534) [23:27:15] PROBLEM - Free space - all mounts on deployment-maps01 is CRITICAL: CRITICAL: deployment-prep.deployment-maps01.diskspace.root.byte_percentfree (<22.22%) [23:36:17] PROBLEM - Puppet errors on deployment-maps01 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [23:42:14] RECOVERY - Free space - all mounts on deployment-maps01 is OK: OK: All targets OK