[00:09:42] Project beta-scap-eqiad build #226100: 04STILL FAILING in 12 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/226100/ [00:09:42] Project beta-scap-eqiad build #226100: 04STILL FAILING in 12 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/226100/ [00:20:06] maintenance-disconnect-full-disks build 15405 integration-slave-jessie-1002: OFFLINE due to disk space [00:20:06] maintenance-disconnect-full-disks build 15405 integration-slave-jessie-1002: OFFLINE due to disk space [00:23:31] Project beta-scap-eqiad build #226101: 04STILL FAILING in 13 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/226101/ [00:23:31] Project beta-scap-eqiad build #226101: 04STILL FAILING in 13 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/226101/ [00:37:30] Project beta-scap-eqiad build #226102: 04STILL FAILING in 13 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/226102/ [00:37:30] Project beta-scap-eqiad build #226102: 04STILL FAILING in 13 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/226102/ [00:40:41] (03CR) 10Legoktm: [C: 032] Add Nathan (me) to CI Whitelist [integration/config] - 10https://gerrit.wikimedia.org/r/469948 (owner: 10Nathan fraignt) [00:40:41] (03CR) 10Legoktm: [C: 032] Add Nathan (me) to CI Whitelist [integration/config] - 10https://gerrit.wikimedia.org/r/469948 (owner: 10Nathan fraignt) [00:42:50] (03Merged) 10jenkins-bot: Add Nathan (me) to CI Whitelist [integration/config] - 10https://gerrit.wikimedia.org/r/469948 (owner: 10Nathan fraignt) [00:42:50] (03Merged) 10jenkins-bot: Add Nathan (me) to CI Whitelist [integration/config] - 10https://gerrit.wikimedia.org/r/469948 (owner: 10Nathan fraignt) [00:44:00] !log deploying https://gerrit.wikimedia.org/r/469948 [00:44:00] !log deploying https://gerrit.wikimedia.org/r/469948 [00:44:02] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [00:44:02] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [00:44:14] * legoktm glares at hashar [00:44:14] ACTION glares at hashar [00:44:43] (03CR) 10Legoktm: "This was left undeployed - I deployed it now." [integration/config] - 10https://gerrit.wikimedia.org/r/470059 (owner: 10Hashar) [00:44:43] (03CR) 10Legoktm: "This was left undeployed - I deployed it now." [integration/config] - 10https://gerrit.wikimedia.org/r/470059 (owner: 10Hashar) [00:44:50] !log also deploying https://gerrit.wikimedia.org/r/470059 [00:44:50] !log also deploying https://gerrit.wikimedia.org/r/470059 [00:44:51] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [00:44:51] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [00:45:06] maintenance-disconnect-full-disks build 15410 integration-slave-jessie-1002: OFFLINE due to disk space [00:45:06] maintenance-disconnect-full-disks build 15410 integration-slave-jessie-1002: OFFLINE due to disk space [00:51:29] Project beta-scap-eqiad build #226103: 04STILL FAILING in 13 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/226103/ [00:51:29] Project beta-scap-eqiad build #226103: 04STILL FAILING in 13 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/226103/ [01:05:21] Project beta-scap-eqiad build #226104: 04STILL FAILING in 13 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/226104/ [01:05:21] Project beta-scap-eqiad build #226104: 04STILL FAILING in 13 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/226104/ [01:10:06] maintenance-disconnect-full-disks build 15415 integration-slave-jessie-1002: OFFLINE due to disk space [01:10:06] maintenance-disconnect-full-disks build 15415 integration-slave-jessie-1002: OFFLINE due to disk space [01:19:01] Project beta-scap-eqiad build #226105: 04STILL FAILING in 12 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/226105/ [01:19:01] Project beta-scap-eqiad build #226105: 04STILL FAILING in 12 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/226105/ [01:33:35] Project beta-scap-eqiad build #226106: 04STILL FAILING in 13 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/226106/ [01:33:35] Project beta-scap-eqiad build #226106: 04STILL FAILING in 13 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/226106/ [01:35:06] maintenance-disconnect-full-disks build 15420 integration-slave-jessie-1002: OFFLINE due to disk space [01:35:06] maintenance-disconnect-full-disks build 15420 integration-slave-jessie-1002: OFFLINE due to disk space [01:47:45] Project beta-scap-eqiad build #226107: 04STILL FAILING in 13 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/226107/ [01:47:45] Project beta-scap-eqiad build #226107: 04STILL FAILING in 13 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/226107/ [02:00:06] maintenance-disconnect-full-disks build 15425 integration-slave-jessie-1002: OFFLINE due to disk space [02:00:06] maintenance-disconnect-full-disks build 15425 integration-slave-jessie-1002: OFFLINE due to disk space [02:01:54] Project beta-scap-eqiad build #226108: 04STILL FAILING in 13 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/226108/ [02:01:54] Project beta-scap-eqiad build #226108: 04STILL FAILING in 13 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/226108/ [02:15:54] Project beta-scap-eqiad build #226109: 04STILL FAILING in 13 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/226109/ [02:15:54] Project beta-scap-eqiad build #226109: 04STILL FAILING in 13 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/226109/ [02:25:07] maintenance-disconnect-full-disks build 15430 integration-slave-jessie-1002: OFFLINE due to disk space [02:25:07] maintenance-disconnect-full-disks build 15430 integration-slave-jessie-1002: OFFLINE due to disk space [02:30:24] Project beta-scap-eqiad build #226110: 04STILL FAILING in 13 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/226110/ [02:30:24] Project beta-scap-eqiad build #226110: 04STILL FAILING in 13 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/226110/ [02:44:38] Project beta-scap-eqiad build #226111: 04STILL FAILING in 13 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/226111/ [02:44:38] Project beta-scap-eqiad build #226111: 04STILL FAILING in 13 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/226111/ [02:50:05] maintenance-disconnect-full-disks build 15435 integration-slave-jessie-1002: OFFLINE due to disk space [02:50:05] maintenance-disconnect-full-disks build 15435 integration-slave-jessie-1002: OFFLINE due to disk space [02:58:40] Project beta-scap-eqiad build #226112: 04STILL FAILING in 13 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/226112/ [02:58:40] Project beta-scap-eqiad build #226112: 04STILL FAILING in 13 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/226112/ [03:12:26] Project beta-scap-eqiad build #226113: 04STILL FAILING in 12 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/226113/ [03:12:26] Project beta-scap-eqiad build #226113: 04STILL FAILING in 12 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/226113/ [03:15:07] maintenance-disconnect-full-disks build 15440 integration-slave-jessie-1002: OFFLINE due to disk space [03:15:07] maintenance-disconnect-full-disks build 15440 integration-slave-jessie-1002: OFFLINE due to disk space [03:26:40] Project beta-scap-eqiad build #226114: 04STILL FAILING in 13 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/226114/ [03:26:41] Project beta-scap-eqiad build #226114: 04STILL FAILING in 13 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/226114/ [03:40:06] maintenance-disconnect-full-disks build 15445 integration-slave-jessie-1002: OFFLINE due to disk space [03:40:06] maintenance-disconnect-full-disks build 15445 integration-slave-jessie-1002: OFFLINE due to disk space [03:40:18] Project beta-scap-eqiad build #226115: 04STILL FAILING in 12 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/226115/ [03:40:18] Project beta-scap-eqiad build #226115: 04STILL FAILING in 12 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/226115/ [03:54:10] Project beta-scap-eqiad build #226116: 04STILL FAILING in 13 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/226116/ [03:54:10] Project beta-scap-eqiad build #226116: 04STILL FAILING in 13 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/226116/ [04:05:06] maintenance-disconnect-full-disks build 15450 integration-slave-jessie-1002: OFFLINE due to disk space [04:05:06] maintenance-disconnect-full-disks build 15450 integration-slave-jessie-1002: OFFLINE due to disk space [04:07:07] !log deployment-deploy01:sudo rm -rf /tmp/scap_l10n_* [04:07:07] !log deployment-deploy01:sudo rm -rf /tmp/scap_l10n_* [04:07:09] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [04:07:09] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [04:12:09] Yippee, build fixed! [04:12:10] Yippee, build fixed! [04:12:10] Project beta-scap-eqiad build #226117: 09FIXED in 17 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/226117/ [04:12:10] Project beta-scap-eqiad build #226117: 09FIXED in 17 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/226117/ [04:30:06] maintenance-disconnect-full-disks build 15455 integration-slave-jessie-1002: OFFLINE due to disk space [04:30:06] maintenance-disconnect-full-disks build 15455 integration-slave-jessie-1002: OFFLINE due to disk space [04:55:06] maintenance-disconnect-full-disks build 15460 integration-slave-jessie-1002: OFFLINE due to disk space [04:55:06] maintenance-disconnect-full-disks build 15460 integration-slave-jessie-1002: OFFLINE due to disk space [05:20:06] maintenance-disconnect-full-disks build 15465 integration-slave-jessie-1002: OFFLINE due to disk space [05:20:06] maintenance-disconnect-full-disks build 15465 integration-slave-jessie-1002: OFFLINE due to disk space [05:45:06] maintenance-disconnect-full-disks build 15470 integration-slave-jessie-1002: OFFLINE due to disk space [05:45:06] maintenance-disconnect-full-disks build 15470 integration-slave-jessie-1002: OFFLINE due to disk space [06:10:06] maintenance-disconnect-full-disks build 15475 integration-slave-jessie-1002: OFFLINE due to disk space [06:10:06] maintenance-disconnect-full-disks build 15475 integration-slave-jessie-1002: OFFLINE due to disk space [06:17:49] (03PS1) 10Legoktm: Configure l10n-bot and l10n-bot-watcher [labs/tools] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/470312 (https://phabricator.wikimedia.org/T208138) [06:17:49] (03PS1) 10Legoktm: Configure l10n-bot and l10n-bot-watcher [labs/tools] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/470312 (https://phabricator.wikimedia.org/T208138) [06:19:02] 10Gerrit, 10Commons-Mass-Description, 10Map-of-monuments, 10Repository-Ownership-Requests, and 3 others: Give access to l10n-bot on several repositories - https://phabricator.wikimedia.org/T208138 (10Legoktm) >>! In T208138#4700956, @hashar wrote: > Similar request has been made for `wikipedia/gadgets` T20... [06:19:02] 10Gerrit, 10Commons-Mass-Description, 10Map-of-monuments, 10Repository-Ownership-Requests, and 3 others: Give access to l10n-bot on several repositories - https://phabricator.wikimedia.org/T208138 (10Legoktm) >>! In T208138#4700956, @hashar wrote: > Similar request has been made for `wikipedia/gadgets` T20... [06:35:06] maintenance-disconnect-full-disks build 15480 integration-slave-jessie-1002: OFFLINE due to disk space [06:35:06] maintenance-disconnect-full-disks build 15480 integration-slave-jessie-1002: OFFLINE due to disk space [07:00:06] maintenance-disconnect-full-disks build 15485 integration-slave-jessie-1002: OFFLINE due to disk space [07:00:06] maintenance-disconnect-full-disks build 15485 integration-slave-jessie-1002: OFFLINE due to disk space [07:25:07] maintenance-disconnect-full-disks build 15490 integration-slave-jessie-1002: OFFLINE due to disk space [07:25:07] maintenance-disconnect-full-disks build 15490 integration-slave-jessie-1002: OFFLINE due to disk space [07:50:07] maintenance-disconnect-full-disks build 15495 integration-slave-jessie-1002: OFFLINE due to disk space [07:50:07] maintenance-disconnect-full-disks build 15495 integration-slave-jessie-1002: OFFLINE due to disk space [08:15:06] maintenance-disconnect-full-disks build 15500 integration-slave-jessie-1002: OFFLINE due to disk space [08:15:06] maintenance-disconnect-full-disks build 15500 integration-slave-jessie-1002: OFFLINE due to disk space [08:33:35] 10Beta-Cluster-Infrastructure, 10DNS, 10Operations, 10Traffic, and 4 others: Ferm's upstream Net::DNS Perl library questionable handling of NOERROR responses without records causing puppet errors when we try to @resolve AAAA in labs - https://phabricator.wikimedia.org/T153468 (10fgiunchedi) [08:33:35] 10Beta-Cluster-Infrastructure, 10DNS, 10Operations, 10Traffic, and 4 others: Ferm's upstream Net::DNS Perl library questionable handling of NOERROR responses without records causing puppet errors when we try to @resolve AAAA in labs - https://phabricator.wikimedia.org/T153468 (10fgiunchedi) [08:36:56] 10Project-Admins: Create group/project for "On-wiki documentation working group" - https://phabricator.wikimedia.org/T201639 (10Aklapper) 05Open>03Resolved a:03Aklapper Ah, thanks! Requested public project #WMF-Documentation-Working-Group has been created: https://phabricator.wikimedia.org/project/view/37... [08:36:56] 10Project-Admins: Create group/project for "On-wiki documentation working group" - https://phabricator.wikimedia.org/T201639 (10Aklapper) 05Open>03Resolved a:03Aklapper Ah, thanks! Requested public project #WMF-Documentation-Working-Group has been created: https://phabricator.wikimedia.org/project/view/37... [08:40:06] maintenance-disconnect-full-disks build 15505 integration-slave-jessie-1002: OFFLINE due to disk space [08:40:06] maintenance-disconnect-full-disks build 15505 integration-slave-jessie-1002: OFFLINE due to disk space [09:04:07] hashar: o/ [09:04:07] hashar: o/ [09:04:10] im having a scap issues [09:04:10] im having a scap issues [09:05:06] maintenance-disconnect-full-disks build 15510 integration-slave-jessie-1002: OFFLINE due to disk space [09:05:06] maintenance-disconnect-full-disks build 15510 integration-slave-jessie-1002: OFFLINE due to disk space [09:10:04] addshore: coming in a few [09:10:04] addshore: coming in a few [09:10:13] ack [09:10:13] ack [09:10:16] * addshore will paste some things here [09:10:16] ACTION will paste some things here [09:10:17] Failure processing (u'/srv/mediawiki/php-1.33.0-wmf.1/cache/l10n', u'l10n_cache-ti.cdb', True) [09:10:17] Failure processing (u'/srv/mediawiki/php-1.33.0-wmf.1/cache/l10n', u'l10n_cache-ti.cdb', True) [09:10:31] full paste at https://phabricator.wikimedia.org/P7729 [09:10:31] full paste at https://phabricator.wikimedia.org/P7729 [09:11:37] that happened during a full scap, as a scap pull failed on mwdebug1002, and i figured a full scap would fix it, [09:11:37] that happened during a full scap, as a scap pull failed on mwdebug1002, and i figured a full scap would fix it, [09:12:05] scap pull on mwdebug1002 now works fine, but I got that error when doing the scap sync, not sure if it is fixed and that is just a left over error or something? [09:12:05] scap pull on mwdebug1002 now works fine, but I got that error when doing the scap sync, not sure if it is fixed and that is just a left over error or something? [09:14:03] i think it has to do with a "scap sync" that we aborted earlier today [09:14:03] i think it has to do with a "scap sync" that we aborted earlier today [09:15:09] addshore: back [09:15:09] addshore: back [09:16:02] AH [09:16:02] AH [09:16:06] ValueError: Unterminated string starting at: line 23529 column 8 (char 2654110) [09:16:06] ValueError: Unterminated string starting at: line 23529 column 8 (char 2654110) [09:16:08] hmm [09:16:08] hmm [09:16:23] sounds like some json is malformed [09:16:23] sounds like some json is malformed [09:18:00] so some json in code / i18n file? or something specific to the localization cache / build etc [09:18:00] so some json in code / i18n file? or something specific to the localization cache / build etc [09:20:50] the file is broken, I have downloaded it and can reproduce locally [09:20:50] the file is broken, I have downloaded it and can reproduce locally [09:21:02] also the .MD5 file is from Oct 25th when the json is from Oct 29th [09:21:02] also the .MD5 file is from Oct 25th when the json is from Oct 29th [09:23:19] hashar: ill let you fix it? :D [09:23:19] hashar: ill let you fix it? :D [09:23:39] so [09:23:39] so [09:23:42] the file is not complete [09:23:42] the file is not complete [09:23:49] it is truncated :-\ [09:23:49] it is truncated :-\ [09:25:33] * hashar creates a task [09:25:33] ACTION creates a task [09:27:12] (03CR) 10Urbanecm: [C: 031] "LGTM" [labs/tools] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/470312 (https://phabricator.wikimedia.org/T208138) (owner: 10Legoktm) [09:27:12] (03CR) 10Urbanecm: [C: 031] "LGTM" [labs/tools] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/470312 (https://phabricator.wikimedia.org/T208138) (owner: 10Legoktm) [09:28:21] 10Deployments, 10Release-Engineering-Team (Kanban), 10MediaWiki-extensions-LocalisationUpdate: l10nupdate aborts due to l10n_cache-ti.cdb.json being truncated - https://phabricator.wikimedia.org/T208196 (10hashar) [09:28:21] 10Deployments, 10Release-Engineering-Team (Kanban), 10MediaWiki-extensions-LocalisationUpdate: l10nupdate aborts due to l10n_cache-ti.cdb.json being truncated - https://phabricator.wikimedia.org/T208196 (10hashar) [09:28:23] addshore: yeah I will dig into it [09:28:23] addshore: yeah I will dig into it [09:28:57] [= [09:28:57] [= [09:29:15] addshore: unrelated, but while you are here, I have managed to craft a job for Wikibase Client and Repo tests. It might do what is expected :] [09:29:15] addshore: unrelated, but while you are here, I have managed to craft a job for Wikibase Client and Repo tests. It might do what is expected :] [09:29:19] the CI change is https://gerrit.wikimedia.org/r/#/c/integration/config/+/470060/ [09:29:20] the CI change is https://gerrit.wikimedia.org/r/#/c/integration/config/+/470060/ [09:29:41] my report is on the task https://phabricator.wikimedia.org/T188717#4698738 with a couple builds I have triggered manually [09:29:41] my report is on the task https://phabricator.wikimedia.org/T188717#4698738 with a couple builds I have triggered manually [09:30:06] so I guess it is now all about verifying the tests being run are the proper ones. [09:30:06] so I guess it is now all about verifying the tests being run are the proper ones. [09:30:06] https://integration.wikimedia.org/ci/job/wikibase-client-docker/3/testReport/ [09:30:06] https://integration.wikimedia.org/ci/job/wikibase-client-docker/3/testReport/ [09:30:06] https://integration.wikimedia.org/ci/job/wikibase-repo-docker/6/testReport/ [09:30:06] maintenance-disconnect-full-disks build 15515 integration-slave-jessie-1002: OFFLINE due to disk space [09:30:06] https://integration.wikimedia.org/ci/job/wikibase-repo-docker/6/testReport/ [09:30:06] maintenance-disconnect-full-disks build 15515 integration-slave-jessie-1002: OFFLINE due to disk space [09:30:07] :] [09:30:07] :] [09:30:19] and if that works, that mean I can then phase out the legacy jobs that are still on Nodepool \o/ [09:30:19] and if that works, that mean I can then phase out the legacy jobs that are still on Nodepool \o/ [09:30:33] ooh, that looks kind of about right [09:30:33] ooh, that looks kind of about right [09:31:01] I have been too lazy on friday night to do a diff between the junit xml files [09:31:01] I have been too lazy on friday night to do a diff between the junit xml files [09:31:22] but someone once wrote a junit differ in integration/junitdiff.git , that might be helpful [09:31:22] but someone once wrote a junit differ in integration/junitdiff.git , that might be helpful [09:44:51] 10Deployments, 10Release-Engineering-Team (Kanban), 10MediaWiki-extensions-LocalisationUpdate: l10nupdate aborts due to l10n_cache-ti.cdb.json being truncated - https://phabricator.wikimedia.org/T208196 (10hashar) In logstash looking for `l10n` and `host: deploy1001` and `_type: scap`: The automatic update... [09:44:52] 10Deployments, 10Release-Engineering-Team (Kanban), 10MediaWiki-extensions-LocalisationUpdate: l10nupdate aborts due to l10n_cache-ti.cdb.json being truncated - https://phabricator.wikimedia.org/T208196 (10hashar) In logstash looking for `l10n` and `host: deploy1001` and `_type: scap`: The automatic update... [09:47:00] 10Deployments, 10Release-Engineering-Team (Kanban), 10MediaWiki-extensions-LocalisationUpdate: l10nupdate aborts due to l10n_cache-ti.cdb.json being truncated - https://phabricator.wikimedia.org/T208196 (10Addshore) It was also done earlier today but it was aborted as I told gilles that they only needed to s... [09:47:00] 10Deployments, 10Release-Engineering-Team (Kanban), 10MediaWiki-extensions-LocalisationUpdate: l10nupdate aborts due to l10n_cache-ti.cdb.json being truncated - https://phabricator.wikimedia.org/T208196 (10Addshore) It was also done earlier today but it was aborted as I told gilles that they only needed to s... [09:50:21] grr [09:50:21] grr [09:50:32] and /var/log/l10nupdatelog is no more generated :\ [09:50:32] and /var/log/l10nupdatelog is no more generated :\ [09:55:06] maintenance-disconnect-full-disks build 15520 integration-slave-jessie-1002: OFFLINE due to disk space [09:55:06] maintenance-disconnect-full-disks build 15520 integration-slave-jessie-1002: OFFLINE due to disk space [10:00:40] 10Deployments, 10Release-Engineering-Team (Kanban), 10MediaWiki-extensions-LocalisationUpdate: l10nupdate aborts due to l10n_cache-ti.cdb.json being truncated - https://phabricator.wikimedia.org/T208196 (10hashar) `/var/log/l10nupdate` is empty. The automatic cronjob has been disabled on Sept 27th by 9482dfc... [10:00:40] 10Deployments, 10Release-Engineering-Team (Kanban), 10MediaWiki-extensions-LocalisationUpdate: l10nupdate aborts due to l10n_cache-ti.cdb.json being truncated - https://phabricator.wikimedia.org/T208196 (10hashar) `/var/log/l10nupdate` is empty. The automatic cronjob has been disabled on Sept 27th by 9482dfc... [10:04:21] !log cleaned disk on integration-slave-jessie-1002 [10:04:21] !log cleaned disk on integration-slave-jessie-1002 [10:04:23] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [10:04:23] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [10:09:05] addshore: so I hve no idea how to trigger the l10nupdate :-\ [10:09:05] addshore: so I hve no idea how to trigger the l10nupdate :-\ [10:10:45] neither [10:10:45] neither [10:10:55] I could try another full scap at some point if I find a slot in the day [10:10:55] I could try another full scap at some point if I find a slot in the day [10:11:07] anyway, sync-file still works fine [10:11:07] anyway, sync-file still works fine [10:11:20] yeah [10:11:20] yeah [10:11:24] I have moved the ti files for now [10:11:24] I have moved the ti files for now [10:12:02] it looks like only 1 thing is scheduled for thw eu swat today [10:12:02] it looks like only 1 thing is scheduled for thw eu swat today [10:12:06] so I could do it then [10:12:06] so I could do it then [10:12:14] * addshore is just about to prepare another bunch of config changes [10:12:15] ACTION is just about to prepare another bunch of config changes [10:12:44] I think l10nupdate is run as part of a full scap [10:12:44] I think l10nupdate is run as part of a full scap [10:14:28] yup [10:14:28] yup [10:14:33] but i think that is the onyl way to run it [10:14:33] but i think that is the onyl way to run it [10:15:23] oh fuck [10:15:23] oh fuck [10:15:37] scap no more have a command to run the l10nupdate [10:15:37] scap no more have a command to run the l10nupdate [10:23:02] 10Deployments, 10Release-Engineering-Team (Kanban), 10MediaWiki-extensions-LocalisationUpdate: l10nupdate aborts due to l10n_cache-ti.cdb.json being truncated - https://phabricator.wikimedia.org/T208196 (10hashar) /srv/mediawiki-staging/php-1.33.0-wmf.1/cache/l10n/upstream/l10n_cache-ti.cdb.json is corrupted... [10:23:02] 10Deployments, 10Release-Engineering-Team (Kanban), 10MediaWiki-extensions-LocalisationUpdate: l10nupdate aborts due to l10n_cache-ti.cdb.json being truncated - https://phabricator.wikimedia.org/T208196 (10hashar) /srv/mediawiki-staging/php-1.33.0-wmf.1/cache/l10n/upstream/l10n_cache-ti.cdb.json is corrupted... [10:23:18] addshore: i give up for l10nupdate. I have no idea how the system works, it has been changed recently [10:23:18] addshore: i give up for l10nupdate. I have no idea how the system works, it has been changed recently [10:23:27] ack [10:23:27] ack [10:23:32] think another full scap should fix it? [10:23:32] think another full scap should fix it? [10:23:49] maybe [10:23:49] maybe [10:24:18] I will poke thcipriani when he is around [10:24:19] I will poke thcipriani when he is around [10:24:25] ac [10:24:25] ac [10:24:26] ack [10:24:26] ack [10:24:37] I 'd rather not break everything :] [10:24:37] I 'd rather not break everything :] [12:37:25] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban): Alert in -releng when permanent hosts have low disk space - https://phabricator.wikimedia.org/T193661 (10hashar) 05Open>03Resolved a:03thcipriani Done as part of T201224 . We now automatically depool the faulty slave... [12:37:25] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban): Alert in -releng when permanent hosts have low disk space - https://phabricator.wikimedia.org/T193661 (10hashar) 05Open>03Resolved a:03thcipriani Done as part of T201224 . We now automatically depool the faulty slave... [12:48:50] 10Continuous-Integration-Config, 10Patch-For-Review: Phase out jobs "pplint-HEAD" and "erblint-HEAD" - https://phabricator.wikimedia.org/T154894 (10hashar) [12:48:50] 10Continuous-Integration-Config, 10Patch-For-Review: Phase out jobs "pplint-HEAD" and "erblint-HEAD" - https://phabricator.wikimedia.org/T154894 (10hashar) [12:48:57] 10Continuous-Integration-Config, 10Release-Engineering-Team (Kanban), 10Operations, 10Patch-For-Review, 10Puppet: Get rid of "import realm.pp" in manifests/site.pp - https://phabricator.wikimedia.org/T154915 (10hashar) 05stalled>03Open The reason for this task is to have PuppetSyntax to whine on a de... [12:48:57] 10Continuous-Integration-Config, 10Release-Engineering-Team (Kanban), 10Operations, 10Patch-For-Review, 10Puppet: Get rid of "import realm.pp" in manifests/site.pp - https://phabricator.wikimedia.org/T154915 (10hashar) 05stalled>03Open The reason for this task is to have PuppetSyntax to whine on a de... [13:06:20] 10Beta-Cluster-Infrastructure, 10cloud-services-team: deployment-prep: instances with puppet issues - https://phabricator.wikimedia.org/T208040 (10aborrero) [13:06:20] 10Beta-Cluster-Infrastructure, 10cloud-services-team: deployment-prep: instances with puppet issues - https://phabricator.wikimedia.org/T208040 (10aborrero) [13:10:06] 10Release-Engineering-Team (Kanban), 10Release Pipeline, 10User-zeljkofilipin: Develop set of metrics to assess incident reports/post mortems - https://phabricator.wikimedia.org/T206622 (10zeljkofilipin) [13:10:06] 10Release-Engineering-Team (Kanban), 10Release Pipeline, 10User-zeljkofilipin: Develop set of metrics to assess incident reports/post mortems - https://phabricator.wikimedia.org/T206622 (10zeljkofilipin) [13:35:16] (03CR) 10Hashar: [C: 032] Run extension-quibble for AnonPrivacy [integration/config] - 10https://gerrit.wikimedia.org/r/469959 (owner: 10Umherirrender) [13:35:16] (03CR) 10Hashar: [C: 032] Run extension-quibble for AnonPrivacy [integration/config] - 10https://gerrit.wikimedia.org/r/469959 (owner: 10Umherirrender) [13:35:18] (03CR) 10Hashar: [C: 032] Run extension-quibble for CountDownClock [integration/config] - 10https://gerrit.wikimedia.org/r/469957 (owner: 10Umherirrender) [13:35:18] (03CR) 10Hashar: [C: 032] Run extension-quibble for CountDownClock [integration/config] - 10https://gerrit.wikimedia.org/r/469957 (owner: 10Umherirrender) [13:36:27] (03CR) 10Hashar: [V: 032 C: 032] Configure l10n-bot and l10n-bot-watcher [labs/tools] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/470312 (https://phabricator.wikimedia.org/T208138) (owner: 10Legoktm) [13:36:27] (03CR) 10Hashar: [V: 032 C: 032] Configure l10n-bot and l10n-bot-watcher [labs/tools] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/470312 (https://phabricator.wikimedia.org/T208138) (owner: 10Legoktm) [13:36:51] 10Gerrit, 10Commons-Mass-Description, 10Map-of-monuments, 10Repository-Ownership-Requests, and 3 others: Give access to l10n-bot on several repositories - https://phabricator.wikimedia.org/T208138 (10hashar) Thanks @Legoktm :) [13:36:51] 10Gerrit, 10Commons-Mass-Description, 10Map-of-monuments, 10Repository-Ownership-Requests, and 3 others: Give access to l10n-bot on several repositories - https://phabricator.wikimedia.org/T208138 (10hashar) Thanks @Legoktm :) [13:37:20] (03Merged) 10jenkins-bot: Run extension-quibble for AnonPrivacy [integration/config] - 10https://gerrit.wikimedia.org/r/469959 (owner: 10Umherirrender) [13:37:20] (03Merged) 10jenkins-bot: Run extension-quibble for AnonPrivacy [integration/config] - 10https://gerrit.wikimedia.org/r/469959 (owner: 10Umherirrender) [13:37:22] (03Merged) 10jenkins-bot: Run extension-quibble for CountDownClock [integration/config] - 10https://gerrit.wikimedia.org/r/469957 (owner: 10Umherirrender) [13:37:22] (03Merged) 10jenkins-bot: Run extension-quibble for CountDownClock [integration/config] - 10https://gerrit.wikimedia.org/r/469957 (owner: 10Umherirrender) [13:51:26] 10Gerrit, 10Commons-Mass-Description, 10Map-of-monuments, 10Repository-Ownership-Requests, and 3 others: Give access to l10n-bot on several repositories - https://phabricator.wikimedia.org/T208138 (10Urbanecm) 05Open>03Resolved a:03Legoktm Looks like resolved to me. [13:51:26] 10Gerrit, 10Commons-Mass-Description, 10Map-of-monuments, 10Repository-Ownership-Requests, and 3 others: Give access to l10n-bot on several repositories - https://phabricator.wikimedia.org/T208138 (10Urbanecm) 05Open>03Resolved a:03Legoktm Looks like resolved to me. [14:30:37] 10Release-Engineering-Team (Kanban), 10Code-Health-Metrics: Code health metrics spike - https://phabricator.wikimedia.org/T207046 (10zeljkofilipin) a:03zeljkofilipin Next steps: - talk with @hashar about installing the SonarQube Scanner for Jenkins [14:30:37] 10Release-Engineering-Team (Kanban), 10Code-Health-Metrics: Code health metrics spike - https://phabricator.wikimedia.org/T207046 (10zeljkofilipin) a:03zeljkofilipin Next steps: - talk with @hashar about installing the SonarQube Scanner for Jenkins [14:31:32] 10Release-Engineering-Team (Kanban), 10Code-Health-Metrics, 10User-zeljkofilipin: Code health metrics spike - https://phabricator.wikimedia.org/T207046 (10zeljkofilipin) [14:31:32] 10Release-Engineering-Team (Kanban), 10Code-Health-Metrics, 10User-zeljkofilipin: Code health metrics spike - https://phabricator.wikimedia.org/T207046 (10zeljkofilipin) [14:33:15] Yippee, build fixed! [14:33:15] Yippee, build fixed! [14:33:15] Project beta-scap-eqiad build #226156: 09FIXED in 15 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/226156/ [14:33:15] Project beta-scap-eqiad build #226156: 09FIXED in 15 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/226156/ [14:52:33] 10Release-Engineering-Team (Kanban), 10Code-Health-Metrics, 10User-zeljkofilipin: Code health metrics spike - https://phabricator.wikimedia.org/T207046 (10hashar) >>! In T207046#4702909, @zeljkofilipin wrote: > - talk with @hashar about installing the SonarQube Scanner for Jenkins The plugin is merely a Jen... [14:52:33] 10Release-Engineering-Team (Kanban), 10Code-Health-Metrics, 10User-zeljkofilipin: Code health metrics spike - https://phabricator.wikimedia.org/T207046 (10hashar) >>! In T207046#4702909, @zeljkofilipin wrote: > - talk with @hashar about installing the SonarQube Scanner for Jenkins The plugin is merely a Jen... [14:55:41] 10Continuous-Integration-Config, 10Patch-For-Review: AssertionError: /src/.git/config has tabs - https://phabricator.wikimedia.org/T208063 (10hashar) 05Open>03Resolved a:03hashar [14:55:41] 10Continuous-Integration-Config, 10Patch-For-Review: AssertionError: /src/.git/config has tabs - https://phabricator.wikimedia.org/T208063 (10hashar) 05Open>03Resolved a:03hashar [15:10:01] (03PS2) 10Hashar: Wikibase client/repo tests with Quibble [integration/config] - 10https://gerrit.wikimedia.org/r/470060 (https://phabricator.wikimedia.org/T188717) [15:10:01] (03PS2) 10Hashar: Wikibase client/repo tests with Quibble [integration/config] - 10https://gerrit.wikimedia.org/r/470060 (https://phabricator.wikimedia.org/T188717) [15:10:22] (03CR) 10Hashar: [C: 032] "Good enough for now, lets deploy it so we can "check experimental" as needed." [integration/config] - 10https://gerrit.wikimedia.org/r/470060 (https://phabricator.wikimedia.org/T188717) (owner: 10Hashar) [15:10:22] (03CR) 10Hashar: [C: 032] "Good enough for now, lets deploy it so we can "check experimental" as needed." [integration/config] - 10https://gerrit.wikimedia.org/r/470060 (https://phabricator.wikimedia.org/T188717) (owner: 10Hashar) [15:14:51] (03Merged) 10jenkins-bot: Wikibase client/repo tests with Quibble [integration/config] - 10https://gerrit.wikimedia.org/r/470060 (https://phabricator.wikimedia.org/T188717) (owner: 10Hashar) [15:14:51] (03Merged) 10jenkins-bot: Wikibase client/repo tests with Quibble [integration/config] - 10https://gerrit.wikimedia.org/r/470060 (https://phabricator.wikimedia.org/T188717) (owner: 10Hashar) [15:27:19] 10MediaWiki-Codesniffer, 10PHP 7.3 support: MediaWiki codesniffer fails on php7.3: PHP Fatal error: Uncaught PHP_CodeSniffer\Exceptions\RuntimeException: "continue" targeting switch is equivalent to "break". - https://phabricator.wikimedia.org/T208234 (10hashar) [15:27:19] 10MediaWiki-Codesniffer, 10PHP 7.3 support: MediaWiki codesniffer fails on php7.3: PHP Fatal error: Uncaught PHP_CodeSniffer\Exceptions\RuntimeException: "continue" targeting switch is equivalent to "break". - https://phabricator.wikimedia.org/T208234 (10hashar) [15:33:47] thcipriani: good morning :] Latest madness from deployment/scap: some l10n json file got corrupted and I could not find out how to retrieve the latest one :\ [15:33:47] thcipriani: good morning :] Latest madness from deployment/scap: some l10n json file got corrupted and I could not find out how to retrieve the latest one :\ [15:33:56] https://phabricator.wikimedia.org/T208196 is da task [15:33:56] https://phabricator.wikimedia.org/T208196 is da task [15:34:08] somehow /srv/mediawiki-staging/php-1.33.0-wmf.1/cache/l10n/upstream/l10n_cache-ti.cdb.json is truncated :\ [15:34:09] somehow /srv/mediawiki-staging/php-1.33.0-wmf.1/cache/l10n/upstream/l10n_cache-ti.cdb.json is truncated :\ [15:35:56] hrm [15:35:56] hrm [15:36:21] ah, so we bailed on scap while that file was being generated [15:36:21] ah, so we bailed on scap while that file was being generated [15:40:09] thcipriani: yus [15:40:09] thcipriani: yus [15:40:25] * addshore assumed scap would be fine and handel it well on the next scap ryun [15:40:26] ACTION assumed scap would be fine and handel it well on the next scap ryun [15:40:39] but you know what they say about assumptions ;) [15:40:39] but you know what they say about assumptions ;) [15:40:48] I think the same thing [15:40:48] I think the same thing [15:46:25] * addshore tries to remember how the -computed db lists work [15:46:25] ACTION tries to remember how the -computed db lists work [15:46:26] .... [15:46:26] .... [15:46:40] in IS.php do I refer to them with their -computed suffix or not? [15:46:40] in IS.php do I refer to them with their -computed suffix or not? [15:50:07] (03PS1) 10Giuseppe Lavagetto: Upgrade operations-puppet CI image to use stretch [integration/config] - 10https://gerrit.wikimedia.org/r/470419 [15:50:07] (03PS1) 10Giuseppe Lavagetto: Upgrade operations-puppet CI image to use stretch [integration/config] - 10https://gerrit.wikimedia.org/r/470419 [15:55:35] 10Deployments, 10Release-Engineering-Team (Kanban), 10MediaWiki-extensions-LocalisationUpdate: l10nupdate aborts due to l10n_cache-ti.cdb.json being truncated - https://phabricator.wikimedia.org/T208196 (10thcipriani) >>! In T208196#4702086, @Addshore wrote: > ``` > 08:35 gilles@deploy1001: sync aborted: T20... [15:55:35] 10Deployments, 10Release-Engineering-Team (Kanban), 10MediaWiki-extensions-LocalisationUpdate: l10nupdate aborts due to l10n_cache-ti.cdb.json being truncated - https://phabricator.wikimedia.org/T208196 (10thcipriani) >>! In T208196#4702086, @Addshore wrote: > ``` > 08:35 gilles@deploy1001: sync aborted: T20... [15:56:42] 10Deployments, 10Release-Engineering-Team (Kanban), 10MediaWiki-extensions-LocalisationUpdate: l10nupdate aborts due to l10n_cache-ti.cdb.json being truncated - https://phabricator.wikimedia.org/T208196 (10Addshore) >>! In T208196#4703264, @thcipriani wrote: > After {D983} we no longer have the code to just... [15:56:42] 10Deployments, 10Release-Engineering-Team (Kanban), 10MediaWiki-extensions-LocalisationUpdate: l10nupdate aborts due to l10n_cache-ti.cdb.json being truncated - https://phabricator.wikimedia.org/T208196 (10Addshore) >>! In T208196#4703264, @thcipriani wrote: > After {D983} we no longer have the code to just... [16:00:14] 10Deployments, 10Release-Engineering-Team (Kanban), 10MediaWiki-extensions-LocalisationUpdate: l10nupdate aborts due to l10n_cache-ti.cdb.json being truncated - https://phabricator.wikimedia.org/T208196 (10thcipriani) >>! In T208196#4703273, @Addshore wrote: >>>! In T208196#4703264, @thcipriani wrote: >> Aft... [16:00:15] 10Deployments, 10Release-Engineering-Team (Kanban), 10MediaWiki-extensions-LocalisationUpdate: l10nupdate aborts due to l10n_cache-ti.cdb.json being truncated - https://phabricator.wikimedia.org/T208196 (10thcipriani) >>! In T208196#4703273, @Addshore wrote: >>>! In T208196#4703264, @thcipriani wrote: >> Aft... [16:05:21] 10Release-Engineering-Team (Kanban), 10Code-Health-Metrics, 10User-zeljkofilipin: Code health metrics spike - https://phabricator.wikimedia.org/T207046 (10zeljkofilipin) p:05Triage>03Normal [16:05:21] 10Release-Engineering-Team (Kanban), 10Code-Health-Metrics, 10User-zeljkofilipin: Code health metrics spike - https://phabricator.wikimedia.org/T207046 (10zeljkofilipin) p:05Triage>03Normal [16:13:46] 10Continuous-Integration-Config, 10Advanced-Search, 10TCB-Team, 10Wikibase-Quality-Constraints, 10Wikidata: Update grunt to 1.0.3 for AdvancedSearch and WikibaseQualityConstraints - https://phabricator.wikimedia.org/T207988 (10gabriel-wmde) Thanks for bringing this up! We decided to postpone the fix for... [16:13:46] 10Continuous-Integration-Config, 10Advanced-Search, 10TCB-Team, 10Wikibase-Quality-Constraints, 10Wikidata: Update grunt to 1.0.3 for AdvancedSearch and WikibaseQualityConstraints - https://phabricator.wikimedia.org/T207988 (10gabriel-wmde) Thanks for bringing this up! We decided to postpone the fix for... [16:22:01] Project beta-scap-eqiad build #226163: 04FAILURE in 14 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/226163/ [16:22:01] Project beta-scap-eqiad build #226163: 04FAILURE in 14 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/226163/ [16:22:10] Project beta-update-databases-eqiad build #29389: 04FAILURE in 2 min 9 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/29389/ [16:22:10] Project beta-update-databases-eqiad build #29389: 04FAILURE in 2 min 9 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/29389/ [16:22:54] Project beta-scap-eqiad build #226164: 04STILL FAILING in 1.3 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/226164/ [16:22:54] Project beta-scap-eqiad build #226164: 04STILL FAILING in 1.3 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/226164/ [16:38:42] Yippee, build fixed! [16:38:42] Yippee, build fixed! [16:38:43] Project beta-scap-eqiad build #226165: 09FIXED in 14 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/226165/ [16:38:43] Project beta-scap-eqiad build #226165: 09FIXED in 14 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/226165/ [16:41:52] :> [16:41:52] :> [16:50:49] 10Project-Admins, 10Wikimedia-Technical-Conference-2018: Create "MediaWiki-Decoupling" phabricator project - https://phabricator.wikimedia.org/T207976 (10bd808) a:05bd808>03None I'm un-licking this cookie but will be more than happy to be the person to create the Phab objects once name & scope are decided. [16:50:49] 10Project-Admins, 10Wikimedia-Technical-Conference-2018: Create "MediaWiki-Decoupling" phabricator project - https://phabricator.wikimedia.org/T207976 (10bd808) a:05bd808>03None I'm un-licking this cookie but will be more than happy to be the person to create the Phab objects once name & scope are decided. [16:54:17] Project beta-scap-eqiad build #226166: 04FAILURE in 14 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/226166/ [16:54:17] Project beta-scap-eqiad build #226166: 04FAILURE in 14 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/226166/ [16:56:32] 10Release-Engineering-Team (Kanban), 10Code-Health-Metrics, 10User-zeljkofilipin: Code health metrics spike - https://phabricator.wikimedia.org/T207046 (10zeljkofilipin) @hashar as far as I understood it, the client runs the analysis locally, and the server stores the data and creates reports. We can use [[... [16:56:32] 10Release-Engineering-Team (Kanban), 10Code-Health-Metrics, 10User-zeljkofilipin: Code health metrics spike - https://phabricator.wikimedia.org/T207046 (10zeljkofilipin) @hashar as far as I understood it, the client runs the analysis locally, and the server stores the data and creates reports. We can use [[... [17:00:55] Project beta-scap-eqiad build #226167: 15ABORTED in 5 min 42 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/226167/ [17:00:55] Project beta-scap-eqiad build #226167: 15ABORTED in 5 min 42 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/226167/ [17:01:01] ^ me [17:01:01] ^ me [17:16:14] 10Release-Engineering-Team, 10User-zeljkofilipin, 10Wikimedia-Incident, 10Wikimedia-production-error: Deployments of MediaWiki with scap cause a spam of "web request took longer than 60 seconds and timed out" - https://phabricator.wikimedia.org/T204871 (10Addshore) [17:16:14] 10Release-Engineering-Team, 10User-zeljkofilipin, 10Wikimedia-Incident, 10Wikimedia-production-error: Deployments of MediaWiki with scap cause a spam of "web request took longer than 60 seconds and timed out" - https://phabricator.wikimedia.org/T204871 (10Addshore) [17:19:58] Hello! I need some help with `git fat pull` command. I'm currently doing WDQS deployment, and I need to pull large binaries with `git fat pull`. When I try to, I get `OSError: [Errno 1] Operation not permitted` error. So the command should pull a binary and replace the current one, but it throws that error instead. The file permission of the current file is set to 665 and I belong in the group as the current [17:19:58] Hello! I need some help with `git fat pull` command. I'm currently doing WDQS deployment, and I need to pull large binaries with `git fat pull`. When I try to, I get `OSError: [Errno 1] Operation not permitted` error. So the command should pull a binary and replace the current one, but it throws that error instead. The file permission of the current file is set to 665 and I belong in the group as the current [17:19:58] file. [17:19:58] file. [17:21:10] https://www.irccloud.com/pastebin/q1ayOJXh/ [17:21:10] https://www.irccloud.com/pastebin/q1ayOJXh/ [17:22:54] Yippee, build fixed! [17:22:54] Yippee, build fixed! [17:22:55] Project beta-update-databases-eqiad build #29390: 09FIXED in 2 min 54 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/29390/ [17:22:55] Project beta-update-databases-eqiad build #29390: 09FIXED in 2 min 54 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/29390/ [17:24:55] thcipriani: any idea about that git-fat permission issues ^^ ? [17:24:55] thcipriani: any idea about that git-fat permission issues ^^ ? [17:26:08] * thcipriani looks [17:26:08] ACTION looks [17:27:02] strangely, that file has 664 permissions [17:27:02] strangely, that file has 664 permissions [17:27:45] so the file exists at ls -lhA /srv/deployment/wdqs/wdqs/.git/fat/objects/5049936153947ba426ce0b3dc0d9939d6c09adae [17:27:45] so the file exists at ls -lhA /srv/deployment/wdqs/wdqs/.git/fat/objects/5049936153947ba426ce0b3dc0d9939d6c09adae [17:28:04] yep, and that one also seems to have reasonable permissions [17:28:04] yep, and that one also seems to have reasonable permissions [17:28:42] so it's trying to change the update time on that file and failing...should be like touching that file, basically [17:28:43] so it's trying to change the update time on that file and failing...should be like touching that file, basically [17:29:22] which I can do... [17:29:22] which I can do... [17:30:03] onimisionipe: was trying to do that wdqs deployment and failed on `git fat update`, it failed for me as well. [17:30:03] onimisionipe: was trying to do that wdqs deployment and failed on `git fat update`, it failed for me as well. [17:30:40] i think I broke my record for number of scaps / syncs today >.> [17:30:40] i think I broke my record for number of scaps / syncs today >.> [17:33:00] gehel: hrm, well I can recreate it [17:33:00] gehel: hrm, well I can recreate it [17:33:26] https://phabricator.wikimedia.org/P7734 [17:33:26] https://phabricator.wikimedia.org/P7734 [17:34:12] thcipriani: it does not make sense :( [17:34:12] thcipriani: it does not make sense :( [17:35:35] https://phabricator.wikimedia.org/P7734#45361 [17:35:35] https://phabricator.wikimedia.org/P7734#45361 [17:35:58] so weird. So I can update the mtime, but not to any specific mtime [17:35:58] so weird. So I can update the mtime, but not to any specific mtime [17:37:07] there is some black magic at work here [17:37:07] there is some black magic at work here [17:39:19] so git-fat updates the mtime to mtime + 1 to ensure mtime has changed so that the subsequent call to git checkout-index will actually update the file [17:39:20] so git-fat updates the mtime to mtime + 1 to ensure mtime has changed so that the subsequent call to git checkout-index will actually update the file [17:39:32] otherwise git uses its file cache [17:39:32] otherwise git uses its file cache [17:39:41] or //might use// it's file cache [17:39:41] or //might use// it's file cache [17:39:48] is this on deployment-* thcipriani ? [17:39:49] is this on deployment-* thcipriani ? [17:39:59] nope, on deploy1001 [17:39:59] nope, on deploy1001 [17:40:01] paladox: production [17:40:01] paladox: production [17:40:04] deploy1001 [17:40:04] deploy1001 [17:40:05] ah ok nvm :) [17:40:05] ah ok nvm :) [17:40:24] * gehel needs to be off in a few for dinner [17:40:24] ACTION needs to be off in a few for dinner [17:40:33] onimisionipe: can you follow up with thcipriani ? [17:40:33] onimisionipe: can you follow up with thcipriani ? [17:42:47] sure [17:42:47] sure [17:44:32] hrm [17:44:32] hrm [17:46:28] Yep..strange [17:46:28] Yep..strange [17:50:12] > Changing timestamps is permitted when: either the process has appropriate privileges, or the effective user ID equals the user ID of the file, or times is NULL and the process has write permission for the file. [17:50:12] > Changing timestamps is permitted when: either the process has appropriate privileges, or the effective user ID equals the user ID of the file, or times is NULL and the process has write permission for the file. [17:50:42] touch -mt $(date +%y%m%d%H%M) blazegraph-service-0.3.1-SNAPSHOT.war gives me permission denied [17:50:42] touch -mt $(date +%y%m%d%H%M) blazegraph-service-0.3.1-SNAPSHOT.war gives me permission denied [17:51:31] I guess because: I'm not the file owner and the times are not null [17:51:31] I guess because: I'm not the file owner and the times are not null [17:51:51] I guess same for me as well? [17:51:51] I guess same for me as well? [17:51:52] if I copy that file, I can set the timestamps, even on the same filesystem [17:51:52] if I copy that file, I can set the timestamps, even on the same filesystem [17:52:11] try: rm-ing the file, git checkout the file, then git fat pull [17:52:11] try: rm-ing the file, git checkout the file, then git fat pull [17:53:02] seems weird if you can write the file, you can change mtime of the file as it is, but you can't forge mtime on the file, I guess. [17:53:02] seems weird if you can write the file, you can change mtime of the file as it is, but you can't forge mtime on the file, I guess. [17:53:13] 10MediaWiki-Codesniffer, 10PHP 7.3 support: MediaWiki codesniffer fails on php7.3: PHP Fatal error: Uncaught PHP_CodeSniffer\Exceptions\RuntimeException: "continue" targeting switch is equivalent to "break". - https://phabricator.wikimedia.org/T208234 (10Legoktm) 05Open>03Invalid v15.0.0 is incredibly old... [17:53:13] 10MediaWiki-Codesniffer, 10PHP 7.3 support: MediaWiki codesniffer fails on php7.3: PHP Fatal error: Uncaught PHP_CodeSniffer\Exceptions\RuntimeException: "continue" targeting switch is equivalent to "break". - https://phabricator.wikimedia.org/T208234 (10Legoktm) 05Open>03Invalid v15.0.0 is incredibly old... [17:53:42] Ok [17:53:42] Ok [17:54:21] trying now [17:54:21] trying now [17:58:36] thcipriani: so I moved them somewhere else, then moved them back, and git fat pull runs well now [17:58:36] thcipriani: so I moved them somewhere else, then moved them back, and git fat pull runs well now [17:59:03] by them I mean there's another file affected by the same issue [17:59:04] by them I mean there's another file affected by the same issue [18:00:30] hrm, we don't run into this often since, I suppose, there is one deploy user that does all the deployment out to targets and mostly the same folks prepping repos on the deployment hosts [18:00:30] hrm, we don't run into this often since, I suppose, there is one deploy user that does all the deployment out to targets and mostly the same folks prepping repos on the deployment hosts [18:01:01] it seems strange to me that you have to be the file owner to update mtime. [18:01:01] it seems strange to me that you have to be the file owner to update mtime. [18:01:19] but that's how I read the man page and how things appear to work :\ [18:01:19] but that's how I read the man page and how things appear to work :\ [18:01:25] very starnge [18:01:25] very starnge [18:01:33] *strange [18:01:33] *strange [18:01:55] do you have a second to file a bug about this? [18:01:55] do you have a second to file a bug about this? [18:02:09] sure! [18:02:09] sure! [18:02:12] thank you! [18:02:12] thank you! [18:02:16] I will do that after my deployment [18:02:16] I will do that after my deployment [18:02:32] no Thank you! [18:02:32] no Thank you! [18:06:07] 10MediaWiki-Codesniffer, 10PHP 7.3 support: MediaWiki codesniffer fails on php7.3: PHP Fatal error: Uncaught PHP_CodeSniffer\Exceptions\RuntimeException: "continue" targeting switch is equivalent to "break". - https://phabricator.wikimedia.org/T208234 (10hashar) They use: ``` name=composer.json { "wik... [18:06:07] 10MediaWiki-Codesniffer, 10PHP 7.3 support: MediaWiki codesniffer fails on php7.3: PHP Fatal error: Uncaught PHP_CodeSniffer\Exceptions\RuntimeException: "continue" targeting switch is equivalent to "break". - https://phabricator.wikimedia.org/T208234 (10hashar) They use: ``` name=composer.json { "wik... [18:18:31] (03CR) 10Jforrester: [C: 031] "Yay." [integration/config] - 10https://gerrit.wikimedia.org/r/469954 (owner: 10Umherirrender) [18:18:31] (03CR) 10Jforrester: [C: 031] "Yay." [integration/config] - 10https://gerrit.wikimedia.org/r/469954 (owner: 10Umherirrender) [18:28:10] 10Phabricator (Upstream), 10Legalpad, 10Upstream: Legalpad access controls are confusing and seemingly broken - https://phabricator.wikimedia.org/T208254 (10mmodell) [18:28:10] 10Phabricator (Upstream), 10Legalpad, 10Upstream: Legalpad access controls are confusing and seemingly broken - https://phabricator.wikimedia.org/T208254 (10mmodell) [18:58:34] 10Phabricator (Upstream), 10Legalpad, 10Upstream: Legalpad access controls are confusing and seemingly broken - https://phabricator.wikimedia.org/T208254 (10Dzahn) let him try again now. it's possible it was just me doing it wrong in the custom policy all this time. [18:58:34] 10Phabricator (Upstream), 10Legalpad, 10Upstream: Legalpad access controls are confusing and seemingly broken - https://phabricator.wikimedia.org/T208254 (10Dzahn) let him try again now. it's possible it was just me doing it wrong in the custom policy all this time. [19:01:17] PROBLEM - DPKG on contint2001 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [19:01:17] PROBLEM - DPKG on contint2001 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [19:11:15] 10Deployments, 10Release-Engineering-Team (Kanban), 10MediaWiki-extensions-LocalisationUpdate: l10nupdate aborts due to l10n_cache-ti.cdb.json being truncated - https://phabricator.wikimedia.org/T208196 (10thcipriani) 05Open>03Resolved a:03thcipriani So it looks like >>! In T208196#4703273, @Addshore... [19:11:15] 10Deployments, 10Release-Engineering-Team (Kanban), 10MediaWiki-extensions-LocalisationUpdate: l10nupdate aborts due to l10n_cache-ti.cdb.json being truncated - https://phabricator.wikimedia.org/T208196 (10thcipriani) 05Open>03Resolved a:03thcipriani So it looks like >>! In T208196#4703273, @Addshore... [19:18:29] (03PS1) 10Stibba: Add Stibba to the Jenkins whitelist [integration/config] - 10https://gerrit.wikimedia.org/r/470453 [19:18:29] (03PS1) 10Stibba: Add Stibba to the Jenkins whitelist [integration/config] - 10https://gerrit.wikimedia.org/r/470453 [19:24:25] (03CR) 10Urbanecm: [C: 031] Add Stibba to the Jenkins whitelist [integration/config] - 10https://gerrit.wikimedia.org/r/470453 (owner: 10Stibba) [19:24:25] (03CR) 10Urbanecm: [C: 031] Add Stibba to the Jenkins whitelist [integration/config] - 10https://gerrit.wikimedia.org/r/470453 (owner: 10Stibba) [19:33:00] Project beta-code-update-eqiad build #223439: 04FAILURE in 0.13 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/223439/ [19:33:00] Project beta-code-update-eqiad build #223439: 04FAILURE in 0.13 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/223439/ [19:43:40] 10Deployments, 10Operations, 10Release: OSError: [Errno 1] Operation not permitted when running git fat pull - https://phabricator.wikimedia.org/T208259 (10Mathew.onipe) [19:43:40] 10Deployments, 10Operations, 10Release: OSError: [Errno 1] Operation not permitted when running git fat pull - https://phabricator.wikimedia.org/T208259 (10Mathew.onipe) [19:43:56] Yippee, build fixed! [19:43:56] Yippee, build fixed! [19:43:56] Project beta-code-update-eqiad build #223440: 09FIXED in 55 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/223440/ [19:43:57] Project beta-code-update-eqiad build #223440: 09FIXED in 55 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/223440/ [19:49:03] 10Project-Admins: Replace tracking task T119681 by creating #MediaWiki-Edit-Summary - https://phabricator.wikimedia.org/T208156 (10MusikAnimal) :+1: to `#MediaWiki-Commenting`. There are many tools that will need updating, and MediaWiki core issues may arise, so I do see this tag coming in handy. [19:49:03] 10Project-Admins: Replace tracking task T119681 by creating #MediaWiki-Edit-Summary - https://phabricator.wikimedia.org/T208156 (10MusikAnimal) :+1: to `#MediaWiki-Commenting`. There are many tools that will need updating, and MediaWiki core issues may arise, so I do see this tag coming in handy. [19:51:07] why is shinken-wm not speaking in here [19:51:07] why is shinken-wm not speaking in here [19:58:12] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Kanban), 10cloud-services-team: Ensure there are no hard-coded IPs in use for beta - https://phabricator.wikimedia.org/T208262 (10thcipriani) [19:58:12] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Kanban), 10cloud-services-team: Ensure there are no hard-coded IPs in use for beta - https://phabricator.wikimedia.org/T208262 (10thcipriani) [19:58:42] thcipriani: beta-scap-eqiad is disabled… was that you? [19:58:42] thcipriani: beta-scap-eqiad is disabled… was that you? [20:02:49] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Kanban), 10cloud-services-team: Ensure there are no hard-coded IPs in use for beta - https://phabricator.wikimedia.org/T208262 (10thcipriani) A `git grep` for `10.68` yields a good amount of stuff related to beta-cluster. FrEx: ``` hieradata/labs/d... [20:02:49] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Kanban), 10cloud-services-team: Ensure there are no hard-coded IPs in use for beta - https://phabricator.wikimedia.org/T208262 (10thcipriani) A `git grep` for `10.68` yields a good amount of stuff related to beta-cluster. FrEx: ``` hieradata/labs/d... [20:15:00] James_F: yes, whoops, sorry missed your message. Troubleshooting some scap problems. Re-enabling now. [20:15:00] James_F: yes, whoops, sorry missed your message. Troubleshooting some scap problems. Re-enabling now. [20:15:15] Thanks! [20:15:15] Thanks! [20:20:15] Project beta-update-databases-eqiad build #29393: 04FAILURE in 14 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/29393/ [20:20:15] Project beta-update-databases-eqiad build #29393: 04FAILURE in 14 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/29393/ [20:23:24] Lame. [20:23:24] Lame. [20:23:57] Oh, hmm, if beta-scap-eqiad is not yet run, maybe beta-update-databases-eqiad is running the old config? [20:23:57] Oh, hmm, if beta-scap-eqiad is not yet run, maybe beta-update-databases-eqiad is running the old config? [20:24:07] hi [20:24:07] hi [20:24:24] ^ me [20:24:24] ^ me [20:24:59] Yeah, looks like. [20:24:59] Yeah, looks like. [20:25:14] * James_F stalks [20:25:14] ACTION stalks  [20:25:18] https://integration.wikimedia.org/ci/view/Beta/job/beta-scap-eqiad/226168/console [20:25:18] https://integration.wikimedia.org/ci/view/Beta/job/beta-scap-eqiad/226168/console [20:27:13] Hello World [20:27:13] Hello World [20:29:29] PROBLEM - Free space - all mounts on deployment-deploy01 is CRITICAL: CRITICAL: deployment-prep.deployment-deploy01.diskspace.root.byte_percentfree (<11.11%) [20:29:29] PROBLEM - Free space - all mounts on deployment-deploy01 is CRITICAL: CRITICAL: deployment-prep.deployment-deploy01.diskspace.root.byte_percentfree (<11.11%) [20:29:35] ok [20:29:36] ok [20:33:35] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Kanban), 10cloud-services-team: Ensure there are no hard-coded IPs in use for beta - https://phabricator.wikimedia.org/T208262 (10Krenair) I did this in T208101 [20:33:35] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Kanban), 10cloud-services-team: Ensure there are no hard-coded IPs in use for beta - https://phabricator.wikimedia.org/T208262 (10Krenair) I did this in T208101 [20:36:14] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Kanban), 10cloud-services-team: Ensure there are no hard-coded IPs in use for beta - https://phabricator.wikimedia.org/T208262 (10Krenair) [20:36:14] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Kanban), 10cloud-services-team: Ensure there are no hard-coded IPs in use for beta - https://phabricator.wikimedia.org/T208262 (10Krenair) [20:36:17] 10Beta-Cluster-Infrastructure, 10Cloud-Services, 10Epic: Migrate deployment-prep to eqiad1 - https://phabricator.wikimedia.org/T208101 (10Krenair) [20:36:17] 10Beta-Cluster-Infrastructure, 10Cloud-Services, 10Epic: Migrate deployment-prep to eqiad1 - https://phabricator.wikimedia.org/T208101 (10Krenair) [20:37:10] Project beta-scap-eqiad build #226168: 04STILL FAILING in 13 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/226168/ [20:37:10] Project beta-scap-eqiad build #226168: 04STILL FAILING in 13 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/226168/ [20:37:11] Project beta-code-update-eqiad build #223445: 04FAILURE in 0.1 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/223445/ [20:37:11] Project beta-code-update-eqiad build #223445: 04FAILURE in 0.1 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/223445/ [20:39:25] heh [20:39:25] heh [20:39:29] James_F: is that us? :P [20:39:29] James_F: is that us? :P [20:39:35] Yes. [20:39:35] Yes. [20:39:37] Well. [20:39:37] Well. [20:40:17] beta-update-databases-eqiad is broken because of us; it's hopefully fixed once beta-scap-eqiad works, but that is broken due to scap issues which are not us. [20:40:18] beta-update-databases-eqiad is broken because of us; it's hopefully fixed once beta-scap-eqiad works, but that is broken due to scap issues which are not us. [20:40:22] (I hope.) [20:40:22] (I hope.) [20:40:33] 13:37:10 20:37:10 Updating LocalisationCache for master using 6 thread(s) [20:40:33] 13:37:10 20:37:10 Updating LocalisationCache for master using 6 thread(s) [20:40:33] 13:37:10 20:37:10 Last output: [20:40:33] 13:37:10 mkdir: cannot create directory ‘/srv/jenkins/home/jenkins-deploy/workspace/beta-scap-eqiad/scap_l10n_2463257910’: Permission denied [20:40:33] 13:37:10 20:37:10 Last output: [20:40:33] 13:37:10 mkdir: cannot create directory ‘/srv/jenkins/home/jenkins-deploy/workspace/beta-scap-eqiad/scap_l10n_2463257910’: Permission denied [20:40:54] ... [20:40:54] ... [20:40:56] hrm [20:40:56] hrm [20:41:15] why does it think the workspace is the tempdir suddenly [20:41:15] why does it think the workspace is the tempdir suddenly [20:41:56] Project beta-scap-eqiad build #226169: 04STILL FAILING in 0.14 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/226169/ [20:41:56] Project beta-scap-eqiad build #226169: 04STILL FAILING in 0.14 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/226169/ [20:42:56] that one was even stranger. [20:42:56] that one was even stranger. [20:42:58] (The DB breakage is T208043.) [20:42:58] (The DB breakage is T208043.) [20:42:58] T208043: Unknown entity type mediainfo - https://phabricator.wikimedia.org/T208043 [20:42:59] T208043: Unknown entity type mediainfo - https://phabricator.wikimedia.org/T208043 [20:43:00] Project beta-code-update-eqiad build #223446: 04STILL FAILING in 0.1 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/223446/ [20:43:00] Project beta-code-update-eqiad build #223446: 04STILL FAILING in 0.1 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/223446/ [20:43:05] * James_F sighs. [20:43:05] ACTION sighs. [20:43:39] db is this https://phabricator.wikimedia.org/T208043? [20:43:39] db is this https://phabricator.wikimedia.org/T208043? [20:43:57] right, let me have a look at that [20:43:57] right, let me have a look at that [20:44:01] looks deployment-deploy01 is full of junk [20:44:01] looks deployment-deploy01 is full of junk [20:44:24] addshore: I kinda-sort-hope that its not broken with master conf. [20:44:24] addshore: I kinda-sort-hope that its not broken with master conf. [20:44:25] sorry [20:44:25] sorry [20:44:40] I was poking at shinken's check for disk space thcipriani [20:44:40] I was poking at shinken's check for disk space thcipriani [20:44:55] gone now [20:44:55] gone now [20:45:25] ah, that explains it, thanks Krenair [20:45:25] ah, that explains it, thanks Krenair [20:45:28] James_F: update.php doesnt break with the master conf [20:45:28] James_F: update.php doesnt break with the master conf [20:45:37] James_F: I guess the updates shouldnt run if the repo is turned on / loaded? [20:45:37] James_F: I guess the updates shouldnt run if the repo is turned on / loaded? [20:45:54] manually re-running beta-scap-eqiad [20:45:54] manually re-running beta-scap-eqiad [20:46:20] It should only trigger when the code is loaded the first time, yeah. [20:46:20] It should only trigger when the code is loaded the first time, yeah. [20:47:16] James_F: https://gerrit.wikimedia.org/r/#/c/470470/ should fix it [20:47:16] James_F: https://gerrit.wikimedia.org/r/#/c/470470/ should fix it [20:48:11] thcipriani, I must say though I'm surprised anyone has noticed [20:48:11] thcipriani, I must say though I'm surprised anyone has noticed [20:48:19] That host has been getting disk space full errors for days now [20:48:20] That host has been getting disk space full errors for days now [20:48:49] I noticed that sunday night :\ [20:48:49] I noticed that sunday night :\ [20:49:03] it's been going a little longer than that :) [20:49:03] it's been going a little longer than that :) [20:49:34] cleaned off a whole bunch of scap tmp files. It's frustrating. These files would be automagically cleaned if the sudo thing didn't keep happening. [20:49:34] cleaned off a whole bunch of scap tmp files. It's frustrating. These files would be automagically cleaned if the sudo thing didn't keep happening. [20:49:52] I haven't found the time to try to fiddle with the scap logic to try to work around it. [20:49:52] I haven't found the time to try to fiddle with the scap logic to try to work around it. [20:50:44] James_F: CI complaied, that MediaInfoEnable wasn't set? [20:50:44] James_F: CI complaied, that MediaInfoEnable wasn't set? [20:54:26] RECOVERY - Free space - all mounts on deployment-deploy01 is OK: OK: All targets OK [20:54:26] RECOVERY - Free space - all mounts on deployment-deploy01 is OK: OK: All targets OK [21:00:17] Yippee, build fixed! [21:00:17] Yippee, build fixed! [21:00:18] Project beta-scap-eqiad build #226170: 09FIXED in 14 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/226170/ [21:00:18] Project beta-scap-eqiad build #226170: 09FIXED in 14 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/226170/ [21:01:18] Yippee, build fixed! [21:01:18] Yippee, build fixed! [21:01:18] Project beta-code-update-eqiad build #223447: 09FIXED in 59 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/223447/ [21:01:18] Project beta-code-update-eqiad build #223447: 09FIXED in 59 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/223447/ [21:20:09] Project beta-update-databases-eqiad build #29394: 04STILL FAILING in 9.2 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/29394/ [21:20:09] Project beta-update-databases-eqiad build #29394: 04STILL FAILING in 9.2 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/29394/ [21:44:57] RECOVERY - DPKG on contint2001 is OK: All packages OK [21:44:58] RECOVERY - DPKG on contint2001 is OK: All packages OK [21:45:24] ^ fixed that [21:45:25] ^ fixed that [21:45:49] (03PS1) 10QChris: Allow “Gerrit Managers” to import history [labs/tools/ipwatcher] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/470504 [21:45:49] (03PS1) 10QChris: Allow “Gerrit Managers” to import history [labs/tools/ipwatcher] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/470504 [21:45:52] (03CR) 10QChris: [V: 032 C: 032] Allow “Gerrit Managers” to import history [labs/tools/ipwatcher] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/470504 (owner: 10QChris) [21:45:52] (03CR) 10QChris: [V: 032 C: 032] Allow “Gerrit Managers” to import history [labs/tools/ipwatcher] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/470504 (owner: 10QChris) [21:46:33] (03PS1) 10QChris: Import done. Revoke import grants [labs/tools/ipwatcher] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/470505 [21:46:33] (03PS1) 10QChris: Import done. Revoke import grants [labs/tools/ipwatcher] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/470505 [21:46:36] (03CR) 10QChris: [V: 032 C: 032] Import done. Revoke import grants [labs/tools/ipwatcher] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/470505 (owner: 10QChris) [21:46:36] (03CR) 10QChris: [V: 032 C: 032] Import done. Revoke import grants [labs/tools/ipwatcher] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/470505 (owner: 10QChris) [21:51:27] 10Phabricator (Upstream), 10Legalpad, 10Upstream: Legalpad access controls are confusing and seemingly broken - https://phabricator.wikimedia.org/T208254 (10mmodell) [21:51:27] 10Phabricator (Upstream), 10Legalpad, 10Upstream: Legalpad access controls are confusing and seemingly broken - https://phabricator.wikimedia.org/T208254 (10mmodell) [21:53:06] (03PS1) 10QChris: Allow “Gerrit Managers” to import history [labs/tools/gerrit-newcomer-bot] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/470506 [21:53:06] (03PS1) 10QChris: Allow “Gerrit Managers” to import history [labs/tools/gerrit-newcomer-bot] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/470506 [21:53:09] (03CR) 10QChris: [V: 032 C: 032] Allow “Gerrit Managers” to import history [labs/tools/gerrit-newcomer-bot] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/470506 (owner: 10QChris) [21:53:09] (03CR) 10QChris: [V: 032 C: 032] Allow “Gerrit Managers” to import history [labs/tools/gerrit-newcomer-bot] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/470506 (owner: 10QChris) [21:53:30] (03PS1) 10QChris: Import done. Revoke import grants [labs/tools/gerrit-newcomer-bot] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/470507 [21:53:30] (03PS1) 10QChris: Import done. Revoke import grants [labs/tools/gerrit-newcomer-bot] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/470507 [21:53:33] (03CR) 10QChris: [V: 032 C: 032] Import done. Revoke import grants [labs/tools/gerrit-newcomer-bot] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/470507 (owner: 10QChris) [21:53:33] (03CR) 10QChris: [V: 032 C: 032] Import done. Revoke import grants [labs/tools/gerrit-newcomer-bot] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/470507 (owner: 10QChris) [22:12:42] legoktm: is there a task for the phpunit-coverage html not being collected /viewable in Jenkins? [22:12:42] legoktm: is there a task for the phpunit-coverage html not being collected /viewable in Jenkins? [22:12:54] for patch coverage [22:12:54] for patch coverage [22:12:54] Krinkle: https://phabricator.wikimedia.org/T206230 [22:12:54] Krinkle: https://phabricator.wikimedia.org/T206230 [22:13:03] I just haven't had time to look into it yet [22:13:03] I just haven't had time to look into it yet [22:13:03] oops [22:13:03] oops [22:13:05] I shuold have known that [22:13:06] I shuold have known that [22:13:20] OK. [22:13:20] OK. [22:20:16] Project beta-update-databases-eqiad build #29395: 04STILL FAILING in 16 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/29395/ [22:20:16] Project beta-update-databases-eqiad build #29395: 04STILL FAILING in 16 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/29395/ [22:51:21] (03CR) 10Legoktm: [C: 032] Add Stibba to the Jenkins whitelist [integration/config] - 10https://gerrit.wikimedia.org/r/470453 (owner: 10Stibba) [22:51:21] (03CR) 10Legoktm: [C: 032] Add Stibba to the Jenkins whitelist [integration/config] - 10https://gerrit.wikimedia.org/r/470453 (owner: 10Stibba) [22:51:25] (03CR) 10Legoktm: [C: 032] Run extension-seccheck for WikibaseMediaInfo [integration/config] - 10https://gerrit.wikimedia.org/r/469954 (owner: 10Umherirrender) [22:51:25] (03CR) 10Legoktm: [C: 032] Run extension-seccheck for WikibaseMediaInfo [integration/config] - 10https://gerrit.wikimedia.org/r/469954 (owner: 10Umherirrender) [22:52:05] (03CR) 10Legoktm: [C: 032] Fix title of check php job [integration/config] - 10https://gerrit.wikimedia.org/r/470241 (https://phabricator.wikimedia.org/T208164) (owner: 10Zoranzoki21) [22:52:05] (03CR) 10Legoktm: [C: 032] Fix title of check php job [integration/config] - 10https://gerrit.wikimedia.org/r/470241 (https://phabricator.wikimedia.org/T208164) (owner: 10Zoranzoki21) [22:52:30] (03CR) 10Legoktm: [C: 032] Whitelist Edward Chernenko [integration/config] - 10https://gerrit.wikimedia.org/r/468900 (owner: 10Paladox) [22:52:31] (03CR) 10Legoktm: [C: 032] Whitelist Edward Chernenko [integration/config] - 10https://gerrit.wikimedia.org/r/468900 (owner: 10Paladox) [22:52:54] (03CR) 10Legoktm: [C: 032] Remove WikibaseQuality from Wikibase, WikibaseQualityConstraints [integration/config] - 10https://gerrit.wikimedia.org/r/469181 (https://phabricator.wikimedia.org/T207655) (owner: 10Tarrow) [22:52:54] (03CR) 10Legoktm: [C: 032] Remove WikibaseQuality from Wikibase, WikibaseQualityConstraints [integration/config] - 10https://gerrit.wikimedia.org/r/469181 (https://phabricator.wikimedia.org/T207655) (owner: 10Tarrow) [22:54:09] !log Manually triggered a beta-update-databases-eqiad run to see if our fix for T208043 has worked. [22:54:10] !log Manually triggered a beta-update-databases-eqiad run to see if our fix for T208043 has worked. [22:54:13] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [22:54:13] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [22:54:13] T208043: Unknown entity type mediainfo - https://phabricator.wikimedia.org/T208043 [22:54:13] T208043: Unknown entity type mediainfo - https://phabricator.wikimedia.org/T208043 [22:54:39] (03Merged) 10jenkins-bot: Add Stibba to the Jenkins whitelist [integration/config] - 10https://gerrit.wikimedia.org/r/470453 (owner: 10Stibba) [22:54:40] (03Merged) 10jenkins-bot: Add Stibba to the Jenkins whitelist [integration/config] - 10https://gerrit.wikimedia.org/r/470453 (owner: 10Stibba) [22:54:42] (03Merged) 10jenkins-bot: Run extension-seccheck for WikibaseMediaInfo [integration/config] - 10https://gerrit.wikimedia.org/r/469954 (owner: 10Umherirrender) [22:54:42] (03Merged) 10jenkins-bot: Run extension-seccheck for WikibaseMediaInfo [integration/config] - 10https://gerrit.wikimedia.org/r/469954 (owner: 10Umherirrender) [22:54:44] (03Merged) 10jenkins-bot: Fix title of check php job [integration/config] - 10https://gerrit.wikimedia.org/r/470241 (https://phabricator.wikimedia.org/T208164) (owner: 10Zoranzoki21) [22:54:45] (03Merged) 10jenkins-bot: Fix title of check php job [integration/config] - 10https://gerrit.wikimedia.org/r/470241 (https://phabricator.wikimedia.org/T208164) (owner: 10Zoranzoki21) [22:54:47] (03Merged) 10jenkins-bot: Whitelist Edward Chernenko [integration/config] - 10https://gerrit.wikimedia.org/r/468900 (owner: 10Paladox) [22:54:47] (03Merged) 10jenkins-bot: Whitelist Edward Chernenko [integration/config] - 10https://gerrit.wikimedia.org/r/468900 (owner: 10Paladox) [22:54:50] (03Merged) 10jenkins-bot: Remove WikibaseQuality from Wikibase, WikibaseQualityConstraints [integration/config] - 10https://gerrit.wikimedia.org/r/469181 (https://phabricator.wikimedia.org/T207655) (owner: 10Tarrow) [22:54:50] (03Merged) 10jenkins-bot: Remove WikibaseQuality from Wikibase, WikibaseQualityConstraints [integration/config] - 10https://gerrit.wikimedia.org/r/469181 (https://phabricator.wikimedia.org/T207655) (owner: 10Tarrow) [22:56:14] Yippee, build fixed! [22:56:14] Yippee, build fixed! [22:56:15] Project beta-update-databases-eqiad build #29396: 09FIXED in 2 min 42 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/29396/ [22:56:15] Project beta-update-databases-eqiad build #29396: 09FIXED in 2 min 42 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/29396/ [22:56:19] It worked. [22:56:19] It worked. [22:56:20] Yay. [22:56:20] Yay. [22:56:25] addshore: Success. [22:56:25] addshore: Success. [22:56:35] !log deployed https://gerrit.wikimedia.org/r/470453 https://gerrit.wikimedia.org/r/469954 https://gerrit.wikimedia.org/r/470241 https://gerrit.wikimedia.org/r/468900 https://gerrit.wikimedia.org/r/469181 [22:56:35] !log deployed https://gerrit.wikimedia.org/r/470453 https://gerrit.wikimedia.org/r/469954 https://gerrit.wikimedia.org/r/470241 https://gerrit.wikimedia.org/r/468900 https://gerrit.wikimedia.org/r/469181 [22:56:36] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [22:56:37] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [22:57:33] 10Continuous-Integration-Config, 10Wikibase-Quality, 10Wikibase-Quality-Constraints, 10Wikidata, and 2 others: WikibaseQualityConstraints and Wikibase CI Job shouldn't load WikibaseQuality - https://phabricator.wikimedia.org/T207655 (10Legoktm) 05Open>03Resolved [22:57:33] 10Continuous-Integration-Config, 10Wikibase-Quality, 10Wikibase-Quality-Constraints, 10Wikidata, and 2 others: WikibaseQualityConstraints and Wikibase CI Job shouldn't load WikibaseQuality - https://phabricator.wikimedia.org/T207655 (10Legoktm) 05Open>03Resolved [23:11:58] 10Continuous-Integration-Config, 10Ipwatcher, 10User-Urbanecm: Lint CI support for IPWatcher - https://phabricator.wikimedia.org/T208277 (10Urbanecm) [23:11:58] 10Continuous-Integration-Config, 10Ipwatcher, 10User-Urbanecm: Lint CI support for IPWatcher - https://phabricator.wikimedia.org/T208277 (10Urbanecm) [23:27:47] (03PS1) 10Urbanecm: Add CI support for IPWatcher [integration/config] - 10https://gerrit.wikimedia.org/r/470521 (https://phabricator.wikimedia.org/T208277) [23:27:47] (03PS1) 10Urbanecm: Add CI support for IPWatcher [integration/config] - 10https://gerrit.wikimedia.org/r/470521 (https://phabricator.wikimedia.org/T208277) [23:54:56] PROBLEM - SSH on integration-slave-docker-1021 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:54:56] PROBLEM - SSH on integration-slave-docker-1021 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:59:45] RECOVERY - SSH on integration-slave-docker-1021 is OK: SSH OK - OpenSSH_6.7p1 Debian-5+deb8u7 (protocol 2.0) [23:59:45] RECOVERY - SSH on integration-slave-docker-1021 is OK: SSH OK - OpenSSH_6.7p1 Debian-5+deb8u7 (protocol 2.0)