[00:45:11] RECOVERY - Free space - all mounts on integration-slave-jessie-1004 is OK: OK: All targets OK [00:51:10] PROBLEM - Free space - all mounts on integration-slave-jessie-1004 is CRITICAL: CRITICAL: integration.integration-slave-jessie-1004.diskspace._srv.byte_percentfree (<11.11%) [04:39:06] PROBLEM - Free space - all mounts on deployment-fluorine02 is CRITICAL: CRITICAL: deployment-prep.deployment-fluorine02.diskspace._srv.byte_percentfree (<11.11%) [06:48:43] 10Phabricator: Phabricator uses a variant of upstream favicon when there are unread message(s) in Conpherence - https://phabricator.wikimedia.org/T182839 (10zhuyifei1999) 05Open>03Resolved Looks like resolved? [06:54:07] RECOVERY - Free space - all mounts on deployment-fluorine02 is OK: OK: All targets OK [08:34:21] 10Gerrit, 10Release-Engineering-Team, 10Patch-For-Review: Make PolyGerrit the default ui - https://phabricator.wikimedia.org/T196812 (10Framawiki) >>! In T196812#4292577, @Johan wrote: >>>! In T196812#4269309, @Framawiki wrote: >> Proposition of summary for #user-notice / tech news: >> New users to Wikimedia... [09:14:31] 10Phabricator, 10Project-Admins, 10Pywikibot-core: Merge #Pywikibot-General to #Pywikibot-core - https://phabricator.wikimedia.org/T195856 (10MarcoAurelio) @Aklapper If we're going to remove (do we need to remove the tag?) #pywikibot-general from a lot of tasks, there's a nice CLI command to do so silently a... [09:49:20] 10Release-Engineering-Team (Kanban), 10MediaWiki-Core-Tests, 10User-zeljkofilipin: Q1 Selenium framework improvements - https://phabricator.wikimedia.org/T198389 (10zeljkofilipin) [09:49:23] 10Release-Engineering-Team (Kanban), 10MediaWiki-Core-Tests, 10User-zeljkofilipin: Sample code in Node.js for repositories that still have Selenium+Ruby tests - https://phabricator.wikimedia.org/T183160 (10zeljkofilipin) [09:50:25] 10Release-Engineering-Team (Kanban), 10MediaWiki-Core-Tests, 10User-zeljkofilipin: Q1 Selenium framework improvements - https://phabricator.wikimedia.org/T198389 (10zeljkofilipin) [09:52:43] 10Release-Engineering-Team, 10MediaWiki-Core-Tests, 10User-zeljkofilipin: Q4 Selenium framework improvements - https://phabricator.wikimedia.org/T190994 (10zeljkofilipin) 05Open>03Resolved The only remaining sub-task {T183162} was moved to {T198389}. [09:52:45] 10Release-Engineering-Team, 10MediaWiki-Core-Tests, 10Epic, 10Tracking, 10User-zeljkofilipin: Selenium framework improvements - https://phabricator.wikimedia.org/T182986 (10zeljkofilipin) [10:54:05] Project beta-scap-eqiad build #213957: 15ABORTED in 1 hr 0 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/213957/ [10:57:38] PROBLEM - Free space - all mounts on deployment-tin is CRITICAL: CRITICAL: deployment-prep.deployment-tin.diskspace._mnt.byte_percentfree (No valid datapoints found)deployment-prep.deployment-tin.diskspace._srv.byte_percentfree (<11.11%) [12:02:13] 10Phabricator: The Edit Task form of an existing task should expose the 'Due Date' field - https://phabricator.wikimedia.org/T193770 (10mmodell) It's just another field on the already long list of fields to fill out when creating a task. [12:05:30] 10Phabricator: Add support for task types - https://phabricator.wikimedia.org/T93499 (10mmodell) a:03mmodell [12:06:36] 10Phabricator: Add support for task types - https://phabricator.wikimedia.org/T93499 (10mmodell) D1074 will add the ability to set task types from herald rules. [13:02:42] 10Phabricator: The Edit Task form of an existing task should expose the 'Due Date' field - https://phabricator.wikimedia.org/T193770 (10dbarratt) >>! In T193770#4326937, @mmodell wrote: >>>! In T193770#4180438, @dbarratt wrote: >> I'm asking why it's distracting... is the field distracting or the display of the... [13:23:59] 10Phabricator, 10Project-Admins, 10Pywikibot-core: Merge #Pywikibot-General to #Pywikibot-core - https://phabricator.wikimedia.org/T195856 (10Aklapper) I don't see a need to "remove" (a feature that does not exist) that tag. [14:56:36] 10Continuous-Integration-Config: mwgate-php70lint is failing mercilessly for AbuseFilter - https://phabricator.wikimedia.org/T198530 (10Daimona) [14:58:08] 10Continuous-Integration-Config: mwgate-php70lint is failing mercilessly for AbuseFilter - https://phabricator.wikimedia.org/T198530 (10Reedy) [15:02:40] 10Continuous-Integration-Config: mwgate-php70lint is failing mercilessly for AbuseFilter - https://phabricator.wikimedia.org/T198530 (10Reedy) `/srv` was full... [15:11:11] RECOVERY - Free space - all mounts on integration-slave-jessie-1004 is OK: OK: All targets OK [17:06:51] 10Continuous-Integration-Config: mwgate-php70lint is failing mercilessly for AbuseFilter - https://phabricator.wikimedia.org/T198530 (10Daimona) 05Open>03Resolved a:03Daimona Solved, many thanks! [17:27:40] 10Release-Engineering-Team, 10Performance-Team: Saving timings up 50% since 2018-06-28 20:53 - https://phabricator.wikimedia.org/T198483 (10Tgr) >>! In T198483#4326191, @greg wrote: > @tgr @daniel could the changes made to address the issue have caused a save timing regression? Not in any real way, I'd guess... [17:45:02] so https://integration.wikimedia.org/ci/job/mediawiki-core-hhvmlint/1338/console fails with 'no space left on device'; you may want to have a look and purge stuff, etc. Thanks. [17:47:12] Hauskatze: Already fixed [17:47:19] Nearly 3 hours ago [17:47:35] Good [17:47:51] I checked https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL but saw no entry [17:57:25] PROBLEM - Free space - all mounts on deployment-tin is CRITICAL: CRITICAL: deployment-prep.deployment-tin.diskspace._mnt.byte_percentfree (No valid datapoints found)deployment-prep.deployment-tin.diskspace._srv.byte_percentfree (<44.44%) [18:05:25] 10Phabricator, 10Project-Admins, 10Pywikibot-core: Merge #Pywikibot-General to #Pywikibot-core - https://phabricator.wikimedia.org/T195856 (10MarcoAurelio) >>! In T195856#4327013, @Aklapper wrote: > I don't see a need to "remove" (a feature that does not exist) that tag. Sorry but this does not help. I thin... [18:20:59] @steward please block 35.229.125.215, open proxy, thanks [18:21:06] Ehh, wrong channel, sorry [18:22:26] PROBLEM - Free space - all mounts on deployment-tin is CRITICAL: CRITICAL: deployment-prep.deployment-tin.diskspace._mnt.byte_percentfree (No valid datapoints found)deployment-prep.deployment-tin.diskspace._srv.byte_percentfree (<44.44%) [19:06:58] who keeps breaking the scap package across beta dammit [19:12:27] PROBLEM - Free space - all mounts on deployment-tin is CRITICAL: CRITICAL: deployment-prep.deployment-tin.diskspace._mnt.byte_percentfree (No valid datapoints found)deployment-prep.deployment-tin.diskspace._srv.byte_percentfree (<44.44%) [19:25:45] 15G /srv/mediawiki [19:25:46] 9.4G /srv/mediawiki-staging [19:25:56] is it normal to have mediawiki be bigger than mediawiki-staging ? [19:51:18] RECOVERY - Puppet errors on deployment-imagescaler01 is OK: OK: Less than 1.00% above the threshold [0.0] [19:52:48] RECOVERY - Puppet errors on deployment-aqs02 is OK: OK: Less than 1.00% above the threshold [0.0] [19:52:52] RECOVERY - Puppet errors on deployment-cassandra3-02 is OK: OK: Less than 1.00% above the threshold [0.0] [19:53:53] RECOVERY - Puppet errors on deployment-cpjobqueue is OK: OK: Less than 1.00% above the threshold [0.0] [19:57:25] PROBLEM - Free space - all mounts on deployment-tin is CRITICAL: CRITICAL: deployment-prep.deployment-tin.diskspace._mnt.byte_percentfree (No valid datapoints found)deployment-prep.deployment-tin.diskspace._srv.byte_percentfree (<44.44%) [19:58:44] RECOVERY - Puppet errors on deployment-aqs03 is OK: OK: Less than 1.00% above the threshold [0.0] [19:58:46] RECOVERY - Puppet errors on deployment-pdfrender02 is OK: OK: Less than 1.00% above the threshold [0.0] [19:59:28] RECOVERY - Puppet errors on deployment-changeprop is OK: OK: Less than 1.00% above the threshold [0.0] [20:00:24] RECOVERY - Puppet errors on deployment-restbase01 is OK: OK: Less than 1.00% above the threshold [0.0] [20:00:48] RECOVERY - Puppet errors on deployment-restbase02 is OK: OK: Less than 1.00% above the threshold [0.0] [20:01:31] RECOVERY - Puppet errors on deployment-tin is OK: OK: Less than 1.00% above the threshold [0.0] [20:01:37] RECOVERY - Puppet errors on deployment-cassandra3-01 is OK: OK: Less than 1.00% above the threshold [0.0] [20:02:25] RECOVERY - Puppet errors on deployment-aqs01 is OK: OK: Less than 1.00% above the threshold [0.0] [20:02:33] RECOVERY - Puppet errors on deployment-sca02 is OK: OK: Less than 1.00% above the threshold [0.0] [20:02:40] RECOVERY - Puppet errors on deployment-zotero01 is OK: OK: Less than 1.00% above the threshold [0.0] [20:02:44] RECOVERY - Puppet errors on deployment-mathoid is OK: OK: Less than 1.00% above the threshold [0.0] [20:02:58] RECOVERY - Puppet errors on deployment-mcs01 is OK: OK: Less than 1.00% above the threshold [0.0] [20:03:24] RECOVERY - Puppet errors on deployment-sca01 is OK: OK: Less than 1.00% above the threshold [0.0] [20:03:30] RECOVERY - Puppet errors on deployment-logstash2 is OK: OK: Less than 1.00% above the threshold [0.0] [20:03:56] RECOVERY - Puppet errors on deployment-parsoid09 is OK: OK: Less than 1.00% above the threshold [0.0] [20:04:00] RECOVERY - Puppet errors on deployment-mediawiki06 is OK: OK: Less than 1.00% above the threshold [0.0] [20:04:02] RECOVERY - Puppet errors on deployment-mira is OK: OK: Less than 1.00% above the threshold [0.0] [20:12:31] 10Continuous-Integration-Config, 10Operations, 10Traffic: Set up CI for new repo operations/software/certcentral.git - https://phabricator.wikimedia.org/T198541 (10Krenair) p:05Triage>03Normal [20:15:35] RECOVERY - Puppet errors on deployment-maps03 is OK: OK: Less than 1.00% above the threshold [0.0] [20:17:36] 10Continuous-Integration-Config, 10Operations, 10Traffic: Set up CI for new repo operations/software/certcentral.git - https://phabricator.wikimedia.org/T198541 (10Krenair) p:05Normal>03Triage [20:33:49] Project beta-code-update-eqiad build #211523: 04FAILURE in 48 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/211523/ [20:35:31] ^ no space left on device + cannot find refs/heads/master? [20:35:37] the last one is weird [20:35:43] * Hauskatze logs and sees [20:37:29] Hauskatze, that's me [20:37:50] ah ok [20:37:53] I was running git gc on deployment-tin to try to free up space and the process used up the remaining space [20:37:56] but [20:38:00] before I ran it, 2G available [20:38:03] now: 5.2G [20:38:13] cause I'm checking and I don't see nothing weird [20:38:19] oh, git gc :D [20:38:30] I use sometimes git gc --aggressive on my local repos [20:40:14] well I'm obviously doing something wrong [20:40:16] maurelio@deployment-tin:/srv$ du -sh jenkins-workspace/ [20:40:17] 4.0K jenkins-workspace/ [20:44:36] Hauskatze, know anything about CI config? [20:44:46] a bit [20:44:58] Yippee, build fixed! [20:44:59] Project beta-code-update-eqiad build #211524: 09FIXED in 1 min 58 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/211524/ [20:45:01] enough to get a new repo set up with basic python checks? [20:45:01] basics [20:45:08] tox-docker? [20:45:13] * Krenair shrugs [20:45:19] whatever we normally use for these repos [20:45:44] let me have a look... looks like it's for certcentral.git? [20:46:04] yes [20:46:17] * Hauskatze puts on his glasses [20:46:47] I had to manually set Verified+1 and submit earlier [20:47:19] was a blast from the past [20:47:47] RECOVERY - Free space - all mounts on deployment-tin is OK: OK: deployment-prep.deployment-tin.diskspace._mnt.byte_percentfree (No valid datapoints found) [20:48:52] so the only two things shinken is complaining about now are puppet cherry-picks and puppet timeouts on -cache-text04 which are probably my fault [20:49:05] also, this doesn't look good: [20:49:06] krenair@deployment-tin:/srv/mediawiki$ sudo git gc --aggressive --prune=now [20:49:06] Counting objects: 221534, done. [20:49:06] Delta compression using up to 8 threads. [20:49:06] error: pack-objects died of signal 99019) [20:49:07] error: failed to run repack [20:52:07] 10Continuous-Integration-Config, 10Operations, 10Traffic: Set up CI for new repo operations/software/certcentral.git - https://phabricator.wikimedia.org/T198541 (10MarcoAurelio) I think we can use the `tox-docker` template but I reckon the repository needs a `tox.ini` file for this to work. [21:06:23] Hauskatze, https://gerrit.wikimedia.org/r/#/c/operations/software/certcentral/+/443291/ [21:06:45] okay, I'll make a patch for zuul and let's see how that works [21:08:01] Hauskatze, if you think that patch is ok, +1 and I'll merge? [21:13:34] maurelio@gerrit.wikimedia.org: Permission denied (publickey). [21:13:46] if I git-review again it dissapears [21:13:52] but it's annoying [21:14:32] (03PS1) 10MarcoAurelio: Add tox-docker tests for operations/software/certcentral.git [integration/config] - 10https://gerrit.wikimedia.org/r/443292 (https://phabricator.wikimedia.org/T198541) [21:14:45] 10Phabricator, 10Project-Admins, 10Pywikibot-core: Merge #Pywikibot-General to #Pywikibot-core - https://phabricator.wikimedia.org/T195856 (10Aklapper) >>! In T195856#4327341, @MarcoAurelio wrote: > Sorry but this does not help. I think the message is clear as to what was proposed: remove the tag from existi... [21:15:18] Hauskatze, ... so running it once gives you publickey problems but the second time it succeeds? [21:19:24] on some newly cloned repos, yes it does [21:21:17] weird [21:25:33] 10Phabricator, 10Project-Admins, 10Pywikibot-core: Merge #Pywikibot-General to #Pywikibot-core - https://phabricator.wikimedia.org/T195856 (10MarcoAurelio) > Where does it say so [...] T195856#4326765 first line: "if we're going to remove [the deprecated tag] from a lot of tasks". That makes it clear what t... [21:30:53] (03CR) 10MarcoAurelio: "recheck" [integration/config] - 10https://gerrit.wikimedia.org/r/443292 (https://phabricator.wikimedia.org/T198541) (owner: 10MarcoAurelio) [21:31:39] (03PS2) 10MarcoAurelio: Add tox-docker tests for operations/software/certcentral.git [integration/config] - 10https://gerrit.wikimedia.org/r/443292 (https://phabricator.wikimedia.org/T198541) [21:33:03] (03CR) 10MarcoAurelio: "recheck" [integration/config] - 10https://gerrit.wikimedia.org/r/443292 (https://phabricator.wikimedia.org/T198541) (owner: 10MarcoAurelio) [21:34:10] something's up with the flake8 version I have locally [21:34:21] I get RecursionError: maximum recursion depth exceeded in there [21:35:53] I don't have tox on my laptop so I cannot check that file [21:44:33] got it and testing that change [21:45:09] some Danish butter cookies in the meanwhile [21:47:41] (03PS3) 10Hashar: Add tox-docker tests for operations/software/certcentral.git [integration/config] - 10https://gerrit.wikimedia.org/r/443292 (https://phabricator.wikimedia.org/T198541) (owner: 10MarcoAurelio) [21:47:59] well 'tox' just ended with lots of warnings and stuff I don't understand [21:48:14] (03CR) 10Hashar: [C: 032] "I have removed the Depends-On so we can configure CI first." [integration/config] - 10https://gerrit.wikimedia.org/r/443292 (https://phabricator.wikimedia.org/T198541) (owner: 10MarcoAurelio) [21:48:45] well, we'll get CI setup [21:49:34] (03Merged) 10jenkins-bot: Add tox-docker tests for operations/software/certcentral.git [integration/config] - 10https://gerrit.wikimedia.org/r/443292 (https://phabricator.wikimedia.org/T198541) (owner: 10MarcoAurelio) [21:52:49] (03PS1) 10Hashar: Fix up certcentral repo name [integration/config] - 10https://gerrit.wikimedia.org/r/443295 (https://phabricator.wikimedia.org/T198541) [22:07:52] 10Continuous-Integration-Config, 10Operations, 10Traffic, 10Patch-For-Review: Set up CI for new repo operations/software/certcentral.git - https://phabricator.wikimedia.org/T198541 (10MarcoAurelio) So the CI is setup, but there are some errors that needs to get fixed or ignored. Unfortunatelly, I cannot he... [22:39:51] 10Continuous-Integration-Config, 10Operations, 10Traffic, 10Patch-For-Review, 10User-MarcoAurelio: Set up CI for new repo operations/software/certcentral.git - https://phabricator.wikimedia.org/T198541 (10Krenair) 05Open>03Resolved a:05Krenair>03MarcoAurelio thanks guys, this looks ok