[00:03:29] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Next), 10Patch-For-Review: Beta puppetmaster cherry-pick process - https://phabricator.wikimedia.org/T135427 (10Krinkle) I've gone through and hash-tagged the Gerrit patches with `beta-picked` and restored any that were marked as abandoned: (03CR) 10Krinkle: [C: 031] "Afaik this is only used as macro from JJB, which invokes it from /srv/deployment/integration/slave-scripts, which... is embedded in Nodepo" [integration/jenkins] - 10https://gerrit.wikimedia.org/r/394907 (https://phabricator.wikimedia.org/T181940) (owner: 10Reedy) [01:11:58] PROBLEM - Free space - all mounts on deployment-tin is CRITICAL: CRITICAL: deployment-prep.deployment-tin.diskspace._mnt.byte_percentfree (No valid datapoints found)deployment-prep.deployment-tin.diskspace._srv.byte_percentfree (<10.00%) WARN: deployment-prep.deployment-tin.diskspace.root.byte_percentfree (<10.00%) [01:16:51] Project beta-scap-eqiad build #214843: 04FAILURE in 3 min 8 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/214843/ [01:23:52] Project beta-scap-eqiad build #214844: 04STILL FAILING in 9 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/214844/ [01:33:52] Project beta-scap-eqiad build #214845: 04STILL FAILING in 11 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/214845/ [01:40:31] deployment-tin is out of space [01:40:39] Krenair thcipriani ^^ [01:43:52] Project beta-scap-eqiad build #214846: 04STILL FAILING in 10 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/214846/ [01:45:24] PROBLEM - Puppet errors on deployment-cache-text04 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [01:52:01] 10Project-Admins, 10Developer-Advocacy (Jul-Sep 2018): Sort out scope/confusion between #Possible-Tech-Projects and #Outreach-Programs-Projects tags - https://phabricator.wikimedia.org/T198101 (10srishakatux) This approach sounds good to me! I've started cleaning up the #outreach-programs-projects and #possibl... [01:53:54] Project beta-scap-eqiad build #214847: 04STILL FAILING in 11 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/214847/ [02:03:53] Project beta-scap-eqiad build #214848: 04STILL FAILING in 11 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/214848/ [02:13:51] Project beta-scap-eqiad build #214849: 04STILL FAILING in 11 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/214849/ [02:23:51] Project beta-scap-eqiad build #214850: 04STILL FAILING in 7.2 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/214850/ [02:33:51] Project beta-scap-eqiad build #214851: 04STILL FAILING in 10 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/214851/ [02:43:52] Project beta-scap-eqiad build #214852: 04STILL FAILING in 10 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/214852/ [02:53:54] Project beta-scap-eqiad build #214853: 04STILL FAILING in 11 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/214853/ [03:03:52] Project beta-scap-eqiad build #214854: 04STILL FAILING in 11 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/214854/ [03:13:52] Project beta-scap-eqiad build #214855: 04STILL FAILING in 12 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/214855/ [03:23:45] Project beta-scap-eqiad build #214856: 04STILL FAILING in 7.1 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/214856/ [03:33:52] Project beta-scap-eqiad build #214857: 04STILL FAILING in 11 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/214857/ [03:38:04] !log deployment-tin:sudo rm -rf /srv/mediawiki/.git [03:38:08] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [03:56:23] Yippee, build fixed! [03:56:23] Project beta-scap-eqiad build #214858: 09FIXED in 12 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/214858/ [04:04:29] PROBLEM - Free space - all mounts on deployment-fluorine02 is CRITICAL: CRITICAL: deployment-prep.deployment-fluorine02.diskspace._srv.byte_percentfree (<55.56%) [04:20:25] 10Gerrit, 10Release-Engineering-Team (Kanban): Gerrit has created duplicate accounts for some users - https://phabricator.wikimedia.org/T197083 (10kaldari) @thcipriani: Some more information... Logging into https://gerrit.wikimedia.org/ with either kaldari or Kaldari fails ("Authentication failed."). sshing... [05:59:19] 10Gerrit: Gerrit's "Show Diffs" does not show all diffs if there are more than two diffs - https://phabricator.wikimedia.org/T199012 (10TerraCodes) [06:12:57] 10Beta-Cluster-Infrastructure, 10Growth-Team (Current Sprint), 10Patch-For-Review: Set up test environment for PageTriage drafts in beta labs - https://phabricator.wikimedia.org/T198898 (10Catrope) It turns out the article creation wizard is already imported on beta labs, but it's an old version. I imported... [07:14:33] RECOVERY - Free space - all mounts on deployment-fluorine02 is OK: OK: All targets OK [08:27:06] 10Beta-Cluster-Infrastructure, 10Cassandra, 10Services: Cassandra broken in beta - https://phabricator.wikimedia.org/T198995 (10Pchelolo) 05Open>03Invalid It's all fine, Cassandra lives on `deployment-cassandra3-01/2` now. We still need to reset the environment properly there, but I think we're waiting t... [10:40:48] (03PS1) 10Umherirrender: Run phan and seccheck for PageTriage [integration/config] - 10https://gerrit.wikimedia.org/r/444350 [10:42:14] (03CR) 10jerkins-bot: [V: 04-1] Run phan and seccheck for PageTriage [integration/config] - 10https://gerrit.wikimedia.org/r/444350 (owner: 10Umherirrender) [10:44:39] (03CR) 10Umherirrender: "recheck" [integration/config] - 10https://gerrit.wikimedia.org/r/444350 (owner: 10Umherirrender) [10:45:59] (03CR) 10jerkins-bot: [V: 04-1] Run phan and seccheck for PageTriage [integration/config] - 10https://gerrit.wikimedia.org/r/444350 (owner: 10Umherirrender) [13:36:33] 10Gerrit: Gerrit's "Show Diffs" does not show all diffs if there are more than two diffs - https://phabricator.wikimedia.org/T199012 (10Aklapper) 05Open>03stalled I do not see a "Show Diffs" button. When creating tasks, please always follow https://mediawiki.org/wiki/How_to_report_a_bug and structure your ta... [13:38:37] 10Gerrit: Gerrit's "Show Diffs" does not show all diffs if there are more than two diffs - https://phabricator.wikimedia.org/T199012 (10Paladox) It's in the polygerrit ui @Aklapper. [16:09:37] PROBLEM - Long lived cherry-picks on puppetmaster on deployment-puppetmaster03 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [17:16:07] 10MediaWiki-Releasing, 10Security: Streamline MW security release process - https://phabricator.wikimedia.org/T196602 (10Reedy) Should we be still sending a pre-announce email so people are aware? I'm guessing ~24H before pushing the fixes to master/branches? >Fix in production >Send pre-release announcement... [17:42:34] PROBLEM - Puppet errors on deployment-memc06 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [18:02:05] 10Phabricator: Rate-limit is too harsh - https://phabricator.wikimedia.org/T198974 (10Isarra) I started my browser and most of my phabricator tabs turned into complaints about too many concurrent connections. Is this the same thing? ``` TOO MANY CONCURRENT CONNECTIONS You (", 10.192.16.138, 10.192.16.138, 1... [18:12:04] PROBLEM - Puppet errors on deployment-webperf11 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [18:14:24] 10Phabricator: Rate-limit is too harsh - https://phabricator.wikimedia.org/T198974 (10Paladox) p:05Triage>03High It's happening to alot of users. I think we need to move the rate limit code further into phabricator where it can tell if your a user and in a group. Triaging as high. [18:14:37] 10Phabricator: Rate-limit is too harsh - https://phabricator.wikimedia.org/T198974 (10Jc86035) Yes, it's the same thing. Maybe it's an issue with the comment preview, which is rendered every second. I'm noticing that it also reloads my profile picture at the same rate when I'm typing. I have no idea why anyone w... [18:17:37] RECOVERY - Puppet errors on deployment-memc06 is OK: OK: Less than 1.00% above the threshold [0.0] [18:26:09] 18:24:14 1) MessageBlobStoreTest::testGetBlobCached [18:26:09] 18:24:14 MessageBlobStore::fetchMessage('example', 'en') was not expected to be called. [18:26:14] ^ Keeps flapping on rel1_29 [18:26:32] I think we disabled that test at some point [18:26:52] 353f09fac98be288d933d73145cffa14773e31d8 ? [18:27:43] Reedy: ^ [18:28:03] It only seems to fail on the tests for v+2 [18:28:05] on merging, it's fine [18:29:03] Cheers, might aswell backport [18:40:41] <3 [18:40:46] I was waiting for jerkins to finish the lot [18:41:00] 10Gerrit: Gerrit's "Show Diffs" button did not expand all diffs for each changed file - https://phabricator.wikimedia.org/T199012 (10Aklapper) [18:42:13] :) [18:47:03] RECOVERY - Puppet errors on deployment-webperf11 is OK: OK: Less than 1.00% above the threshold [0.0] [18:48:21] 10Phabricator, 10Wikibugs, 10Patch-For-Review: wikibugs hits Phabricator's rate limiting and hence is unreliable - https://phabricator.wikimedia.org/T198915 (10Legoktm) Why are read-only requests being rate limited? That doesn't really make sense. Only write requests should be rate limited... [19:12:22] PROBLEM - Puppet errors on deployment-aqs01 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [19:41:06] (03CR) 10Hashar: [C: 032] Run phan for Cite [integration/config] - 10https://gerrit.wikimedia.org/r/444320 (owner: 10Umherirrender) [19:42:43] (03Merged) 10jenkins-bot: Run phan for Cite [integration/config] - 10https://gerrit.wikimedia.org/r/444320 (owner: 10Umherirrender) [19:44:26] 10Continuous-Integration-Config, 10MathSearch, 10Patch-For-Review: mwext-testextension-zend should load extension mathsearch after math - https://phabricator.wikimedia.org/T117659 (10hashar) Next issue: InvalidArgumentException from line 40 of extensions/MathSearch/ContentMathFormatter.php: Unsupported... [19:47:06] 10Beta-Cluster-Infrastructure, 10Services (later): "invalid locale" warning on deployment-restbase02.deployment-prep.eqiad.wmflabs - https://phabricator.wikimedia.org/T195709 (10Krinkle) [19:47:21] RECOVERY - Puppet errors on deployment-aqs01 is OK: OK: Less than 1.00% above the threshold [0.0] [19:47:43] 10Beta-Cluster-Infrastructure, 10Services: RESTBase errors on logstash-beta - https://phabricator.wikimedia.org/T186994 (10Krinkle) [19:54:53] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Someday), 10Technical-Debt: Remove deployment.wikimedia.beta.wmflabs.org wiki (deploymentwiki) - https://phabricator.wikimedia.org/T198673 (10MarcoAurelio) >>! In T198673#4404490, @Krenair wrote: >>>! In T198673#4399005, @MarcoAurelio wrote: >> Can w... [19:56:56] RECOVERY - Free space - all mounts on deployment-tin is OK: OK: deployment-prep.deployment-tin.diskspace._mnt.byte_percentfree (No valid datapoints found) [20:29:08] (03PS1) 10Hashar: Migrate Math to Quibble [integration/config] - 10https://gerrit.wikimedia.org/r/444383 (https://phabricator.wikimedia.org/T183512) [20:29:12] (03PS1) 10Hashar: Migrate LifeWeb to Quibble [integration/config] - 10https://gerrit.wikimedia.org/r/444384 (https://phabricator.wikimedia.org/T183512) [20:29:15] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10releng-201718-q3, 10Epic, 10Patch-For-Review: [EPIC] Migrate Mediawiki jobs from Nodepool to Docker - https://phabricator.wikimedia.org/T183512 (10hashar) [20:29:31] (03CR) 10Hashar: [C: 032] Migrate LifeWeb to Quibble [integration/config] - 10https://gerrit.wikimedia.org/r/444384 (https://phabricator.wikimedia.org/T183512) (owner: 10Hashar) [20:29:35] (03CR) 10Hashar: [C: 032] Migrate Math to Quibble [integration/config] - 10https://gerrit.wikimedia.org/r/444383 (https://phabricator.wikimedia.org/T183512) (owner: 10Hashar) [20:31:07] (03Merged) 10jenkins-bot: Migrate Math to Quibble [integration/config] - 10https://gerrit.wikimedia.org/r/444383 (https://phabricator.wikimedia.org/T183512) (owner: 10Hashar) [20:31:10] (03Merged) 10jenkins-bot: Migrate LifeWeb to Quibble [integration/config] - 10https://gerrit.wikimedia.org/r/444384 (https://phabricator.wikimedia.org/T183512) (owner: 10Hashar) [20:39:44] (03PS1) 10Hashar: Migrate PageDisqus to Quibble without Selenium [integration/config] - 10https://gerrit.wikimedia.org/r/444386 (https://phabricator.wikimedia.org/T183512) [20:39:56] (03CR) 10Hashar: [C: 032] Migrate PageDisqus to Quibble without Selenium [integration/config] - 10https://gerrit.wikimedia.org/r/444386 (https://phabricator.wikimedia.org/T183512) (owner: 10Hashar) [20:40:18] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10releng-201718-q3, 10Epic, 10Patch-For-Review: [EPIC] Migrate Mediawiki jobs from Nodepool to Docker - https://phabricator.wikimedia.org/T183512 (10hashar) [20:42:00] (03Merged) 10jenkins-bot: Migrate PageDisqus to Quibble without Selenium [integration/config] - 10https://gerrit.wikimedia.org/r/444386 (https://phabricator.wikimedia.org/T183512) (owner: 10Hashar) [20:45:57] 10Continuous-Integration-Config, 10Release-Engineering-Team (Someday), 10MediaWiki-extensions-SendGrid: Extensions with PHP 5.6+ as requirements making Jenkins to fail on merge when CR+2 - https://phabricator.wikimedia.org/T185451 (10hashar) 05Open>03declined php5.5 is no more supported in master. We now... [20:48:17] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10releng-201718-q3, 10Epic, 10Patch-For-Review: [EPIC] Migrate Mediawiki jobs from Nodepool to Docker - https://phabricator.wikimedia.org/T183512 (10hashar) [20:48:32] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10releng-201718-q3, 10Epic, 10Patch-For-Review: [EPIC] Migrate Mediawiki jobs from Nodepool to Docker - https://phabricator.wikimedia.org/T183512 (10hashar) [21:03:41] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10releng-201718-q3, 10Epic, 10Patch-For-Review: [EPIC] Migrate Mediawiki jobs from Nodepool to Docker - https://phabricator.wikimedia.org/T183512 (10hashar) [21:09:13] 10Continuous-Integration-Config, 10Release-Engineering-Team (Someday), 10MediaWiki-extensions-SendGrid: Extensions with PHP 5.6+ as requirements making Jenkins to fail on merge when CR+2 - https://phabricator.wikimedia.org/T185451 (10D3r1ck01) Perfect, that makes sense as the CI tests of the extension has st... [21:15:13] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10releng-201718-q3, 10Epic, 10Patch-For-Review: [EPIC] Migrate Mediawiki jobs from Nodepool to Docker - https://phabricator.wikimedia.org/T183512 (10hashar) [21:24:26] (03PS1) 10Hashar: Migrate GoogleAppEngine to Quibble without Selenium [integration/config] - 10https://gerrit.wikimedia.org/r/444418 (https://phabricator.wikimedia.org/T196346) [21:24:52] 10Release-Engineering-Team (Kanban), 10MediaWiki-extensions-GoogleAppEngine, 10Patch-For-Review: [GoogleAppEngine] exception when saving page - https://phabricator.wikimedia.org/T196346 (10hashar) 05Open>03declined CI is not Google App Engine, so I have disabled the selenium test in CI. [21:25:00] (03CR) 10Hashar: [C: 032] Migrate GoogleAppEngine to Quibble without Selenium [integration/config] - 10https://gerrit.wikimedia.org/r/444418 (https://phabricator.wikimedia.org/T196346) (owner: 10Hashar) [21:26:18] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10releng-201718-q3, 10Epic, 10Patch-For-Review: [EPIC] Migrate Mediawiki jobs from Nodepool to Docker - https://phabricator.wikimedia.org/T183512 (10hashar) [21:27:20] (03Merged) 10jenkins-bot: Migrate GoogleAppEngine to Quibble without Selenium [integration/config] - 10https://gerrit.wikimedia.org/r/444418 (https://phabricator.wikimedia.org/T196346) (owner: 10Hashar) [21:35:09] (03PS1) 10Hashar: Quibble template without Selenium but with Composer [integration/config] - 10https://gerrit.wikimedia.org/r/444420 [21:36:05] (03PS1) 10Hashar: Migrate reCaptcha to Quibble without Selenium [integration/config] - 10https://gerrit.wikimedia.org/r/444421 (https://phabricator.wikimedia.org/T183512) [21:36:28] (03CR) 10Hashar: [C: 032] Quibble template without Selenium but with Composer [integration/config] - 10https://gerrit.wikimedia.org/r/444420 (owner: 10Hashar) [21:36:31] (03CR) 10Hashar: [C: 032] Migrate reCaptcha to Quibble without Selenium [integration/config] - 10https://gerrit.wikimedia.org/r/444421 (https://phabricator.wikimedia.org/T183512) (owner: 10Hashar) [21:37:12] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10releng-201718-q3, 10Epic, 10Patch-For-Review: [EPIC] Migrate Mediawiki jobs from Nodepool to Docker - https://phabricator.wikimedia.org/T183512 (10hashar) [21:37:57] (03Merged) 10jenkins-bot: Quibble template without Selenium but with Composer [integration/config] - 10https://gerrit.wikimedia.org/r/444420 (owner: 10Hashar) [21:37:59] (03Merged) 10jenkins-bot: Migrate reCaptcha to Quibble without Selenium [integration/config] - 10https://gerrit.wikimedia.org/r/444421 (https://phabricator.wikimedia.org/T183512) (owner: 10Hashar) [23:11:51] PROBLEM - SSH on integration-slave-docker-1017 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:16:41] RECOVERY - SSH on integration-slave-docker-1017 is OK: SSH OK - OpenSSH_6.7p1 Debian-5+deb8u4 (protocol 2.0) [23:22:09] 10Gerrit: Gerrit's "Show Diffs" button did not expand all diffs for each changed file - https://phabricator.wikimedia.org/T199012 (10TerraCodes) huh, now its happening again on https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/443881/2 {F23438881} [23:26:16] 10Gerrit: Gerrit's "Show Diffs" button did not expand all diffs for each changed file - https://phabricator.wikimedia.org/T199012 (10TerraCodes) [23:26:53] 10Gerrit: Gerrit's "Show Diffs" button did not expand all diffs for each changed file - https://phabricator.wikimedia.org/T199012 (10TerraCodes) 05stalled>03Open [23:26:59] 10Gerrit: Gerrit's "Show Diffs" button did not expand all diffs for each changed file - https://phabricator.wikimedia.org/T199012 (10Paladox) I could reproduce for a minute but then it worked. It's because that file is large. [23:31:27] 10Gerrit: Gerrit's "Show Diffs" button did not expand all diffs for each changed file - https://phabricator.wikimedia.org/T199012 (10TerraCodes) >>! In T199012#4405912, @Paladox wrote: > I could reproduce for a minute but then it worked. It's because that file is large. > > So i would call that expected behavio...