[01:55:13] 10Beta-Cluster-Infrastructure: Captchas sent with wrong mime type on beta - https://phabricator.wikimedia.org/T164047#3219881 (10Bawolff) [02:08:50] 10Beta-Cluster-Infrastructure: Captchas sent with wrong mime type on beta - https://phabricator.wikimedia.org/T164047#3219896 (10Bawolff) I mean its being sent as `Content-Type: application/x-www-form-urlencoded` [05:32:13] 10Deployment-Systems, 10Scap (Scap3-MediaWiki-MVP), 06Operations, 13Patch-For-Review, 15User-Joe: Install conftool on deployment masters - https://phabricator.wikimedia.org/T163565#3219967 (10Joe) [07:59:55] 10Deployment-Systems, 06Operations, 10Ops-Access-Requests: Enable keyholder for ORES deployments - https://phabricator.wikimedia.org/T163939#3220070 (10akosiaris) @halfak, you don't get to know the appropriate passphrase. No deployer ever does. keyholder gets armed by an ops person once (after every deployme... [08:08:33] 06Release-Engineering-Team, 10Wikidata, 07Story: [Story] Use composer-merge-plugin to include Wikidata components in mediawiki-vendor - https://phabricator.wikimedia.org/T95663#3220074 (10hoo) Any updates here? For us it's good enough if we can manually update the vendor "build" when needed, we don't need an... [08:22:27] 10Gerrit, 05Security: bouncycastle information disclosure [DSA 3829-1] [CVE-2015-6644] (and make Gerrit use Debian package) - https://phabricator.wikimedia.org/T163185#3220114 (10Peachey88) [09:20:41] 10Scap: Scap3 canonical target list - https://phabricator.wikimedia.org/T148992#2738970 (10akosiaris) See https://gerrit.wikimedia.org/r/#/c/306431 for a first (and old) attempt at this. [11:19:51] PROBLEM - Puppet errors on deployment-sca03 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [12:31:42] I see https://www.mediawiki.org/wiki/Release_checklist covering tarballs. Is there a similar RelEng doc for covering REL_1_xx branching? [12:32:55] ...cause I'd love to document that @ReleaseTaggerBot in Phab gets unhappy if you do not also create a #MW-1.xx-release project in Phab after branching REL_1_xx in Git. [13:12:10] 10Browser-Tests-Infrastructure, 07JavaScript, 15User-zeljkofilipin: WebdriverIO CREDIT showcase - https://phabricator.wikimedia.org/T164070#3220651 (10zeljkofilipin) [13:13:25] 10Browser-Tests-Infrastructure, 10MediaWiki-General-or-Unknown, 07JavaScript, 05MW-1.29-release (WMF-deploy-2017-03-21_(1.29.0-wmf.17)), and 4 others: Port Selenium tests from Ruby to Node.js - https://phabricator.wikimedia.org/T139740#2821253 (10zeljkofilipin) [13:39:17] 06Release-Engineering-Team, 06Operations, 05Goal, 06Services (designing), and 2 others: Prepare and maintain base container images - https://phabricator.wikimedia.org/T162042#3220763 (10MoritzMuehlenhoff) a:03MoritzMuehlenhoff [13:45:55] 06Release-Engineering-Team, 06Operations, 05Goal, 06Services (designing), and 2 others: Prepare and maintain base container images - https://phabricator.wikimedia.org/T162042#3220775 (10MoritzMuehlenhoff) I don't think we need a trusty image. The remaining services run on trusty are in the process of migra... [13:53:29] 10Deployment-Systems, 10Scap, 13Patch-For-Review: Update Debian Package for Scap3 - https://phabricator.wikimedia.org/T127762#3220788 (10fgiunchedi) 05Open>03Resolved @thcipriani yep, all done! [13:58:38] greg-g: Just a heads-up on potential regressions: https://phabricator.wikimedia.org/T164049 / https://phabricator.wikimedia.org/T164068 (FlaggedRevs/RC regression) and https://phabricator.wikimedia.org/T164059 (Watchlist 503) make me a bit nervous today [14:04:04] I see this https://phabricator.wikimedia.org/rEFLRad612350b61786b00897f1dec44bdfac5c8ab349 which is related to both FlaggedRevs and Watchlist. [14:05:13] 10Browser-Tests-Infrastructure, 07JavaScript, 15User-zeljkofilipin: WebdriverIO CREDIT showcase - https://phabricator.wikimedia.org/T164070#3220827 (10zeljkofilipin) Applied for CREDIT showcase: https://etherpad.wikimedia.org/p/CREDIT [14:06:44] True (and thanks!), could be related [14:07:39] Your welcome :), i've triaged one as high priority as someone else confirmed it. I am unsure about the second one. [14:09:02] it's only one person affected so far. [14:09:09] we'll see if there are more issues. [14:09:40] Yep. [14:30:04] Yippee, build fixed! [14:30:05] Project language-screenshots-VisualEditor ยป chrome,Windows 10,ci-jessie-wikimedia build #57: 09FIXED in 20 min: https://integration.wikimedia.org/ci/job/language-screenshots-VisualEditor/BROWSER=chrome,PLATFORM=Windows%2010,label=ci-jessie-wikimedia/57/ [14:31:02] 10Beta-Cluster-Infrastructure, 10media-storage, 13Patch-For-Review, 15User-fgiunchedi: Migrate beta cluster Swift cluster from Trusty to Jessie - https://phabricator.wikimedia.org/T162247#3220937 (10fgiunchedi) For reference, switching mw to talk to deployment-ms-fe02 the configuration is here: https://hor... [14:56:21] if we have monitors for ores beta, they will start screaming soon [14:56:53] that's me [15:06:33] RainbowSprinkles hi, remeber what we were talking about a few months ago about you wanting to be able to stop email notifications after merging since sometimes users have discussions on the change even after merging [15:06:45] Yes [15:07:04] RainbowSprinkles https://gerrit-review.googlesource.com/#/c/104938/ [15:07:10] is that what you wanted? [15:07:16] It adds an ignore and unignore button [15:07:57] Yep [15:08:02] :) [15:29:52] RECOVERY - Puppet errors on deployment-sca03 is OK: OK: Less than 1.00% above the threshold [0.0] [15:30:28] greg-g: urgh. and potentially https://phabricator.wikimedia.org/T164045 (no thumbnails for recent pdf files). sigh [15:43:33] andre__: If you're looking at PDF bugs, look for the ones that have been filed in #logspam recently with the words SyntaxError [15:44:00] eg T157644 [15:44:01] T157644: Spikes in "Syntax Error: Top-level pages object is wrong type (null)" and "Syntax Error: Invalid XRef entry" errors in production - https://phabricator.wikimedia.org/T157644 [15:46:22] uhm, wasn't aware of them, thanks [15:59:11] 06Release-Engineering-Team, 06Operations, 10Phabricator, 10ops-eqiad, 13Patch-For-Review: setup/install phab1001.eqiad.wmnet - https://phabricator.wikimedia.org/T163938#3221288 (10Cmjohnson) [16:01:06] 06Release-Engineering-Team, 10Wikidata, 07Story: [Story] Use composer-merge-plugin to include Wikidata components in mediawiki-vendor - https://phabricator.wikimedia.org/T95663#3221292 (10Reedy) Can't see much reason that you can't just do that... [16:12:15] 06Release-Engineering-Team, 06Operations, 05Goal, 13Patch-For-Review, and 3 others: Prepare and maintain base container images - https://phabricator.wikimedia.org/T162042#3150656 (10mobrovac) >>! In T162042#3220775, @MoritzMuehlenhoff wrote: > I don't think we need a trusty image. The remaining services ru... [16:14:26] PROBLEM - Long lived cherry-picks on puppetmaster on deployment-puppetmaster02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [16:33:13] 06Release-Engineering-Team, 06Operations, 10Phabricator, 10ops-eqiad, 13Patch-For-Review: setup/install phab1001.eqiad.wmnet - https://phabricator.wikimedia.org/T163938#3221494 (10Cmjohnson) [16:33:29] 06Release-Engineering-Team, 06Operations, 10Phabricator, 13Patch-For-Review: setup/install phab1001.eqiad.wmnet - https://phabricator.wikimedia.org/T163938#3215445 (10Cmjohnson) [16:42:51] 10Gerrit, 10Developer-Wishlist (2017), 07Documentation, 07Upstream: Enable and document "WIP" workflow status in Gerrit - https://phabricator.wikimedia.org/T135245#3221521 (10Paladox) Upstream have merged some changes that add support for WIP status. This will be in gerrit 2.15.See https://gerrit-review.go... [16:51:41] 10Beta-Cluster-Infrastructure, 10Edit-Review-Improvements-RC-Page, 10ORES, 06Revision-Scoring-As-A-Service, and 4 others: ORES errors on beta sites (dewp; zhwp) - https://phabricator.wikimedia.org/T163873#3221530 (10Etonkovidova) Checked in betalabs - dewiki, zhwiki, cawiki - Contributions and Preferences... [16:56:23] Heh theres also a mute button now RainbowSprinkles [16:56:44] that does what ignore does except it will still send emails though but removes it from your dashbored. [17:05:39] PROBLEM - Puppet errors on buildlog is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [17:05:45] PROBLEM - Puppet errors on swift is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [17:06:43] PROBLEM - Puppet errors on deployment-urldownloader is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [17:07:17] PROBLEM - Puppet errors on swift-storage-01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [17:07:23] PROBLEM - Puppet errors on deployment-ircd is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [17:08:46] PROBLEM - Puppet errors on deployment-phab01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [17:16:47] Amir1, what's going on in beta? [17:17:11] halfak: the thing is the puppet configurations on horizon can't find ores in production [17:17:26] I don't know what's going on there but I will get it done soon [17:37:18] Weird. [17:37:20] OK. [17:37:28] (sorry had audio off. I missed the ping.) [21:20:21] (03PS1) 10Umherirrender: [MultiBoilerplate] Add npm job [integration/config] - 10https://gerrit.wikimedia.org/r/350911 [21:58:49] (03PS1) 10Jdlrobson: Add experimental Node selenium job [integration/config] - 10https://gerrit.wikimedia.org/r/350941 (https://phabricator.wikimedia.org/T164024) [22:05:07] (03CR) 10jerkins-bot: [V: 04-1] Add experimental Node selenium job [integration/config] - 10https://gerrit.wikimedia.org/r/350941 (https://phabricator.wikimedia.org/T164024) (owner: 10Jdlrobson)