[00:19:02] 10Diffusion, 10Repository-Ownership-Requests, 10Research, 10Research-landing-page: Mirror Github repo to Diffusion - https://phabricator.wikimedia.org/T188887#4022729 (10bmansurov) [00:31:16] PROBLEM - Long lived cherry-picks on puppetmaster on deployment-puppetmaster02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [01:24:39] 10Beta-Cluster-Infrastructure: Requesting PCR on enwiki beta - https://phabricator.wikimedia.org/T188873#4022412 (10Reedy) https://en.wikipedia.org/wiki/Special:UserRights/sau226 ``` (change visibility) 09:25, 30 January 2018 Widr (talk | contribs | block) changed group membership for Sau226 from extended confi... [01:45:23] PROBLEM - App Server Main HTTP Response on deployment-mediawiki06 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:45:31] PROBLEM - App Server Main HTTP Response on deployment-mediawiki04 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:45:55] PROBLEM - English Wikipedia Mobile Main page on beta-cluster is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:46:19] PROBLEM - English Wikipedia Main page on beta-cluster is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:50:20] RECOVERY - App Server Main HTTP Response on deployment-mediawiki06 is OK: HTTP OK: HTTP/1.1 200 OK - 47174 bytes in 5.472 second response time [01:50:24] RECOVERY - App Server Main HTTP Response on deployment-mediawiki04 is OK: HTTP OK: HTTP/1.1 200 OK - 47134 bytes in 3.752 second response time [01:50:48] RECOVERY - English Wikipedia Mobile Main page on beta-cluster is OK: HTTP OK: HTTP/1.1 200 OK - 36268 bytes in 3.460 second response time [01:51:12] RECOVERY - English Wikipedia Main page on beta-cluster is OK: HTTP OK: HTTP/1.1 200 OK - 47760 bytes in 3.610 second response time [03:16:23] 10Beta-Cluster-Infrastructure: Requesting Pending Changes Reviewer on enwiki beta - https://phabricator.wikimedia.org/T188873#4022796 (10Sau226) [04:43:06] 10Diffusion, 10Repository-Admins, 10Research, 10Research-landing-page: Mirror Github repo to Diffusion - https://phabricator.wikimedia.org/T188887#4022816 (10Legoktm) #repository-ownership-requests is for Gerrit, so I'm going to remove that project. + #repository-admins since that's for Diffusion. [04:44:26] 10Phabricator: Phame blog posts don't have 'published' metadata, only 'updated' - https://phabricator.wikimedia.org/T188890#4022822 (10Samwilson) [04:49:31] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [10.0] [05:14:31] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [10.0] [05:20:21] 10Diffusion, 10Repository-Admins, 10Research, 10Research-landing-page: Mirror Github repo to Diffusion - https://phabricator.wikimedia.org/T188887#4022845 (10bmansurov) Thanks, @Legoktm. [05:50:48] (03CR) 10Legoktm: "Awesome :D Would it be possible to add a warning if the identifier is deprecated?" [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/416337 (owner: 10Umherirrender) [06:14:49] !log legoktm@integration-slave-jessie-1001:/srv/jenkins-workspace/workspace$ sudo rm -rf * [06:14:52] paladox: ^ [06:14:56] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [06:27:24] RECOVERY - Free space - all mounts on integration-slave-jessie-1001 is OK: OK: integration.integration-slave-jessie-1001.diskspace._mnt.byte_percentfree (No valid datapoints found) [07:05:41] 10Gerrit, 10Release-Engineering-Team (Next), 10DBA, 10Operations, 10Patch-For-Review: Gerrit is failing to connect to db on gerrit2001 thus preventing systemd from working - https://phabricator.wikimedia.org/T176532#3628916 (10elukey) Just added a week of downtime to gerrit2001 since icinga was spamming. [07:38:20] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10Release Pipeline, 10Patch-For-Review: On CI, upgrade docker-ce from 17.06.2 to 17.12.1 - https://phabricator.wikimedia.org/T177499#4022931 (10hashar) Magic! Thanks @akosiaris. [07:45:11] Project mwext-phpunit-coverage-publish build #1764: 04FAILURE in 27 sec: https://integration.wikimedia.org/ci/job/mwext-phpunit-coverage-publish/1764/ [08:05:23] Yippee, build fixed! [08:05:24] Project mwext-phpunit-coverage-publish build #1765: 09FIXED in 1 min 1 sec: https://integration.wikimedia.org/ci/job/mwext-phpunit-coverage-publish/1765/ [08:17:48] 10Release-Engineering-Team (Watching / External), 10Analytics-Kanban, 10Operations, 10Patch-For-Review, and 2 others: Deprecation of mw.errors.* metrics - https://phabricator.wikimedia.org/T188749#4022964 (10elukey) [08:34:24] 10Gerrit, 10translatewiki.net: Mark as read-only the gerrit copies of GitHub-hosted repos that are only open for i18n now that's done there - https://phabricator.wikimedia.org/T131718#4022992 (10Nikerabbit) What's the status of this task? Was this done? [09:01:22] Thanks legoktm :) [09:20:59] (03CR) 10Chad: Grant Edit hashtag right to everyone. (031 comment) [All-Projects] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/416357 (owner: 10Paladox) [09:30:50] 10Gerrit, 10translatewiki.net: Mark as read-only the gerrit copies of GitHub-hosted repos that are only open for i18n now that's done there - https://phabricator.wikimedia.org/T131718#4023118 (10demon) 05Open>03Resolved a:03demon Looks like it. All the repos are read-only except WhatsNearby, which looks... [09:31:34] 10Release-Engineering-Team (Watching / External), 10Operations, 10Patch-For-Review: Provide cross-dc redundancy (active-active or active-passive) to all important misc services - https://phabricator.wikimedia.org/T156937#4023123 (10ArielGlenn) [09:33:44] PROBLEM - Free space - all mounts on integration-slave-jessie-1004 is CRITICAL: CRITICAL: integration.integration-slave-jessie-1004.diskspace._srv.byte_percentfree (<30.00%) [09:55:01] 10Phabricator (Upstream), 10Upstream: Phame blog posts don't have 'published' metadata, only 'updated' - https://phabricator.wikimedia.org/T188890#4023172 (10Peachey88) [10:03:41] RECOVERY - Free space - all mounts on integration-slave-jessie-1004 is OK: OK: All targets OK [10:14:54] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Find a few people interested in reviewing Selenium patches - https://phabricator.wikimedia.org/T188744#4018049 (10Jdrewniak) Feel free to add me to any selenium patches as well 👍 [10:54:08] PROBLEM - Host deployment-videoscaler01 is DOWN: CRITICAL - Host Unreachable (10.68.19.130) [10:54:52] PROBLEM - Host deployment-tmh01 is DOWN: CRITICAL - Host Unreachable (10.68.16.211) [11:30:39] 10Diffusion, 10Repository-Admins, 10Research, 10Research-landing-page: Mirror Github repo to Diffusion - https://phabricator.wikimedia.org/T188887#4023319 (10MarcoAurelio) 05Open>03Resolved p:05Triage>03Normal {rWRES} created mirroring changes from https://github.com/wikimedia/wikimedia-research. [12:30:54] PROBLEM - Puppet errors on deployment-ores01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [13:27:44] (03PS2) 10Paladox: Grant Edit hashtag right to everyone. [All-Projects] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/416357 [13:27:55] (03CR) 10Paladox: Grant Edit hashtag right to everyone. (031 comment) [All-Projects] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/416357 (owner: 10Paladox) [13:32:38] (03CR) 10Thiemo Kreuz (WMDE): [C: 031] "The WikibaseQualityExternalValidation extension is not deployed anywhere, and there are no plans to invest any time on it. Please disable " [integration/config] - 10https://gerrit.wikimedia.org/r/414774 (https://phabricator.wikimedia.org/T185697) (owner: 10Lucas Werkmeister (WMDE)) [13:36:29] 10Beta-Cluster-Infrastructure, 10Operations: Beta cluster Obama page often responds with 503 - https://phabricator.wikimedia.org/T188913#4023646 (10Aklapper) [13:45:30] RECOVERY - Mediawiki Error Rate on graphite-labs is OK: OK: Less than 1.00% above the threshold [1.0] [14:05:47] 10Scap, 10Navigation-Popups: scap fails on .eslintrc.json: invalid JSON - https://phabricator.wikimedia.org/T188914#4023698 (10hashar) [14:23:05] 10Release-Engineering-Team (Watching / External), 10Analytics-Kanban, 10Operations, 10Patch-For-Review, and 2 others: Deprecation of mw.errors.* metrics - https://phabricator.wikimedia.org/T188749#4023747 (10fgiunchedi) >>! In T188749#4021055, @elukey wrote: > Thanks @Krinkle! @fgiunchedi I think we are re... [14:29:49] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Find a few people interested in reviewing Selenium patches - https://phabricator.wikimedia.org/T188744#4023764 (10zeljkofilipin) Thanks @WMDE-Fisch, @Ladsgroup and @Jdrewniak! 🎉 At the moment I am blocked on T185094/[[ https://gerrit.wikimedia.org/r/... [14:32:06] 10Scap, 10Navigation-Popups: scap fails on .eslintrc.json: invalid JSON - https://phabricator.wikimedia.org/T188914#4023766 (10hashar) [14:35:01] 10Release-Engineering-Team (Watching / External), 10Analytics-Kanban, 10Operations, 10Patch-For-Review, and 2 others: Deprecation of mw.errors.* metrics - https://phabricator.wikimedia.org/T188749#4023773 (10elukey) [14:36:25] 10Release-Engineering-Team (Kanban), 10MediaWiki-Core-Tests, 10Patch-For-Review, 10User-zeljkofilipin: Update page object pattern in Selenium tests - https://phabricator.wikimedia.org/T185094#4023776 (10zeljkofilipin) [14:41:34] 10Release-Engineering-Team (Kanban), 10MediaWiki-Core-Tests, 10Patch-For-Review, 10User-zeljkofilipin: Update page object pattern in Selenium tests - https://phabricator.wikimedia.org/T185094#4023786 (10zeljkofilipin) [14:45:05] 10Release-Engineering-Team (Kanban), 10MediaWiki-Core-Tests, 10Patch-For-Review, 10User-zeljkofilipin: Update page object pattern in Selenium tests - https://phabricator.wikimedia.org/T185094#4023794 (10zeljkofilipin) @WMDE-Fisch, @Ladsgroup and @Jdrewniak: I have updated task description. Any help is appr... [14:47:38] 10Release-Engineering-Team (Kanban), 10MediaWiki-Core-Tests, 10Patch-For-Review, 10User-zeljkofilipin: Update page object pattern in Selenium tests - https://phabricator.wikimedia.org/T185094#4023802 (10zeljkofilipin) [14:53:15] PROBLEM - Host deployment-puppetdb01 is DOWN: CRITICAL - Host Unreachable (10.68.23.76) [15:36:35] Hm… looks to me like mwscript still uses php5 so I need it to run maintenance scripts even on hhvm systems… does that sound right? [15:37:36] Unfortunately, yes [15:37:46] I think no_justification was clearing some of that up [15:38:01] andrewbogott: you can, of course, just symlink php5 somewhere else [15:38:37] since there aren't php5 packages on Stretch… any thought on what the 'correct' solution is there? [15:39:49] symlink php5 to php7.0 or php [15:39:50] andrewbogott: https://gerrit.wikimedia.org/r/#/c/358896/ [15:40:15] or export PHP=php [15:41:28] Reedy: that looks promising [16:31:43] Project beta-scap-eqiad build #198326: 04FAILURE in 18 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/198326/ [16:32:49] PROBLEM - Free space - all mounts on deployment-mediawiki04 is CRITICAL: CRITICAL: deployment-prep.deployment-mediawiki04.diskspace.root.byte_percentfree (<55.56%) [16:33:17] PROBLEM - Puppet errors on deployment-logstash2 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [16:34:22] Well this is fun [16:35:52] !log removed 2G temp folder from /srv/mediawiki/php-master/cache/l10n/upstream [16:35:57] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:37:02] !log that was from deployment-mediawiki04 [16:37:08] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:37:16] !log removed pre 2018 hhvm error logs from deployment-mediawiki04 [16:37:22] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:38:22] !log deleted the stack traces too [16:38:28] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:38:47] 2.6G free [16:39:07] The last Puppet run was at Thu Jan 18 15:59:14 UTC 2018 (66279 minutes ago). [16:39:17] Anyone know why puppet hasn't run on deployment-logstash2? [16:40:26] (03PS1) 10Hashar: Remove tools/bundler submodule [integration/jenkins] - 10https://gerrit.wikimedia.org/r/416469 [16:40:50] zeljkof: easy +2 ^^ :] [16:40:54] Yippee, build fixed! [16:40:54] Project beta-scap-eqiad build #198327: 09FIXED in 8 min 20 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/198327/ [16:41:25] (03PS2) 10Hashar: Remove tools/bundler submodule [integration/jenkins] - 10https://gerrit.wikimedia.org/r/416469 [16:42:09] hashar: looking :) [16:42:32] that get rid of some old bundler snapshot :) [16:42:53] /dev/mapper/vd-second--local--disk on /var/lib/elasticsearch type ext4 (rw,relatime,data=ordered) [16:42:53] /dev/mapper/vd-second--local--disk on /srv type ext4 (rw,relatime,data=ordered) [16:42:54] weird [16:43:07] hehe [16:43:17] (03PS3) 10Zfilipin: Remove tools/bundler submodule [integration/jenkins] - 10https://gerrit.wikimedia.org/r/416469 (owner: 10Hashar) [16:43:23] Error: Could not retrieve catalog from remote server: Error 500 on SERVER: Server Error: Evaluation Error: Error while evaluating a Resource Statement, Evaluation Error: Error while evaluating a Function Call, undefined method `[]' for nil:NilClass at /etc/puppet/modules/role/manifests/logstash/eventlogging.pp:11:21 on node deployment-logstash2.deployment-prep.eqiad.wmflabs [16:43:24] (03CR) 10Zfilipin: [C: 032] Remove tools/bundler submodule [integration/jenkins] - 10https://gerrit.wikimedia.org/r/416469 (owner: 10Hashar) [16:43:27] Reedy: conflicts in Puppet resources I guess [16:43:46] hashar: done [16:43:52] zeljkof: danke [16:44:02] bitte :) [16:44:06] (03Merged) 10jenkins-bot: Remove tools/bundler submodule [integration/jenkins] - 10https://gerrit.wikimedia.org/r/416469 (owner: 10Hashar) [16:45:02] 10Beta-Cluster-Infrastructure: Puppet broken ondeployment-logstash2.deployment-prep.eqiad.wmflabs - https://phabricator.wikimedia.org/T188921#4024103 (10Reedy) [16:47:48] RECOVERY - Free space - all mounts on deployment-mediawiki04 is OK: OK: All targets OK [16:49:16] (03CR) 10Hashar: "Yup. I haven't started looking at that beast yet, I am more leaning toward a single container that is able to match most/all uses cases. E" [integration/config] - 10https://gerrit.wikimedia.org/r/416213 (owner: 10Legoktm) [16:52:54] (03CR) 10Hashar: [C: 04-1] "The reason for the php5 is that Zend php 5.5 is too slow so we only run those jobs on gate-and-submit." [integration/config] - 10https://gerrit.wikimedia.org/r/313213 (https://phabricator.wikimedia.org/T144872) (owner: 10Paladox) [16:53:17] (03Abandoned) 10Paladox: Add php7 pipeline for zuul [integration/config] - 10https://gerrit.wikimedia.org/r/313213 (https://phabricator.wikimedia.org/T144872) (owner: 10Paladox) [16:54:31] (03CR) 10Hashar: "The job has been migrated to Docker. It merges the job template phpunit-coverage-docker-publish ( releng/php:0.1.1 ) but has a 90 minutes " [integration/config] - 10https://gerrit.wikimedia.org/r/386679 (https://phabricator.wikimedia.org/T179055) (owner: 10Legoktm) [16:55:56] (03Abandoned) 10Hashar: pywikibot-tests-beta-cluster pass PYWIKIBOT2_DIR to tox [integration/config] - 10https://gerrit.wikimedia.org/r/388091 (owner: 10Hashar) [16:56:58] (03Abandoned) 10Hashar: Install postgresql on ci-image-jessie [integration/config] - 10https://gerrit.wikimedia.org/r/316475 (https://phabricator.wikimedia.org/T39602) (owner: 10Paladox) [16:59:50] 10Continuous-Integration-Config, 10Release-Engineering-Team, 10GitHub-Mirrors, 10Repository-Admins: Set up CI and github sync for new extra-analysis repo - https://phabricator.wikimedia.org/T188686#4024154 (10greg) [17:02:13] 10Continuous-Integration-Config, 10Release-Engineering-Team, 10GitHub-Mirrors, 10Repository-Admins: Set up CI and github sync for new extra-analysis repo - https://phabricator.wikimedia.org/T188686#4016268 (10Reedy) Empty https://github.com/wikimedia/search-extra-analysis created [17:15:46] 10Release-Engineering-Team (Watching / External), 10Operations, 10ops-eqiad, 10Patch-For-Review: setup/install/deploy deploy1001 as deployment server - https://phabricator.wikimedia.org/T175288#4024193 (10Dzahn) installer is in install loop, fails at partioner (like bast1002 T186623#4024188) [17:22:16] 10Continuous-Integration-Config, 10Release-Engineering-Team (Kanban), 10MediaWiki-Core-Tests, 10Patch-For-Review, 10User-zeljkofilipin: Use a more complex password for WikiAdmin in selenium tests - https://phabricator.wikimedia.org/T188520#4010763 (10zeljkofilipin) This is on hold until @hashar has the t... [17:25:36] <_joe_> how can I figure out which branches I have to backport a patch from mediawiki/core master to, so that current and future-deployed branches have it? [17:26:41] https://noc.wikimedia.org/conf/ [17:26:45] Currently active MediaWiki versions: 1.31.0-wmf.23 [17:27:04] https://tools.wmflabs.org/versions/ [17:27:12] As for future branches.... It's best that you get it merged into master [17:27:12] that's a prettier view of the data in noc [17:27:15] 10Beta-Cluster-Infrastructure, 10Wikidata: Stack overflow in WikibaseRepo initialization on Wikidata Beta - https://phabricator.wikimedia.org/T188924#4024243 (10Lucas_Werkmeister_WMDE) [17:27:21] So mean, bd808 [17:28:18] I have honestly thought about making noc prettier, and then realized that nobody but me would care ;) [17:28:53] <_joe_> Reedy: it's already merged into master [17:28:59] So... [17:29:09] At worst you need to cherry pick it to .23 if you want it now [17:29:17] It'll automagically be in .24 as that won't have been cut [17:29:29] <_joe_> Reedy: that's already done, I wasn't sure if .24 or .25 were cut [17:29:33] <_joe_> cool, thanks [17:30:07] .24 probably won't be cut till a few hours before the window tomorrow [17:34:18] <_joe_> ok, thanks [17:35:40] 10Diffusion, 10Repository-Admins, 10Research, 10Research-landing-page: Mirror Github repo to Diffusion - https://phabricator.wikimedia.org/T188887#4024300 (10MarcoAurelio) Please note that I've not done anything with the Gerrit repo so in fact you have two. The GitHub one which Diffusion will mirror, and t... [18:01:25] 10Phabricator (Upstream), 10Upstream: Project tags can not be found via the fulltext search index - https://phabricator.wikimedia.org/T92630#4024419 (10mmodell) [18:20:15] 10Release-Engineering-Team (Kanban), 10Release Pipeline: install kubectl on integration agents - https://phabricator.wikimedia.org/T188933#4024498 (10thcipriani) [18:21:01] 10Continuous-Integration-Infrastructure, 10Wikidata, 10Wikidata Query UI, 10Jenkins, 10Patch-For-Review: wikidata/query/gui CI job lacks PhantomJS / proper browsers - https://phabricator.wikimedia.org/T183831#4024508 (10Lydia_Pintscher) [18:21:26] 10Release-Engineering-Team (Kanban), 10Release Pipeline: Verify functionality of the 'production' image in the context of an isolated k8s deployment - https://phabricator.wikimedia.org/T183165#4024512 (10dduvall) [18:21:28] 10Release-Engineering-Team (Kanban), 10Release Pipeline: install kubectl on integration agents - https://phabricator.wikimedia.org/T188933#4024511 (10dduvall) [18:21:30] 10Release-Engineering-Team (Kanban), 10Release Pipeline: install helm on integration agents - https://phabricator.wikimedia.org/T188934#4024513 (10thcipriani) [18:21:48] 10Release-Engineering-Team (Kanban), 10Release Pipeline: Verify functionality of the 'production' image in the context of an isolated k8s deployment - https://phabricator.wikimedia.org/T183165#3845669 (10dduvall) [18:21:53] 10Release-Engineering-Team (Kanban), 10Release Pipeline: install helm on integration agents - https://phabricator.wikimedia.org/T188934#4024525 (10dduvall) [18:26:37] (03PS1) 10MarcoAurelio: Initial configuration. [extensions/NamespaceSidebar] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/416485 [18:26:42] PROBLEM - Free space - all mounts on deployment-tin is CRITICAL: CRITICAL: deployment-prep.deployment-tin.diskspace._mnt.byte_percentfree (No valid datapoints found)deployment-prep.deployment-tin.diskspace._srv.byte_percentfree (<10.00%) [18:26:52] (03CR) 10MarcoAurelio: [V: 032 C: 032] Initial configuration. [extensions/NamespaceSidebar] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/416485 (owner: 10MarcoAurelio) [18:27:19] can somebody submit https://gerrit.wikimedia.org/r/#/c/416485/ for me? [18:27:25] don't have enough privs [18:30:48] Reedy: ^^ [18:31:46] I wonder if the submit rule on https://gerrit.wikimedia.org/r/#/admin/projects/mediawiki/extensions,access is overiding it from All-Projects [18:32:15] Hauskatze: Once I submit I won't be able to afterwards. :-) [18:32:43] James_F you should still be able to after :) [18:32:51] James_F: it inherits from mediawiki/extensions, you should be fine I guess [18:32:53] Hauskatze: Done. And the .gitreview one. [18:32:54] because mediawiki is applied from parents [18:32:58] Hauskatze: Ah, OK, in that case yay. [18:34:02] (03CR) 10Paladox: Initial configuration. (031 comment) [extensions/NamespaceSidebar] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/416485 (owner: 10MarcoAurelio) [18:34:37] PROBLEM - App Server Main HTTP Response on deployment-mediawiki07 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 hphp_invoke - string 'Wikipedia' not found on 'http://en.wikipedia.beta.wmflabs.org:80/wiki/Main_Page?debug=true' - 287 bytes in 0.007 second response time [18:38:15] paladox: recently I saw jenkins being unable to submit w/o that so I prefer a bit of redundancy just in case [18:38:23] it was not under mediawiki/extensions though [18:38:24] oh [18:39:14] and wrt to submit for gerrit managers, it doesn't work because Gerrit ignores it as precaution. The only owners of all repos are administrators. [18:41:43] track=1 doesn't seem to be needed either? [18:54:28] PROBLEM - Puppet errors on deployment-secureredirexperiment is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [19:29:13] do I have to add that to mediawiki/extensions? [19:29:27] NamespaceSidebar I mean [19:37:59] 10Continuous-Integration-Config, 10Release-Engineering-Team, 10GitHub-Mirrors, 10Repository-Admins: Set up CI and github sync for new extra-analysis repo - https://phabricator.wikimedia.org/T188686#4024739 (10TJones) Thanks, @Reedy! Does anything else need to be done for the github synchronization? [19:38:46] 10Continuous-Integration-Config, 10Release-Engineering-Team, 10GitHub-Mirrors, 10Repository-Admins: Set up CI and github sync for new extra-analysis repo - https://phabricator.wikimedia.org/T188686#4024741 (10Reedy) IIRC, just some activity in the source repo, and then gerrit will try and replicate it all [19:40:57] AndyRussG: I notice that in beta, CentralNotice has $wgCentralBannerRecorder = "//meta.wikimedia.beta.wmflabs.org/w/index.php?title=Special:RecordImpression"; [19:41:09] AndyRussG: Is that still needed, or could it match prod setting, which is /beacon/impression? [19:41:14] Or does that not yet work in beta? [19:41:22] Krinkle: one sec, in a meeting, done in a few [19:41:29] k :) [19:41:31] or rather 15 min about, I should say [19:41:33] thx! [19:45:54] 10Continuous-Integration-Config, 10Release-Engineering-Team, 10GitHub-Mirrors, 10Repository-Admins: Set up CI and github sync for new extra-analysis repo - https://phabricator.wikimedia.org/T188686#4024754 (10TJones) Cool. Hopefully I'll be committing soon. Thanks again! [19:57:05] greg-g: can I grab a deploy window 17-18 PST? I want to backport a bunch of changes in ReadingLists so that beta testing can start a week sooner [19:57:22] not exposed to users yet so it should be fairly low-risk [20:01:42] PROBLEM - Free space - all mounts on deployment-tin is CRITICAL: CRITICAL: deployment-prep.deployment-tin.diskspace._mnt.byte_percentfree (No valid datapoints found)deployment-prep.deployment-tin.diskspace._srv.byte_percentfree (<10.00%) [20:01:44] If I have a Mediawiki config change that I'd like to have deployed, is that something that RelEng would usually look at? If so, is there a specific person who I should tag on the review? [20:02:18] 10Beta-Cluster-Infrastructure, 10Wikidata: Stack overflow in WikibaseRepo initialization on Wikidata Beta - https://phabricator.wikimedia.org/T188924#4024813 (10Addshore) p:05Triage>03High [20:02:44] Project selenium-Wikibase-chrome » chrome,beta,Linux,DebianJessie && contintLabsSlave build #133: 04FAILURE in 15 min: https://integration.wikimedia.org/ci/job/selenium-Wikibase-chrome/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=DebianJessie%20&&%20contintLabsSlave/133/ [20:02:48] 10Beta-Cluster-Infrastructure, 10Wikidata: Stack overflow in WikibaseRepo initialization on Wikidata Beta - https://phabricator.wikimedia.org/T188924#4024243 (10Addshore) Marked as high so that we investigate this, in theory the cause would be in this weeks train. [20:05:31] marlier: if you don't specifically need review, you can just add it to a SWAT window: https://wikitech.wikimedia.org/wiki/SWAT_deploys [20:06:16] and then the person doing the SWAT deploy will review it (but ideally it should have a +1 by that point) [20:06:22] Cool, will do. Thanks! [20:07:51] 10Beta-Cluster-Infrastructure, 10Wikidata: Stack overflow in WikibaseRepo initialization on Wikidata Beta - https://phabricator.wikimedia.org/T188924#4024866 (10Addshore) I see the following in the logs after hitting https://wikidata.beta.wmflabs.org/wiki/Wikidata:Main_Page a few times ``` Fatal error: Stack... [20:08:17] 10Beta-Cluster-Infrastructure, 10Wikidata: Stack overflow in WikibaseRepo initialization on Wikidata Beta - https://phabricator.wikimedia.org/T188924#4024868 (10Addshore) [20:08:19] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments: 1.31.0-wmf.24 deployment blockers - https://phabricator.wikimedia.org/T183963#4024867 (10Addshore) [20:11:34] Krinkle: hey! good question... I don't know if we have any beacon endpoints on the beta cluster [20:12:48] I'd not owrry too much about it, I guess? I mean, with the pending move to EventLogging...? [20:14:50] PROBLEM - Free space - all mounts on deployment-mediawiki04 is CRITICAL: CRITICAL: deployment-prep.deployment-mediawiki04.diskspace.root.byte_percentfree (<11.11%) [20:19:12] (03CR) 10Krinkle: "composer.json enforces PHP 5.5, not 5.6, but that's fine given this method was added in PHP 5.4.0. We should consider making the requireme" [integration/docroot] - 10https://gerrit.wikimedia.org/r/414216 (owner: 10Legoktm) [20:24:47] RECOVERY - Free space - all mounts on deployment-mediawiki04 is OK: OK: All targets OK [20:25:24] AndyRussG: OK. Question is though, is the current setting override for beta useful to you? [20:25:38] E.g. I suspect you probably don't consume the data from beta in any meaningful way, only the frontend testing? [20:25:56] But in that case, neither might work, or maybe the special page does work currently in beta? [20:26:26] AndyRussG: There's an easier bit of clean up that I suspect will be easier to review first - https://gerrit.wikimedia.org/r/#/c/416500/ [20:26:44] I can do the second bit at a later point, and we can test together and/or let me know if the special page does anything right now in beta for you. [20:26:50] If not, then we can drop the override I suppose? [20:41:51] tgr: sure [20:57:00] Krinkle: cool, thanks! I'll check it out. Yeah, on beta we don't usually consume the data from that pipeline, just on the client sometimes we check that the impression recording call sent correctly... [21:03:26] 10Deployments: Make Jenkins upload wmfXX branch release notes to mw.org or doc.wikimedia.org - https://phabricator.wikimedia.org/T64331#4025041 (10Framawiki) [21:06:41] 10Deployments, 10Jouncebot, 10Tools, 10Patch-For-Review, 10User-bd808: Jouncebot not joining #wikimedia-operations - https://phabricator.wikimedia.org/T144189#4025045 (10Framawiki) [21:07:28] 10Continuous-Integration-Infrastructure, 10Jouncebot, 10Patch-For-Review, 10User-bd808: Status of Jouncebot and dropping the yamllint Jenkins job - https://phabricator.wikimedia.org/T95894#4025050 (10Framawiki) [21:07:45] 10Deployments, 10Cloud-Services, 10Jouncebot: Add release engineering people to tools.jouncebot user group - https://phabricator.wikimedia.org/T144175#4025052 (10Framawiki) [21:08:00] 10Release-Engineering-Team, 10Jouncebot, 10Tools, 10Patch-For-Review, 10User-Zppix: Jouncebot: Add functionality to change Nick from Jouncebot_ to Jouncebot automatically - https://phabricator.wikimedia.org/T150916#4025053 (10Framawiki) [21:08:18] 10Release-Engineering-Team, 10DBA, 10Jouncebot, 10MediaWiki-Maintenance-scripts, and 3 others: Add section for long-running tasks on the Deployment page (specially for database maintenance) - https://phabricator.wikimedia.org/T144661#4025055 (10Framawiki) [21:08:20] 10Deployments, 10Jouncebot, 10Patch-For-Review: Jouncebot should not mix SWATters and patch owners - https://phabricator.wikimedia.org/T101329#4025056 (10Framawiki) [21:08:21] PROBLEM - Puppet errors on deployment-mx02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [21:15:44] 10Diffusion, 10Repository-Admins, 10Research, 10Research-landing-page: Mirror Github repo to Diffusion - https://phabricator.wikimedia.org/T188887#4025079 (10bmansurov) Thanks, @MarcoAurelio! [21:15:49] Krinkle: so, nothing specific to the special page... I'll check out the patch :) [21:16:18] AndyRussG: Right, the varnish log processing we used to have for the speical page is no longer live in prod, and presumably not in beta either. [21:25:43] 10Release-Engineering-Team (Kanban), 10Release Pipeline (Blubber): Blubber should implement a better file permissions convention - https://phabricator.wikimedia.org/T187372#4025091 (10dduvall) 05Open>03Resolved [21:29:30] Krinkle: Ah hmmmm certainly we should make sure we're not consuming unneeded resources in beta [21:29:57] There are browser tests running on beta [21:50:26] Project mwext-phpunit-coverage-publish build #1798: 04FAILURE in 34 sec: https://integration.wikimedia.org/ci/job/mwext-phpunit-coverage-publish/1798/ [21:51:07] Yippee, build fixed! [21:51:07] Project mwext-phpunit-coverage-publish build #1799: 09FIXED in 40 sec: https://integration.wikimedia.org/ci/job/mwext-phpunit-coverage-publish/1799/ [22:47:17] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team: Zuul Clone frequently fails in php70-phan-docker job - https://phabricator.wikimedia.org/T188958#4025305 (10Krinkle) [22:49:43] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team: Zuul Clone frequently fails in php70-phan-docker job - https://phabricator.wikimedia.org/T188958#4025320 (10Krinkle) [23:15:30] 10Release-Engineering-Team, 10MediaWiki-extensions-Other: Add mediawiki/extensions/NamespaceSidebar to mediawiki/extensions - https://phabricator.wikimedia.org/T188940#4025385 (10MarcoAurelio) [23:19:32] 10Beta-Cluster-Infrastructure: Puppet broken on deployment-logstash2.deployment-prep.eqiad.wmflabs - https://phabricator.wikimedia.org/T188921#4025390 (10MarcoAurelio) [23:30:00] 10Release-Engineering-Team, 10DNS, 10Operations, 10Traffic, and 2 others: Move Foundation Wiki to new URL when new Wikimedia Foundation website launches - https://phabricator.wikimedia.org/T188776#4019041 (10Platonides) Please remember to 301 /wiki/.* to the new url…