[00:30:07] (03PS1) 10Awight: Enable code health reports for ext-Cite [integration/config] - 10https://gerrit.wikimedia.org/r/554391 [01:04:45] PROBLEM - Host deployment-logstash2 is DOWN: CRITICAL - Host Unreachable (172.16.5.22) [02:49:54] I'm trying to re-understand how the train works… can someone tell me when https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/OpenStackManager/+/549214/ is likely to be deployed to wikitech? (I see that group1 is updated tomorrow but I don't think I know what group wikitech is in) [03:16:45] 10Release-Engineering-Team, 10serviceops, 10Patch-For-Review: switch prod Phabricator from phab1003 to phab1001 - https://phabricator.wikimedia.org/T238956 (10mmodell) Phab1001 disk I/O seems a lot slower than phab1003. Running `lshw -class storage` yields one obvious difference: phab1001 is running in lega... [03:22:09] 10Release-Engineering-Team, 10serviceops, 10Patch-For-Review: switch prod Phabricator from phab1003 to phab1001 - https://phabricator.wikimedia.org/T238956 (10mmodell) Slow disk is manifesting with tasks blocked for extended periods of time waiting for I/O: `name=/var/log/kern.log Dec 4 02:08:53 phab1001... [03:42:11] 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO, 10Operations, 10Jenkins: Add latest jenkins debian packages to apt.wikimedia.org and upgrade jenkins to latest LTS (2.190.3) - https://phabricator.wikimedia.org/T239586 (10colewhite) p:05Triage→03Normal [04:55:04] 10Release-Engineering-Team, 10serviceops, 10Patch-For-Review: switch prod Phabricator from phab1003 to phab1001 - https://phabricator.wikimedia.org/T238956 (10Dzahn) > @Dzahn: Can we switch it over to AHCI in the bios? Do we need DC-Ops for that? @mmodell I did reboot into BIOS and looked at it but when swi... [05:05:01] 10Release-Engineering-Team, 10serviceops, 10Patch-For-Review: switch prod Phabricator from phab1003 to phab1001 - https://phabricator.wikimedia.org/T238956 (10Dzahn) [05:58:18] 10Release-Engineering-Team, 10serviceops, 10Patch-For-Review: switch prod Phabricator from phab1003 to phab1001 - https://phabricator.wikimedia.org/T238956 (10Mholloway) https://integration.wikimedia.org/ci/job/mobileapps-periodic-test/ has been failing since this happened. I'm guessing that is not a coinci... [06:25:45] PROBLEM - Puppet staleness on deployment-kafka-main-1 is CRITICAL: (Service Check Timed Out) [06:25:48] PROBLEM - Puppet staleness on deployment-restbase01 is CRITICAL: (Service Check Timed Out) [06:25:50] PROBLEM - Puppet errors on deployment-eventlog05 is CRITICAL: (Service Check Timed Out) [06:25:50] PROBLEM - Puppet staleness on integration-agent-docker-1013 is CRITICAL: (Service Check Timed Out) [08:15:54] 10Release-Engineering-Team (Unit & Int & System Tooling), 10Gerrit-Privilege-Requests, 10Operations, 10Wikidata, 10Wikidata-Query-Service: Push rights on https://gerrit.wikimedia.org/r/admin/projects/wikidata/query/blazegraph for onimisionipe - https://phabricator.wikimedia.org/T238733 (10hashar) We have... [08:18:22] 10Phabricator: Grant hashar ability to delete comments in Phabricator - https://phabricator.wikimedia.org/T239479 (10hashar) Thank you @mmodell [08:19:53] PROBLEM - English Wikipedia Mobile Main page on beta-cluster is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Backend fetch failed - string 'Wikipedia' not found on 'https://en.m.wikipedia.beta.wmflabs.org:443/wiki/Main_Page?debug=true' - 2555 bytes in 2.201 second response time [08:21:22] PROBLEM - App Server Main HTTP Response on deployment-mediawiki-09 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:24:49] RECOVERY - English Wikipedia Mobile Main page on beta-cluster is OK: HTTP OK: HTTP/1.1 200 OK - 37078 bytes in 0.483 second response time [08:33:16] 10Gerrit, 10Operations, 10vm-requests: Gerrit VM to test data migration - https://phabricator.wikimedia.org/T239151 (10hashar) The way I understand the message: the virtualization servers in group `row_A` lack free memory to allocate a VM. But maybe another group would have memory available? You should be... [08:33:59] (03CR) 10Thiemo Kreuz (WMDE): [C: 03+1] Enable code health reports for ext-Cite [integration/config] - 10https://gerrit.wikimedia.org/r/554391 (owner: 10Awight) [08:47:25] 10Diffusion, 10Phabricator: Viewing MediaWiki repository in diffusion results in an Unhandled Exception ("CommandException") - https://phabricator.wikimedia.org/T239786 (10Mainframe98) [08:56:41] 10Gerrit, 10Operations, 10vm-requests: Gerrit VM to test data migration - https://phabricator.wikimedia.org/T239151 (10MoritzMuehlenhoff) The old puppetdb hosts (puppetdb1001) should be ready to go away, @jbond merged the patches to stop broadcasting to it last week. It also has 16G RAM, so those would be fr... [09:00:12] 10Gerrit: Install rename-project plugin - https://phabricator.wikimedia.org/T201953 (10hashar) 05Declined→03Open Reopening so. Thank you both :] [09:00:14] 10Gerrit, 10Release-Engineering-Team (Development services): Support renaming repositories in Gerrit - https://phabricator.wikimedia.org/T239693 (10hashar) [09:09:16] 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO, 10Operations, 10Jenkins: Add latest jenkins debian packages to apt.wikimedia.org and upgrade jenkins to latest LTS (2.190.3) - https://phabricator.wikimedia.org/T239586 (10MoritzMuehlenhoff) 05Open→03Resolved a:03Mori... [09:16:12] RECOVERY - App Server Main HTTP Response on deployment-mediawiki-09 is OK: HTTP OK: HTTP/1.1 200 OK - 49262 bytes in 0.494 second response time [09:50:19] (03CR) 10Hashar: [C: 03+2] Enable code health reports for ext-Cite [integration/config] - 10https://gerrit.wikimedia.org/r/554391 (owner: 10Awight) [09:51:06] (03Merged) 10jenkins-bot: Enable code health reports for ext-Cite [integration/config] - 10https://gerrit.wikimedia.org/r/554391 (owner: 10Awight) [09:51:38] (03CR) 10Hashar: [C: 03+2] "Deployed!" [integration/config] - 10https://gerrit.wikimedia.org/r/554391 (owner: 10Awight) [09:51:48] Thanks :-} [10:07:21] PROBLEM - App Server Main HTTP Response on deployment-mediawiki-09 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:27:13] RECOVERY - App Server Main HTTP Response on deployment-mediawiki-09 is OK: HTTP OK: HTTP/1.1 200 OK - 49295 bytes in 0.765 second response time [10:43:21] PROBLEM - App Server Main HTTP Response on deployment-mediawiki-09 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:18:12] RECOVERY - App Server Main HTTP Response on deployment-mediawiki-09 is OK: HTTP OK: HTTP/1.1 200 OK - 49217 bytes in 0.969 second response time [11:46:17] btw I am not running my uploads/captions/depicts, i ran them for a short period of time a couple hours ago and stopped because of the above errors [13:45:30] 10Release-Engineering-Team, 10serviceops, 10Patch-For-Review: switch prod Phabricator from phab1003 to phab1001 - https://phabricator.wikimedia.org/T238956 (10Lucas_Werkmeister_WMDE) The recently added Gerrit integration directly beneath a tasks description seems to be gone, could that be related to this swi... [13:51:44] 10Continuous-Integration-Infrastructure, 10Product-Infrastructure-Team-Backlog: mobileapps-periodic-test failing since 2019-12-04 01:00 UTC due to failing git fetches - https://phabricator.wikimedia.org/T239815 (10Mholloway) [14:39:10] (03PS1) 10Ottomata: Add CI for (WIP) EventStreamConfig MW extension [integration/config] - 10https://gerrit.wikimedia.org/r/554512 (https://phabricator.wikimedia.org/T233634) [15:05:08] hi. I am looking for Antoine and the contact page says their nick is "hashar" but I can't seem to find them here :) [15:09:55] sukhe: He comes and goes during the day... Depending on whether he's actually working or not [15:10:13] But if he's working he's usually here (so might be busy with his kids or something at this time :)) [15:10:41] Reedy: thanks! I just assume everyone has no life like me and is on IRC all the time :P [15:10:47] I will wait for him [15:10:51] Some of us just have a bouncer ;) [15:11:00] [15:10:55] [NickServ] Last seen : Dec 03 16:39:08 2019 (22h 31m 47s ago) [15:11:07] so it does look like he's not been on today [15:11:35] yeah. it's not urgent so I will just wait. [15:11:36] Reedy: thanks! [15:12:02] Other people might be able to help depending on what it is... He looks to have some meetings in a few hours so might appear then [15:12:42] I will just paste here: I am looking for some help with setting up CI for T238977 and the last comment indicates that hashar is the expert [15:12:47] T238977: Add operations/software/censorship-monitoring.git to CI - https://phabricator.wikimedia.org/T238977 [15:14:21] tox-docker sounds right ot me [15:14:25] Easy enough to try [15:16:44] (03PS1) 10Reedy: Add tox-docker to operations/software/censorship-monitoring [integration/config] - 10https://gerrit.wikimedia.org/r/554530 (https://phabricator.wikimedia.org/T238977) [15:17:36] 10Continuous-Integration-Config, 10Patch-For-Review: Add operations/software/censorship-monitoring.git to CI - https://phabricator.wikimedia.org/T238977 (10Reedy) [15:17:44] ah thank you Reedy! this is the first time I am doing this and while I did find documentation, I wanted to run it by someone else [15:17:46] 10Continuous-Integration-Config, 10Patch-For-Review: Add operations/software/censorship-monitoring.git to CI - https://phabricator.wikimedia.org/T238977 (10Reedy) [15:18:21] sukhe: If you need different OS packages and stuff installing, you might need a custom docker image... Which is more work and stuff [15:18:32] (03CR) 10Reedy: [C: 03+2] Add tox-docker to operations/software/censorship-monitoring [integration/config] - 10https://gerrit.wikimedia.org/r/554530 (https://phabricator.wikimedia.org/T238977) (owner: 10Reedy) [15:18:37] This should get thing going at least :) [15:18:52] which is great for now! [15:19:23] (03Merged) 10jenkins-bot: Add tox-docker to operations/software/censorship-monitoring [integration/config] - 10https://gerrit.wikimedia.org/r/554530 (https://phabricator.wikimedia.org/T238977) (owner: 10Reedy) [15:19:58] !log Reloading Zuul to deploy https://gerrit.wikimedia.org/r/554530 [15:19:59] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [15:20:41] sukhe: https://integration.wikimedia.org/ci/job/tox-docker/9084/ [15:21:04] You've got a C+2 on https://gerrit.wikimedia.org/r/#/c/operations/software/censorship-monitoring/+/553385/ [15:21:09] Whether that's right or useful.. :D [15:21:58] Reedy: this has been a learning experience for when I next deploy the code :D [15:22:06] 15:21:36 [117] /src$ /src/.tox/flake8/bin/flake8 [15:22:06] 15:21:36 ./ioda/iodafetch.py:33:1: E303 too many blank lines (3) [15:22:06] 15:21:36 1 E303 too many blank lines (3) [15:22:08] heh [15:22:16] ha yeah [15:22:16] Sounds like it's vaguely doing like we'd expect :) [15:22:50] I had hooked tox to my post-commit hook in the meantime but that happened after this E303 [15:24:24] thanks for your help. you sorted this out in seconds! [15:24:27] np :) [15:24:48] If it was much more complex you might've needed Hashar or someone else from releng.. [15:25:03] But simple stuff, numerous of us can usually fuddle their way through it :D [15:36:49] 10Continuous-Integration-Config, 10Patch-For-Review: Add operations/software/censorship-monitoring.git to CI - https://phabricator.wikimedia.org/T238977 (10Reedy) 05Open→03Resolved a:03Reedy [15:42:09] (03CR) 10Jforrester: Enable code health reports for ext-Cite (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/554391 (owner: 10Awight) [15:43:37] 10Release-Engineering-Team-TODO (201912): Arrange intra-team PGP keysigning at Atlanta offsite - https://phabricator.wikimedia.org/T232990 (10LarsWirzenius) [15:43:39] 10Release-Engineering-Team-TODO (201912), 10Repository-Admins: Create Gerrit repository for EngProd PGP keys (and later maybe others) - https://phabricator.wikimedia.org/T239422 (10LarsWirzenius) 05Open→03Resolved https://gerrit.wikimedia.org/r/admin/projects/pgp-public-keys exists now. [15:45:10] 10Release-Engineering-Team-TODO (201912): Write howto for signing keys for keysigning party - https://phabricator.wikimedia.org/T239829 (10LarsWirzenius) [15:49:46] (03CR) 10Jforrester: [C: 04-1] layout: add wikibase/vuejs-components & build via pipeline (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/551829 (owner: 10Pablo Grass (WMDE)) [15:55:56] (03CR) 10Jforrester: [C: 03+2] Add CI for (WIP) EventStreamConfig MW extension [integration/config] - 10https://gerrit.wikimedia.org/r/554512 (https://phabricator.wikimedia.org/T233634) (owner: 10Ottomata) [15:56:58] (03Merged) 10jenkins-bot: Add CI for (WIP) EventStreamConfig MW extension [integration/config] - 10https://gerrit.wikimedia.org/r/554512 (https://phabricator.wikimedia.org/T233634) (owner: 10Ottomata) [15:57:33] !log Zuul: Add CI for EventStreamConfig MW extension T233634 [15:57:36] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [15:57:36] T233634: Modern Event Platform: Stream Configuration: Implementation - https://phabricator.wikimedia.org/T233634 [16:05:50] 10Phabricator, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO, 10Security-Team, 10User-MModell: Publish the source for phabricator-antivandalism - https://phabricator.wikimedia.org/T202080 (10chasemp) @mmodell @greg I'm echoing what @sbassett said above. We are all fo... [16:09:17] 10Phabricator, 10Office-IT, 10Security-Team: create security-test@ google group and test functionality - https://phabricator.wikimedia.org/T239834 (10chasemp) [16:21:08] 10Release-Engineering-Team, 10serviceops, 10Patch-For-Review: switch prod Phabricator from phab1003 to phab1001 - https://phabricator.wikimedia.org/T238956 (10Mainframe98) I suspect that this move inadvertently caused {T239786} as I only started seeing it today. [16:34:37] 10Continuous-Integration-Infrastructure, 10Mobile-Content-Service, 10Product-Infrastructure-Team-Backlog: mobileapps-periodic-test failing since 2019-12-04 01:00 UTC due to failing git fetches - https://phabricator.wikimedia.org/T239815 (10Jhernandez) [16:38:28] (03PS1) 10Volans: Setup CI for operations/software/netbox-extra [integration/config] - 10https://gerrit.wikimedia.org/r/554560 (https://phabricator.wikimedia.org/T233183) [16:43:14] 10Phabricator, 10Security-Team: PoC for security@ email tracking - https://phabricator.wikimedia.org/T239144 (10chasemp) notice T230951 will probably tie in here [16:43:25] 10Continuous-Integration-Infrastructure, 10Page Content Service, 10Product-Infrastructure-Team-Backlog: mobileapps-periodic-test failing since 2019-12-04 01:00 UTC due to failing git fetches - https://phabricator.wikimedia.org/T239815 (10Mholloway) [16:46:44] (03CR) 10Jforrester: [C: 03+2] Setup CI for operations/software/netbox-extra [integration/config] - 10https://gerrit.wikimedia.org/r/554560 (https://phabricator.wikimedia.org/T233183) (owner: 10Volans) [16:47:35] (03Merged) 10jenkins-bot: Setup CI for operations/software/netbox-extra [integration/config] - 10https://gerrit.wikimedia.org/r/554560 (https://phabricator.wikimedia.org/T233183) (owner: 10Volans) [16:48:59] !log Zuul: Add CI for operations/software/netbox-extra T233183 [16:49:03] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:49:03] T233183: Automate generation of Management DNS records from Netbox - https://phabricator.wikimedia.org/T233183 [16:56:39] !log MariaDB [wikidatawiki]> delete * from wb_changes_dispatch; (T237551) [16:56:47] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:56:48] T237551: beta wikidata trying to dispatch to wikis that don't exist - https://phabricator.wikimedia.org/T237551 [16:56:51] 10Scap, 10MediaWiki-Internationalization, 10Performance-Team (Radar): Use static php array files for l10n cache instead of CDB - https://phabricator.wikimedia.org/T99740 (10Jdforrester-WMF) [17:01:20] 10Beta-Cluster-Infrastructure, 10Wikidata, 10Wikidata-Campsite: beta wikidata trying to dispatch to wikis that don't exist - https://phabricator.wikimedia.org/T237551 (10Ladsgroup) This was fun. The client, the list, everything for that was fine, the problem was that somehow `wb_changes_dispatch` table got p... [17:02:26] (03PS1) 10Clarakosi: Update Quibble to use api-testing npm package [integration/quibble] - 10https://gerrit.wikimedia.org/r/554571 (https://phabricator.wikimedia.org/T236680) [17:03:10] (03CR) 10jerkins-bot: [V: 04-1] Update Quibble to use api-testing npm package [integration/quibble] - 10https://gerrit.wikimedia.org/r/554571 (https://phabricator.wikimedia.org/T236680) (owner: 10Clarakosi) [17:10:26] (03PS2) 10Clarakosi: Update Quibble to use api-testing npm package [integration/quibble] - 10https://gerrit.wikimedia.org/r/554571 (https://phabricator.wikimedia.org/T236680) [17:16:05] 10Continuous-Integration-Config, 10Release-Engineering-Team-TODO (201912), 10CPT Initiatives (API Integration Tests), 10Code-Health, and 2 others: Enable API integration tests in CI for MediaWiki core - https://phabricator.wikimedia.org/T236680 (10Clarakosi) [17:28:00] 10Gerrit, 10ORES, 10Scoring-platform-team (Current): Write a cookbook for the workaround for getting LFS to gerrit - https://phabricator.wikimedia.org/T226055 (10Halfak) a:03Halfak [17:40:37] !log lucaswerkmeister-wmde@deployment-deploy01:~$ mwscript createAndPromote.php dewiki 'Lucas Werkmeister (WMDE)' --force --custom-groups=bureaucrat # make myself bureaucrat to confirm users [17:40:39] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [17:50:15] 10Continuous-Integration-Infrastructure (phase-out-jessie), 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO, 10Cloud-VPS (Debian Jessie Deprecation): "integration" Cloud VPS project jessie deprecation - https://phabricator.wikimedia.org/T236576 (10bd808) The Cloud Servic... [18:01:31] (03PS5) 10Pablo Grass (WMDE): layout: add wikibase/vuejs-components & build via pipeline [integration/config] - 10https://gerrit.wikimedia.org/r/551829 [18:02:35] (03CR) 10Pablo Grass (WMDE): layout: add wikibase/vuejs-components & build via pipeline (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/551829 (owner: 10Pablo Grass (WMDE)) [18:11:02] 10Continuous-Integration-Config, 10Release-Engineering-Team, 10Release-Engineering-Team-TODO, 10MobileFrontend, 10Readers-Web-Backlog: Add curl support to mwext-node10-rundoc-docker - https://phabricator.wikimedia.org/T239246 (10hashar) a:03hashar [18:19:08] 10Continuous-Integration-Infrastructure, 10Page Content Service, 10Product-Infrastructure-Team-Backlog: mobileapps-periodic-test failing since 2019-12-04 01:00 UTC due to failing git fetches - https://phabricator.wikimedia.org/T239815 (10mmodell) Something is wrong with this repo. I spent a couple of hours t... [18:36:04] 10MediaWiki-Releasing, 10Security: Streamline MW security release process - https://phabricator.wikimedia.org/T196602 (10mmodell) {T156445} [18:38:40] 10Continuous-Integration-Infrastructure, 10Page Content Service, 10Product-Infrastructure-Team-Backlog: mobileapps-periodic-test failing since 2019-12-04 01:00 UTC due to failing git fetches - https://phabricator.wikimedia.org/T239815 (10Mholloway) Thanks for investigating, @mmodell. That's strange; a shall... [18:49:25] 10Continuous-Integration-Infrastructure, 10Page Content Service, 10Product-Infrastructure-Team-Backlog: mobileapps-periodic-test failing since 2019-12-04 01:00 UTC due to failing git fetches - https://phabricator.wikimedia.org/T239815 (10mmodell) It should be as simple as switching the url to gerrit. General... [18:50:50] 10Continuous-Integration-Infrastructure, 10Page Content Service, 10Product-Infrastructure-Team-Backlog: mobileapps-periodic-test failing since 2019-12-04 01:00 UTC due to failing git fetches - https://phabricator.wikimedia.org/T239815 (10Jdforrester-WMF) >>! In T239815#5713380, @mmodell wrote: > It should be... [18:54:32] 10Phabricator, 10Security-Team: PoC for security@ email tracking - https://phabricator.wikimedia.org/T239144 (10mmodell) So far the phabricator hardware migration has interfered with me really working on this, however, it's a simple patch which was already written last week. I will try to get it deployed this... [19:04:51] 10Continuous-Integration-Infrastructure (phase-out-jessie): Migrate debian-glue jobs to Stretch instances - https://phabricator.wikimedia.org/T224943 (10hashar) [19:04:56] 10Continuous-Integration-Config, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (201912), 10Wikidata, and 3 others: Update Doxygen in CI to 1.8.15 or greater - https://phabricator.wikimedia.org/T239482 (10hashar) [19:06:10] 10Continuous-Integration-Infrastructure (phase-out-jessie): Migrate debian-glue jobs to Stretch instances - https://phabricator.wikimedia.org/T224943 (10hashar) Another use case is `operations/debs/doxygen` which uses pristine-tar for the upstream tarball but the version on Jessie does not support delta version... [19:12:29] (03PS1) 10Hashar: Stop using zuul-cloner in debian-glue jobs [integration/config] - 10https://gerrit.wikimedia.org/r/554588 (https://phabricator.wikimedia.org/T224943) [19:12:57] (03CR) 10Hashar: "Gotta be tested first, non-voting jobs would do I guess." [integration/config] - 10https://gerrit.wikimedia.org/r/554588 (https://phabricator.wikimedia.org/T224943) (owner: 10Hashar) [19:13:25] 10Continuous-Integration-Infrastructure (phase-out-jessie), 10Patch-For-Review: Migrate debian-glue jobs to Stretch instances - https://phabricator.wikimedia.org/T224943 (10hashar) a:03hashar [19:13:32] (03CR) 10Jforrester: "In many ways this is actually simpler…" [integration/config] - 10https://gerrit.wikimedia.org/r/554588 (https://phabricator.wikimedia.org/T224943) (owner: 10Hashar) [19:14:05] 10Release-Engineering-Team, 10serviceops, 10Patch-For-Review: switch prod Phabricator from phab1003 to phab1001 - https://phabricator.wikimedia.org/T238956 (10Dzahn) Unfortunately we have to switch back to the server before, change a BIOS setting in the current server, reimage it and then switch back a third... [19:16:52] (03CR) 10Jforrester: [C: 03+2] layout: add wikibase/vuejs-components & build via pipeline (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/551829 (owner: 10Pablo Grass (WMDE)) [19:17:41] (03Merged) 10jenkins-bot: layout: add wikibase/vuejs-components & build via pipeline [integration/config] - 10https://gerrit.wikimedia.org/r/551829 (owner: 10Pablo Grass (WMDE)) [19:21:00] (03PS1) 10Jforrester: jjb: [mobileapps-periodic-test] Point at gerrit, not diffusion [integration/config] - 10https://gerrit.wikimedia.org/r/554595 (https://phabricator.wikimedia.org/T239815) [19:22:53] !log Zuul: Add CI for wikibase/vuejs-components [19:22:55] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [19:23:30] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team-TODO (201912), 10Page Content Service, 10Product-Infrastructure-Team-Backlog, 10Patch-For-Review: mobileapps-periodic-test failing since 2019-12-04 01:00 UTC due to failing git fetches - https://phabricator.wikimedia.org/T239815 (10Jdforr... [19:26:50] (03PS2) 10Jforrester: jjb: [mobileapps-periodic-test] Point at gerrit, not diffusion [integration/config] - 10https://gerrit.wikimedia.org/r/554595 (https://phabricator.wikimedia.org/T239815) [19:27:05] (03CR) 10Jforrester: [C: 03+2] "Deployed, this bit is now working." [integration/config] - 10https://gerrit.wikimedia.org/r/554595 (https://phabricator.wikimedia.org/T239815) (owner: 10Jforrester) [19:27:57] (03Merged) 10jenkins-bot: jjb: [mobileapps-periodic-test] Point at gerrit, not diffusion [integration/config] - 10https://gerrit.wikimedia.org/r/554595 (https://phabricator.wikimedia.org/T239815) (owner: 10Jforrester) [19:35:00] (03PS1) 10Volans: Setup default permissions [software/netbox-extras] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/554598 [19:42:52] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team-TODO (201912), 10Page Content Service, 10Product-Infrastructure-Team-Backlog, 10Patch-For-Review: mobileapps-periodic-test failing since 2019-12-04 01:00 UTC due to failing git fetches - https://phabricator.wikimedia.org/T239815 (10Jdforr... [19:43:21] 10Release-Engineering-Team-TODO, 10Core Platform Team, 10Front-end-Standards-Group, 10Product-Infrastructure-Team-Backlog, and 2 others: Should npm packages maintained by Wikimedia be scoped or unscoped? - https://phabricator.wikimedia.org/T239742 (10daniel) Pinging TechCom to have a look. Maybe an RFC is... [19:46:23] 10Phabricator, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO, 10Availability, and 3 others: Deploy phabricator to phab2001.codfw.wmnet - https://phabricator.wikimedia.org/T137928 (10mmodell) 05Open→03Resolved >>! In T137928#5676819, @Dzahn wrote: > - phab2001 has be... [19:46:26] 10Phabricator, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO, 10Operations, 10serviceops: Reimage both phab1001 and phab2001 to stretch / buster - https://phabricator.wikimedia.org/T190568 (10mmodell) [19:46:31] 10Phabricator, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO, 10DBA: Switch phabricator production to codfw - https://phabricator.wikimedia.org/T164810 (10mmodell) [19:46:34] 10Phabricator, 10RelEng-Archive-FY201718-Q1, 10Operations: reinstall iridium (phabricator) as phab1001 with jessie - https://phabricator.wikimedia.org/T152129 (10mmodell) [20:01:16] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team-TODO (201912), 10Page Content Service, 10Product-Infrastructure-Team-Backlog, 10Patch-For-Review: mobileapps-periodic-test failing since 2019-12-04 01:00 UTC due to failing git fetches - https://phabricator.wikimedia.org/T239815 (10Mhollo... [20:10:06] (03CR) 10Kosta Harlan: Update Quibble to use api-testing npm package (031 comment) [integration/quibble] - 10https://gerrit.wikimedia.org/r/554571 (https://phabricator.wikimedia.org/T236680) (owner: 10Clarakosi) [20:19:56] 10Release-Engineering-Team-TODO (201912), 10User-greg, 10Wikimedia-extension-review-queue: Investigate and make improvements to the extension review process - https://phabricator.wikimedia.org/T195244 (10Mholloway) >>! In T195244#4413456, @bd808 wrote: > Flipping this redirect around the other direction woul... [20:20:00] 10Release-Engineering-Team, 10Developer-Advocacy (Oct-Dec-2016), 10Documentation: Merge Wikimedia's "Deployment checklist for new extensions" doc pages - https://phabricator.wikimedia.org/T142081 (10Jdforrester-WMF) Hey, I think this was a bad change (especially in how it implicitly excludes non-extensions),... [21:16:43] (03CR) 10Hashar: "Quite impressive. I think we are fine hardcoding those specific settings for now. If later the framework adds more settings that would ne" (031 comment) [integration/quibble] - 10https://gerrit.wikimedia.org/r/554571 (https://phabricator.wikimedia.org/T236680) (owner: 10Clarakosi) [21:45:56] (03PS2) 10Hashar: Stop using zuul-cloner in debian-glue jobs [integration/config] - 10https://gerrit.wikimedia.org/r/554588 (https://phabricator.wikimedia.org/T224943) [21:47:33] 10Continuous-Integration-Config, 10Parsoid-PHP: Set up extension tests for Parsoid repo - https://phabricator.wikimedia.org/T227352 (10Arlolra) p:05Triage→03High Since Parsoid/PHP is in production, a train rollout without proper integration testing could have dire effects. [21:47:59] !log Updated debian glue non voting jobs to use git instead of zuul-cloner | https://gerrit.wikimedia.org/r/#/c/integration/config/+/554588/ | T224943 [21:48:01] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [21:48:02] T224943: Migrate debian-glue jobs to Stretch instances - https://phabricator.wikimedia.org/T224943 [21:48:27] (03CR) 10Hashar: "I have updated the non-voting jobs." [integration/config] - 10https://gerrit.wikimedia.org/r/554588 (https://phabricator.wikimedia.org/T224943) (owner: 10Hashar) [21:57:24] brennen and/or twentyafterfour, tomorrow's train (if I understand the deployment cadence correctly) is going to role out a very dramatic change on wikitech. It's also happening at 4AM my time. [21:57:59] Can I ask one of you to spot-check wikitech after the deployment? And ping me here if things go poorly? [22:00:18] andrewbogott: I'll be about (and helped do the damage) if they want to ping me too [22:01:05] oh great, thanks Reedy. [22:01:07] andrewbogott, Reedy: can do. [22:01:26] The thing that's most likely to break is logins, so 'spot check' probably means 'log out and back in again' [22:01:38] if it breaks account creation that can maybe wait for me to wake up. [22:01:46] thank you brennen [22:02:02] andrewbogott: is there a task for the change that i should track / modify? [22:02:34] (03CR) 10Clarakosi: "> Patch Set 2:" (031 comment) [integration/quibble] - 10https://gerrit.wikimedia.org/r/554571 (https://phabricator.wikimedia.org/T236680) (owner: 10Clarakosi) [22:02:39] brennen: [22:02:40] https://phabricator.wikimedia.org/T161553 [22:02:59] thx [22:12:49] 10Beta-Cluster-Infrastructure, 10Operations, 10Wikimedia-Logstash: Logstash in Beta Cluster stopped ingesting messages from MediaWiki - https://phabricator.wikimedia.org/T239868 (10Krinkle) [22:13:32] 10Beta-Cluster-Infrastructure, 10Operations, 10Wikimedia-Logstash: Logstash in Beta Cluster stopped ingesting messages from MediaWiki - https://phabricator.wikimedia.org/T239868 (10Krinkle) >>! @colewhite wrote at : > seeing rsyslog complaining about "omkafka: kafka del... [22:14:19] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team-TODO, 10Wikimedia-Logstash, 10observability: logstash-beta.wmflabs.org does not receive any mediawiki events - https://phabricator.wikimedia.org/T233134 (10Krinkle) [22:14:21] 10Beta-Cluster-Infrastructure, 10Operations, 10Wikimedia-Logstash: Logstash in Beta Cluster stopped ingesting messages from MediaWiki - https://phabricator.wikimedia.org/T239868 (10Krinkle) [22:14:30] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team-TODO, 10Wikimedia-Logstash, 10observability: logstash-beta.wmflabs.org does not receive any mediawiki events - https://phabricator.wikimedia.org/T233134 (10Krinkle) >>! @colewhite wrote at : > seeing rsyslog co... [22:14:33] 10Beta-Cluster-Infrastructure, 10Operations, 10Wikimedia-Logstash: Logstash in Beta Cluster stopped ingesting messages from MediaWiki - https://phabricator.wikimedia.org/T239868 (10Jdforrester-WMF) See also {T211984} and {T233134}. [22:28:41] 10Continuous-Integration-Infrastructure (phase-out-jessie), 10Patch-For-Review: Migrate debian-glue jobs to Stretch instances - https://phabricator.wikimedia.org/T224943 (10hashar) I have created a Stretch instance `integration-agent-pkgbuilder-1001` https://horizon.wikimedia.org/project/instances/b09e917a-c92... [22:35:57] !log integration: added Stretch instance integration-agent-pkgbuilder-1001 for Debian glue jobs # T224943 [22:35:59] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [22:36:00] T224943: Migrate debian-glue jobs to Stretch instances - https://phabricator.wikimedia.org/T224943 [22:50:08] Heads-up for people that care but don't follow the SAL: I've semi-permanently disabled Parsoid (and thus VE) on wikitech as it won't work in PHP land until the code moves into MW. [22:51:33] 10Gerrit, 10Operations, 10vm-requests: Gerrit VM to test data migration - https://phabricator.wikimedia.org/T239151 (10Dzahn) >>! In T239151#5711306, @hashar wrote: > The way I understand the message: the virtualization servers in group `row_A` lack free memory to allocate a VM. But maybe another group woul... [22:52:58] 10Continuous-Integration-Infrastructure (phase-out-jessie), 10Patch-For-Review: Migrate debian-glue jobs to Stretch instances - https://phabricator.wikimedia.org/T224943 (10hashar) I gave it a try with the Doxygen package. It eventually fails due to autopkgtest not being recognized by jenkins-debian-glue. That... [22:55:59] 10Gerrit, 10Operations, 10vm-requests: Gerrit VM to test data migration - https://phabricator.wikimedia.org/T239151 (10Dzahn) >>! In T239151#5711346, @MoritzMuehlenhoff wrote: > The old puppetdb hosts (puppetdb1001) should be ready to go away, @jbond merged the patches to stop broadcasting to it last week. I... [22:58:27] 10Phabricator, 10Developer-Advocacy (Oct-Dec 2019): Decrease issues with assignee field set for years without progress (aka cookie licking) - https://phabricator.wikimedia.org/T228575 (10Dzahn) Don't they have to be nagged to .. do the review.. though? [23:00:29] 10Phabricator, 10Developer-Advocacy (Oct-Dec 2019): Decrease issues with assignee field set for years without progress (aka cookie licking) - https://phabricator.wikimedia.org/T228575 (10Dzahn) >>! In T228575#5688336, @thiemowmde wrote: > One could argue that it's even less of a problem to unassign people from... [23:03:25] PROBLEM - Host integration-agent-pkgbuilder-1001 is DOWN: CRITICAL - Host Unreachable (172.16.0.61) [23:12:15] (03PS3) 10Hashar: Stop using zuul-cloner in debian-glue jobs [integration/config] - 10https://gerrit.wikimedia.org/r/554588 (https://phabricator.wikimedia.org/T224943) [23:12:17] (03PS1) 10Hashar: jjb: use a single label for debian packaging jobs [integration/config] - 10https://gerrit.wikimedia.org/r/554649 (https://phabricator.wikimedia.org/T224943) [23:14:25] 10Continuous-Integration-Infrastructure (phase-out-jessie), 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO, 10Cloud-VPS (Debian Jessie Deprecation): "integration" Cloud VPS project jessie deprecation - https://phabricator.wikimedia.org/T236576 (10thcipriani) a:03hashar... [23:20:59] 10Continuous-Integration-Infrastructure (phase-out-jessie), 10Patch-For-Review: Migrate debian-glue jobs to Stretch instances - https://phabricator.wikimedia.org/T224943 (10hashar) integration-agent-pkgbuilder-1001 rebuild with Buster. [23:21:05] 10Phabricator, 10Developer-Advocacy (Oct-Dec 2019): Decrease issues with assignee field set for years without progress (aka cookie licking) - https://phabricator.wikimedia.org/T228575 (10Peachey88) Perhaps we generate a report to see how many tasks it would effect first before doing any changes? [23:23:09] RECOVERY - Host integration-agent-pkgbuilder-1001 is UP: PING OK - Packet loss = 0%, RTA = 0.63 ms [23:25:42] (03CR) 10Jforrester: [C: 03+1] jjb: use a single label for debian packaging jobs [integration/config] - 10https://gerrit.wikimedia.org/r/554649 (https://phabricator.wikimedia.org/T224943) (owner: 10Hashar) [23:33:08] PROBLEM - Puppet staleness on integration-agent-pkgbuilder-1001 is CRITICAL: (Service Check Timed Out) [23:34:22] PROBLEM - App Server Main HTTP Response on deployment-mediawiki-09 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:34:58] PROBLEM - App Server Main HTTP Response on deployment-mediawiki-07 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:35:15] PROBLEM - English Wikipedia Main page on beta-cluster is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:36:33] PROBLEM - App Server Main HTTP Response on deployment-mediawiki-parsoid10 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:37:36] PROBLEM - Puppet errors on deployment-acme-chief04 is CRITICAL: (Service Check Timed Out) [23:39:14] RECOVERY - App Server Main HTTP Response on deployment-mediawiki-09 is OK: HTTP OK: HTTP/1.1 200 OK - 49231 bytes in 3.804 second response time [23:39:48] RECOVERY - App Server Main HTTP Response on deployment-mediawiki-07 is OK: HTTP OK: HTTP/1.1 200 OK - 49245 bytes in 0.608 second response time [23:40:09] RECOVERY - English Wikipedia Main page on beta-cluster is OK: HTTP OK: HTTP/1.1 200 OK - 49763 bytes in 0.587 second response time [23:41:25] RECOVERY - App Server Main HTTP Response on deployment-mediawiki-parsoid10 is OK: HTTP OK: HTTP/1.1 200 OK - 49303 bytes in 1.857 second response time [23:42:45] 10Phabricator, 10Core Platform Team: Provide field for Actual Story Points to be captured - https://phabricator.wikimedia.org/T239870 (10Aklapper) [23:50:26] 10Phabricator, 10Developer-Advocacy (Oct-Dec 2019): Decrease issues with assignee field set for years without progress (aka cookie licking) - https://phabricator.wikimedia.org/T228575 (10Aklapper) >>! In T228575#5714093, @Dzahn wrote: > Don't they have to be nagged to .. do the review.. though? Yes, but the p... [23:51:36] 10Phabricator, 10Developer-Advocacy (Oct-Dec 2019): Decrease issues with assignee field set for years without progress (aka cookie licking) - https://phabricator.wikimedia.org/T228575 (10Aklapper) For the last three years (`94608000`) we talk about 965 tasks. For the last two years (`63072000`) we talk about 1...