[03:25:52] PROBLEM - Free space - all mounts on deployment-fluorine02 is CRITICAL: CRITICAL: deployment-prep.deployment-fluorine02.diskspace._srv.byte_percentfree (<10.00%) [06:32:39] 10Continuous-Integration-Config, 10Operations: Fix operations/puppet.git "rebase hell" - https://phabricator.wikimedia.org/T224033 (10Joe) I am 100% against having ci handle merges of ops/puppet. Think of the case ci is down and we need puppet for anything. I am also all for moving to rebase-if-necessary and... [07:05:52] RECOVERY - Free space - all mounts on deployment-fluorine02 is OK: OK: All targets OK [08:24:57] (03CR) 10Gabriel Birke: [C: 03+1] Make it more clear that the codehealth patch job is non-voting [integration/config] - 10https://gerrit.wikimedia.org/r/512443 (owner: 10Kosta Harlan) [08:57:30] 10Release-Engineering-Team, 10Operations, 10SRE-Access-Requests: Request access to analytics cluster for Alaa Sarhan - https://phabricator.wikimedia.org/T223697 (10alaa_wmde) Hi @Nuria yes access to both would be ideal .. thank you [09:11:50] 10Release-Engineering-Team, 10Operations, 10SRE-Access-Requests: Request access to deployment cluster for Alaa Sarhan - https://phabricator.wikimedia.org/T223698 (10alaa_wmde) Thanks @Volans I've fully and carefully read the Server Access Responsibilities document, and signed it. I believe we are awaiting on... [10:21:02] 10Release-Engineering-Team (Kanban), 10MediaWiki-Core-Testing, 10Quibble, 10MW-1.34-notes (1.34.0-wmf.7; 2019-05-28), and 2 others: Quibble should run `npm install` and `npm run selenium-test` for each extension/skin that has Selenium tests - https://phabricator.wikimedia.org/T199116 (10awight) @Jdforreste... [10:28:21] 10Release-Engineering-Team (Kanban), 10Operations, 10SRE-Access-Requests, 10Patch-For-Review, and 3 others: Requesting access to production for SWAT deploy for Urbanecm - https://phabricator.wikimedia.org/T192830 (10Volans) As per docs added `Urbanecm` to the `wmf-deployment` group in Gerrit. [10:56:53] 10Phabricator, 10Documentation: Document subtypes / task types in Phabricator - https://phabricator.wikimedia.org/T224417 (10Aklapper) [10:59:22] 10Phabricator, 10CommRel-Design, 10CommRel-Internals: Create Phabricator form for CommRel-Design and Comms requests and add a link to it in the "Star" dropdown - https://phabricator.wikimedia.org/T223102 (10Aklapper) I'm currently blocked on T223102#5202623 to set up a second form. After that I'd consider th... [11:03:41] 10Phabricator, 10Project-Admins, 10MediaWiki-General-or-Unknown, 10Developer-Wishlist (2017): Allow to search tasks about MediaWiki core and core only (create MediaWiki umbrella project?) - https://phabricator.wikimedia.org/T76942 (10Aklapper) Also note that nowadays, a lot of MediaWiki core tasks are unde... [11:12:17] 10Phabricator, 10WMSE-Bug-Reporting-and-Translation-2019: "Move Tasks to Column..." opens dialogue for moving to project - https://phabricator.wikimedia.org/T220585 (10Aklapper) @Sebastian_Berlin-WMSE: AFAIU that's intended and a feature. The dialog for moving tasks preselects the existing project. After that... [11:13:36] 10Phabricator: move_project script allows using --keep-members for milestones which cannot have members - https://phabricator.wikimedia.org/T224420 (10Aklapper) [11:15:55] 10Phabricator: move_project script does not check if task is in both future parent and future subproject and does not remove future parent project - https://phabricator.wikimedia.org/T224421 (10Aklapper) [11:16:24] 10Phabricator, 10Pywikibot, 10Proposal, 10Pywikibot-RfCs: RFC: Add pywikibot-core to all new pywikibot-script/module/class tasks automatically - https://phabricator.wikimedia.org/T170090 (10Aklapper) [11:16:28] 10Phabricator, 10Project-Admins, 10Pywikibot: Move #Pywikibot-Other-scripts to #Pywikibot-Scripts and transform it into parent project - https://phabricator.wikimedia.org/T195855 (10Aklapper) 05Open→03Resolved Filed the issue in the last comment as T224421. No reply to my previous comment, hence closing... [11:25:19] 10Release-Engineering-Team (Kanban), 10Operations, 10SRE-Access-Requests, 10Patch-For-Review, and 3 others: Requesting access to production for SWAT deploy for Urbanecm - https://phabricator.wikimedia.org/T192830 (10Volans) I've added `urbanecm` to the LDAP group `nda` as per request above given that it's... [11:30:53] 10Release-Engineering-Team (Kanban), 10Operations, 10SRE-Access-Requests, 10Patch-For-Review, and 3 others: Requesting access to production for SWAT deploy for Urbanecm - https://phabricator.wikimedia.org/T192830 (10zeljkofilipin) Thanks! I'll resolve it after the first successful deployment. [11:33:59] 10Phabricator (Upstream), 10Upstream: Public log for Phabricator user account role change (e.g. deactivation, admin rights) - https://phabricator.wikimedia.org/T102576 (10Aklapper) I'm personally not convinced that this is needed. If really wanted I'd rather expect Phab admins to be able to amend/overwrite a u... [11:44:06] 10Release-Engineering-Team, 10Operations, 10Release Pipeline, 10Wikidata, and 5 others: Introduce wikidata termbox SSR to kubernetes - https://phabricator.wikimedia.org/T220402 (10mobrovac) >>! In T220402#5199730, @Tarrow wrote: > Should we be using http rather than https internally? Yes, indeed, sorry th... [11:49:51] 10Release-Engineering-Team, 10Operations, 10Release Pipeline, 10Wikidata, and 5 others: Introduce wikidata termbox SSR to kubernetes - https://phabricator.wikimedia.org/T220402 (10Tarrow) @mobrovac Thanks! I think we've now taken most of this onboard and merged it. @akosiaris could you take a look at out... [11:50:24] (03PS1) 10Pwirth: Add BlueSpiceSocial [integration/config] - 10https://gerrit.wikimedia.org/r/512661 [11:51:59] 10Phabricator, 10Project-Admins, 10Pywikibot: Move #Pywikibot-Other-scripts to #Pywikibot-Scripts and transform it into parent project - https://phabricator.wikimedia.org/T195855 (10Dvorapa) - catfiles: ok, let's leave it as is - everything seems okay now [11:52:58] (03PS2) 10Pwirth: Add BlueSpiceSocial [integration/config] - 10https://gerrit.wikimedia.org/r/512661 [11:58:13] 10Phabricator, 10Documentation: Document subtypes / task types in Phabricator - https://phabricator.wikimedia.org/T224417 (10Aklapper) also see https://phabricator.wikimedia.org/phame/post/view/145/phab_phebruary/ [12:34:19] Hi! For images built using the pipeline is there a way to get the latest build image? [12:34:33] looks like the latest tag isn't written automatically [12:38:47] We have a "beta" version of our service running on labs and we'd like to always pull the latest built image [12:40:42] 10Phabricator, 10Project-Admins, 10MediaWiki-General-or-Unknown, 10Developer-Wishlist (2017): Allow to search tasks about MediaWiki core and core only (create MediaWiki umbrella project?) - https://phabricator.wikimedia.org/T76942 (10Tgr) >>! In T76942#5214363, @Aklapper wrote: > Also note that nowadays, a... [13:14:57] (03CR) 10Lars Wirzenius: [C: 03+2] "LGTM" [blubber] - 10https://gerrit.wikimedia.org/r/511784 (owner: 10Dduvall) [13:58:03] 10Beta-Cluster-Infrastructure, 10Editing-team, 10Release Pipeline, 10serviceops, and 3 others: Migrate Beta cluster services to use Kubernetes - https://phabricator.wikimedia.org/T220235 (10Maintenance_bot) [14:44:24] 10Phabricator, 10WMSE-Bug-Reporting-and-Translation-2019: "Move Tasks to Column..." opens dialogue for moving to project - https://phabricator.wikimedia.org/T220585 (10Sebastian_Berlin-WMSE) It does indeed do what it's supposed to. I never tried continuing because the label says "Move to Project", which I read... [15:02:25] 10Phabricator (Upstream), 10WMSE-Bug-Reporting-and-Translation-2019, 10Upstream: "Move Tasks to Column..." opens dialogue for moving to project - https://phabricator.wikimedia.org/T220585 (10Aklapper) p:05Triage→03Lowest Hmm. Well, you "move" the tasks to the very same project... I'm not sure how upstrea... [15:06:04] 10Project-Admins: Requests for addition to the #acl*Project-Admins group (in comments) - https://phabricator.wikimedia.org/T706 (10Aklapper) @aezell, @Susannaanas: Added you. The usual disclaimer to [please set good project descriptions](https://www.mediawiki.org/wiki/Phabricator/Creating_and_renaming_projects#G... [15:06:19] 10Project-Admins: Project for Wikimedia Finland - https://phabricator.wikimedia.org/T222442 (10Aklapper) 05Open→03Resolved a:03Aklapper No reply by @Susannaanas ([who can create projects now](https://phabricator.wikimedia.org/T706#5210475)), hence closing as resolved. [15:50:11] 10Phabricator, 10Documentation: Document subtypes / task types in Phabricator - https://phabricator.wikimedia.org/T224417 (10Quiddity) I made screenshots, but I don't know enough to write the words. https://commons.wikimedia.org/wiki/File:Phabricator_task_subtypes.jpg https://commons.wikimedia.org/wiki/File:Ph... [16:23:52] PROBLEM - Work requests waiting in Zuul Gearman server on contint1001 is CRITICAL: CRITICAL: 57.14% of data above the critical threshold [140.0] https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1 [16:26:56] (03PS1) 10Florianschmidtwelzow: [Passwordless Login] Jobs for extension [integration/config] - 10https://gerrit.wikimedia.org/r/512729 (https://phabricator.wikimedia.org/T224371) [16:27:43] PROBLEM - Free space - all mounts on integration-slave-docker-1021 is CRITICAL: CRITICAL: integration.integration-slave-docker-1021.diskspace.root.byte_percentfree (<22.22%) [16:28:15] (03PS2) 10Florianschmidtwelzow: [Passwordless Login] Jobs for extension [integration/config] - 10https://gerrit.wikimedia.org/r/512729 (https://phabricator.wikimedia.org/T224371) [16:28:25] (03CR) 10Florianschmidtwelzow: "This change is ready for review." [integration/config] - 10https://gerrit.wikimedia.org/r/512729 (https://phabricator.wikimedia.org/T224371) (owner: 10Florianschmidtwelzow) [16:39:12] RECOVERY - Work requests waiting in Zuul Gearman server on contint1001 is OK: OK: Less than 30.00% above the threshold [90.0] https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1 [17:18:44] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Backlog), 10Release Pipeline: contint1001:/var/lib/docker growth - https://phabricator.wikimedia.org/T207702 (10greg) [17:18:46] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Operations, 10Release Pipeline: Switch CI Docker Storage Driver to its own partition and to use devicemapper - https://phabricator.wikimedia.org/T178663 (10greg) [17:18:49] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Operations: contint1001 store docker images on separate partition or disk - https://phabricator.wikimedia.org/T207707 (10greg) 05Open→03Stalled stalled until the disks are installed [18:49:20] 10Release-Engineering-Team (Watching / External), 10MediaWiki-Database, 10Performance-Team, 10MW-1.34-notes (1.34.0-wmf.7; 2019-05-28), and 4 others: 1.34.0-wmf.3 generating lots of temporary tables on MySQL slaves - https://phabricator.wikimedia.org/T223978 (10Krinkle) 05Open→03Resolved a:03aaron Re... [18:49:48] 10Release-Engineering-Team (Watching / External), 10MediaWiki-Database, 10Performance-Team, 10MW-1.34-notes (1.34.0-wmf.7; 2019-05-28), and 2 others: 1.34.0-wmf.3 generating lots of temporary tables on MySQL slaves - https://phabricator.wikimedia.org/T223978 (10Krinkle) [18:50:00] 10Release-Engineering-Team (Watching / External), 10MediaWiki-Database, 10Performance-Team, 10MW-1.34-notes (1.34.0-wmf.7; 2019-05-28), and 2 others: 1.34.0-wmf.3 generating lots of temporary tables on MySQL slaves - https://phabricator.wikimedia.org/T223978 (10Krinkle) (Untagging from release, given the p... [20:00:15] 10Release-Engineering-Team, 10Operations, 10SRE-Access-Requests: Request access to analytics cluster for Alaa Sarhan - https://phabricator.wikimedia.org/T223697 (10Nuria) @alaa_wmde Can you be a bit more specific as to what data you need access to? So you know Logstash and hadoop do not share any data, mayb... [20:15:43] (03PS1) 10QChris: Allow “Gerrit Managers” to import history [labs/tools/shorturls] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/512764 [20:15:45] (03CR) 10QChris: [V: 03+2 C: 03+2] Allow “Gerrit Managers” to import history [labs/tools/shorturls] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/512764 (owner: 10QChris) [20:15:50] (03PS1) 10QChris: Import done. Revoke import grants [labs/tools/shorturls] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/512765 [20:15:52] (03CR) 10QChris: [V: 03+2 C: 03+2] Import done. Revoke import grants [labs/tools/shorturls] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/512765 (owner: 10QChris) [20:34:47] 10Release-Engineering-Team (Watching / External), 10MediaWiki-Database, 10Performance-Team, 10MW-1.34-notes (1.34.0-wmf.6; 2019-05-21), and 2 others: 1.34.0-wmf.3 generating lots of temporary tables on MySQL slaves - https://phabricator.wikimedia.org/T223978 (10Marostegui) Thank you! [20:52:43] RECOVERY - Free space - all mounts on integration-slave-docker-1021 is OK: OK: All targets OK [22:04:35] 10Beta-Cluster-Infrastructure, 10Sentry: deployment-sentry01 puppet has a broken exec - https://phabricator.wikimedia.org/T224447 (10Krenair) [23:01:06] 10Project-Admins: Create project for Wiki Techstorm 2019 - https://phabricator.wikimedia.org/T222929 (10Aklapper) To summarize my understanding: * rename `#Wikistorm` to `#Wiki-Techstorm-2018` * create `#Wiki-Techstorm-2019` * re-add `#Wiki-Techstorm-2018` to https://phabricator.wikimedia.org/maniphest/?ids=2081... [23:18:14] 10Gerrit, 10Release-Engineering-Team, 10serviceops: Gerrit http threads stuck behind sendemail thread - https://phabricator.wikimedia.org/T224448 (10thcipriani) [23:18:35] 10Gerrit, 10Release-Engineering-Team, 10serviceops: Gerrit http threads stuck behind sendemail thread - https://phabricator.wikimedia.org/T224448 (10thcipriani) p:05Triage→03Normal [23:31:23] 10Gerrit, 10Release-Engineering-Team: Investigate gerrit session expiration - https://phabricator.wikimedia.org/T222472 (10thcipriani) Restarted Gerrit 2019-05-27T23:10, `show-cache` before and after: ` 2019-05-27T23:00:01+00:00.txt:D web_sessions | 364 1676 690.93k| | 96% 0%| 2... [23:33:59] 10Phabricator: Subscriber "Unknown Object (MLST)" can't be removed: "Edges are not available for objects of type 'MLST'!" - https://phabricator.wikimedia.org/T122677 (10Aklapper) @mmodell: Do you think that updating the DB as proposed in my previous comment would be worth a shot, or do you consider that too risk...