[00:48:48] !log Restarting Jenkins. [00:48:49] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [00:59:05] 10Gerrit, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)): Test upgrade of Gerrit to 2.16.13 on the ganeti test instance - https://phabricator.wikimedia.org/T243981 (10Jdforrester-WMF) [00:59:15] 10Gerrit, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)): Test u - https://phabricator.wikimedia.org/T243982 (10Jdforrester-WMF) [00:59:54] 10Gerrit, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)): Test backup and restore of on the ganeti test instance - https://phabricator.wikimedia.org/T243982 (10Jdforrester-WMF) a:05thcipriani→03None [01:00:53] 10Gerrit, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10serviceops: Add second virtual hard disk to ganeti gerrit test instance - https://phabricator.wikimedia.org/T243983 (10Jdforrester-WMF) [01:06:32] James_F backup/restore the db? [01:11:55] paladox|UKInEU: Yes. [01:13:07] i thought the user we are using for the ganeti instance is read only? Or do we want to back it up in case on the upgrade day? [01:15:07] paladox|UKInEU: We decided that we wanted to do a full dress rehearsal of the backup/restore process in test before we find out that our runbook doesn't work and we need it in anger for production. :-) [01:15:22] heh [01:16:13] James_F wouldn't we need to also have a backup of git repos? (latest) :) [01:16:22] Since the repos act as a db too [01:21:33] the db only really contains groups + there's another table. [03:26:47] 10Gerrit, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)): Test backup and restore of Gerrit on the ganeti test instance - https://phabricator.wikimedia.org/T243982 (10Dzahn) [03:28:07] paladox|UKInEU is right. the test instance uses a readonly DB user [03:28:21] (on purpose so that it can't mess with prod data no matter what) [03:28:31] bbl [08:14:23] PROBLEM - Free space - all mounts on deployment-logstash03 is CRITICAL: CRITICAL: deployment-prep.deployment-logstash03.diskspace.root.byte_percentfree (<100.00%) [09:30:04] (03PS1) 10Ricordisamoa: New email [integration/config] - 10https://gerrit.wikimedia.org/r/569188 [09:33:52] (03CR) 10Addshore: [C: 03+1] Prepare to run browser suites in parallel [integration/quibble] - 10https://gerrit.wikimedia.org/r/545661 (https://phabricator.wikimedia.org/T226869) (owner: 10Awight) [09:39:17] (03CR) 10Ricordisamoa: "TBH not much is 'Pywikibot related' these days..." (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/569188 (owner: 10Ricordisamoa) [10:38:01] (03CR) 10MarcoAurelio: [C: 03+1] "LGTM from the code POV." [integration/config] - 10https://gerrit.wikimedia.org/r/569188 (owner: 10Ricordisamoa) [11:47:05] !log lucaswerkmeister-wmde@deployment-deploy01:~$ mwscript createAndPromote.php enwiki 'Lucas Werkmeister (WMDE)' --force --custom-groups=bureaucrat # make myself bureaucrat to make colleagues sysops [11:47:06] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [11:48:26] hm, that doesn’t seem to have worked [11:52:23] it didn’t print an error, though [11:52:26] strange [11:53:36] tried it again with --bureaucrat instead of --custom-groups=bureaucrat, “account exists and nothing to do” [11:54:31] okay, I’m bureaucrat in the API but not on Special:UserRights? [11:55:37] but I was able to sysop my colleague through the API, so, 🤷 [12:00:17] I can grant you rights there if needed Lucas_WMDE [12:03:06] hmpf this is funny [12:03:28] so special:userrights at deploymentwiki shows you're an admin, bureaucrat and confirmed user [12:03:42] hauskatze: I’m creating a Phab task right now, something’s broken here [12:03:42] special:userrights at enwiki beta shows you're just an admin and confirmed user [12:03:53] weird [12:04:15] Lucas_WMDE: appreciated. Beta and broken are synonyms :-) [12:16:14] https://phabricator.wikimedia.org/T244019 [12:16:16] 10Beta-Cluster-Infrastructure, 10MediaWiki-User-management: Inconsistent user permissions on Beta English Wikipedia - https://phabricator.wikimedia.org/T244019 (10Lucas_Werkmeister_WMDE) [12:16:48] 10Beta-Cluster-Infrastructure, 10MediaWiki-User-management: Inconsistent user permissions on Beta English Wikipedia - https://phabricator.wikimedia.org/T244019 (10Lucas_Werkmeister_WMDE) [12:16:51] 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10Release, 10Train Deployments: 1.35.0-wmf.18 deployment blockers - https://phabricator.wikimedia.org/T233866 (10Lucas_Werkmeister_WMDE) [12:17:17] 10Beta-Cluster-Infrastructure, 10MediaWiki-User-management: Inconsistent user permissions on Beta English Wikipedia - https://phabricator.wikimedia.org/T244019 (10Lucas_Werkmeister_WMDE) Tentatively marking this as a train blocker, in case it’s a MediaWiki bug – we wouldn’t want something like this to hit prod... [12:29:59] 10Beta-Cluster-Infrastructure, 10MediaWiki-User-management: Inconsistent user permissions on Beta English Wikipedia - https://phabricator.wikimedia.org/T244019 (10Lucas_Werkmeister_WMDE) Database says I’m a bureaucrat: ` MariaDB [enwiki]> SELECT ug_group FROM user_groups WHERE ug_user = (SELECT user_id FROM u... [12:34:49] 10Beta-Cluster-Infrastructure, 10MediaWiki-User-management: Inconsistent user permissions on Beta English Wikipedia - https://phabricator.wikimedia.org/T244019 (10Lucas_Werkmeister_WMDE) PHP also agrees: `lang=shell-session lucaswerkmeister-wmde@deployment-deploy01:~$ mwscript eval.php enwiki <<< 'var_export(... [12:41:45] 10Beta-Cluster-Infrastructure, 10Browser-Support-Firefox: en.wikipedia.beta.wmflabs.org not accessible in Firefox, due to something with OCSP - https://phabricator.wikimedia.org/T243881 (10Kaartic) I can't reproduce this in Firefox 72.0.2 on Windows 10. Though, we're facing an issue in the Wikimedia Commons An... [16:39:11] 10Phabricator, 10User-LokalProfil, 10WMSE (IT): Move subproject into a user project - https://phabricator.wikimedia.org/T240710 (10Lokal_Profil) >>! In T240710#5820389, @mmodell wrote: > @Lokal_Profil I'll attempt to move the project today if I can. Will report back after it's done. Looks like it didn't wor... [16:41:19] 10Beta-Cluster-Infrastructure, 10MediaWiki-User-management: Inconsistent user permissions on Beta English Wikipedia - https://phabricator.wikimedia.org/T244019 (10Ammarpad) >But Special:UserRights/Lucas Werkmeister (WMDE) doesn’t show the group: >>Member of: Confirmed users, Administrators >>Implicit member of... [16:42:24] 10Beta-Cluster-Infrastructure, 10MediaWiki-User-management: Inconsistent user permissions on Beta English Wikipedia - https://phabricator.wikimedia.org/T244019 (10Lucas_Werkmeister_WMDE) Now I’m seeing it as well… are those cached somewhere? [17:44:21] 10Beta-Cluster-Infrastructure, 10Browser-Support-Firefox: en.wikipedia.beta.wmflabs.org not accessible in Firefox, due to something with OCSP - https://phabricator.wikimedia.org/T243881 (10Krenair) `alex@alex-laptop:~$ ssh deployment-cache-text05 Linux deployment-cache-text05 4.9.0-8-amd64 #1 SMP Debian 4.9.14... [18:14:20] PROBLEM - Work requests waiting in Zuul Gearman server on contint1001 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [150.0] https://www.mediawiki.org/wiki/Continuous_integration/Zuul https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1 [20:40:44] (03Abandoned) 10MarcoAurelio: Grant `wikidata-query` people more access in wikidata/query/rdf.git [wikidata/query/rdf] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/567112 (https://phabricator.wikimedia.org/T243431) (owner: 10MarcoAurelio) [20:41:41] (03PS1) 10MarcoAurelio: Grant `wikidata-query` people more access in wikidata/query/rdf.git [wikidata/query/rdf] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/569280 (https://phabricator.wikimedia.org/T243431) [20:41:52] (03Abandoned) 10MarcoAurelio: Grant `wikidata-query` people more access in wikidata/query/rdf.git [wikidata/query/rdf] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/569280 (https://phabricator.wikimedia.org/T243431) (owner: 10MarcoAurelio) [20:43:56] (03PS1) 10MarcoAurelio: Grant `wikidata-query` people more access in wikidata/query/rdf.git [wikidata/query/rdf] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/569281 (https://phabricator.wikimedia.org/T243431) [20:45:11] (03Restored) 10MarcoAurelio: Grant `wikidata-query` people more access in wikidata/query/rdf.git [wikidata/query/rdf] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/569280 (https://phabricator.wikimedia.org/T243431) (owner: 10MarcoAurelio) [20:45:41] (03PS2) 10MarcoAurelio: Wrong empty commit [wikidata/query/rdf] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/569280 [20:45:43] (03Abandoned) 10MarcoAurelio: Wrong empty commit [wikidata/query/rdf] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/569280 (owner: 10MarcoAurelio) [20:50:18] 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10User-zeljkofilipin: Arrange PGP keysigning at All Hands 2020 - https://phabricator.wikimedia.org/T242340 (10Dzahn) What is the time of the party today? [20:50:32] RECOVERY - Work requests waiting in Zuul Gearman server on contint1001 is OK: OK: Less than 100.00% above the threshold [90.0] https://www.mediawiki.org/wiki/Continuous_integration/Zuul https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1 [20:52:36] 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10User-zeljkofilipin: Arrange PGP keysigning at All Hands 2020 - https://phabricator.wikimedia.org/T242340 (10LarsWirzenius) I happened already. I'm always happy to exchange key fingerprints personally, though.