[00:24:49] 10Gerrit, 10LDAP: Gerrit login failure for user SMcCandlish - https://phabricator.wikimedia.org/T196597#4262796 (10bd808) [00:33:57] 10Release-Engineering-Team, 10DNS, 10Operations, 10Traffic, and 2 others: Move Foundation Wiki to new URL when new Wikimedia Foundation website launches - https://phabricator.wikimedia.org/T188776#4262808 (10Varnent) @greg - July 30 has been set as target launch date. [01:30:20] 10Beta-Cluster-Infrastructure, 10Operations: confd broken on deployment-redis hosts - https://phabricator.wikimedia.org/T196596#4262846 (10Reedy) Brandon reckons it's something to do with `confd::srv_dns` not being set correctly on beta I'm guessing the config got broken a little while ago, and because it did... [01:34:34] twentyafterfour: phab email seems to be broken [01:34:37] Same with gerrit [01:34:43] 10Gerrit, 10Phabricator, 10Mail, 10Operations: Phab and Gerrit emails stopped at around 1900 UTC 6th June - https://phabricator.wikimedia.org/T196598#4262848 (10Reedy) [01:35:18] 10Gerrit, 10Phabricator, 10Mail, 10Operations: Phab and Gerrit emails stopped at around 1900 UTC 6th June - https://phabricator.wikimedia.org/T196598#4262858 (10Reedy) p:05Triage>03High [01:41:34] 10Gerrit, 10Phabricator, 10Mail, 10Operations: Phab and Gerrit emails stopped at around 1900 UTC 6th June - https://phabricator.wikimedia.org/T196598#4262848 (10Legoktm) https://grafana.wikimedia.org/dashboard/db/mail?refresh=5m&orgId=1&from=now-2d&to=now suggests all mail is down, but I'm still getting ma... [01:52:38] 10Gerrit, 10Phabricator, 10Mail, 10Operations: Phab and Gerrit emails stopped at around 1900 UTC 6th June - https://phabricator.wikimedia.org/T196598#4262868 (10Reedy) Mail seems to be generally flowing through codfw though... https://grafana.wikimedia.org/dashboard/db/mail?refresh=5m&orgId=1&from=now-2d&t... [02:12:11] 10Gerrit, 10Phabricator, 10Mail, 10Operations: Phab and Gerrit emails stopped at around 1900 UTC 6th June - https://phabricator.wikimedia.org/T196598#4262872 (10faidon) 05Open>03Resolved a:03faidon The cause was the prep for T175361, in combination with a couple of unexpected misconfigurations/SPOFs,... [06:04:50] (03CR) 10Chad: [C: 04-1] "Wrote a magnum opus in the inline comment" (031 comment) [tools/release] - 10https://gerrit.wikimedia.org/r/437867 (owner: 10Legoktm) [06:11:21] (03CR) 10Chad: [C: 032] makerelease: Don't run composer [tools/release] - 10https://gerrit.wikimedia.org/r/437867 (owner: 10Legoktm) [06:11:56] (03Merged) 10jenkins-bot: makerelease: Don't run composer [tools/release] - 10https://gerrit.wikimedia.org/r/437867 (owner: 10Legoktm) [06:23:37] 10MediaWiki-Releasing, 10Security: Streamline MW security release process - https://phabricator.wikimedia.org/T196602#4262983 (10greg) [06:40:19] 10Phabricator, 10Release-Engineering-Team: Phabricator replica functionality broken, needs port configuration - https://phabricator.wikimedia.org/T196604#4263010 (10jcrespo) [06:45:35] 10Release-Engineering-Team (Kanban), 10Wiki-Setup (Close): Close chairwiki - https://phabricator.wikimedia.org/T184961#4263043 (10demon) a:05demon>03None [06:45:47] 10Deployments, 10Release-Engineering-Team (Kanban), 10Operations, 10Patch-For-Review, 10User-Addshore: cannot delete non-empty directory: php-1.29.0-wmf.3 messages on 'scap sync' on mwdebug1002 - https://phabricator.wikimedia.org/T157030#4263044 (10demon) a:05demon>03None [07:59:45] (03PS1) 10Hashar: Migrate ShortUrl to Quibble [integration/config] - 10https://gerrit.wikimedia.org/r/437920 (https://phabricator.wikimedia.org/T183512) [08:00:03] (03CR) 10Hashar: [C: 032] Migrate ShortUrl to Quibble [integration/config] - 10https://gerrit.wikimedia.org/r/437920 (https://phabricator.wikimedia.org/T183512) (owner: 10Hashar) [08:00:08] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10releng-201718-q3, 10Epic, 10Patch-For-Review: [EPIC] Migrate Mediawiki jobs from Nodepool to Docker - https://phabricator.wikimedia.org/T183512#4263144 (10hashar) [08:02:11] (03Merged) 10jenkins-bot: Migrate ShortUrl to Quibble [integration/config] - 10https://gerrit.wikimedia.org/r/437920 (https://phabricator.wikimedia.org/T183512) (owner: 10Hashar) [08:15:55] 10Continuous-Integration-Config, 10Release-Engineering-Team (Kanban), 10ContentTranslation, 10ContentTranslation-Deployments, and 3 others: Add Backport Repository to Jenkins Package Build - https://phabricator.wikimedia.org/T196037#4263184 (10KartikMistry) 05Open>03Resolved [08:17:05] 10Continuous-Integration-Config, 10Release-Engineering-Team (Kanban), 10ContentTranslation, 10ContentTranslation-Deployments, and 2 others: Add Backport Repository to Jenkins Package Build - https://phabricator.wikimedia.org/T196037#4263190 (10Petar.petkovic) [08:20:15] Hi https://en.wikipedia.beta.wmflabs.org/ seems to be down [08:20:27] (Someone reported that in #wikipedia-en ) [08:20:43] If you report this error to the Wikimedia System Administrators, please include the details below. [08:20:44] PHP fatal error: [08:20:44] Stack overflow [08:22:15] 10Continuous-Integration-Config, 10Release-Engineering-Team (Kanban), 10ContentTranslation, 10ContentTranslation-Deployments, and 2 others: Add Backport Repository to Jenkins Package Build - https://phabricator.wikimedia.org/T196037#4263197 (10hashar) [08:43:46] 10Release-Engineering-Team, 10Cleanup, 10GitHub-Mirrors, 10Repository-Admins, 10User-MarcoAurelio: Archive the CategorySlideShow extension - https://phabricator.wikimedia.org/T186969#4263232 (10MarcoAurelio) `:-)` [08:58:34] 10Continuous-Integration-Config: Add 'noop' CI to mediawiki/(extensions|skins} - https://phabricator.wikimedia.org/T196615#4263308 (10MarcoAurelio) [09:13:32] https://wikidata.beta.wmflabs.org/ also down [09:14:41] anyone aware of what's wrong/what should we do? [09:15:11] !log added wmde-leszek to deployment-prep [09:15:13] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [09:26:23] Stack Overflow! [09:30:11] paladox: stack overflows, stack overflows everywhere [09:31:20] paladox: is there a ticket? [09:47:24] addshore: heh I think there is an old task [09:47:46] addshore: https://phabricator.wikimedia.org/T185055 [09:49:36] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Lexicographical data, 10Math, and 3 others: MediaWiki core's selenium tests flaky when run as part of mwext-mw-selenium-node-composer-jessie job - https://phabricator.wikimedia.org/T191537#4263460 (10hashar) Looking at [[ https... [09:56:57] (03Draft1) 10MarcoAurelio: Mark repository as read only [extensions/WikivoteMapsYandex] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/437942 (https://phabricator.wikimedia.org/T193844) [09:57:01] (03PS2) 10MarcoAurelio: Mark repository as read only [extensions/WikivoteMapsYandex] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/437942 (https://phabricator.wikimedia.org/T193844) [10:09:44] OHH [10:09:54] addshore: zeljkof: I got quibble to run the Wikibase phpunit tests :] [10:10:11] Tests: 9156, Assertions: 57864, Skipped: 207. [10:10:19] hashar: yeah! :D [10:10:27] and it does not require any extension dependency ahha [10:10:40] (I guess that is why tests are skipped) [10:11:15] (03PS1) 10Hashar: Prepend mediawiki configuration [integration/quibble] - 10https://gerrit.wikimedia.org/r/437943 (https://phabricator.wikimedia.org/T196013) [10:11:57] going out for a walk to celebrate [10:13:56] 10Release-Engineering-Team (Kanban), 10MinervaNeue, 10Readers-Web-Backlog, 10MW-1.30-release-notes (WMF-deploy-2017-08-29 (1.30.0-wmf.16)), and 3 others: Minerva Ruby and Node.js browser tests running side by side - https://phabricator.wikimedia.org/T190710#4263495 (10zeljkofilipin) a:05Jdrewniak>03zelj... [10:29:37] (03CR) 10Hashar: [C: 032] Prepend mediawiki configuration [integration/quibble] - 10https://gerrit.wikimedia.org/r/437943 (https://phabricator.wikimedia.org/T196013) (owner: 10Hashar) [10:30:11] (03CR) 10Hashar: [C: 04-2] Prepend mediawiki configuration [integration/quibble] - 10https://gerrit.wikimedia.org/r/437943 (https://phabricator.wikimedia.org/T196013) (owner: 10Hashar) [10:31:28] (03PS2) 10Hashar: Prepend mediawiki configuration [integration/quibble] - 10https://gerrit.wikimedia.org/r/437943 (https://phabricator.wikimedia.org/T196013) [10:34:02] (03PS1) 10Hashar: Fix comments in mediawiki.d files [integration/quibble] - 10https://gerrit.wikimedia.org/r/437946 [10:34:18] (03CR) 10Hashar: [C: 032] "Updated mediawiki.d/README" [integration/quibble] - 10https://gerrit.wikimedia.org/r/437943 (https://phabricator.wikimedia.org/T196013) (owner: 10Hashar) [10:34:23] (03CR) 10Hashar: [C: 032] Fix comments in mediawiki.d files [integration/quibble] - 10https://gerrit.wikimedia.org/r/437946 (owner: 10Hashar) [10:35:12] (03Merged) 10jenkins-bot: Prepend mediawiki configuration [integration/quibble] - 10https://gerrit.wikimedia.org/r/437943 (https://phabricator.wikimedia.org/T196013) (owner: 10Hashar) [10:35:14] (03Merged) 10jenkins-bot: Fix comments in mediawiki.d files [integration/quibble] - 10https://gerrit.wikimedia.org/r/437946 (owner: 10Hashar) [10:35:48] (03CR) 10jenkins-bot: Prepend mediawiki configuration [integration/quibble] - 10https://gerrit.wikimedia.org/r/437943 (https://phabricator.wikimedia.org/T196013) (owner: 10Hashar) [10:36:18] (03CR) 10jenkins-bot: Fix comments in mediawiki.d files [integration/quibble] - 10https://gerrit.wikimedia.org/r/437946 (owner: 10Hashar) [10:37:53] (03PS1) 10Hashar: docker: quibble 0.0.18 [integration/config] - 10https://gerrit.wikimedia.org/r/437947 (https://phabricator.wikimedia.org/T196013) [10:39:19] (03PS1) 10Hashar: Bump Quibble jobs to 0.0.18 [integration/config] - 10https://gerrit.wikimedia.org/r/437948 (https://phabricator.wikimedia.org/T196013) [10:39:29] (03CR) 10Hashar: [C: 032] docker: quibble 0.0.18 [integration/config] - 10https://gerrit.wikimedia.org/r/437947 (https://phabricator.wikimedia.org/T196013) (owner: 10Hashar) [10:39:50] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10MediaWiki-extensions-WikibaseRepository, 10Wikidata, 10Patch-For-Review: Get Wikibase + dependencies to run with Quibble - https://phabricator.wikimedia.org/T196013#4263546 (10hashar) a:03hashar [10:40:52] (03Merged) 10jenkins-bot: docker: quibble 0.0.18 [integration/config] - 10https://gerrit.wikimedia.org/r/437947 (https://phabricator.wikimedia.org/T196013) (owner: 10Hashar) [10:41:22] !log Building Docker images for Quibble 0.0.18 [10:41:24] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [10:59:13] (03CR) 10Thiemo Kreuz (WMDE): [C: 032] deploy-promote: Remove extra newline from results [tools/release] - 10https://gerrit.wikimedia.org/r/437798 (owner: 10Thcipriani) [10:59:48] (03Merged) 10jenkins-bot: deploy-promote: Remove extra newline from results [tools/release] - 10https://gerrit.wikimedia.org/r/437798 (owner: 10Thcipriani) [11:15:15] 10Beta-Cluster-Infrastructure, 10Operations, 10Performance-Team (Radar), 10Puppet: Define scap::sources in a way that is shared between prod and beta - https://phabricator.wikimedia.org/T196034#4263635 (10Krinkle) [11:22:48] (03CR) 10Hashar: [C: 032] Bump Quibble jobs to 0.0.18 [integration/config] - 10https://gerrit.wikimedia.org/r/437948 (https://phabricator.wikimedia.org/T196013) (owner: 10Hashar) [11:22:59] !log Switching quibble jenkins jobs to 0.0.18 images [11:23:01] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [11:23:11] took 40 minutes to build the images :/ [11:24:52] (03Merged) 10jenkins-bot: Bump Quibble jobs to 0.0.18 [integration/config] - 10https://gerrit.wikimedia.org/r/437948 (https://phabricator.wikimedia.org/T196013) (owner: 10Hashar) [11:26:16] dcausse, ebernhardson: I had a question at twitter (of all places), somebody wants to run CirrusSearch tests (tests/integration) but there is no readme file explaining how to do it [11:26:42] readme file in root says "testing - see tests" but there is no readme file there :D [11:32:01] figured it out, will document :D [11:36:28] 10Continuous-Integration-Infrastructure: CI: upgrade tox, currently running 2.6.0 - https://phabricator.wikimedia.org/T196628#4263671 (10Volans) [11:36:31] zeljkof: thanks! I was sure we had a README un tests/integration, I must have dreamed probably :( [11:36:51] :D [11:37:16] I'll make the simplest readme ever, but enough for me to remind me the next time I need it :D [11:37:41] ok :) [11:40:36] I wonder who's trying to run our integration tests, they must be brave :) [11:46:07] it's Chris :D https://twitter.com/zeljkofilipin/status/1004367483132882946 [11:46:20] dcausse: ^ [11:48:52] dcausse: https://gerrit.wikimedia.org/r/#/c/437969/ [11:48:53] (03PS1) 10Hashar: Do not git clean language-screenshots-VisualEditor [integration/config] - 10https://gerrit.wikimedia.org/r/437970 (https://phabricator.wikimedia.org/T196374) [11:51:40] zeljkof: oh that makes sense, they want examples. Running them might be a bit difficult because you need to setup MW with vagrant/cirrus/elastic/custom plugins & co [11:52:09] worked for me, just with cirrussearch role provisioned [11:52:20] (running the tests worked) [11:52:42] but yes, I think he is mostly interested in example code, but it's nice to be able to run it [11:52:43] ah yes I'm stupid, mw-vagrant should setup this all for you [11:52:53] indeed [11:53:47] until we have something better, mw-vagrant does the job, most of the time :'D [12:11:24] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10MediaWiki-extensions-WikibaseRepository, 10Wikidata, 10Patch-For-Review: Get Wikibase + dependencies to run with Quibble - https://phabricator.wikimedia.org/T196013#4263710 (10hashar) @addshore first build ever pas... [12:11:34] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10MediaWiki-extensions-WikibaseRepository, 10Wikidata, 10Patch-For-Review: Get Wikibase + dependencies to run with Quibble - https://phabricator.wikimedia.org/T196013#4263712 (10hashar) [12:11:38] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Lexicographical data, 10Math, and 3 others: MediaWiki core's selenium tests flaky when run as part of mwext-mw-selenium-node-composer-jessie job - https://phabricator.wikimedia.org/T191537#4109191 (10hashar) [12:13:41] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Lexicographical data, 10Math, and 3 others: MediaWiki core's selenium tests flaky when run as part of mwext-mw-selenium-node-composer-jessie job - https://phabricator.wikimedia.org/T191537#4263719 (10hashar) I have completed fi... [12:21:39] hashar: race condition? :O [12:21:59] addshore: yeah somehow a user session is lost for the WikiAdmin when trying to undelete a page :( [12:22:06] https://phabricator.wikimedia.org/T191537 is the messy task [12:22:24] hmm, thats something i have seen in browser tests before [12:22:36] oh yes that ticket [12:22:42] i never did figure out wtf was happening there [12:22:49] 10Phabricator: Dot overlaps with string "Notification server not enabled" at the bottom of the notification bar - https://phabricator.wikimedia.org/T166448#4263748 (10Kaartic) 05Open>03Invalid [12:23:04] 10Phabricator: Dot overlaps with string "Notification server not enabled" at the bottom of the notification bar - https://phabricator.wikimedia.org/T166448#3296479 (10Kaartic) Unable to see the weird dot these days. [12:23:56] well, good luck hashar :D [12:23:59] beta cluster wikis are down, maybe this is expected? [12:24:00] [aa7ea6670b9ab6339930954f] [no req] Error from line 35 of /home/hashar/projects/mediawiki/extensions/Wikibase/lib/WikibaseLib.entitytypes.php: Class 'Wikibase\DataModel\Entity\ItemId' not found [12:24:01] :( [12:24:21] hashar: interesting, when did that happen? [12:24:28] locally :] [12:24:31] awight: indeed, been seeing "stack overflow" for quite some hours now [12:25:02] addshore: gotcha, thanks! [12:25:17] * awight wonders if WP was sold to SO [12:27:22] addshore: I have just forgot to add Wikibase to composer.local.json haha [12:28:48] 10Release-Engineering-Team (Kanban), 10VisualEditor, 10Patch-For-Review, 10User-zeljkofilipin: language-screenshots-VisualEditor fails with `stderr: warning: failed to remove cache` - https://phabricator.wikimedia.org/T196374#4263769 (10zeljkofilipin) a:03zeljkofilipin [12:29:40] 10Release-Engineering-Team (Kanban), 10VisualEditor, 10Patch-For-Review, 10User-zeljkofilipin: language-screenshots-VisualEditor fails with `stderr: warning: failed to remove cache` - https://phabricator.wikimedia.org/T196374#4254001 (10zeljkofilipin) a:05zeljkofilipin>03hashar @hashar is fixing the jo... [12:30:30] ahahahahaha [12:30:34] I reproduced on my first try [12:30:37] I am such a hacker [12:39:21] zeljkof: and i found https://www.npmjs.com/package/wdio-screenshot :] [12:40:08] hashar: um, we already take screenshots... [12:40:15] what did you reproduce? [12:40:39] wikibase selenium failures? [12:43:17] 10Release-Engineering-Team (Kanban), 10VisualEditor, 10Patch-For-Review, 10User-zeljkofilipin: language-screenshots-VisualEditor fails with `stderr: warning: failed to remove cache` - https://phabricator.wikimedia.org/T196374#4263800 (10zeljkofilipin) [[ https://integration.wikimedia.org/ci/job/language-sc... [12:44:32] zeljkof: yes [12:44:38] 10Release-Engineering-Team (Kanban), 10VisualEditor, 10Patch-For-Review, 10User-zeljkofilipin: language-screenshots-VisualEditor fails with `stderr: warning: failed to remove cache` - https://phabricator.wikimedia.org/T196374#4263804 (10zeljkofilipin) [[ https://integration.wikimedia.org/ci/job/language-sc... [12:44:40] but somehow my wiki no more logs anything bah [12:45:14] :D [12:47:09] maybe it was the GDPR extension that stopped loggin on the 25th :p j/k [12:48:47] zeljkof: ah no I was using MW_USERNAME instead of MEDIAWIKI_USER :] [12:50:45] mutante: that's hilarious :'D [12:55:23] Something is always broken when you go to create new wikis : [12:55:23] :P [12:56:21] 10Release-Engineering-Team (Kanban), 10Quibble, 10Patch-For-Review: Error: 1071 Specified key was too long; max key length is 767 bytes - https://phabricator.wikimedia.org/T193222#4263831 (10hashar) [12:56:27] 10Release-Engineering-Team (Kanban), 10Quibble, 10Patch-For-Review: Error: 1071 Specified key was too long; max key length is 767 bytes - https://phabricator.wikimedia.org/T193222#4163311 (10hashar) [13:06:06] 10Phabricator, 10Release-Engineering-Team: Phabricator replica functionality broken, needs port configuration - https://phabricator.wikimedia.org/T196604#4263848 (10jcrespo) [13:07:20] 10Beta-Cluster-Infrastructure, 10CirrusSearch, 10Discovery, 10Discovery-Search, 10Beta-Cluster-reproducible: Exception from line 653 of MetaStoreIndex.php: meta store does not exist, you must index your data first - https://phabricator.wikimedia.org/T196632#4263850 (10Reedy) [13:13:53] 10Phabricator, 10Release-Engineering-Team: Phabricator replica functionality broken, needs port configuration - https://phabricator.wikimedia.org/T196604#4263869 (10jcrespo) Also seen as: ``` Traceback (most recent call last): File "/srv/phab/tools/public_task_dump.py", line 25, in mdb = dbcon... [13:15:38] 10Beta-Cluster-Infrastructure, 10CirrusSearch, 10Discovery, 10Discovery-Search, 10Beta-Cluster-reproducible: Exception from line 653 of MetaStoreIndex.php: meta store does not exist, you must index your data first - https://phabricator.wikimedia.org/T196632#4263875 (10Reedy) 05Open>03Resolved a:03dc... [13:16:49] 10Beta-Cluster-Infrastructure, 10CirrusSearch, 10Discovery, 10Discovery-Search, 10Beta-Cluster-reproducible: Exception from line 653 of MetaStoreIndex.php: meta store does not exist, you must index your data first - https://phabricator.wikimedia.org/T196632#4263881 (10dcausse) for the record the script t... [13:30:43] somehow my apache runs php file_put_contents() just fine [13:30:48] but nothing get written to disk :( [13:33:08] https://en.wikipedia.beta.wmflabs.org/ [13:33:10] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Lexicographical data, 10Math, and 3 others: MediaWiki core's selenium tests flaky when run as part of mwext-mw-selenium-node-composer-jessie job - https://phabricator.wikimedia.org/T191537#4263920 (10hashar) I have tried locall... [13:33:12] PHP fatal error: [13:33:12] Stack overflow [13:39:33] FATAL ERROR [13:39:41] https://stackoverflow.com [13:39:49] Is the answer on there? [13:39:55] maybe [13:39:56] ;) [13:40:05] all of the beta sites have been broken all day :( [13:40:20] Sessions were broken last night... [13:40:24] And it seems an offshoot of that [13:40:30] suspicious [13:40:34] 10Release-Engineering-Team (Kanban), 10Scap, 10Patch-For-Review: Support git-lfs in scap - https://phabricator.wikimedia.org/T180627#4263974 (10awight) [13:41:06] the stackoverflow is from not being able to connect to redis... [13:41:09] which is daaaft [13:42:02] hah [13:48:55] 10Beta-Cluster-Infrastructure, 10Operations: confd broken on deployment-redis hosts - https://phabricator.wikimedia.org/T196596#4264001 (10Joe) This is not really an issue and redis is correctly working on these servers: ``` deployment-redis05:~$ systemctl status redis-instance-tcp_6379.service ● redis-insta... [13:49:06] 10Beta-Cluster-Infrastructure, 10Operations: confd broken on deployment-redis hosts - https://phabricator.wikimedia.org/T196596#4264002 (10Joe) 05Open>03Invalid [13:54:15] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Lexicographical data, 10Math, and 3 others: MediaWiki core's selenium tests flaky when run as part of mwext-mw-selenium-node-composer-jessie job - https://phabricator.wikimedia.org/T191537#4264025 (10Anomie) >>! In T191537#4263... [14:02:34] (03CR) 10Hashar: [V: 032 C: 032] Mark repository as read only [extensions/WikivoteMapsYandex] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/437942 (https://phabricator.wikimedia.org/T193844) (owner: 10MarcoAurelio) [14:03:49] !log github: deleting https://github.com/wikimedia/mediawiki-extensions-WikivoteMapsYandex | T193844 [14:03:52] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [14:03:52] T193844: Archive the WikivoteMapsYandex extension - https://phabricator.wikimedia.org/T193844 [14:10:31] 10Release-Engineering-Team (Watching / External), 10Operations, 10Patch-For-Review: setup/install/deploy deploy1001 as deployment server - https://phabricator.wikimedia.org/T175288#4264071 (10Dzahn) [14:11:04] 10Release-Engineering-Team (Watching / External), 10Operations, 10Patch-For-Review: setup/install/deploy deploy1001 as deployment server - https://phabricator.wikimedia.org/T175288#4073640 (10Dzahn) [14:16:13] <_joe_> !log killed "puppet master" process in depoloyment-prep [14:16:14] (03PS1) 10Hashar: Migrate Sentry to Quibble [integration/config] - 10https://gerrit.wikimedia.org/r/438014 (https://phabricator.wikimedia.org/T183512) [14:16:15] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [14:16:41] <_joe_> !log restarted apache2 in deployment-puppetmaster02 in deployment-prep, which is the correct way to run the puppetmaster there [14:16:42] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [14:16:44] (03CR) 10Hashar: [C: 032] Migrate Sentry to Quibble [integration/config] - 10https://gerrit.wikimedia.org/r/438014 (https://phabricator.wikimedia.org/T183512) (owner: 10Hashar) [14:18:29] (03Merged) 10jenkins-bot: Migrate Sentry to Quibble [integration/config] - 10https://gerrit.wikimedia.org/r/438014 (https://phabricator.wikimedia.org/T183512) (owner: 10Hashar) [14:18:59] <_joe_> so, apparently we changed puppetmaster, but the old one is still active and refers to itself [14:19:20] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10releng-201718-q3, 10Epic, 10Patch-For-Review: [EPIC] Migrate Mediawiki jobs from Nodepool to Docker - https://phabricator.wikimedia.org/T183512#4264096 (10hashar) [14:20:40] thcipriani: deployment-tin is still alive i here.. didnt you say you already deleted it [14:20:53] or did it need jessie for some reason [14:20:56] and was recreated [14:24:27] mutante: I didn't delete deployment-tin, last update from my side was: https://phabricator.wikimedia.org/T192561#4244792 Also, I think deployment-deploy1001 needs some other roles applied in horizon (looking at it vs deployment-tin) [14:25:09] er deployment-deploy01. I deleted deployment-deploy1001 since it was smaller than we wanted. [14:27:38] <_joe_> !log force-run puppet on deployment-prep appservers, restarted nutcracker after reconfigruation [14:27:40] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [14:30:26] thcipriani: ooh, right. yea, makes sense. thank you [14:35:20] sure thing! let me know what your plans are on that task. I can fiddle with it and make the swap next week if I'm needed. [14:42:30] ok, cool, will do. thanks [15:03:52] 10Scap: update-interwiki-cache failed: 'Namespace' object has no attribute 'force' - https://phabricator.wikimedia.org/T196642#4264272 (10Reedy) [15:23:09] 10Beta-Cluster-Infrastructure, 10Operations, 10Performance-Team (Radar), 10Puppet: Define scap::sources in a way that is shared between prod and beta - https://phabricator.wikimedia.org/T196034#4244858 (10Imarlier) Can I propose an alternative? 1. Get rid of that hiera thing 2. Write a small script that r... [15:25:01] zeljkof: I tried in a local vagrant box and neither X11 forwarding not video creation via xvfb seems to work [15:25:39] the first results in a bunch of timeouts, the second runs but doesn't create any files as far as I can see [15:27:20] tgr: on the phone now, can you please create a phab task? [15:27:39] sure [15:29:10] Thanks! [15:29:50] Reedy: did you end up making an account for 'cscott' on beta? [15:30:00] Nope.. You might be able to create your own now [15:30:10] As login is at least fixed... Not sure about captchas yet [15:30:12] today it says the username is taken, and i don't think it was yesterday (or maybe the beta problems yesterday just meant that it wasn't properly reporting it as taken) [15:31:18] cscott: You seem to have an account from 2014 [15:31:39] With an email address set you your personal domain [15:31:44] ah, thx [15:31:51] that was the next thing i had to guess ;) [15:32:03] from 2014... i think that was before I had a @wikimedia address [15:32:37] heh [15:33:18] :) [15:35:59] actually, i guess i started in 2013, so i don't know why i used my personal email. [15:36:13] anyway, logged in! now to set pagelanguage on a sandbox page.... [15:40:53] 10Release-Engineering-Team, 10MediaWiki-Vagrant, 10Browser-Tests: Running selenium inside Vagrant with xvfb or X11 does not work - https://phabricator.wikimedia.org/T196646#4264427 (10Tgr) [15:41:26] Let me know if you need rights granting anyway [15:41:28] *anywhere [15:44:06] 10MediaWiki-Releasing, 10MediaWiki-Installer, 10MW-1.31-release: Expand the set of bundled extensions and skins in MediaWiki 1.31 - https://phabricator.wikimedia.org/T196648#4264457 (10CCicalese_WMF) p:05Triage>03Normal [15:44:57] 10MediaWiki-Releasing, 10MediaWiki-Installer, 10MW-1.31-release: Expand the set of bundled extensions and skins in MediaWiki 1.31 - https://phabricator.wikimedia.org/T196648#4264470 (10CCicalese_WMF) [15:45:07] 10MediaWiki-Releasing, 10MediaWiki-extensions-CategoryTree, 10MW-1.31-release: Bundle CategoryTree extension with MW 1.31 - https://phabricator.wikimedia.org/T191735#4264469 (10CCicalese_WMF) [15:45:10] 10MediaWiki-Releasing, 10MediaWiki-Installer, 10MW-1.31-release, 10Patch-For-Review: Expand the set of bundled extensions to achieve a default MediaWiki experience that's comparable to Wikimedia sites - https://phabricator.wikimedia.org/T178349#4264471 (10CCicalese_WMF) [15:45:22] 10MediaWiki-Releasing, 10MediaWiki-Installer, 10MW-1.31-release: Expand the set of bundled extensions and skins in MediaWiki 1.31 - https://phabricator.wikimedia.org/T196648#4264457 (10CCicalese_WMF) [15:45:25] 10MediaWiki-Releasing, 10MediaWiki-extensions-ReplaceText, 10MW-1.31-release, 10MW-1.31-release-notes: Bundle Replace Text extension with MW 1.31 - https://phabricator.wikimedia.org/T191741#4264487 (10CCicalese_WMF) [15:45:28] 10Release-Engineering-Team, 10MediaWiki-Vagrant, 10Browser-Tests: Running selenium inside Vagrant with xvfb or X11 does not work - https://phabricator.wikimedia.org/T196646#4264490 (10Tgr) [15:45:31] 10MediaWiki-Releasing, 10MediaWiki-Installer, 10MW-1.31-release, 10Patch-For-Review: Expand the set of bundled extensions to achieve a default MediaWiki experience that's comparable to Wikimedia sites - https://phabricator.wikimedia.org/T178349#4109974 (10CCicalese_WMF) [15:45:42] 10MediaWiki-Releasing, 10MediaWiki-Installer, 10MW-1.31-release: Expand the set of bundled extensions and skins in MediaWiki 1.31 - https://phabricator.wikimedia.org/T196648#4264457 (10CCicalese_WMF) [15:45:47] 10MediaWiki-Releasing, 10CodeEditor, 10MW-1.31-release, 10Patch-For-Review: Bundle CodeEditor extension with MW 1.31 - https://phabricator.wikimedia.org/T191742#4264491 (10CCicalese_WMF) [15:45:50] 10MediaWiki-Releasing, 10MediaWiki-Installer, 10MW-1.31-release, 10Patch-For-Review: Expand the set of bundled extensions to achieve a default MediaWiki experience that's comparable to Wikimedia sites - https://phabricator.wikimedia.org/T178349#4110031 (10CCicalese_WMF) [15:45:55] 10MediaWiki-Releasing, 10MediaWiki-Installer, 10MW-1.31-release: Expand the set of bundled extensions and skins in MediaWiki 1.31 - https://phabricator.wikimedia.org/T196648#4264457 (10CCicalese_WMF) [15:46:00] 10MediaWiki-Releasing, 10MediaWiki-extensions-MultimediaViewer, 10Multimedia, 10MW-1.31-release, 10Patch-For-Review: Bundle Multimedia Viewer extension with MW 1.31 - https://phabricator.wikimedia.org/T191744#4264494 (10CCicalese_WMF) [15:46:04] 10MediaWiki-Releasing, 10MediaWiki-Installer, 10MW-1.31-release, 10Patch-For-Review: Expand the set of bundled extensions to achieve a default MediaWiki experience that's comparable to Wikimedia sites - https://phabricator.wikimedia.org/T178349#4110146 (10CCicalese_WMF) [15:46:06] (03PS1) 10Thiemo Kreuz (WMDE): Don't warn on @license in "namespace … { … }" scopes [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/438031 [15:46:16] 10MediaWiki-Releasing, 10MediaWiki-Installer, 10MW-1.31-release: Expand the set of bundled extensions and skins in MediaWiki 1.31 - https://phabricator.wikimedia.org/T196648#4264457 (10CCicalese_WMF) [15:46:20] 10MediaWiki-Releasing, 10MW-1.31-release, 10Patch-For-Review: Bundle OATHAuth extension with MW 1.31 - https://phabricator.wikimedia.org/T191745#4264497 (10CCicalese_WMF) [15:46:23] 10MediaWiki-Releasing, 10MediaWiki-Installer, 10MW-1.31-release, 10Patch-For-Review: Expand the set of bundled extensions to achieve a default MediaWiki experience that's comparable to Wikimedia sites - https://phabricator.wikimedia.org/T178349#4110196 (10CCicalese_WMF) [15:46:53] (03CR) 10jerkins-bot: [V: 04-1] Don't warn on @license in "namespace … { … }" scopes [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/438031 (owner: 10Thiemo Kreuz (WMDE)) [15:47:14] Reedy: i think i do, just trying to dig up which rights i need [15:49:33] Reedy: https://www.mediawiki.org/wiki/Manual:Language#Page_content_language indicates I need the "pagelang" permission [15:50:02] but also that $wgPageLanguageUseDB needs to be true, dunno if that's the case [15:50:06] cscott: Which wiki do you want it on? [15:50:14] I did create the srwiki and crhwiki btw [15:50:19] https://deployment.wikimedia.beta.wmflabs.org/wiki/User:Cscott/SrTest [15:50:19] If you just want to use those.... [15:50:29] oh, but maybe it's easier just to use srwiki in labs, thx [15:50:43] 10MediaWiki-Releasing, 10MediaWiki-Installer, 10Epic, 10MW-1.32-release: Expand the set of bundled extensions and skins in MediaWiki 1.32 - https://phabricator.wikimedia.org/T196650#4264512 (10CCicalese_WMF) p:05Triage>03Normal [15:52:15] 10MediaWiki-Releasing, 10MediaWiki-Installer, 10Epic, 10MW-1.31-release: Expand the set of bundled extensions and skins in MediaWiki 1.31 - https://phabricator.wikimedia.org/T196648#4264528 (10CCicalese_WMF) [15:53:47] 10MediaWiki-Releasing, 10MediaWiki-Installer, 10Epic, 10MW-1.32-release: Expand the set of bundled extensions and skins in MediaWiki 1.32 - https://phabricator.wikimedia.org/T196650#4264533 (10CCicalese_WMF) [15:53:54] 10MediaWiki-Releasing, 10MediaWiki-extensions-LoginNotify, 10MW-1.31-release: Bundle LoginNotify extension with MW 1.31 - https://phabricator.wikimedia.org/T191746#4264532 (10CCicalese_WMF) [15:53:57] 10MediaWiki-Releasing, 10MediaWiki-Installer, 10MW-1.31-release, 10Patch-For-Review: Expand the set of bundled extensions to achieve a default MediaWiki experience that's comparable to Wikimedia sites - https://phabricator.wikimedia.org/T178349#4264534 (10CCicalese_WMF) [15:54:02] 10MediaWiki-Releasing, 10MediaWiki-extensions-LoginNotify, 10MW-1.31-release: Bundle LoginNotify extension with MW 1.32 - https://phabricator.wikimedia.org/T191746#4115206 (10CCicalese_WMF) [15:54:20] 10MediaWiki-Releasing, 10AntiSpoof, 10MW-1.31-release: Bundle AntiSpoof extension with MW 1.31 - https://phabricator.wikimedia.org/T191736#4264541 (10CCicalese_WMF) [15:54:24] 10MediaWiki-Releasing, 10MediaWiki-Installer, 10MW-1.31-release, 10Patch-For-Review: Expand the set of bundled extensions to achieve a default MediaWiki experience that's comparable to Wikimedia sites - https://phabricator.wikimedia.org/T178349#4110367 (10CCicalese_WMF) [15:54:29] 10MediaWiki-Releasing, 10AntiSpoof, 10MW-1.31-release: Bundle AntiSpoof extension with MW 1.32 - https://phabricator.wikimedia.org/T191736#4115112 (10CCicalese_WMF) [15:54:41] 10MediaWiki-Releasing, 10AntiSpoof, 10MW-1.32-release: Bundle AntiSpoof extension with MW 1.32 - https://phabricator.wikimedia.org/T191736#4115112 (10CCicalese_WMF) [15:55:14] 10MediaWiki-Releasing, 10MediaWiki-extensions-LoginNotify, 10MW-1.32-release: Bundle LoginNotify extension with MW 1.32 - https://phabricator.wikimedia.org/T191746#4115206 (10CCicalese_WMF) [15:55:37] 10MediaWiki-Releasing, 10MediaWiki-extensions-Scribunto, 10MW-1.31-release, 10MW-1.31-release-notes (WMF-deploy-2018-04-10 (1.31.0-wmf.29)), 10Patch-For-Review: Bundle Scribunto extension with MW 1.31 - https://phabricator.wikimedia.org/T191737#4264554 (10CCicalese_WMF) [15:55:40] 10MediaWiki-Releasing, 10MediaWiki-Installer, 10MW-1.31-release, 10Patch-For-Review: Expand the set of bundled extensions to achieve a default MediaWiki experience that's comparable to Wikimedia sites - https://phabricator.wikimedia.org/T178349#4110368 (10CCicalese_WMF) [15:55:49] 10MediaWiki-Releasing, 10MediaWiki-extensions-Scribunto, 10MW-1.31-release-notes (WMF-deploy-2018-04-10 (1.31.0-wmf.29)), 10MW-1.32-release, 10Patch-For-Review: Bundle Scribunto extension with MW 1.32 - https://phabricator.wikimedia.org/T191737#4115121 (10CCicalese_WMF) [15:56:01] 10MediaWiki-Releasing, 10MediaWiki-Installer, 10Epic, 10MW-1.32-release: Expand the set of bundled extensions and skins in MediaWiki 1.32 - https://phabricator.wikimedia.org/T196650#4264562 (10CCicalese_WMF) [15:56:06] 10MediaWiki-Releasing, 10Collaboration-Team-Triage, 10Thanks, 10MW-1.31-release: Bundle Thanks extension with MW 1.31 - https://phabricator.wikimedia.org/T191739#4264561 (10CCicalese_WMF) [15:56:12] 10MediaWiki-Releasing, 10MediaWiki-Installer, 10MW-1.31-release, 10Patch-For-Review: Expand the set of bundled extensions to achieve a default MediaWiki experience that's comparable to Wikimedia sites - https://phabricator.wikimedia.org/T178349#4264563 (10CCicalese_WMF) [15:56:15] 10MediaWiki-Releasing, 10Collaboration-Team-Triage, 10Thanks, 10MW-1.32-release: Bundle Thanks extension with MW 1.31 - https://phabricator.wikimedia.org/T191739#4115139 (10CCicalese_WMF) [15:56:21] 10MediaWiki-Releasing, 10Collaboration-Team-Triage, 10Thanks, 10MW-1.32-release: Bundle Thanks extension with MW 1.32 - https://phabricator.wikimedia.org/T191739#4115139 (10CCicalese_WMF) [15:56:31] 10MediaWiki-Releasing, 10MediaWiki-Installer, 10Epic, 10MW-1.32-release: Expand the set of bundled extensions and skins in MediaWiki 1.32 - https://phabricator.wikimedia.org/T196650#4264512 (10CCicalese_WMF) [15:56:34] 10MediaWiki-Releasing, 10Collaboration-Team-Triage, 10Notifications, 10MW-1.31-release: Bundle Echo extension with MW 1.31 - https://phabricator.wikimedia.org/T191738#4264566 (10CCicalese_WMF) [15:56:38] 10MediaWiki-Releasing, 10MediaWiki-Installer, 10MW-1.31-release, 10Patch-For-Review: Expand the set of bundled extensions to achieve a default MediaWiki experience that's comparable to Wikimedia sites - https://phabricator.wikimedia.org/T178349#4110375 (10CCicalese_WMF) [15:56:44] 10MediaWiki-Releasing, 10Collaboration-Team-Triage, 10Notifications, 10MW-1.32-release: Bundle Echo extension with MW 1.32 - https://phabricator.wikimedia.org/T191738#4115130 (10CCicalese_WMF) [15:56:57] 10MediaWiki-Releasing, 10MediaWiki-Installer, 10Epic, 10MW-1.32-release: Expand the set of bundled extensions and skins in MediaWiki 1.32 - https://phabricator.wikimedia.org/T196650#4264512 (10CCicalese_WMF) [15:57:00] 10MediaWiki-Releasing, 10AbuseFilter, 10MW-1.31-release, 10MW-1.31-release-notes (WMF-deploy-2018-04-10 (1.31.0-wmf.29)): Bundle AbuseFilter extension with MW 1.31 - https://phabricator.wikimedia.org/T191740#4264573 (10CCicalese_WMF) [15:57:09] 10MediaWiki-Releasing, 10MediaWiki-Installer, 10MW-1.31-release, 10Patch-For-Review: Expand the set of bundled extensions to achieve a default MediaWiki experience that's comparable to Wikimedia sites - https://phabricator.wikimedia.org/T178349#4110392 (10CCicalese_WMF) [15:57:47] 10MediaWiki-Releasing, 10AbuseFilter, 10MW-1.31-release-notes (WMF-deploy-2018-04-10 (1.31.0-wmf.29)), 10MW-1.32-release: Bundle AbuseFilter extension with MW 1.32 - https://phabricator.wikimedia.org/T191740#4115148 (10CCicalese_WMF) [15:57:52] 10MediaWiki-Releasing, 10MediaWiki-Installer, 10Epic, 10MW-1.32-release: Expand the set of bundled extensions and skins in MediaWiki 1.32 - https://phabricator.wikimedia.org/T196650#4264512 (10CCicalese_WMF) [15:57:55] 10MediaWiki-Releasing, 10MinervaNeue, 10MW-1.31-release, 10Readers-Web-Backlog (Tracking): Bundle Minerva Neue skin with MW 1.31 - https://phabricator.wikimedia.org/T191743#4264582 (10CCicalese_WMF) [15:57:58] 10MediaWiki-Releasing, 10MediaWiki-Installer, 10MW-1.31-release, 10Patch-For-Review: Expand the set of bundled extensions to achieve a default MediaWiki experience that's comparable to Wikimedia sites - https://phabricator.wikimedia.org/T178349#4110411 (10CCicalese_WMF) [15:58:14] 10MediaWiki-Releasing, 10MinervaNeue, 10MW-1.32-release, 10Readers-Web-Backlog (Tracking): Bundle Minerva Neue skin with MW 1.32 - https://phabricator.wikimedia.org/T191743#4115175 (10CCicalese_WMF) [15:58:21] 10MediaWiki-Releasing, 10MediaWiki-Installer, 10Epic, 10MW-1.32-release: Expand the set of bundled extensions and skins in MediaWiki 1.32 - https://phabricator.wikimedia.org/T196650#4264589 (10CCicalese_WMF) [15:58:25] 10MediaWiki-Releasing, 10MobileFrontend, 10MW-1.31-release, 10MW-1.31-release-notes (WMF-deploy-2018-04-17 (1.31.0-wmf.30)), and 2 others: Bundle MobileFrontend extension with MW 1.31 - https://phabricator.wikimedia.org/T191734#4264588 (10CCicalese_WMF) [15:58:28] 10MediaWiki-Releasing, 10MediaWiki-Installer, 10MW-1.31-release, 10Patch-For-Review: Expand the set of bundled extensions to achieve a default MediaWiki experience that's comparable to Wikimedia sites - https://phabricator.wikimedia.org/T178349#4264590 (10CCicalese_WMF) [15:58:44] 10MediaWiki-Releasing, 10MobileFrontend, 10MW-1.31-release-notes (WMF-deploy-2018-04-17 (1.31.0-wmf.30)), 10MW-1.32-release, and 2 others: Bundle MobileFrontend extension with MW 1.32 - https://phabricator.wikimedia.org/T191734#4115094 (10CCicalese_WMF) [16:00:04] 10Beta-Cluster-Infrastructure, 10Operations, 10Performance-Team (Radar), 10Puppet: Define scap::sources in a way that is shared between prod and beta - https://phabricator.wikimedia.org/T196034#4264607 (10Imarlier) Gets all scap-enabled projects -- looking for a checkout and cloning if not present wouldn't... [16:00:13] Right, time to see if i can still run these selenium tests... [16:01:26] 10MediaWiki-Releasing, 10MediaWiki-Installer, 10Epic, 10MW-1.32-release: Expand the set of bundled extensions and skins in MediaWiki 1.32 - https://phabricator.wikimedia.org/T196650#4264611 (10CCicalese_WMF) [16:01:29] 10MediaWiki-Releasing, 10MediaWiki-Installer, 10MW-1.32-release: Expand the set of bundled skins - https://phabricator.wikimedia.org/T194266#4264612 (10CCicalese_WMF) [16:03:11] 10MediaWiki-Releasing, 10MediaWiki-Installer, 10Epic, 10MW-1.31-release: Expand the set of bundled extensions and skins in MediaWiki 1.31 - https://phabricator.wikimedia.org/T196648#4264617 (10CCicalese_WMF) [16:05:48] 10MediaWiki-Releasing, 10MediaWiki-Installer, 10Epic, 10MW-1.32-release: Expand the set of bundled extensions and skins in MediaWiki 1.32 - https://phabricator.wikimedia.org/T196650#4264621 (10CCicalese_WMF) [16:05:53] 10MediaWiki-Releasing, 10MediaWiki-Installer, 10MW-1.31-release, 10Patch-For-Review: Expand the set of bundled extensions to achieve a default MediaWiki experience that's comparable to Wikimedia sites - https://phabricator.wikimedia.org/T178349#4264622 (10CCicalese_WMF) [16:09:57] (03Draft1) 10MarcoAurelio: Mark repository as read only [extensions/InterwikiIntegration] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/438038 (https://phabricator.wikimedia.org/T196442) [16:10:01] (03PS2) 10MarcoAurelio: Mark repository as read only [extensions/InterwikiIntegration] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/438038 (https://phabricator.wikimedia.org/T196442) [16:24:19] zeljkof: I filed 2 small patches ofr selenium stuff :D [16:28:03] 10MediaWiki-Releasing, 10MediaWiki-Installer, 10Epic, 10MW-1.32-release: Expand the set of bundled extensions and skins in MediaWiki 1.32 - https://phabricator.wikimedia.org/T196650#4264702 (10CCicalese_WMF) [16:28:58] make that 3 :D [16:29:24] but now selenium tests work for me just using the "npm run selenium" command :D [16:37:57] 10Continuous-Integration-Config: Add 'noop' CI to mediawiki/(extensions|skins} - https://phabricator.wikimedia.org/T196615#4264722 (10MarcoAurelio) @hashar Do you think `noop` would work here or maybe other test template? [16:43:43] paladox: 2.15.x supports muting notifications from a change, right? [16:43:49] yep [16:43:53] Good. [16:43:54] and also WIP! [16:44:06] So to stop getting annoying e-mails, I don't need to hack the database I need to just upgrade! [16:44:07] Yay [16:44:12] * no_justification has motivation now [16:44:14] lol [16:44:23] xD [16:44:31] I want to see 2.15 too [16:44:46] no_justification it has two buttons actually [16:44:53] one hides it from the dashboard [16:44:55] and the other mutes [16:46:22] you can also move changes to other branches [16:48:02] There's no outstanding 2.15.x config prep we needed right? [16:48:09] no_justification /me checks [16:48:23] actually nvm, nope [16:48:29] Everything I see is just nice to have stuff [16:48:36] yeh [16:48:45] 10Release-Engineering-Team (Kanban), 10Operations: Add CI namespace in staging k8s cluster - https://phabricator.wikimedia.org/T196654#4264759 (10thcipriani) [16:49:01] Did you see I uploaded the 2.15.2 artifacts? [16:49:04] 10Release-Engineering-Team (Kanban), 10Operations, 10Release Pipeline: Add CI namespace in staging k8s cluster - https://phabricator.wikimedia.org/T196654#4264773 (10thcipriani) [16:49:06] And prepped the change [16:49:30] heh i tryed it last night [16:49:33] but woulden't download [16:49:35] trying it now :) [16:50:05] Oh wait, I couldn't uploaded them [16:50:06] I remember now [16:50:10] I got an error [16:50:13] But got distracted [16:50:28] heh [16:50:37] : No such file or directory (2)\nrsync: link_stat "/git-fat/0648d102ca70e6c2a3118dbb6ced6c9eb12e7f2d" (in archiva) [16:51:38] no_justification and emojies work too! [16:52:14] Heh, once we do the notedb migration [16:52:26] But we'll start off with just reviewdb, let that settle, then start migrating [16:52:31] yeh [16:52:42] it will automatically start once you start gerrit with 2.15 [16:52:45] on one thread [16:52:51] so it wont overload the db i think [16:52:53] which is good :) [16:55:36] https://gerrit-review.googlesource.com/Documentation/note-db.html#online-migration [16:57:42] 10MediaWiki-Releasing, 10MediaWiki-Installer: Expand the set of bundled extensions to achieve a default MediaWiki experience that's comparable to Wikimedia sites - https://phabricator.wikimedia.org/T178349#4264813 (10CCicalese_WMF) [16:57:51] 10MediaWiki-Releasing, 10MediaWiki-Installer, 10Epic: Expand the set of bundled extensions to achieve a default MediaWiki experience that's comparable to Wikimedia sites - https://phabricator.wikimedia.org/T178349#4112048 (10CCicalese_WMF) [17:00:56] greg-g, hey, FYI [17:01:05] with the new beta wikis created today, beta is at its max number of wikis [17:01:58] no more can be created without either dropping the zero domains or moving the LE stuff to using DNS validation [17:02:15] alex@alex-laptop:~$ openssl s_client -connect en.wikipedia.beta.wmflabs.org:443 2>&1 | openssl x509 -text -noout | grep DNS: | sed -e 's/, DNS:/\n/g' | wc -l [17:02:15] 100 [17:02:31] what's the technical limitation? [17:02:33] Oh yeah! I remember why I couldn't upload the artifacts to archiva. [17:02:33] I didn't have maven installed on the laptop I was using :p [17:02:36] paladox: ^^ Heh [17:02:44] lol [17:03:07] 10Release-Engineering-Team, 10Multi-Content-Revisions, 10User-Addshore: Investigate possibility of having some MCR related patches on test / group0 for an extended period - https://phabricator.wikimedia.org/T196585#4264822 (10CCicalese_WMF) [17:03:07] greg-g, https://letsencrypt.org/docs/rate-limits/ says "If you have a lot of subdomains, you may want to combine them into a single certificate, up to a limit of 100 Names per Certificate" [17:03:10] * paladox has maven installed [17:03:14] i used homebrew i think [17:03:22] 10Gerrit, 10Release-Engineering-Team: Gerrit plugin "zuul" failed to load - https://phabricator.wikimedia.org/T195176#4218757 (10MarcoAurelio) This was happening to me as well. I filed {T193990} but closed it once it dissapeared. It seems it was not me the only one with the problem :-) [17:03:30] Krenair: aha [17:03:57] I'd be willing to have a go at moving to DNS validation, but [17:04:58] I need ops to do https://phabricator.wikimedia.org/T194998 first to do it relatively securely [17:07:04] 10Release-Engineering-Team, 10Multi-Content-Revisions, 10User-Addshore: Investigate possibility of having some MCR related patches on test / group0 for an extended period - https://phabricator.wikimedia.org/T196585#4264833 (10greg) >>! In T196585#4262440, @Addshore wrote: > One idea would be to create an ext... [17:07:26] 10Release-Engineering-Team (Kanban), 10Release Pipeline: Refactor pipeline build step to be more isolated/secure/scalable - https://phabricator.wikimedia.org/T195050#4264835 (10thcipriani) p:05Triage>03Normal We talked about this at the [[ https://www.mediawiki.org/wiki/Wikimedia_Release_Engineering_Team/D... [17:08:25] 10Release-Engineering-Team (Kanban), 10Operations, 10Release Pipeline: Refactor pipeline build step to be more isolated/secure/scalable - https://phabricator.wikimedia.org/T195050#4264844 (10thcipriani) actually adding operations [17:13:27] 10Release-Engineering-Team, 10Multi-Content-Revisions, 10User-Addshore: Investigate possibility of having some MCR related patches on test / group0 for an extended period - https://phabricator.wikimedia.org/T196585#4264878 (10Addshore) >>! In T196585#4264833, @greg wrote: > I think this is a reasonable plan... [17:14:25] greg approves :D [17:14:29] ;) [17:16:33] Ummm [17:16:39] #2​10​82​4 [17:16:39] (pending—Waiting for next available executor on deployment-tin.eqiad; castor02 doesn’t have label BetaClusterBastion; ci-jessie-wikimedia-1067374 doesn’t have label BetaClusterBastion; ci-jessie-wikimedia-1067376 doesn’t have label [17:16:48] Looks a bit broken [17:17:06] Is this the one where I need to disconnect it and reconnect it? [17:17:50] * Reedy tries https://phabricator.wikimedia.org/T72597#4146162 [17:22:13] hashar: ^ [17:22:25] Done it twice and it's still doing the waiting for executor.. [17:24:40] * Reedy disables some stuff [17:24:42] Looks to have helped [17:24:55] !log unstuck jenkins deployments to beta [17:24:57] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [17:25:54] Woo [17:26:08] 10Beta-Cluster-Infrastructure, 10Technical-Debt: Set up LVS in beta like prod - https://phabricator.wikimedia.org/T196662#4264970 (10Krenair) [17:28:07] 10Release-Engineering-Team, 10Scap, 10ORES, 10Scoring-platform-team (Current): ORES deployments blocked by mysterious tin.eqiad.wmnet error - https://phabricator.wikimedia.org/T196663#4264985 (10awight) p:05Triage>03High [17:34:18] * paladox waits for no_justification to upload them :) [17:38:13] 10MediaWiki-Releasing, 10MediaWiki-Installer, 10Epic, 10MW-1.31-release: Expand the set of bundled extensions and skins in MediaWiki 1.31 - https://phabricator.wikimedia.org/T196648#4264457 (10Jdforrester-WMF) Is this now Resolved? [17:41:20] 10Beta-Cluster-Infrastructure, 10MediaWiki-JobQueue, 10Operations, 10Beta-Cluster-reproducible, 10Performance-Team (Radar): Stack overflow when Redis is down - https://phabricator.wikimedia.org/T185055#4265112 (10Reedy) [17:48:40] 10MediaWiki-Releasing, 10MediaWiki-Installer, 10Epic, 10MW-1.31-release: Expand the set of bundled extensions and skins in MediaWiki 1.31 - https://phabricator.wikimedia.org/T196648#4265136 (10CCicalese_WMF) I believe so. @Legoktm? [17:49:45] 10Release-Engineering-Team, 10Scap, 10ORES, 10Scoring-platform-team (Current): ORES deployments blocked by mysterious tin.eqiad.wmnet error - https://phabricator.wikimedia.org/T196663#4265139 (10thcipriani) p:05High>03Unbreak! This is due to running `git fetch` (without `--no-recurse-submodules`) in th... [17:53:10] 10MediaWiki-Releasing, 10MediaWiki-Installer, 10MW-1.32-release: Enhance installer to allow skins and extensions to depend upon one another - https://phabricator.wikimedia.org/T196670#4265149 (10CCicalese_WMF) p:05Triage>03Normal [17:54:07] 10MediaWiki-Releasing, 10MediaWiki-Installer, 10MW-1.32-release: Enhance installer to allow skins and extensions to depend upon one another - https://phabricator.wikimedia.org/T196670#4265149 (10CCicalese_WMF) [17:54:27] (03CR) 10Legoktm: "The intent of this check is to have the license be in file level comments. Why would you put the tag at a class level?" [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/438031 (owner: 10Thiemo Kreuz (WMDE)) [17:54:43] 10MediaWiki-Releasing, 10MediaWiki-Installer, 10MW-1.32-release: Enhance installer to support dependencies between skins and extensions - https://phabricator.wikimedia.org/T196670#4265149 (10CCicalese_WMF) [17:58:13] paladox: Oh duh, I forgot to give deploy_artifacts.py a --version [17:58:20] lol [17:58:28] So "unauthorized" wasn't "you're not auth'd" [17:58:31] It's "you can't do that" [17:58:33] oh [17:59:03] https://archiva.wikimedia.org/#browse/com.googlesource.gerrit.2.15.2 [17:59:06] just showed up! [17:59:09] :) [17:59:11] no_justification ^^ [18:00:41] 10Beta-Cluster-Infrastructure, 10Technical-Debt: Set up LVS in beta like prod - https://phabricator.wikimedia.org/T196662#4265215 (10Krenair) [18:01:56] I just uploaded the plugins too [18:02:01] :) [18:07:46] no_justification it deployed now :) [18:07:51] just doing the jar init step now [18:07:56] So yeah, if you can install with those artifacts (yes, I know gerrit.war is -dirty, that's quota plugin fault [18:08:05] There's something in it that ends up not being .gitignore'd [18:08:11] ok [18:08:23] no_justification also the git repo will need to be chowned as gerrit2 after updating :) [18:08:29] otherwise logins will fail heh [18:08:55] motd, not quota [18:09:15] Ideally, we should be able to run init as gerrit2 as well [18:09:21] hmm *** Gerrit Code Review 2.14.2-4760-g70f5858937-dirty [18:09:27] no_justification ^^ [18:09:31] Crap. Did I build the wrong branch?! [18:09:41] lol quite possibly :) [18:09:45] Gahhhh [18:10:11] Can you check out wmf/stable-2.15 on our gerrit fork? [18:10:19] commit 70f5858937b604e1a4229c53037ffa9f5da2e53b (HEAD -> wmf/stable-2.15, origin/wmf/stable-2.15) [18:10:29] ok [18:10:44] i have this checked cc37652b0ec26c259ed25b872ef7b98a9cd121a9 out [18:10:47] which is 2.15 wmf [18:10:52] Gerrit 2.15.2 wmf build [18:11:09] branch * stable-2.15 [18:11:34] That...seems wrong? [18:11:44] Bad object on my end [18:12:01] hmm [18:12:05] * paladox re does it [18:12:19] git reset --hard origin/stable-2.15 [18:12:19] HEAD is now at 68e85d7 WIP: 2.15.1 branch for wikimedia [18:12:25] now will check out your change [18:13:00] Not the deploy repo [18:13:02] The fork repo [18:13:43] The deploy repo is just stable-* [18:13:49] The fork uses wmf/stable-* [18:13:55] That's probably needlessly confusing [18:14:15] oh [18:14:16] i see [18:14:35] https://gerrit.wikimedia.org/r/#/admin/projects/operations/software/gerrit/gerrit [18:14:41] Lemme redo this actually. [18:15:03] ok [18:15:11] It's needlessly convoluted :p [18:15:41] heh [18:16:06] Is anyone able to SSH into deployment-deploy-01? [18:16:24] it won't let me in even as root [18:16:28] isen't that deployment-deploy01 now? [18:16:41] krenair@deployment-puppetmaster03:~$ ping deployment-deploy-01 [18:16:41] PING deployment-deploy-01.deployment-prep.eqiad.wmflabs (10.68.22.177) 56(84) bytes of data. [18:16:41] 64 bytes from deployment-deploy-01.deployment-prep.eqiad.wmflabs (10.68.22.177): icmp_seq=1 ttl=64 time=1.22 ms [18:16:42] no [18:19:07] it's running and responding to ping but refusing SSH and HTTP connections [18:20:38] Krenair maybe restart it through the horizion ui, could it be the firewall rejecting this (ferm?) [18:35:18] paladox: Does this layout make sense? https://gerrit.wikimedia.org/r/#/admin/projects/operations/software/gerrit,branches [18:35:28] It's the software itself, but the deploy pieces are in deploy/wmf/* [18:35:45] I think that's far simpler [18:35:50] ah [18:35:50] yeh [18:35:55] that looks better [18:36:01] deploy/ for when we deploy to prod [18:36:01] So just one repo instead of two [18:36:04] Yep [18:36:07] and tje branches were we build from [18:36:14] the [18:36:15] Yeah [18:36:28] stable-2.14, stable-2.15 and master are straight from upstream [18:36:34] yep [18:36:34] (manually sync'd as needed) [18:37:17] yep [18:45:24] I'll rebuild this over lunch and see what we get [18:46:38] ok :) [18:46:57] no_justification anyways the font issue dosen't seem to be the way you built it. [18:47:00] it works in chrome [18:47:20] so font issue is minor and shoulden't block deployment [18:47:33] as all it does is make the text look bolder [18:51:01] !log deployment-deploy-01 rejecting all connections, rebooting [18:51:03] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [19:39:45] no_justification lol i am about to receive alot of phab notifcations for the gerrit deploy repo :) [19:39:48] like https://phabricator.wikimedia.org/rGERRITDEPLOYbf634810a713 [19:53:38] no_justification ohhhhh [19:53:47] it seems to have the correct polygerrit version [19:53:50] but has 2.14 lol [19:53:58] (by correct i mean for 2.15) [19:54:11] so seems some how it mixed up 2.15 and 2.14) [20:05:21] PROBLEM - Puppet errors on deployment-cpjobqueue is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [20:05:21] PROBLEM - Puppet errors on deployment-mx is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [20:05:21] PROBLEM - Puppet errors on deployment-maps03 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [20:05:22] PROBLEM - Host deployment-puppetmaster02 is DOWN: CRITICAL - Host Unreachable (10.68.21.200) [20:05:22] PROBLEM - Puppet errors on deployment-deploy-01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [20:05:22] PROBLEM - Long lived cherry-picks on puppetmaster on deployment-puppetmaster03 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [20:06:03] 10Phabricator, 10Developer-Relations (Apr-Jun-2018), 10Patch-For-Review: Try to identify new developers (via assignee field) in Phab tasks and potentially follow up - https://phabricator.wikimedia.org/T195780#4265614 (10Aklapper) Hmm, interesting idea, thanks. Looks like the 4th value under `[info]` is the... [20:06:22] PROBLEM - Puppet errors on deployment-elastic07 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [20:06:54] RECOVERY - Host deployment-puppetmaster02 is UP: PING OK - Packet loss = 0%, RTA = 3.21 ms [20:07:11] RECOVERY - Free space - all mounts on deployment-tin is OK: OK: deployment-prep.deployment-tin.diskspace._mnt.byte_percentfree (No valid datapoints found) [20:07:11] PROBLEM - Free space - all mounts on deployment-fluorine02 is CRITICAL: CRITICAL: deployment-prep.deployment-fluorine02.diskspace._srv.byte_percentfree (<100.00%) [20:07:57] RECOVERY - Puppet errors on deployment-logstash2 is OK: OK: Less than 1.00% above the threshold [0.0] [20:08:09] RECOVERY - Puppet errors on deployment-elastic07 is OK: OK: Less than 1.00% above the threshold [0.0] [20:08:41] RECOVERY - Puppet errors on deployment-fluorine02 is OK: OK: Less than 1.00% above the threshold [0.0] [20:08:59] RECOVERY - Puppet errors on deployment-mx02 is OK: OK: Less than 1.00% above the threshold [0.0] [20:09:07] RECOVERY - Host deployment-mediawiki-09 is UP: PING OK - Packet loss = 0%, RTA = 2.29 ms [20:09:19] RECOVERY - Host integration-slave-docker-1013 is UP: PING OK - Packet loss = 0%, RTA = 1.65 ms [20:09:38] ^ I just brought the shinken service back online so it's noisey [20:09:47] PROBLEM - SSH on deployment-deploy-01 is CRITICAL: Connection refused [20:10:18] RECOVERY - Puppet errors on deployment-jobrunner03 is OK: OK: Less than 1.00% above the threshold [0.0] [20:12:52] PROBLEM - Host deployment-deploy-01 is DOWN: CRITICAL - Host Unreachable (10.68.22.177) [20:13:38] Reedy: https://gerrit.wikimedia.org/r/438079 enables Translate and pig latin on deploymentwiki in beta, which should make testing language converter even easier [20:13:56] not a high priority patch atm, but it would be convenient [20:20:21] paladox: I think I'm going to nuke the wmf/stable-2.15 and deploy/wmf/stable-2.15 branches and start fresh [20:20:26] So we have a clean history from 2.15.2 [20:20:30] ok :) [20:20:38] Clean up the archiva artifacts as well [20:20:48] :) [20:26:16] https://gerrit-review.googlesource.com/c/plugins/motd/+/183470 will stop motd having an untracked file and the build ending up -dirty [20:26:37] :) [20:28:44] (03PS1) 10Legoktm: More aggressively skip things based on which stages are run [integration/quibble] - 10https://gerrit.wikimedia.org/r/438084 [20:29:37] (03CR) 10jerkins-bot: [V: 04-1] More aggressively skip things based on which stages are run [integration/quibble] - 10https://gerrit.wikimedia.org/r/438084 (owner: 10Legoktm) [20:29:45] RECOVERY - Host deployment-deploy-01 is UP: PING OK - Packet loss = 0%, RTA = 3.47 ms [20:38:37] Ok, so my new version has plugins all at correct versions. All are either tracking their stable-2.15 branch or pointing at some tag related to 2.15.x [20:38:37] Except for 5 tracking master: avatars-external, quota, motd, wikimedia, zuul (these are fine, I guess) [20:39:25] :) [20:43:51] PROBLEM - Puppet errors on integration-slave-jessie-android is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [21:02:29] twentyafterfour: you around? [21:09:27] paladox: Ok, new wmf/stable-2.15 [21:09:33] :) [21:09:37] woo [21:09:43] https://gerrit.wikimedia.org/r/#/c/438097/ [21:10:33] Is it written what trains are running over the next few weeks? I need to create more MW1_32 milestones in Phab and don't know which are correct… [21:12:39] reminder that each week gets a wmf.XX even when there's no train happening. Week of June 18th and July 2nd there are no trains due to SRE summit and Fourth of July [21:12:57] greg-g: But they don't get milestones. [21:13:05] greg-g: Milestones represent actual releases. [21:13:11] gotcha [21:13:21] Awesome, thanks. [21:13:24] (03CR) 10Hashar: [C: 031] "We should definitely make Quibble smarter/faster, I am afraid though that all those if/else conditions will end up being very hard to main" [integration/quibble] - 10https://gerrit.wikimedia.org/r/438084 (owner: 10Legoktm) [21:14:56] (for the record, it is written on [[wikitech:Deployments]] if you scroll down) [21:15:11] but, week of july 2nd is as far as I've thought about :) [21:15:17] Oh, right. [21:15:38] OK, https://phabricator.wikimedia.org/project/subprojects/3328/ now has out 'til wmf.12 on 2018-07-10+. [21:15:41] paladox: Wtf is up with these entries in archiva....? https://archiva.wikimedia.org/#browse~releases/com.googlesource.gerrit.plugins.gerrit.plugins.gerrit.plugins.gerrit.plugins.gerrit.plugins.gerrit.plugins.gerrit.plugins.gerrit.plugins.gerrit.plugins.gerrit.plugins.gerrit.plugins [21:15:49] * paladox looks [21:15:58] James_F: gracias [21:16:05] oh [21:16:06] wow [21:16:17] greg-g: Thanks as always to the brilliant RelEng team for the smoothly-running ship. [21:16:33] (Ship? Train? Bird? Plane? Let's call the whole thing… on. Umm.) [21:16:43] zuul seems to be the last one [21:16:44] heh [21:16:55] but that is a long pathhhhh [21:17:40] what's archiva? [21:17:47] a femenine archive? [21:17:48] maven [21:17:58] it stores stuff created by maven [21:19:20] no_justification it's removed now :) [21:19:29] I'm trying to clean it up a little [21:19:33] But it's not letting me delete stuff [21:19:40] ok [21:20:07] so are we updating to 2.15 today? [21:20:13] 10MediaWiki-Releasing, 10Security: Streamline MW security release process - https://phabricator.wikimedia.org/T196602#4265941 (10greg) ``` 20:53:41 no_justification | https://github.com/Kentzo/git-archive-all 20:53:43 no_justi... [21:21:01] James_F: :) :) [21:22:11] greg-g: got a minute? https://gerrit.wikimedia.org/r/#/c/438032/ ? [21:22:20] or James_F [21:22:29] Oh, I think I did nuke it all, but it hasn't reindexed? [21:22:56] Done. [21:23:18] no_justification yep [21:23:23] looks like you nuked it all :) [21:23:26] James_F: :) [21:23:29] gitiles [21:23:39] Yay [21:23:40] and webhooks are only available [21:23:58] they are now hidden too [21:24:00] (removed) [21:24:03] Yep, I forced a reindex [21:24:05] Yay ok [21:24:08] :) [21:24:57] Ok, rebuilding 2.15.2 fork against what I merged [21:24:59] And see what we get [21:25:14] :) [21:32:46] Hauskatze not sure, that would be for no_justification :) [21:35:20] (03CR) 10Jforrester: [V: 032 C: 032] Mark repository as read only [extensions/InterwikiIntegration] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/438038 (https://phabricator.wikimedia.org/T196442) (owner: 10MarcoAurelio) [21:49:37] Hauskatze: around now [21:51:05] twentyafterfour: hi, issue handled already; sorry for disturbing [21:57:36] no_justification i think https://archiva.wikimedia.org/#welcome will need another forced reindex to find your newly uploaded jars :) [21:57:40] 10Release-Engineering-Team (Kanban), 10Operations, 10SRE-Access-Requests: Add thcipriani and hashar to gerrit-root - https://phabricator.wikimedia.org/T196702#4266006 (10greg) [21:58:54] 10Release-Engineering-Team (Kanban), 10Operations, 10SRE-Access-Requests: Add thcipriani and hashar to gerrit-root - https://phabricator.wikimedia.org/T196702#4266006 (10greg) Rational: reducing SPOF for future maintenance and upgrades of Gerrit. [22:02:30] 10Release-Engineering-Team (Kanban), 10Operations, 10SRE-Access-Requests: Add thcipriani to phabricator-roots - https://phabricator.wikimedia.org/T196703#4266022 (10greg) [22:10:33] paladox, greg-g: https://wikitech.wikimedia.org/w/index.php?title=Gerrit&type=revision&diff=1794235&oldid=1786483 :D [22:10:38] I think that reflects reality now [22:11:00] :) [22:14:36] hah, mostly deletions, good work :) [22:15:05] That's more or less what we've done for 2.13 and 2.14 and 2.15 :) [22:17:50] no_justification except https://wikitech.wikimedia.org/wiki/Gerrit#Disabling_/_Blocking_an_account [22:17:54] that wont work in 2.15 :) [22:17:59] I wasn't worried about that stuff [22:18:02] ok [22:18:26] Well, we can just remove the manual sql stuff [22:18:27] paladox: what should be done then? [22:18:30] Since we have --inactive and such [22:18:39] Hauskatze what do you mean? :) [22:18:47] no_justification yeh [22:18:53] though gsql is basically deprecated [22:18:54] to block an account in gerrit [22:19:04] ah [22:19:10] Hauskatze to block a account you can use [22:19:13] the rest api [22:19:17] or the ssh commands [22:19:53] we could create a polygerrit plugin to implement this but not sure how easy that would be [22:21:13] the ssh interface seems good enough [22:21:24] yeh [22:21:29] 10Release-Engineering-Team (Kanban), 10Operations, 10SRE-Access-Requests: Add thcipriani and hashar to gerrit-root - https://phabricator.wikimedia.org/T196702#4266071 (10RobH) This requires SRE team meeting review. Moving to the appropriate column for that. Next meeting is Monday, June 11th, 2018. [22:21:34] though we will now have log history of who does it! [22:21:56] no fun blocks anymore :P [22:22:01] lol [22:23:56] 10Release-Engineering-Team (Kanban), 10Operations, 10SRE-Access-Requests: Add thcipriani to phabricator-roots - https://phabricator.wikimedia.org/T196703#4266089 (10RobH) This requires SRE team meeting review. Moving to the appropriate column. The next meeting is Monday, June 11th, 2018. [22:24:09] Updated blocking info [22:24:38] no_justification thanks! [22:26:31] +1 [22:27:11] Ok, all the 2.15.2 versions uploaded to archiva [22:27:16] (the new 2.15.2 builds) [22:28:26] * paladox scaps it :) [22:28:47] deploy-local failed: {u'full_cmd': u'/usr/bin/git fat pull', u'stderr': u'\nrsync: link_stat "/git-fat/b5986e2781a98c2ac88c06ccafbca0478e570a8c" [22:28:53] Try again [22:28:57] Last file was finishing :p [22:28:57] ok [22:29:35] ok [22:30:03] no_justification still errors out [22:30:20] Bleh [22:32:37] Ewwww https://archiva.wikimedia.org/#advancedsearch/~~2.15.2~~~ [22:32:42] I blame my script [22:33:31] heh [22:33:47] I guess I'll delete the script and go back to doing them by hand :\ [22:34:14] I'm kinda tired of doing this song and danceeeeeeee [22:34:23] * no_justification takes a 10 min break [22:35:14] heh [22:36:40] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team, 10Operations, 10Patch-For-Review: Upgrade deployment-prep deployment servers to stretch - https://phabricator.wikimedia.org/T192561#4266117 (10Krenair) Unfortunately this new -deploy-01 instance went into emergency mode after the security reboots y... [22:39:25] 10Release-Engineering-Team, 10Scap, 10ORES, 10Scoring-platform-team (Current): ORES deployments blocked by mysterious tin.eqiad.wmnet error - https://phabricator.wikimedia.org/T196663#4264985 (10mmodell) Can we manually fix this to unbreak prior to a new scap version release? [22:42:41] 10Release-Engineering-Team (Kanban), 10Operations, 10SRE-Access-Requests: Add thcipriani to phabricator-roots - https://phabricator.wikimedia.org/T196703#4266122 (10RobH) p:05Triage>03Normal [22:42:47] 10Release-Engineering-Team (Kanban), 10Operations, 10SRE-Access-Requests: Add thcipriani and hashar to gerrit-root - https://phabricator.wikimedia.org/T196702#4266123 (10RobH) p:05Triage>03Normal [22:46:20] 10Beta-Cluster-Infrastructure, 10Operations: confd broken on deployment-redis hosts - https://phabricator.wikimedia.org/T196596#4262770 (10Krenair) >>! In T196596#4262846, @Reedy wrote: > Brandon reckons it's something to do with `confd::srv_dns` not being set correctly on beta That's weird, I thought that us... [22:47:14] 10Beta-Cluster-Infrastructure, 10Operations: confd broken on deployment-redis hosts - https://phabricator.wikimedia.org/T196596#4266133 (10Krenair) https://wikitech.wikimedia.org/w/index.php?title=Hiera:Deployment-prep&diff=next&oldid=1754779 [22:47:26] paladox: There we go https://archiva.wikimedia.org/#advancedsearch/~~2.15.2~~~ [22:47:33] yay! [22:49:09] I think I have bad logic in my script, as it iterates, it keeps appending .gerrit.plugins.gerrit.plugins, etc [22:49:11] Bad me! [22:49:24] heh [22:49:35] deploy-local failed: {u'full_cmd': u'/usr/bin/git fat pull', u'stderr': u'\nrsync: link_stat "/git-fat/c97c489fe61891fca04ad715b4a55547a4d74cd3" [22:49:44] i guess maybe archiva needs a full reindex? [22:52:34] 10Beta-Cluster-Infrastructure, 10Operations: confd broken on deployment-redis hosts - https://phabricator.wikimedia.org/T196596#4266141 (10Krenair) More session issues: T172560 T173646 [22:52:43] no_justification ^^ [22:53:24] I hate archiva with a passion [22:53:33] heh yeh [22:53:39] we could deploy with git-lfs [22:53:43] but needs stretch [22:54:16] rabbit hole alert [22:54:59] lol [22:57:15] I don't think we have time to reimage the gerrit server before you leave :) [22:58:05] lol [23:00:20] Reindex the releases repository [23:00:24] *reindexed [23:01:44] 10Scap, 10Operations: Update Debian Package for Scap3 to 3.8.2-1 - https://phabricator.wikimedia.org/T196710#4266164 (10thcipriani) [23:03:51] 10Release-Engineering-Team, 10Scap, 10ORES, 10Scoring-platform-team (Current): ORES deployments blocked by mysterious tin.eqiad.wmnet error - https://phabricator.wikimedia.org/T196663#4266179 (10thcipriani) [23:03:53] 10Scap, 10Operations, 10Patch-For-Review: Update Debian Package for Scap3 to 3.8.2-1 - https://phabricator.wikimedia.org/T196710#4266180 (10thcipriani) [23:04:46] PROBLEM - Puppet errors on deployment-sca02 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [23:06:25] Reedy, I think I found our redis problem [23:06:36] krenair@deployment-mediawiki06:~$ redis-cli -a $password_here -s /var/run/nutcracker/redis_eqiad.sock get enwiki:captcha [23:06:36] (error) ERR Connection refused [23:06:48] I restarted the nutcracker service [23:06:49] and now it works [23:07:02] I rebooted the host last night [23:07:12] PROBLEM - Puppet errors on deployment-changeprop is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [23:07:25] Joe did make a puppet and mw-config patch [23:07:32] Cause originally it was still using the old hosts [23:07:36] also wow what, why is cumin broken? did someone forget to arm keyholder? [23:07:47] Reedy, yeah there's an open task and gerrit change about that [23:08:11] https://phabricator.wikimedia.org/T179371#4111142 [23:08:36] PROBLEM - Puppet errors on deployment-cassandra3-01 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [23:08:42] https://gerrit.wikimedia.org/r/438010 [23:09:36] https://gerrit.wikimedia.org/r/#/c/438012/ [23:10:52] so how much of filippo's changes are now obsolete [23:10:57] pass [23:11:01] PROBLEM - Puppet errors on deployment-sca01 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [23:11:06] also what the hell is the password to arm keyholder on deployment-cumin [23:11:37] oh lol [23:11:57] nope, thought it accepted a blank pass, just did nothing actually [23:11:59] PROBLEM - Puppet errors on deployment-aqs01 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [23:12:26] okay got it [23:12:50] was at deployment-puppetmaster03:/var/lib/git/labs/private/files/ssh/tin/cumin_rsa.passphrase [23:12:54] not sure why it's in the tin directory [23:13:20] !log armed keyholder on deployment-cumin after reboot, found passphrase at deployment-puppetmaster03:/var/lib/git/labs/private/files/ssh/tin/cumin_rsa.passphrase [23:13:22] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [23:14:07] PROBLEM - Puppet errors on deployment-mira is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [23:15:16] PROBLEM - Puppet errors on deployment-imagescaler01 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [23:17:20] PROBLEM - Puppet errors on deployment-aqs02 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [23:17:22] PROBLEM - Puppet errors on deployment-tin is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [23:17:54] meh no_justification still says [23:17:54] deploy-local failed: {u'full_cmd': u'/usr/bin/git fat pull', u'stderr': u'\nrsync: link_stat "/git-fat/c97c489fe61891fca04ad715b4a55547a4d74cd3" (in archiva) failed: [23:18:16] oh [23:18:18] PROBLEM - Puppet errors on deployment-cassandra3-02 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [23:18:23] did you upload gerrit.war no_justification ? [23:18:47] Yes. First file I did [23:18:59] rsync: link_stat "/git-fat/c97c489fe61891fca04ad715b4a55547a4d74cd3" (in archiva) failed: No such file or directory (2) [23:18:59] rsync: link_stat "/git-fat/efb4437ddea2d48079a4d950b074ddd4f3c95938" (in archiva) failed: No such file or directory (2) [23:19:24] That's not the sha1 I thought? [23:19:36] hmm it seems so [23:19:41] the one that you merged [23:19:46] 099aa59079aae82178a93ba122da5a1244ca9233 [23:21:00] but strange that it fails on those two [23:21:06] does it pull the rest if it fails? [23:21:13] PROBLEM - Puppet errors on deployment-mathoid is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [23:22:11] PROBLEM - Puppet errors on deployment-mediawiki06 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [23:22:16] no_justification ^^ [23:22:53] Blah. I'm getting a headache... [23:23:33] no_justification i guess we should just append .10 to 2.15.2 (maybe that will work?) [23:25:19] PROBLEM - Puppet errors on deployment-restbase02 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [23:26:12] looks like restarting nutcracker didn't fix problems on all hosts [23:26:58] paladox: Could tag it something like 2.15.2-wmf.1? [23:27:03] yeh [23:27:10] that should work [23:27:38] PROBLEM - Puppet errors on deployment-zotero01 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [23:28:42] PROBLEM - Puppet errors on deployment-restbase01 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [23:29:34] PROBLEM - Puppet errors on deployment-pdfrender02 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [23:30:12] PROBLEM - Puppet errors on deployment-mcs01 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [23:33:57] PROBLEM - Puppet errors on deployment-logstash2 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [23:34:17] I don't wanna do anymore on this today, I've been fighting this for HOURS [23:34:47] I was hoping to *upgrade* the dang thing today [23:34:59] But I can't, because archiva hates me [23:35:04] heh [23:35:34] anyone seen this from nutcracker before? [2018-06-07 23:28:00.725] nc_redis.c:1092 parsed unsupported command 'COMMAND' [23:36:08] PROBLEM - Puppet errors on deployment-parsoid09 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [23:36:59] no_justification i carn't see c97c489fe61891fca04ad715b4a55547a4d74cd3 in any of the files [23:37:00] anyways [23:37:06] PROBLEM - Puppet errors on deployment-aqs03 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [23:38:01] Well this is all very bizarre [23:38:59] yeh [23:39:22] https://phabricator.wikimedia.org/P7226 [23:39:40] ah [23:39:43] no_justification qouta [23:40:09] hmm nope [23:40:14] though it seems different [23:40:24] Ugh. Where did I mess up? [23:40:52] well [23:40:55] i found one [23:41:01] no_justification removing quota.jar fixed one of them [23:41:18] * paladox checks [23:41:26] Hmm, ok so quota is wrong [23:41:43] https://phabricator.wikimedia.org/source/operations-software-gerrit/browse/deploy%252Fwmf%252Fstable-2.15/plugins/quota.jar [23:41:57] it looked like this [23:41:57] -rwxr-xr-x 1 paladox wikidev 74 Jun 7 23:13 quota.jar [23:41:58] for me [23:42:07] when editing with vi the plugins dir [23:42:14] Actually, we don't even use it right now. We could drop it from the deploy branch for now (keep it in the build branch) [23:42:16] that was at the top with the * next to it [23:42:18] Easy fix :p [23:42:33] now i need to find where this one [23:42:33] rsync: link_stat "/git-fat/c97c489fe61891fca04ad715b4a55547a4d74cd3" (in archiva) failed: No such file or directory (2) [23:42:34] leads to [23:43:20] ah [23:43:21] no_justification ./plugins/wikimedia.jar:1:#$# git-fat c97c489fe61891fca04ad715b4a55547a4d74cd3 219919 [23:44:13] Quota dropped [23:44:14] no_justification both of those plugins were not udpated in https://gerrit.wikimedia.org/r/#/c/438115/ [23:44:18] so that's why :) [23:44:32] Weird, I did `git add -A` [23:44:52] Wikimedia plugin is still just a playground [23:45:01] I never made it /do/ anything, right? [23:45:24] except add top menus [23:45:33] nothing really useful yet except top menus :) [23:45:45] woo [23:45:48] and it worked no_justification [23:45:52] once i dropped those two [23:46:12] Ok, lemme drop that [23:46:24] ok [23:46:42] Done & merged [23:46:52] no_justification yay https://gerrit.git.wmflabs.org/r/dashboard/self [23:46:54] it worked! [23:46:57] 2.15 shown too [23:46:59] Ok so HEAD is 73241403d84b905de9528a8b8380349fa709bf1b [23:47:06] ok [23:47:22] And that version shows 2.15.2-2-g....... right? [23:48:17] yep [23:48:17] 2.15.2-2-g33ec53f938 [23:48:38] i git reset and am now on your new uploaded changes! [23:48:52] and worked! [23:51:55] Yay [23:59:45] 10Beta-Cluster-Infrastructure, 10Operations: confd broken on deployment-redis hosts - https://phabricator.wikimedia.org/T196596#4266291 (10Krenair) I found that the nutcracker sockets on some of mediawiki hosts were refusing connections when I tried to run `redis-cli -a $password_here -s /var/run/nutcracker/re...