[00:24:13] PROBLEM - Host deployment-xhgui02 is DOWN: CRITICAL - Host Unreachable (172.16.1.202) [01:12:27] !log Cherry picking CL 608973 into beta puppet, should only affect webperf hosts. [01:12:28] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [01:29:15] RECOVERY - Host deployment-xhgui02 is UP: PING OK - Packet loss = 0%, RTA = 1.71 ms [01:39:13] PROBLEM - Host deployment-xhgui02 is DOWN: CRITICAL - Host Unreachable (172.16.1.202) [01:57:23] performance-xhgui02 is dead, long live performance-xhgui03. Anyone know how I tell shinken as much? [01:57:32] s/performance/deployment/g [02:06:18] Answer from #wikimedia-cloud is that it should eventually figure it out. [03:33:57] 10Release-Engineering-Team (Pipeline), 10Release-Engineering-Team-TODO (Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1))), 10Release Pipeline: Make pipelinelib able to update deployment chart image tags - https://phabricator.wikimedia.org/T255835 (10jeena) a:03jeena [04:33:17] 10Beta-Cluster-Infrastructure, 10The-Wikipedia-Library: Email spam from Betawiki - https://phabricator.wikimedia.org/T256297 (10Majavah) Is it sent via MediaWiki's "email this user" or in some other way? If it is, which account are they coming from? I can block that account from sending e-mail [05:34:54] 10Gerrit, 10DBA, 10Patch-For-Review: Make sure both `reviewdb-test` (used forgerrit upgrade testing) and `reviewdb` (formerly production) databases get torn down - https://phabricator.wikimedia.org/T255715 (10Marostegui) 05Stalled→03Open [05:37:53] 10Scap: scap deploy --init on deployment server fails on first puppet run - https://phabricator.wikimedia.org/T257317 (10jcrespo) Related to this, although a slightly different issue, `scap deploy --init` additionally failed on the non-primary deployment server because scap is disabled there. I think scap syncro... [05:57:10] 10Gerrit, 10DBA, 10Patch-For-Review: Make sure both `reviewdb-test` (used forgerrit upgrade testing) and `reviewdb` (formerly production) databases get torn down - https://phabricator.wikimedia.org/T255715 (10Dzahn) >>! In T255715#6284613, @Marostegui wrote: > I would suggest we start by renaming the tables... [05:58:55] 10Gerrit, 10DBA, 10Patch-For-Review: Make sure both `reviewdb-test` (used forgerrit upgrade testing) and `reviewdb` (formerly production) databases get torn down - https://phabricator.wikimedia.org/T255715 (10Marostegui) There seem to be two parallel conversations, one here and one on the patch :-) I don't m... [06:39:30] 10Release-Engineering-Team, 10Prod-Kubernetes, 10serviceops, 10Kubernetes: CI pipeline/job to build and release helm chart artifacts - https://phabricator.wikimedia.org/T257333 (10JMeybohm) I'm writing a (python) tool that is able to perform the three steps described above in first place. [07:44:16] RECOVERY - Host deployment-xhgui02 is UP: PING OK - Packet loss = 0%, RTA = 1.51 ms [07:54:13] PROBLEM - Host deployment-xhgui02 is DOWN: CRITICAL - Host Unreachable (172.16.1.202) [08:50:45] (03PS1) 10Tarrow: layout: [mediawiki/extensions/Wikibase] Enable experimental apitest [integration/config] - 10https://gerrit.wikimedia.org/r/610713 (https://phabricator.wikimedia.org/T257549) [08:51:21] 10Beta-Cluster-Infrastructure, 10The-Wikipedia-Library: Email spam from Betawiki - https://phabricator.wikimedia.org/T256297 (10Samwalton9) Sorry about this - we're not actively working on this extension right now so perhaps we should disable it on beta wiki for the time being. Who could do that @Catrope @Akla... [08:51:57] (03CR) 10Tarrow: "This change is ready for review." [integration/config] - 10https://gerrit.wikimedia.org/r/610713 (https://phabricator.wikimedia.org/T257549) (owner: 10Tarrow) [09:01:16] (03CR) 10Kosta Harlan: [C: 03+1] "nice! It looks reasonable to me 😊" [integration/config] - 10https://gerrit.wikimedia.org/r/610361 (https://phabricator.wikimedia.org/T48148) (owner: 10Hashar) [09:19:02] 10Phabricator: Can't access phabricator from my server - https://phabricator.wikimedia.org/T257507 (10Aklapper) Hmm, OVH provider.... can imagine that some ranges might be blocked due to past vandalism. Can you run `curl -s -D - -o /dev/null https://phabricator.wikimedia.org` by any chance? [09:25:47] !log contint2001: manually pulled all Docker images referenced in jjb. [09:25:49] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [09:37:10] (03PS1) 10QChris: Allow “Gerrit Managers” to import history [extensions/FlexDiagrams] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/610753 [09:37:12] (03CR) 10QChris: [V: 03+2 C: 03+2] Allow “Gerrit Managers” to import history [extensions/FlexDiagrams] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/610753 (owner: 10QChris) [09:37:20] (03PS1) 10QChris: Import done. Revoke import grants [extensions/FlexDiagrams] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/610754 [09:37:22] (03CR) 10QChris: [V: 03+2 C: 03+2] Import done. Revoke import grants [extensions/FlexDiagrams] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/610754 (owner: 10QChris) [09:45:16] 10Beta-Cluster-Infrastructure, 10The-Wikipedia-Library: Email spam from Betawiki - https://phabricator.wikimedia.org/T256297 (10Aklapper) @Samwalton9: Which extension? [09:48:34] 10Beta-Cluster-Infrastructure, 10The-Wikipedia-Library: Email spam from Betawiki - https://phabricator.wikimedia.org/T256297 (10Samwalton9) `TheWikipediaLibrary`, as implemented in T132084. [09:59:25] (03CR) 10Ladsgroup: [C: 03+1] layout: [mediawiki/extensions/Wikibase] Enable experimental apitest [integration/config] - 10https://gerrit.wikimedia.org/r/610713 (https://phabricator.wikimedia.org/T257549) (owner: 10Tarrow) [10:05:53] 10Continuous-Integration-Infrastructure: Audit stretch-backports usage in CI images - https://phabricator.wikimedia.org/T257553 (10hashar) [10:17:03] 10Gerrit, 10Release-Engineering-Team, 10Jenkins: Jenkins is sometimes not picking up +2 votes, some patches don't get merged - https://phabricator.wikimedia.org/T257556 (10daniel) [10:18:58] (03CR) 10Lucas Werkmeister (WMDE): [C: 03+1] layout: [mediawiki/extensions/Wikibase] Enable experimental apitest [integration/config] - 10https://gerrit.wikimedia.org/r/610713 (https://phabricator.wikimedia.org/T257549) (owner: 10Tarrow) [10:38:49] (03CR) 10Addshore: [C: 03+2] layout: [mediawiki/extensions/Wikibase] Enable experimental apitest [integration/config] - 10https://gerrit.wikimedia.org/r/610713 (https://phabricator.wikimedia.org/T257549) (owner: 10Tarrow) [10:39:50] (03Merged) 10jenkins-bot: layout: [mediawiki/extensions/Wikibase] Enable experimental apitest [integration/config] - 10https://gerrit.wikimedia.org/r/610713 (https://phabricator.wikimedia.org/T257549) (owner: 10Tarrow) [10:42:49] !log reload zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/610713 T257549 [10:42:52] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [10:42:52] T257549: Enable API testing for Wikibase - https://phabricator.wikimedia.org/T257549 [11:25:30] (03CR) 10Hashar: "I plan on deploying this change later tonight (19:00UTC or something). Should work at least based on the gerrit command line generated by" [integration/config] - 10https://gerrit.wikimedia.org/r/610361 (https://phabricator.wikimedia.org/T48148) (owner: 10Hashar) [11:26:58] (03CR) 10Jforrester: "Neat." [integration/config] - 10https://gerrit.wikimedia.org/r/610361 (https://phabricator.wikimedia.org/T48148) (owner: 10Hashar) [11:45:41] (03PS1) 10Mvolz: Add new window for citoid [tools/release] - 10https://gerrit.wikimedia.org/r/610778 (https://phabricator.wikimedia.org/T91646) [11:48:41] (03CR) 10Mvolz: "I tried this this week manually and it worked well. The problem with the deployment schedule right now is both of the services windows wer" [tools/release] - 10https://gerrit.wikimedia.org/r/610778 (https://phabricator.wikimedia.org/T91646) (owner: 10Mvolz) [12:40:56] 10Beta-Cluster-Infrastructure, 10The-Wikipedia-Library: Email spam from Betawiki - https://phabricator.wikimedia.org/T256297 (10Aklapper) @Samwalton9: See https://www.mediawiki.org/wiki/Developers/Maintainers for stewards of that codebase [13:35:40] <_joe_> hi, can someone remind me what processes we have that need to run automated deployments? just l10update? [13:36:39] <_joe_> James_F: maybe? [13:47:31] Do we even run l10update these days? [13:48:43] hmm, https://github.com/wikimedia/puppet/blob/b347052863d4d2e87b37d6c2d9f44f833cfd9dc2/modules/scap/manifests/l10nupdate.pp suggests we do as of my last change... [13:49:04] oh, no [13:49:11] https://github.com/wikimedia/puppet/blob/b4b9093ee6ee8b8d59122734e9a24b38f6daee43/modules/scap/manifests/master.pp#L50 [13:49:17] Cron is seemingly disabled [13:49:33] rofl [13:49:40] Yeah, no automated deploys right now. [13:49:42] https://github.com/wikimedia/puppet/commit/9482dfc3f5d4d805eb437465a615fb0e2d4c96b7 [13:49:52] Disabled temporarily September 2018 [13:49:54] "temporarily" permanently disabled. [13:50:06] Thankfully. [13:56:09] Hey folks. I'm having a weird problem with "git review". Errors: https://phabricator.wikimedia.org/P11820 [13:56:43] The gist is that when I try to run git-review on a repo where I have the rights to create a branch, it just creates the branch and does not set up a patchset [13:56:52] halfak: You need to update your version of git review to 1.27+. [13:56:58] lol [13:56:59] Aha! Thanks James_F [13:57:03] The new gerrit server requires it. [13:57:09] Because why would it Just Work™? [13:57:14] halfak: Sorry for the trouble! [13:57:20] Or give a reasonable error :) [13:57:24] Or that. :-) [13:57:25] Thanks for the quick help! [13:57:40] Sadly you're not the only one who ran into the issue. [13:59:02] I was struggling to figure out how to google "git-review creates a branch and otherwise does nothing" [14:27:55] 10Release-Engineering-Team (Onboarding), 10Release-Engineering-Team-TODO, 10Quality-and-Test-Engineering-Team (QTE), 10User-Soham, and 2 others: Create MediaWiki-Docker documentation for all repositories with Selenium tests - https://phabricator.wikimedia.org/T256239 (10zeljkofilipin) [14:33:47] 10Gerrit, 10fundraising-tech-ops: move main fundraising puppet repo to gerrit - https://phabricator.wikimedia.org/T222840 (10Dwisehaupt) p:05Low→03Medium [14:38:01] 10Continuous-Integration-Infrastructure: Audit stretch-backports usage in CI images - https://phabricator.wikimedia.org/T257553 (10hashar) [14:43:02] 10Release-Engineering-Team, 10Vue.js (Vue.js-Search): Work with RelEng to add PipelineBot to Vector - https://phabricator.wikimedia.org/T257582 (10Niedzielski) [14:45:35] (03CR) 10Thcipriani: [C: 03+2] "LGTM!" (031 comment) [tools/release] - 10https://gerrit.wikimedia.org/r/610778 (https://phabricator.wikimedia.org/T91646) (owner: 10Mvolz) [14:45:52] 10Release-Engineering-Team, 10Analytics, 10ChangeProp, 10Core Platform Team, and 2 others: Run EventBus tests in MediaWiki core CI - https://phabricator.wikimedia.org/T257583 (10Pchelolo) [14:46:19] (03Merged) 10jenkins-bot: Add new window for citoid [tools/release] - 10https://gerrit.wikimedia.org/r/610778 (https://phabricator.wikimedia.org/T91646) (owner: 10Mvolz) [14:50:47] 10Continuous-Integration-Infrastructure: Audit stretch-backports usage in CI images - https://phabricator.wikimedia.org/T257553 (10hashar) 05Open→03Resolved a:03hashar docker-registry.discovery.wmnet/releng/tox-eventlogging:0.4.0 that is for librdkafka and it is wikimedia main already ( per https://gerrit... [14:51:57] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1))): Audit stretch-backports usage in CI images - https://phabricator.wikimedia.org/T257553 (10hashar) [15:00:28] 10phan-taint-check-plugin: Create a web demo - https://phabricator.wikimedia.org/T257301 (10Daimona) p:05Triage→03Low Tried this on [[https://toolsadmin.wikimedia.org/tools/id/taintcheck | toolforge ]]. It's not straightforward, because: - We need to import a few dependencies (most notably ace editor) due t... [15:08:40] PROBLEM - Host deployment-maps05 is DOWN: CRITICAL - Host Unreachable (172.16.4.168) [15:14:27] 10Phabricator, 10Operations: Can't access phabricator from my server - https://phabricator.wikimedia.org/T257507 (10greg) [15:28:44] PROBLEM - Host deployment-chromium02 is DOWN: CRITICAL - Host Unreachable (172.16.4.14) [15:37:33] 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1))): Refine Seakeeper proposal for Security/SRE review - https://phabricator.wikimedia.org/T243436 (10chasemp) Stepping #security-team back from this as our understand... [17:27:59] hrm, beta-scap-eqiad stuck [17:30:21] !log unstick beta-scap-eqiad by cancelling stuck job [17:30:22] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [17:30:50] lol [17:31:03] So it worked for a couple of hours after I unstuck it ~26-27H ago [17:43:47] PROBLEM - App Server Main HTTP Response on deployment-mediawiki-07 is CRITICAL: HTTP CRITICAL: HTTP/1.1 302 Found - string 'Wikipedia' not found on 'http://en.wikipedia.beta.wmflabs.org:80/wiki/Main_Page?debug=true' - 715 bytes in 0.085 second response time [17:44:28] PROBLEM - App Server Main HTTP Response on deployment-mediawiki-09 is CRITICAL: HTTP CRITICAL: HTTP/1.1 302 Found - string 'Wikipedia' not found on 'http://en.wikipedia.beta.wmflabs.org:80/wiki/Main_Page?debug=true' - 716 bytes in 0.111 second response time [17:44:59] 10VPS-project-codesearch, 10Analytics: Add analytics/* gerrit repos to code search - https://phabricator.wikimedia.org/T249318 (10Milimetric) 05Open→03Resolved Thank you very much for doing it better than I was going to! I don't use uBlock so I have no opinions, but it seems like the main request here is... [17:46:58] (03PS1) 10Yaron Koren: Add CI for mediawiki/extensions/PageExchange [integration/config] - 10https://gerrit.wikimedia.org/r/610730 [17:49:24] 10Phabricator, 10Operations: Can't access phabricator from my server - https://phabricator.wikimedia.org/T257507 (10Dzahn) > might be blocked due to past vandalism. Yes, confirmed. This IP is in a range that is blocked by us. It from T218589 most likely. CC: @chasemp > why isn't there a... more informati... [17:50:01] 10Phabricator, 10Operations, 10Security-Team: Can't access phabricator from my server - https://phabricator.wikimedia.org/T257507 (10Dzahn) [17:53:13] (03CR) 10Yaron Koren: "This change is ready for review." [integration/config] - 10https://gerrit.wikimedia.org/r/610730 (owner: 10Yaron Koren) [17:54:16] RECOVERY - Host deployment-xhgui02 is UP: PING OK - Packet loss = 0%, RTA = 1.24 ms [17:55:45] (03PS2) 10Yaron Koren: Add CI for mediawiki/extensions/PageExchange [integration/config] - 10https://gerrit.wikimedia.org/r/610730 [17:56:31] (03PS3) 10Yaron Koren: Add CI for mediawiki/extensions/FlexDiagrams [integration/config] - 10https://gerrit.wikimedia.org/r/610730 [17:59:13] PROBLEM - Host deployment-xhgui02 is DOWN: CRITICAL - Host Unreachable (172.16.1.202) [18:14:26] 10Release-Engineering-Team (Onboarding), 10Release-Engineering-Team-TODO, 10Quality-and-Test-Engineering-Team (QTE), 10User-Soham, and 2 others: Create MediaWiki-Docker documentation for all repositories with Selenium tests - https://phabricator.wikimedia.org/T256239 (10Vidhi-Mody) [18:34:19] 10Gerrit, 10Toolforge (Software install/update), 10cloud-services-team (Kanban): toolforge needs git-review 1.27+ to work with new Gerrit deploy - https://phabricator.wikimedia.org/T257496 (10Dzahn) [18:35:24] 10Gerrit, 10Toolforge (Software install/update), 10cloud-services-team (Kanban): toolforge needs git-review 1.27+ to work with new Gerrit deploy - https://phabricator.wikimedia.org/T257496 (10Dzahn) This now needs someone to install it (the right way) on the toolforge bastions. (fwiw, this also made us real... [18:40:48] 10Gerrit, 10Analytics-Radar, 10Operations: update git-review to >= 1.27 on all stretch hosts across the board - https://phabricator.wikimedia.org/T257609 (10Dzahn) [18:42:47] 10Gerrit, 10Analytics-Radar, 10Operations: update git-review to >= 1.27 on all stretch hosts across the board - https://phabricator.wikimedia.org/T257609 (10Dzahn) per T257496#6294709 I imported the buster 1.27 package to stretch (first tested on mwdebug1001, then imported on apt.wikimedia.org with included... [18:47:42] 10Gerrit, 10Analytics-Radar, 10Operations: upgrade git-review to >= 1.27 on all stretch hosts across the board - https://phabricator.wikimedia.org/T257609 (10Dzahn) [18:48:37] 10Gerrit, 10Analytics-Radar, 10Operations: upgrade git-review to >= 1.27 on all stretch hosts across the board - https://phabricator.wikimedia.org/T257609 (10Dzahn) 05Open→03Resolved a:03Dzahn Upgraded the package on stat1004, stat1006 and stat1007 (apt-get update, apt-get install git-review). https:/... [18:49:10] 10Gerrit, 10Toolforge (Software install/update), 10cloud-services-team (Kanban): toolforge needs git-review 1.27+ to work with new Gerrit deploy - https://phabricator.wikimedia.org/T257496 (10Dzahn) fixed for stat hosts in T257609 [19:01:35] 10Gerrit, 10Toolforge (Software install/update), 10cloud-services-team (Kanban): toolforge needs git-review 1.27+ to work with new Gerrit deploy - https://phabricator.wikimedia.org/T257496 (10bd808) 05Open→03Resolved a:03Dzahn `lang=shell-session tools-sgebastion-08.tools:~$ apt show git-review Package... [19:15:20] (03CR) 10Hashar: [C: 03+2] "train complete. Lets go ;)" [integration/config] - 10https://gerrit.wikimedia.org/r/610361 (https://phabricator.wikimedia.org/T48148) (owner: 10Hashar) [19:15:49] !log Reloading Zuul to have messages to gerrit tagged with 'autogenerated:ci' | https://gerrit.wikimedia.org/r/c/integration/config/+/610361 | T48148 [19:15:51] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [19:15:51] T48148: Allow hiding of non-discussion comments in Gerrit - https://phabricator.wikimedia.org/T48148 [19:16:12] 10Release-Engineering-Team (Onboarding), 10Release-Engineering-Team-TODO (Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1))), 10Scap: Remove unused `scap swat` command - https://phabricator.wikimedia.org/T254787 (10dancy) Updated deployment.eqiad.wmnet:/srv/mediawiki-staging/ and tested. [19:16:15] (yes that is a bug from March 2013) [19:16:25] (03Merged) 10jenkins-bot: zuul: let Gerrit filterout CI reviews [integration/config] - 10https://gerrit.wikimedia.org/r/610361 (https://phabricator.wikimedia.org/T48148) (owner: 10Hashar) [19:16:35] 10Release-Engineering-Team (Onboarding), 10Release-Engineering-Team-TODO (Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1))), 10Scap: Remove unused `scap swat` command - https://phabricator.wikimedia.org/T254787 (10dancy) 05Open→03Resolved [19:23:01] I have made Zuul to send reviews with --tags autogenerated:ci [19:23:15] which should lets user filter them out in Gerrit web interface [19:27:48] 10Continuous-Integration-Infrastructure, 10Gerrit, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1))), and 2 others: Allow hiding of non-discussion comments in Gerrit - https://phabricator.wikimedia.org/T48148 (... [19:28:23] 10Phabricator, 10Operations, 10vm-requests: eqiad: 1 VM request for aphlict - https://phabricator.wikimedia.org/T257617 (10Dzahn) [19:34:34] 10Continuous-Integration-Infrastructure, 10Gerrit, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1))), and 2 others: Allow hiding of non-discussion comments in Gerrit - https://phabricator.wikimedia.org/T48148 (... [19:34:50] 10Continuous-Integration-Infrastructure, 10Gerrit, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1))), and 2 others: Allow hiding of non-discussion comments in Gerrit - https://phabricator.wikimedia.org/T48148 (... [19:37:35] 10Continuous-Integration-Infrastructure, 10Gerrit, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1))), and 2 others: Allow hiding of non-discussion comments in Gerrit - https://phabricator.wikimedia.org/T48148 (... [19:54:42] 10Beta-Cluster-Infrastructure, 10Core Platform Team: Beta cluster changeprop job queue down since 2020-06-18 - https://phabricator.wikimedia.org/T257620 (10Mholloway) [20:16:38] 10Phabricator, 10Operations, 10Security-Team: Can't access phabricator from my server - https://phabricator.wikimedia.org/T257507 (10chasemp) >>! In T257507#6294524, @Dzahn wrote: >> might be blocked due to past vandalism. > > Yes, confirmed. This IP is in a range that is blocked by us. It comes from T218... [20:25:15] 10Phabricator, 10Operations, 10Security-Team: Can't access phabricator from my server - https://phabricator.wikimedia.org/T257507 (10Dzahn) Aha, thanks Chase! [20:26:23] 10Phabricator, 10Operations, 10Security-Team: Can't access phabricator from my server - https://phabricator.wikimedia.org/T257507 (10Dzahn) [20:26:29] 10Phabricator, 10Operations, 10Security-Team: Can't access phabricator from my server - https://phabricator.wikimedia.org/T257507 (10Isarra) Okay, that would explain it, at least. T229620 seems pretty major in particular and I'd argue describes the //bare minimum// of what needs to be resolved here, as when... [20:27:23] 10Phabricator, 10Operations, 10Security-Team: Can't access phabricator from my server - https://phabricator.wikimedia.org/T257507 (10Dzahn) [20:29:57] 10Phabricator, 10Operations, 10Traffic: Accessing Phabricator from Tor - https://phabricator.wikimedia.org/T254568 (10Dzahn) [20:32:12] 10Phabricator, 10Operations, 10Traffic: Accessing Phabricator from Tor - https://phabricator.wikimedia.org/T254568 (10Dzahn) T253632 should be seen as a parent task of this. Also see T257507#6294524 [20:32:55] 10Release-Engineering-Team (Onboarding), 10Release Pipeline (Blubber): Blubber: Allow more than one layer of copying - https://phabricator.wikimedia.org/T254629 (10jeena) [20:33:02] 10Phabricator, 10Operations, 10Traffic: Accessing Phabricator from Tor (some ranges blocked but not others) - https://phabricator.wikimedia.org/T254568 (10Dzahn) [20:53:19] (03PS1) 10Jeena Huneidi: Support injecting env vars when running images [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/610922 (https://phabricator.wikimedia.org/T257526) [20:59:17] RECOVERY - Host deployment-xhgui02 is UP: PING OK - Packet loss = 0%, RTA = 1.31 ms [21:04:15] PROBLEM - Host deployment-xhgui02 is DOWN: CRITICAL - Host Unreachable (172.16.1.202) [21:17:51] (03PS2) 10Jeena Huneidi: Support injecting env vars when running images [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/610922 (https://phabricator.wikimedia.org/T257526) [21:34:09] qchris: paladox: Google abandons the 'check' Gerrit plugin and are rebooting it : https://docs.google.com/document/d/1v2ETifhRXpuYlahtnfIK-1KL3zStERvbnEa8sS1pTwk/edit [21:34:20] Yes. [21:34:23] I saw that today. [21:34:36] I was kind of shocked to read that. [21:34:44] so looks like Google has a whole team working on it ? [21:35:00] On the other hand, I'm glad we pulled off the CI table thing without the checks API. [21:35:21] I've got no clue about the team. But I find the switch awkward. [21:35:32] the switch? [21:35:43] The switch away from the checks API. [21:35:51] I mean ... that's still young. [21:35:54] ahh, well the google doc lists a few pain points [21:36:04] Sure. There are pain points. [21:36:15] But why kill the whole thing altogether? [21:36:41] well it is usable as is [21:36:42] It's not that they want to improve on the checks API. They're building a new thing from ground up, aren't they? [21:37:05] by stop work I guess they will not put much time on it anymore but if there is a critical bug/security issue I guess they will gladly review it [21:37:15] and Iam still reading the paper ;) [21:38:41] Whoa. It did gather some comments in the meantime. When I read it, there was only 1 comment by mordred :-) [21:39:42] good monty ;] [21:41:24] (03PS1) 10Lokal Profil: Review access change [services/wikispeech/wikispeech-server] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/610735 [21:42:20] (03PS2) 10Lokal Profil: Restore access settings to default [services/wikispeech/wikispeech-server] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/610735 [21:42:45] the comments are priceless. Speed of light IS an issue :\ [21:43:18] (03PS3) 10Lokal Profil: Restore access settings to default [services/wikispeech/wikispeech-server] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/610735 (https://phabricator.wikimedia.org/T253851) [21:43:39] for everything else, MasterCard hashar [21:46:46] Just got randomly logged out of enwiki - were tokens reset again? [21:47:36] people on -en talking about that [21:54:15] RECOVERY - Host deployment-xhgui02 is UP: PING OK - Packet loss = 0%, RTA = 0.46 ms [21:55:45] hauskatze: haha I have loved those memes back in the time! [21:56:25] DannyS712|away: hauskatze: from -operations : all sessions have been invalidated due to T256395 [21:56:36] yup, checking the ticket [21:56:53] I am off, have a good rest of the day! [21:57:28] I'm off too in 5 [21:58:59] I wanted to check that ticket, but its private :( [21:59:13] PROBLEM - Host deployment-xhgui02 is DOWN: CRITICAL - Host Unreachable (172.16.1.202) [21:59:21] DannyS712|away: yeah sorry I just copy pasted the whole message [21:59:34] it must be private for a good reason [21:59:39] Would you be willing to add me to the ticket? [22:00:06] too late for me to handle that request ;) [22:00:14] i guess if you have a NDA that should be fine [22:00:27] you could ask tgr in #wikimedia-operations [22:00:34] then folks are working on the task already [22:00:48] I don't have an nda (yet) - still waiting for https://phabricator.wikimedia.org/T256367 [22:01:19] \o/ [22:01:40] I guess you'd need #security access too DannyS712|away [22:02:04] anyone know who Daniel Kinzler's manager is? My understanding is I need an nda to request security access [22:02:27] I don't know about the process though [22:02:31] or who makes the final call [22:02:51] in my time it was a phone call and a scan of passport/id card ;D [22:03:04] and +1 from the kabal. But things have changed/improved since then hehe [22:03:05] DannyS712|away: the ticket has a bunch of personal details but otherwise won't really tell you much you wouldn't know from the email announcement [22:03:07] so really I don't know [22:04:53] The process is a mess, there's also another NDA that is handled by Wikimedia-Legal and it's not the L2 doc. [22:05:00] Daniel's manager is Mandy Mooney, I think. [22:05:15] But apparently for #wmf-nda that is not required and just L2? [22:05:23] yeah volunteer NDAs are a pain. [22:06:19] I once had third the timespan of an Outreachy project eaten up just by giving access. [22:06:30] in any case that task is restricted to acl*security so wmf-nda won't let you see it [22:07:45] good luck everyone, see you later. I am sleeping now [22:08:18] bonne nuite [22:26:51] 10Phabricator (Upstream), 10Upstream: Feature request: - https://phabricator.wikimedia.org/T257633 (10MarcoAurelio) [22:27:23] 10Phabricator (Upstream), 10Upstream: Feature request: New notification option for when somebody @mentions you in a Task - https://phabricator.wikimedia.org/T257633 (10MarcoAurelio) [22:38:27] (03CR) 10Karl Wettin (WMSE): [V: 03+1] "Seems right to me!" [services/wikispeech/wikispeech-server] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/610735 (https://phabricator.wikimedia.org/T253851) (owner: 10Lokal Profil) [22:38:48] 10Phabricator: [Phabricator] Issue uploading screenshots on phab tickets - https://phabricator.wikimedia.org/T257635 (10egardner) [23:01:00] 10Phabricator (Upstream), 10Upstream: Feature request: New notification option for when somebody @mentions you in a Task - https://phabricator.wikimedia.org/T257633 (10JJMC89) [23:03:57] 10Phabricator: [Phabricator] Issue uploading screenshots on phab tickets - https://phabricator.wikimedia.org/T257635 (10Ammarpad) This is duplicate of T155130. It seems this is a known issue with no solution [23:10:57] 10Phabricator (Upstream), 10Upstream: Unclear error message when uploading a larger attachment: "Exception: No configured storage engine can store this file." - https://phabricator.wikimedia.org/T155130 (10Ammarpad) [23:10:59] 10Phabricator: [Phabricator] Issue uploading screenshots on phab tickets - https://phabricator.wikimedia.org/T257635 (10Ammarpad)