[00:34:40] (03PS1) 10EBernhardson: Edit Project Config [wikidata/query/rdf] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/551687 [00:34:58] (03PS1) 10EBernhardson: Edit Project Config [wikidata/query] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/551688 [02:07:53] 10Gerrit: reviewer-bot sometimes not working - https://phabricator.wikimedia.org/T222983 (10Tgr) 05Stalled→03Open Seems to be defunct most of the time these days. E.g. [[https://www.mediawiki.org/wiki/Git/Reviewers#mediawiki/extensions/GrowthExperiments|notifications for GrowthExperiments]] pretty consistent... [02:53:04] 10Release-Engineering-Team (Code Health), 10Release-Engineering-Team-TODO (201908), 10Code-Stewardship-Reviews, 10Graphoid, and 3 others: graphoid: Code stewardship request - https://phabricator.wikimedia.org/T211881 (10Milimetric) >>! In T211881#5615716, @kaldari wrote: >>I guess its not clear to me what... [06:08:04] new batch of 1k uploads/captions/depicts to beta commons just kicked off. since 9 seconds delay seemed to be just fine, I'm cutting that down to 8 seconds [06:08:12] still expect no trouble [06:22:41] 10Gerrit: reviewer-bot sometimes not working - https://phabricator.wikimedia.org/T222983 (10valhallasw) Thanks for spotting that -- I think something has changed in the API response used to get the reviewers: ` {'created': '2019-11-02 00:39:20.000000000', 'updated': '2019-11-04 18:43:33.000000000', 'status': 'N... [06:51:20] (03CR) 10MaxSem: [C: 03+2] Add a sniff for methods deprecated in PHPUnit 8 [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/547958 (https://phabricator.wikimedia.org/T192167) (owner: 10Daimona Eaytoy) [06:51:48] (03Merged) 10jenkins-bot: Add a sniff for methods deprecated in PHPUnit 8 [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/547958 (https://phabricator.wikimedia.org/T192167) (owner: 10Daimona Eaytoy) [08:18:30] 10Project-Admins: Create project Restbase-MySQL - https://phabricator.wikimedia.org/T238631 (10Lens0021) [08:28:20] (03CR) 10Thiemo Kreuz (WMDE): [C: 03+1] Avoid use of isset( ..., ..., [...] ) function construct / syntax [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/551639 (https://phabricator.wikimedia.org/T219139) (owner: 10Umherirrender) [08:29:20] 10Project-Admins: Create project RESTBase-MySQL - https://phabricator.wikimedia.org/T238631 (10Lens0021) [08:51:11] 10Phabricator, 10serviceops, 10Patch-For-Review: Phabricator downtime due to aphlict and websockets (aphlict current disabled) - https://phabricator.wikimedia.org/T238593 (10ema) p:05Triage→03Normal [08:51:22] 10Phabricator, 10Operations, 10Traffic, 10serviceops, 10Patch-For-Review: Phabricator downtime due to aphlict and websockets (aphlict current disabled) - https://phabricator.wikimedia.org/T238593 (10ema) [08:57:53] 10Project-Admins: Create project RESTBase-MySQL - https://phabricator.wikimedia.org/T238631 (10mobrovac) 05Open→03Resolved a:03mobrovac Project #restbase-mysql created. [09:11:34] uploads completed, captions started [09:23:11] (03PS5) 10Awight: Coverage report for PHP extensions [integration/config] - 10https://gerrit.wikimedia.org/r/542071 (https://phabricator.wikimedia.org/T185583) [09:23:27] (03CR) 10Awight: "PS 5: manual rebase" [integration/config] - 10https://gerrit.wikimedia.org/r/542071 (https://phabricator.wikimedia.org/T185583) (owner: 10Awight) [09:24:04] (03CR) 10jerkins-bot: [V: 04-1] Coverage report for PHP extensions [integration/config] - 10https://gerrit.wikimedia.org/r/542071 (https://phabricator.wikimedia.org/T185583) (owner: 10Awight) [09:26:26] (03PS6) 10Awight: Coverage report for PHP extensions [integration/config] - 10https://gerrit.wikimedia.org/r/542071 (https://phabricator.wikimedia.org/T185583) [10:00:08] 10Phabricator, 10Operations, 10Traffic, 10serviceops, 10Patch-For-Review: Phabricator downtime due to aphlict and websockets (aphlict current disabled) - https://phabricator.wikimedia.org/T238593 (10ema) By going through SAL and the irc logs on #wikimedia-operations I've reconstructed the events as follo... [11:58:50] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10observability, 10Patch-For-Review: Alert when Zuul/Gearman queue is stalled - https://phabricator.wikimedia.org/T70113 (10akosiaris) >>! In T70113#5669912, @hashar wrote: > The alarm got removed via https://gerrit.wikimedia.org/r/#/c/op... [12:11:39] captions are done, depicts have started [12:41:41] 10Continuous-Integration-Config, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata, 10Wikidata-Campsite: Wikibase Travis build runs twice for each commit - https://phabricator.wikimedia.org/T236347 (10Addshore) Pinging @hashar and #release-engineering-team Any idea about how to fix this? @Jdforrester... [12:48:02] 10Phabricator, 10Operations, 10Traffic, 10serviceops, 10Patch-For-Review: Phabricator downtime due to aphlict and websockets (aphlict current disabled) - https://phabricator.wikimedia.org/T238593 (10Joe) The problem is most apache workers ended up being stuck talking to aphlict via `proxy_wstunnel` which... [13:10:33] PROBLEM - Puppet errors on deployment-hadoop-test-1 is CRITICAL: (Service Check Timed Out) [13:10:33] PROBLEM - Puppet staleness on deployment-cumin02 is CRITICAL: (Service Check Timed Out) [13:10:36] PROBLEM - Puppet staleness on deployment-deploy01 is CRITICAL: (Service Check Timed Out) [13:10:36] PROBLEM - Puppet staleness on deployment-elastic05 is CRITICAL: (Service Check Timed Out) [13:10:37] PROBLEM - Puppet staleness on integration-agent-docker-1014 is CRITICAL: (Service Check Timed Out) [13:10:37] PROBLEM - Puppet errors on deployment-elastic05 is CRITICAL: (Service Check Timed Out) [13:30:14] 10Phabricator, 10Release-Engineering-Team-TODO, 10Dumps-Generation, 10Phlogiston: Phlogiston dump is missing projects - https://phabricator.wikimedia.org/T236507 (10Aklapper) > The Phlogiston dump has not been actively used in 2019, AFAIK, and the reports have been empty and/or erroring for many months, so... [13:55:47] 10Phabricator, 10Release-Engineering-Team-TODO, 10Dumps-Generation, 10Phlogiston: Phlogiston dump is missing projects - https://phabricator.wikimedia.org/T236507 (10ArielGlenn) Who was using it in the past? [14:20:44] (03PS1) 10Pablo Grass (WMDE): layout: add wikibase/vuejs-components & build via pipeline [integration/config] - 10https://gerrit.wikimedia.org/r/551829 [14:23:53] (03CR) 10Pablo Grass (WMDE): "We'll probably want to add the blubber file to the repo before actually handing this in for review." [integration/config] - 10https://gerrit.wikimedia.org/r/551829 (owner: 10Pablo Grass (WMDE)) [14:44:39] 10Phabricator, 10Operations, 10Traffic, 10serviceops, 10Patch-For-Review: Phabricator downtime due to aphlict and websockets (aphlict current disabled) - https://phabricator.wikimedia.org/T238593 (10akosiaris) >>! In T238593#5674571, @ema wrote: > By going through SAL and the irc logs on #wikimedia-opera... [14:50:02] (03PS1) 10Ottomata: Remove unused eventgate-ci zuul config [integration/config] - 10https://gerrit.wikimedia.org/r/551837 (https://phabricator.wikimedia.org/T229111) [14:54:31] 10Release-Engineering-Team (Pipeline), 10Release-Engineering-Team-TODO, 10Operations, 10Release Pipeline, and 3 others: Migrate EventStreams to k8s deployment pipeline - https://phabricator.wikimedia.org/T238658 (10Ottomata) [14:55:01] 10Release-Engineering-Team (Pipeline), 10Release-Engineering-Team-TODO, 10Operations, 10Release Pipeline, and 3 others: Migrate production services to kubernetes using the pipeline - https://phabricator.wikimedia.org/T198901 (10Ottomata) [14:55:15] (03PS1) 10Ottomata: Setup eventstreams service-pipeline-test-and-publish [integration/config] - 10https://gerrit.wikimedia.org/r/551838 (https://phabricator.wikimedia.org/T238658) [15:11:53] 10Release-Engineering-Team (Pipeline), 10Analytics, 10Release Pipeline, 10Patch-For-Review, 10Services (watching): Migrate EventStreams to k8s deployment pipeline - https://phabricator.wikimedia.org/T238658 (10Ottomata) [15:21:39] (03CR) 10Awight: [C: 03+2] Create PHPUnitAssertEquals sniff to find problematic assertEquals() (031 comment) [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/547721 (owner: 10Thiemo Kreuz (WMDE)) [15:22:05] (03Merged) 10jenkins-bot: Create PHPUnitAssertEquals sniff to find problematic assertEquals() [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/547721 (owner: 10Thiemo Kreuz (WMDE)) [15:28:45] 10Phabricator, 10Release-Engineering-Team-TODO, 10Dumps-Generation, 10Phlogiston: Phlogiston dump is missing projects - https://phabricator.wikimedia.org/T236507 (10JAufrecht) It has no active users at the moment and has been broken for months. However, I would like to re-activate it for further reporting... [15:38:21] depicts complete, that's the end of the batch for today [15:42:27] 10Project-Admins: Request project for parliamentdiagram - https://phabricator.wikimedia.org/T230460 (10Aklapper) 05Open→03Declined Unfortunately closing this Phabricator task as no further information has been provided. @Slashme: After you have provided the information asked for and if this is still wanted,... [15:44:38] PROBLEM - Puppet staleness on deployment-docker-citoid01 is CRITICAL: (Service Check Timed Out) [15:44:39] PROBLEM - Puppet staleness on deployment-logstash2 is CRITICAL: (Service Check Timed Out) [15:44:44] PROBLEM - Puppet staleness on integration-agent-docker-1001 is CRITICAL: (Service Check Timed Out) [15:44:45] PROBLEM - Puppet errors on deployment-aqs02 is CRITICAL: (Service Check Timed Out) [15:46:19] PROBLEM - Puppet errors on deployment-parsoid09 is CRITICAL: (Service Check Timed Out) [15:46:20] PROBLEM - Puppet staleness on integration-agent-docker-1010 is CRITICAL: (Service Check Timed Out) [15:46:24] PROBLEM - Puppet errors on saucelabs-02 is CRITICAL: (Service Check Timed Out) [15:49:13] PROBLEM - Puppet staleness on deployment-eventgate-3 is CRITICAL: (Service Check Timed Out) [15:49:22] PROBLEM - Puppet errors on integration-agent-docker-1011 is CRITICAL: (Service Check Timed Out) [15:49:22] PROBLEM - Puppet staleness on deployment-kafka-main-2 is CRITICAL: (Service Check Timed Out) [15:50:19] RECOVERY - Free space - all mounts on deployment-imagescaler03 is OK: OK: All targets OK [15:50:22] RECOVERY - Free space - all mounts on integration-agent-docker-1001 is OK: OK: All targets OK [15:50:22] RECOVERY - Free space - all mounts on deployment-mediawiki-09 is OK: OK: All targets OK [15:50:34] RECOVERY - Free space - all mounts on integration-agent-docker-1003 is OK: OK: All targets OK [16:14:14] 10Phabricator, 10Operations, 10Traffic, 10serviceops: Phabricator downtime due to aphlict and websockets (aphlict current disabled) - https://phabricator.wikimedia.org/T238593 (10Dzahn) >>! In T238593#5674571, @ema wrote: > - 2019-11-15 17:30 SAL: `mutante: phabricator - -started phd service`. @Dzahn It's... [16:20:31] 10Phabricator, 10Operations, 10Traffic, 10serviceops: Phabricator downtime due to aphlict and websockets (aphlict current disabled) - https://phabricator.wikimedia.org/T238593 (10ema) >>! In T238593#5675662, @Dzahn wrote: > > You can entirely disregard that, i was on phab1001 and not phab1003 by accident.... [16:24:29] (03Abandoned) 10EBernhardson: Edit Project Config [wikidata/query/rdf] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/551687 (owner: 10EBernhardson) [16:24:43] (03Abandoned) 10EBernhardson: Edit Project Config [wikidata/query] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/551688 (owner: 10EBernhardson) [16:25:57] 10Phabricator, 10Operations, 10Traffic, 10serviceops: Phabricator downtime due to aphlict and websockets (aphlict current disabled) - https://phabricator.wikimedia.org/T238593 (10Dzahn) >>! In T238593#5675686, @ema wrote: > Was there any other admin action between the page and when @joe disabled proxy_wstu... [16:35:00] Greetings! I have a couple of questions regarding gerrit, it should trigger Jenkins builds on CR +1 and 'recheck' comments, right? For some reason it does not trigger CI for me, is it some issue with rights granted? I work as a supplier for Search team, and the projects in question are wikidata/query/blazegraph and wikidata/query/rdf [16:35:22] igor_kim: that does sound like a rights issue, i know gerrit ignores 'recheck' unless the correct rights are avaliable [16:35:58] not sure who can fix those these days though outside releng and releng is on a team offsite this week [16:36:25] so the question is what are these rights and if they might be granted to me? [16:36:56] Oh, OK, I could ask later if nobody could answer that these days. Thanks! [16:38:49] igor_kim: my best guess is you need to be added to this group, but i don't have access to do that: https://gerrit.wikimedia.org/r/#/admin/groups/995,members [16:42:29] 10Release-Engineering-Team (Pipeline), 10Analytics, 10Release Pipeline, 10Patch-For-Review, 10Services (watching): Migrate EventStreams to k8s deployment pipeline - https://phabricator.wikimedia.org/T238658 (10Ottomata) I can't seem to develop this locally due to a barrage of version problems. From what... [16:42:36] 10Phabricator, 10Operations, 10Traffic, 10serviceops: Phabricator downtime due to aphlict and websockets (aphlict current disabled) - https://phabricator.wikimedia.org/T238593 (10Dzahn) Regarding the puppetization: There is `hiera('phabricator_aphlict_enabled'.` which is now set to false. What this does:... [16:58:35] ebernhardson thanks! that's a start to ask later on. [17:35:08] 10Gerrit, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO, 10Analytics, and 3 others: Delete eventgate-ci repository from gerrit - https://phabricator.wikimedia.org/T229111 (10MarcoAurelio) Considering that the `delete-project` gerrit plugin does cause some unwanted effec... [17:47:53] PROBLEM - Parsoid on deployment-mediawiki-parsoid10 is CRITICAL: connect to address 172.16.0.141 and port 8000: Connection refused [17:47:53] PROBLEM - Parsoid on deployment-parsoid09 is CRITICAL: connect to address 172.16.5.63 and port 8000: Connection refused [18:11:30] 10Continuous-Integration-Config, 10Pywikibot, 10Documentation, 10Pywikibot-Documentation, 10good first bug: Pywikibot documentation build throws many warnings to console - https://phabricator.wikimedia.org/T187009 (10Morgan11235) a:03Morgan11235 Going to try to make progress on this one with my teammat... [18:39:52] PROBLEM - English Wikipedia Mobile Main page on beta-cluster is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Backend fetch failed - string 'Wikipedia' not found on 'https://en.m.wikipedia.beta.wmflabs.org:443/wiki/Main_Page?debug=true' - 2555 bytes in 7.469 second response time [18:42:35] PROBLEM - App Server Main HTTP Response on deployment-mediawiki-09 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:44:47] RECOVERY - English Wikipedia Mobile Main page on beta-cluster is OK: HTTP OK: HTTP/1.1 200 OK - 37000 bytes in 0.680 second response time [19:20:32] (03CR) 10Jforrester: [C: 03+2] Setup eventstreams service-pipeline-test-and-publish [integration/config] - 10https://gerrit.wikimedia.org/r/551838 (https://phabricator.wikimedia.org/T238658) (owner: 10Ottomata) [19:21:33] (03Merged) 10jenkins-bot: Setup eventstreams service-pipeline-test-and-publish [integration/config] - 10https://gerrit.wikimedia.org/r/551838 (https://phabricator.wikimedia.org/T238658) (owner: 10Ottomata) [19:23:04] (03CR) 10Jforrester: "Hmm, There's no eventstreams.git? Did you mean to add this to wikimedia/services/eventstreams.git? (Sorry, should have done the sanity che" [integration/config] - 10https://gerrit.wikimedia.org/r/551838 (https://phabricator.wikimedia.org/T238658) (owner: 10Ottomata) [19:27:27] RECOVERY - App Server Main HTTP Response on deployment-mediawiki-09 is OK: HTTP OK: HTTP/1.1 200 OK - 49452 bytes in 0.764 second response time [19:54:10] (03PS1) 10Daimona Eaytoy: Release 29.0.0 [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/551911 [19:54:57] (03CR) 10Daimona Eaytoy: [C: 04-2] "Per commit message" [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/551911 (owner: 10Daimona Eaytoy) [19:55:44] (03PS2) 10Daimona Eaytoy: Release 29.0.0 [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/551911 [19:56:26] Hellooo releng - Wondering if anyone can help me with a question regarding mediawiki/core integration tests. I’m curious how the tests being run in jenkins get values that would be set in LocalSettings.php from? e.g. a job like mediawiki-quibble-api-testing-vendor-sqlite-php72-docker ? [19:58:20] (03CR) 10Ottomata: "Ah, yes! patch coming..." [integration/config] - 10https://gerrit.wikimedia.org/r/551838 (https://phabricator.wikimedia.org/T238658) (owner: 10Ottomata) [19:59:39] (03PS1) 10Ottomata: Correct eventstreams repo name to mediawiki/services/eventstreams [integration/config] - 10https://gerrit.wikimedia.org/r/551914 (https://phabricator.wikimedia.org/T238658) [20:26:35] https://groups.google.com/forum/#!topic/repo-discuss/AlsqzByYfvY [20:26:44] gerrit survery ^^ [20:26:48] *survey [20:46:41] (03CR) 10Jforrester: [C: 03+2] Correct eventstreams repo name to mediawiki/services/eventstreams [integration/config] - 10https://gerrit.wikimedia.org/r/551914 (https://phabricator.wikimedia.org/T238658) (owner: 10Ottomata) [20:47:09] once per org it says [20:47:15] are you the one person for our org? [20:47:29] (03Merged) 10jenkins-bot: Correct eventstreams repo name to mediawiki/services/eventstreams [integration/config] - 10https://gerrit.wikimedia.org/r/551914 (https://phabricator.wikimedia.org/T238658) (owner: 10Ottomata) [20:48:01] (03PS1) 10Jeena Huneidi: Add blubberoid service [releng/local-charts] - 10https://gerrit.wikimedia.org/r/551918 [20:49:06] (03CR) 10Brennen Bearnes: [C: 03+2] Documentation for how to contribute to local-charts [releng/local-charts] - 10https://gerrit.wikimedia.org/r/546319 (https://phabricator.wikimedia.org/T236587) (owner: 10Jeena Huneidi) [20:49:12] (03CR) 10Brennen Bearnes: [V: 03+2 C: 03+2] Documentation for how to contribute to local-charts [releng/local-charts] - 10https://gerrit.wikimedia.org/r/546319 (https://phabricator.wikimedia.org/T236587) (owner: 10Jeena Huneidi) [20:49:53] 10Continuous-Integration-Config, 10Pywikibot, 10Documentation, 10Pywikibot-Documentation, 10good first bug: Pywikibot documentation build throws many warnings to console - https://phabricator.wikimedia.org/T187009 (10Dvorapa) >>! In T187009#5676107, @Morgan11235 wrote: > Going to try to make progress on... [20:52:51] apergos: paladox: I'd coordinate with thcipriani on that [20:53:00] greg-g thanks! [20:53:07] yup [21:23:02] 10Phabricator, 10Operations, 10Traffic, 10serviceops: Phabricator downtime due to aphlict and websockets (aphlict current disabled) - https://phabricator.wikimedia.org/T238593 (10Dzahn) The Hiera key now does all the things, also stopped the service and unloaded the httpd module wstunnel. After that Phabr... [21:27:41] (03PS1) 10Jforrester: project.config: Allow wikimedia/production.git inheritance [extensions] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/551928 [21:28:49] 10Gerrit, 10Phabricator: define requirements for new devtools cloud vps project to replace "git" and "phabricator" - https://phabricator.wikimedia.org/T236309 (10Andrew) I haven't been following this task, but @hashar says I should ask y'all what the story is with the 'gerrit' project. It's currently unclaime... [21:29:16] 10Gerrit: reviewer-bot sometimes not working - https://phabricator.wikimedia.org/T222983 (10valhallasw) The bot is running again -- the issue was an empty `file_regexp` on the configuration page and the bot crashing because of that. I'll add some extra monitoring as well as checks for the config page over the co... [21:29:23] (03CR) 10Jforrester: [V: 03+2 C: 03+2] project.config: Allow wikimedia/production.git inheritance [extensions] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/551928 (owner: 10Jforrester) [21:31:59] (03PS1) 10Jforrester: project.config: Allow wikimedia/production.git inheritance [vendor] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/551931 [21:33:29] (03CR) 10Paladox: [C: 03+2] project.config: Allow wikimedia/production.git inheritance [vendor] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/551931 (owner: 10Jforrester) [21:35:27] (03PS1) 10Jforrester: project.config: Allow wikimedia/production.git inheritance [skins] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/551933 [21:36:29] (03CR) 10Jforrester: [V: 03+2] project.config: Allow wikimedia/production.git inheritance [vendor] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/551931 (owner: 10Jforrester) [21:36:53] (03CR) 10Jforrester: [V: 03+2 C: 03+2] project.config: Allow wikimedia/production.git inheritance [skins] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/551933 (owner: 10Jforrester) [21:40:07] (03PS1) 1020after4: Allow wikimedia/production superproject [core] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/551934 [21:41:49] (03CR) 10Jforrester: [V: 03+2 C: 03+2] Allow wikimedia/production superproject [core] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/551934 (owner: 1020after4) [21:47:40] 10Gerrit: reviewer-bot sometimes not working - https://phabricator.wikimedia.org/T222983 (10Tgr) Thanks a lot! [22:24:09] 10Phabricator, 10Research: Fix Research component tags to make them non-mutually-exclusive - https://phabricator.wikimedia.org/T238258 (10leila) @Aklapper thanks for doing all the work. Some comments below. >>! In T238258#5669162, @Aklapper wrote: > > * **SORRY / QUESTION**: I screwed up which task was in whi... [23:06:36] 10Phabricator, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO (201911), 10serviceops, and 3 others: Make PHD run on the backup phabricator server (phab2001, currently) - https://phabricator.wikimedia.org/T232883 (10Dzahn) - It's now possible to start the phd service with... [23:13:06] 10Phabricator, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO (201911), 10serviceops, and 2 others: Make PHD run on the backup phabricator server (phab2001, currently) - https://phabricator.wikimedia.org/T232883 (10Dzahn) [23:17:09] 10Phabricator, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO, 10Availability, and 3 others: Deploy phabricator to phab2001.codfw.wmnet - https://phabricator.wikimedia.org/T137928 (10Dzahn) - phab2001 has been upgraded to buster. - puppet run shows no errors (unless wh... [23:26:31] Project beta-scap-eqiad build #276349: 04FAILURE in 2 min 4 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/276349/ [23:36:39] Yippee, build fixed! [23:36:40] Project beta-scap-eqiad build #276350: 09FIXED in 2 min 11 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/276350/