[00:02:02] PROBLEM - Mathoid on deployment-mathoid is CRITICAL: connect to address 172.16.5.73 and port 10042: Connection refused [02:22:04] RECOVERY - Mathoid on deployment-mathoid is OK: HTTP OK: HTTP/1.1 200 OK - 925 bytes in 0.037 second response time [02:28:03] PROBLEM - Mathoid on deployment-mathoid is CRITICAL: connect to address 172.16.5.73 and port 10042: Connection refused [02:31:00] 03:30:08 cmdline: git fetch --tags -v origin [02:31:00] 03:30:08 stderr: 'fatal: early EOF [02:31:00] 03:30:08 fatal: index-pack failed' [02:31:02] silly gerrit [02:31:38] Heh [02:31:41] Gc? [02:31:52] Seeing as gc ran at 2am utc [02:32:02] Potentially something like that yeah [02:32:14] I'm guessing it's gonna be transient, rather than "omg everything is broken" [02:32:20] Yup [02:58:05] RECOVERY - Mathoid on deployment-mathoid is OK: HTTP OK: HTTP/1.1 200 OK - 925 bytes in 0.030 second response time [03:14:05] PROBLEM - Mathoid on deployment-mathoid is CRITICAL: connect to address 172.16.5.73 and port 10042: Connection refused [03:49:04] RECOVERY - Mathoid on deployment-mathoid is OK: HTTP OK: HTTP/1.1 200 OK - 925 bytes in 0.036 second response time [04:00:06] PROBLEM - Mathoid on deployment-mathoid is CRITICAL: connect to address 172.16.5.73 and port 10042: Connection refused [05:18:53] PROBLEM - Free space - all mounts on deployment-fluorine02 is CRITICAL: CRITICAL: deployment-prep.deployment-fluorine02.diskspace._srv.byte_percentfree (<40.00%) [06:48:52] RECOVERY - Free space - all mounts on deployment-fluorine02 is OK: OK: All targets OK [07:13:31] PROBLEM - Citoid on deployment-sca01 is CRITICAL: connect to address 172.16.5.13 and port 1970: Connection refused [08:25:38] RECOVERY - Mathoid on deployment-mathoid is OK: HTTP OK: HTTP/1.1 200 OK - 925 bytes in 0.548 second response time [08:31:04] PROBLEM - Mathoid on deployment-mathoid is CRITICAL: connect to address 172.16.5.73 and port 10042: Connection refused [08:46:05] RECOVERY - Mathoid on deployment-mathoid is OK: HTTP OK: HTTP/1.1 200 OK - 925 bytes in 0.025 second response time [08:52:04] PROBLEM - Mathoid on deployment-mathoid is CRITICAL: connect to address 172.16.5.73 and port 10042: Connection refused [10:02:33] RECOVERY - Citoid on deployment-sca01 is OK: HTTP OK: HTTP/1.1 200 OK - 921 bytes in 0.027 second response time [10:13:43] PROBLEM - Citoid on deployment-sca01 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:23:30] RECOVERY - Citoid on deployment-sca01 is OK: HTTP OK: HTTP/1.1 200 OK - 921 bytes in 0.024 second response time [11:38:12] 10Continuous-Integration-Config, 10Pywikibot, 10Documentation, 10Patch-For-Review, and 2 others: Pywikibot documentation build throws many warnings to console - https://phabricator.wikimedia.org/T187009 (10Dvorapa) p:05Normal→03Low [11:38:44] 10Continuous-Integration-Config, 10Pywikibot, 10Documentation, 10Patch-For-Review, and 2 others: Pywikibot documentation build throws many warnings to console - https://phabricator.wikimedia.org/T187009 (10Dvorapa) [11:44:23] 10Continuous-Integration-Config, 10Pywikibot, 10Documentation, 10Patch-For-Review, and 2 others: Pywikibot documentation build throws many warnings to console - https://phabricator.wikimedia.org/T187009 (10Dvorapa) [12:01:06] 10Continuous-Integration-Config, 10Pywikibot, 10Documentation, 10Patch-For-Review, and 2 others: Pywikibot documentation build throws many warnings to console - https://phabricator.wikimedia.org/T187009 (10Dvorapa) [16:47:05] 10Continuous-Integration-Config, 10Pywikibot, 10Documentation, 10Pywikibot-Documentation, 10good first bug: Pywikibot documentation build throws many warnings to console - https://phabricator.wikimedia.org/T187009 (10Dvorapa) [16:56:19] PROBLEM - Citoid on deployment-sca02 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:56:40] 10Gerrit: Cannot log into gerrit: 'Cannot assign user name "want" to account 7082; name already in use.' - https://phabricator.wikimedia.org/T222336 (10Aklapper) [17:11:07] RECOVERY - Citoid on deployment-sca02 is OK: HTTP OK: HTTP/1.1 200 OK - 921 bytes in 0.028 second response time [17:11:45] 10Project-Admins: Create tags for all Wikimedia projects - https://phabricator.wikimedia.org/T154549 (10Aklapper) 05Open→03Stalled It is still unclear to me which problem would get solved by this, as this task lacks //specific examples//. That's why it is stalled. (I'm not generally against creating specif... [17:15:05] 10Project-Admins: A way to track when a ticket was created with a pre-filled template - https://phabricator.wikimedia.org/T220568 (10Aklapper) 05Open→03Declined > I think there would still have to be a stamp of some kind, such as "This ticket was made via template" but assuming Maniphest can handle the searc... [17:17:09] PROBLEM - Citoid on deployment-sca02 is CRITICAL: connect to address 172.16.5.112 and port 1970: Connection refused [17:19:35] 10Project-Admins: New tag: #personal-goal - https://phabricator.wikimedia.org/T138451 (10Aklapper) 05Open→03Declined IMHO this is sufficiently covered by using #Goal together with an assignee set, and/or using your `#User-NAME` project if existing. Boldly declining. Or potentially using a personal flag (tho... [17:22:08] RECOVERY - Citoid on deployment-sca02 is OK: HTTP OK: HTTP/1.1 200 OK - 921 bytes in 0.025 second response time [17:28:11] 10Project-Admins, 10Developer-notice, 10Notice: Using Phabricator for gadget-related tasks - https://phabricator.wikimedia.org/T85433 (10Aklapper) [17:28:13] 10Project-Admins: A #Templates tag for tasks related with creating/fixing MediaWiki templates? - https://phabricator.wikimedia.org/T101362 (10Aklapper) [18:07:47] 10Beta-Cluster-Infrastructure, 10Release Pipeline, 10serviceops, 10Core Platform Team Backlog (Next), and 2 others: Migrate Beta cluster services to use Kubernetes - https://phabricator.wikimedia.org/T220235 (10Krenair) Is there anything preventing us from running that same container in beta on docker? We... [18:19:39] 10Beta-Cluster-Infrastructure: Migrate away from Debian Jessie to Debian Stretch - https://phabricator.wikimedia.org/T218729 (10Krenair) [18:24:33] 10Scap: `scap update-interwiki-cache` needs unbreaking - https://phabricator.wikimedia.org/T219103 (10Reedy) 05Open→03Resolved a:03thcipriani >>! In T219103#5053360, @thcipriani wrote: > HTTPS push hasn't be disabled, only HTTP tokens have been disabled. You can login using your ldap password. WFM (not gr... [18:36:21] 10Beta-Cluster-Infrastructure, 10Release Pipeline, 10serviceops, 10Core Platform Team Backlog (Next), and 2 others: Migrate Beta cluster services to use Kubernetes - https://phabricator.wikimedia.org/T220235 (10Krinkle) The status quo is that services always run their code in beta before it reaches product... [18:37:01] 10Beta-Cluster-Infrastructure, 10Editing-team, 10Release Pipeline, 10serviceops, and 3 others: Migrate Beta cluster services to use Kubernetes - https://phabricator.wikimedia.org/T220235 (10Krinkle) [22:43:07] 10Phabricator-Sprint-Extension: Call to undefined method SprintProjectProfilePanelEngine::buildNavigation() - when accessing "Burndown" - https://phabricator.wikimedia.org/T222586 (10Vlaza-servoy-com) I have tracked the issue here: https://secure.phabricator.com/w/changelog/2019.16/ and more specifically here: P... [23:19:53] PROBLEM - Free space - all mounts on deployment-fluorine02 is CRITICAL: CRITICAL: deployment-prep.deployment-fluorine02.diskspace._srv.byte_percentfree (<50.00%) [23:52:04] RECOVERY - Mathoid on deployment-mathoid is OK: HTTP OK: HTTP/1.1 200 OK - 925 bytes in 0.027 second response time [23:58:06] PROBLEM - Mathoid on deployment-mathoid is CRITICAL: connect to address 172.16.5.73 and port 10042: Connection refused