[01:03:44] PROBLEM - Puppet staleness on deployment-logstash2 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [43200.0] [01:05:59] 10Continuous-Integration-Config, 10Release-Engineering-Team, 10Wikimedia-Site-requests, 10serviceops: Consider creating a puppet-compiler equivalent for mediawiki-config.git - https://phabricator.wikimedia.org/T220775 (10Jdforrester-WMF) Initial stabs at this starting in https://gerrit.wikimedia.org/r/c/op... [01:41:56] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments: 1.34.0-wmf.3 deployment blockers - https://phabricator.wikimedia.org/T220728 (10Jdforrester-WMF) [02:51:05] PROBLEM - Mathoid on deployment-mathoid is CRITICAL: connect to address 172.16.5.73 and port 10042: Connection refused [03:16:03] RECOVERY - Mathoid on deployment-mathoid is OK: HTTP OK: HTTP/1.1 200 OK - 925 bytes in 0.023 second response time [03:32:04] PROBLEM - Mathoid on deployment-mathoid is CRITICAL: connect to address 172.16.5.73 and port 10042: Connection refused [04:10:09] PROBLEM - Citoid on deployment-sca02 is CRITICAL: connect to address 172.16.5.112 and port 1970: Connection refused [04:15:11] RECOVERY - Citoid on deployment-sca02 is OK: HTTP OK: HTTP/1.1 200 OK - 921 bytes in 0.026 second response time [04:27:03] RECOVERY - Mathoid on deployment-mathoid is OK: HTTP OK: HTTP/1.1 200 OK - 925 bytes in 0.025 second response time [04:53:04] PROBLEM - Mathoid on deployment-mathoid is CRITICAL: connect to address 172.16.5.73 and port 10042: Connection refused [05:13:04] RECOVERY - Mathoid on deployment-mathoid is OK: HTTP OK: HTTP/1.1 200 OK - 925 bytes in 0.025 second response time [05:19:04] PROBLEM - Mathoid on deployment-mathoid is CRITICAL: connect to address 172.16.5.73 and port 10042: Connection refused [05:44:05] RECOVERY - Mathoid on deployment-mathoid is OK: HTTP OK: HTTP/1.1 200 OK - 925 bytes in 0.024 second response time [06:08:45] 10Continuous-Integration-Infrastructure, 10Shinken, 10Patch-For-Review: Shinken keeps alerting about long gone instances - https://phabricator.wikimedia.org/T218146 (10hashar) 05Open→03Resolved I am no more receiving ghost notifications, so I guess the Shinken configuration has been properly fixed. Addi... [06:26:03] PROBLEM - Free space - all mounts on deployment-fluorine02 is CRITICAL: (Service Check Timed Out) [07:00:51] RECOVERY - Free space - all mounts on deployment-fluorine02 is OK: OK: All targets OK [07:11:27] (03CR) 10WMDE-leszek: [C: 03+1] Update for WikibaseSchema → EntitySchema rename [integration/config] - 10https://gerrit.wikimedia.org/r/507298 (https://phabricator.wikimedia.org/T222189) (owner: 10Lucas Werkmeister (WMDE)) [07:28:06] 10Continuous-Integration-Infrastructure, 10Tracking-Neverending: doc.wikimedia.org: Generate documentation for release tags (tracking) - https://phabricator.wikimedia.org/T73062 (10hashar) [07:28:12] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Zuul, 10Patch-For-Review, 10Upstream: Allow ref-updated listener to filter out tag deletions - https://phabricator.wikimedia.org/T96390 (10hashar) 05Open→03Resolved Fixed upstream which ignores reference deletions by defa... [07:34:24] 10Release-Engineering-Team, 10Developer Productivity, 10Browser-Tests, 10Epic: TEC12:O1:O1.4 Goal – Support running Selenium browser tests in the docker local development environment - https://phabricator.wikimedia.org/T222234 (10hashar) [07:57:50] PROBLEM - Content Translation Server on deployment-sca01 is CRITICAL: connect to address 172.16.5.13 and port 8080: Connection refused [08:02:49] RECOVERY - Content Translation Server on deployment-sca01 is OK: HTTP OK: HTTP/1.1 200 OK - 904 bytes in 0.032 second response time [08:04:31] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10puppet-compiler: Puppet catalog compiler - increasing max concurrent jobs - https://phabricator.wikimedia.org/T221969 (10hashar) Indeed there is just two `m1.large` instances which have: | 4 | vCPUs | 8 GB | RAM | 80 GB | disk I would a... [08:13:24] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10puppet-compiler: Puppet catalog compiler - increasing max concurrent jobs - https://phabricator.wikimedia.org/T221969 (10hashar) p:05Triage→03Normal [09:22:45] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Kanban), 10Scap: On deployment-prep scap cache_git_info takes 12 minutes (that is too slow) - https://phabricator.wikimedia.org/T204762 (10hashar) I went with a very basic script ` lang=python,name=/mnt/home/jenkins-deploy/scap_git.py #!/usr/bin/pyth... [09:29:14] 10Continuous-Integration-Infrastructure: Add pre-commit hook that does basic checks like php -l - https://phabricator.wikimedia.org/T201778 (10Simetrical) @daniel @tstarling Does it make sense to you that I should work on this? I have most of the work done already for my own productivity, it's just a question of... [09:45:37] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Kanban), 10Scap: On deployment-prep scap cache_git_info takes 12 minutes (that is too slow) - https://phabricator.wikimedia.org/T204762 (10hashar) The issue lays somewhere into scap/sh.py :/ One would notice the slowdown in log messages: ` 00:04:00.5... [09:56:47] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments: 1.34.0-wmf.3 deployment blockers - https://phabricator.wikimedia.org/T220728 (10Aklapper) [10:48:37] PROBLEM - Content Translation Server on deployment-sca02 is CRITICAL: connect to address 172.16.5.112 and port 8080: Connection refused [10:58:40] RECOVERY - Content Translation Server on deployment-sca02 is OK: HTTP OK: HTTP/1.1 200 OK - 904 bytes in 0.024 second response time [11:09:36] PROBLEM - Content Translation Server on deployment-sca02 is CRITICAL: connect to address 172.16.5.112 and port 8080: Connection refused [11:24:06] PROBLEM - Mathoid on deployment-mathoid is CRITICAL: connect to address 172.16.5.73 and port 10042: Connection refused [11:24:40] RECOVERY - Content Translation Server on deployment-sca02 is OK: HTTP OK: HTTP/1.1 200 OK - 904 bytes in 0.030 second response time [12:06:39] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Kanban), 10Scap: On deployment-prep scap cache_git_info takes 12 minutes (that is too slow) - https://phabricator.wikimedia.org/T204762 (10hashar) Back in September I have noticed a spam of close() calls ( T204762#4610309 ). Before executing the proc... [12:23:48] (03PS1) 10Hashar: beta: limit scap to 512 max file descriptors [integration/config] - 10https://gerrit.wikimedia.org/r/507773 (https://phabricator.wikimedia.org/T204762) [12:28:45] (03CR) 10Hashar: "I have deployed the job and eventually:" [integration/config] - 10https://gerrit.wikimedia.org/r/507773 (https://phabricator.wikimedia.org/T204762) (owner: 10Hashar) [12:29:05] RECOVERY - Mathoid on deployment-mathoid is OK: HTTP OK: HTTP/1.1 200 OK - 925 bytes in 0.037 second response time [12:29:30] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Kanban), 10Scap, 10Patch-For-Review: On deployment-prep scap cache_git_info takes 12 minutes (that is too slow) - https://phabricator.wikimedia.org/T204762 (10hashar) a:03hashar [12:31:49] 10Continuous-Integration-Config, 10phan-taint-check-plugin: Upgrade php-ast to 1.0.1 - https://phabricator.wikimedia.org/T218719 (10Daimona) @Legoktm Per T216974#5143707, would it be possible to implement the same solution used for the phan job? [12:40:03] PROBLEM - Mathoid on deployment-mathoid is CRITICAL: connect to address 172.16.5.73 and port 10042: Connection refused [12:41:31] PROBLEM - Citoid on deployment-sca01 is CRITICAL: connect to address 172.16.5.13 and port 1970: Connection refused [12:42:14] 10Continuous-Integration-Config, 10Release-Engineering-Team (Kanban), 10Fundraising-Backlog, 10MediaWiki-extensions-DonationInterface, 10Patch-For-Review: Fundraising should fall back to non master - https://phabricator.wikimedia.org/T199130 (10hashar) 05Open→03Resolved Assuming it is fixed properly... [12:45:04] RECOVERY - Mathoid on deployment-mathoid is OK: HTTP OK: HTTP/1.1 200 OK - 925 bytes in 0.042 second response time [12:46:30] RECOVERY - Citoid on deployment-sca01 is OK: HTTP OK: HTTP/1.1 200 OK - 921 bytes in 0.027 second response time [12:59:05] (03CR) 10Hashar: [C: 03+2] "Lets try!" [integration/config] - 10https://gerrit.wikimedia.org/r/506889 (https://phabricator.wikimedia.org/T202030) (owner: 10Hashar) [13:00:48] (03Merged) 10jenkins-bot: Add MinvervaNeue and Vector to gate [integration/config] - 10https://gerrit.wikimedia.org/r/506889 (https://phabricator.wikimedia.org/T202030) (owner: 10Hashar) [13:01:04] PROBLEM - Mathoid on deployment-mathoid is CRITICAL: connect to address 172.16.5.73 and port 10042: Connection refused [13:18:30] PROBLEM - Long lived cherry-picks on puppetmaster on deployment-puppetmaster03 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [13:38:25] 10Continuous-Integration-Config, 10Release-Engineering-Team, 10Epic: Have dependencies of gated extensions in the gate - https://phabricator.wikimedia.org/T204252 (10hashar) [13:38:28] 10Continuous-Integration-Config, 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10Readers-Web-Backlog (Tracking): CI: Minerva PHPUnit tests should be included in shared extension gate job - https://phabricator.wikimedia.org/T202030 (10hashar) 05Open→03Resolved Should be good now! [13:40:17] PROBLEM - Host deployment-conf03 is DOWN: CRITICAL - Host Unreachable (172.16.5.30) [13:42:31] 10Release-Engineering-Team (Kanban), 10docker-pkg, 10serviceops, 10Patch-For-Review: Some HEAD requests to docker-registry yields 405 Method not allowed - https://phabricator.wikimedia.org/T214441 (10hashar) 05Open→03Resolved a:03hashar [13:42:48] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Kanban), 10Operations, 10HHVM, 10Patch-For-Review: hhvm systemd service on deployment-prep reports: hhvm.service: Ignoring invalid environment assignment 'RUN_AS_GROUP=www-data - https://phabricator.wikimedia.org/T209946 (10hashar) 05Open→03De... [13:46:04] RECOVERY - Mathoid on deployment-mathoid is OK: HTTP OK: HTTP/1.1 200 OK - 925 bytes in 0.034 second response time [13:48:55] 10Continuous-Integration-Infrastructure (Slipway), 10Release-Engineering-Team (Kanban), 10Lexicographical data, 10Wikidata, and 3 others: Migrate selenium-Wikibase-chrome selenium-WikibaseLexeme-chrome to Docker containers - https://phabricator.wikimedia.org/T210285 (10hashar) a:03hashar [13:57:06] PROBLEM - Mathoid on deployment-mathoid is CRITICAL: connect to address 172.16.5.73 and port 10042: Connection refused [14:03:39] 10Continuous-Integration-Infrastructure, 10Wikidata, 10Wikidata-Campsite, 10User-zeljkofilipin: Run browser tests as part of "npm test" of wikidata/query/gui - https://phabricator.wikimedia.org/T222200 (10zeljkofilipin) [14:04:05] 10Continuous-Integration-Infrastructure, 10Wikidata, 10Wikidata-Campsite, 10User-zeljkofilipin: Run browser tests as part of "npm test" of wikidata/query/gui - https://phabricator.wikimedia.org/T222200 (10zeljkofilipin) p:05Triage→03Normal [14:06:17] 10Release-Engineering-Team (Kanban), 10MediaWiki-extensions-MultimediaViewer, 10Multimedia, 10Patch-For-Review, 10User-zeljkofilipin: Misconfigured -- Unsupported OS/browser/version/device combo - https://phabricator.wikimedia.org/T214389 (10zeljkofilipin) a:05zeljkofilipin→03None [14:06:44] 10Release-Engineering-Team (Kanban), 10Multimedia, 10SDC Engineering, 10Multimedia-Current-Work, and 2 others: Jenkins job to run core tests against commons.wikimedia.beta.wmflabs.org - https://phabricator.wikimedia.org/T220621 (10zeljkofilipin) a:05zeljkofilipin→03None [14:08:57] 10Release-Engineering-Team, 10MobileFrontend, 10Browser-Tests, 10MW-1.34-notes (1.34.0-wmf.3; 2019-04-30), and 4 others: AssertionError: false === true at thereShouldBeALinkToCreateMyUserPage on wmf-quibble PHP jobs - https://phabricator.wikimedia.org/T221860 (10zeljkofilipin) [14:15:47] 10Release-Engineering-Team, 10Developer Productivity, 10Browser-Tests, 10Epic, 10User-zeljkofilipin: TEC12:O1:O1.4 Goal – Support running Selenium browser tests in the docker local development environment - https://phabricator.wikimedia.org/T222234 (10zeljkofilipin) [14:17:42] 10Continuous-Integration-Config, 10Release-Engineering-Team (Watching / External), 10Analytics: Status of analytics/limn-*-data git repositories? - https://phabricator.wikimedia.org/T221064 (10Milimetric) We decided to merge the repositories into our main reportupdater-queries repository. We will use that g... [14:31:34] (03PS3) 10Hashar: zuul: skip test/test-prio for CR+2 changes [integration/config] - 10https://gerrit.wikimedia.org/r/368154 (https://phabricator.wikimedia.org/T105474) [14:33:08] (03CR) 10jerkins-bot: [V: 04-1] zuul: skip test/test-prio for CR+2 changes [integration/config] - 10https://gerrit.wikimedia.org/r/368154 (https://phabricator.wikimedia.org/T105474) (owner: 10Hashar) [14:33:45] (03PS4) 10Hashar: zuul: skip test/test-prio for CR+2 changes [integration/config] - 10https://gerrit.wikimedia.org/r/368154 (https://phabricator.wikimedia.org/T105474) [14:34:39] 10Continuous-Integration-Config, 10Release-Engineering-Team (Kanban), 10Patch-For-Review: 'recheck' on a CR+2 patch should trigger gate-and-submit, not test - https://phabricator.wikimedia.org/T105474 (10hashar) a:03hashar [14:37:04] RECOVERY - Mathoid on deployment-mathoid is OK: HTTP OK: HTTP/1.1 200 OK - 925 bytes in 0.024 second response time [14:43:06] PROBLEM - Mathoid on deployment-mathoid is CRITICAL: connect to address 172.16.5.73 and port 10042: Connection refused [14:48:03] RECOVERY - Mathoid on deployment-mathoid is OK: HTTP OK: HTTP/1.1 200 OK - 925 bytes in 0.038 second response time [15:00:57] 10Continuous-Integration-Infrastructure, 10Operations: Jessie rsyslog_8.1901.0-1~bpo8+wmf1_amd64.deb package fails to upgrade - https://phabricator.wikimedia.org/T222166 (10fgiunchedi) >>! In T222166#5149390, @hashar wrote: > The workaround kind of make sense, however whenever we provision a new instance we wo... [15:11:52] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments: 1.34.0-wmf.3 deployment blockers - https://phabricator.wikimedia.org/T220728 (10Lucas_Werkmeister_WMDE) [15:18:23] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Kanban), 10Scap, 10Patch-For-Review: On deployment-prep scap cache_git_info takes 12 minutes (that is too slow) - https://phabricator.wikimedia.org/T204762 (10hashar) Courtesy of Tyler: {F28902832 size=full} antoine-approve [15:18:44] (03CR) 10Greg Grossmeier: [C: 03+1] "Nice: https://phabricator.wikimedia.org/F28902830" [integration/config] - 10https://gerrit.wikimedia.org/r/507773 (https://phabricator.wikimedia.org/T204762) (owner: 10Hashar) [15:20:39] (03CR) 10Thcipriani: [C: 03+2] "WOW! Nice work digging on this one!" [integration/config] - 10https://gerrit.wikimedia.org/r/507773 (https://phabricator.wikimedia.org/T204762) (owner: 10Hashar) [15:23:28] (03Merged) 10jenkins-bot: beta: limit scap to 512 max file descriptors [integration/config] - 10https://gerrit.wikimedia.org/r/507773 (https://phabricator.wikimedia.org/T204762) (owner: 10Hashar) [15:26:33] 10Scap: scap: look at removing scap/sh.py - https://phabricator.wikimedia.org/T222372 (10hashar) [15:27:01] 10Release-Engineering-Team (Backlog), 10Scap: scap: look at removing scap/sh.py - https://phabricator.wikimedia.org/T222372 (10hashar) p:05Triage→03Low [15:27:52] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Kanban), 10Scap, 10Patch-For-Review: On deployment-prep scap cache_git_info takes 12 minutes (that is too slow) - https://phabricator.wikimedia.org/T204762 (10hashar) 05Open→03Resolved T222372 would be able dropping scap/sh.git entirely, but I... [15:37:32] (03CR) 10Hashar: "Ideally, EventLogging would flag the content as being JSON and SyntaxHilight would automatically enable itself for pages being json conten" [integration/config] - 10https://gerrit.wikimedia.org/r/507707 (owner: 10Umherirrender) [15:41:35] kostajh: hello, in case you are around yet. Idid talk about your quibble change to add parsoid to it :) [15:42:09] with brennen and longma iirc who are working on kubernetes / helm charts for mediawiki and its services [15:42:22] hashar: OK [15:42:38] but we havent written down any conclusion nor did we have any good outcome [15:42:55] beside that adding that to Quibble might well end up be creating a tech debt for the future [15:43:18] since probbly one would then want to add restbase and other services (citoid, graphoid etcoid ...oid) [15:43:33] which sounds better done in the next grand future of CI with the docker pipeline [15:43:37] that being said hmm [15:43:38] too many *oids [15:43:40] yeah [15:43:41] :- [15:43:41] ( [15:44:11] maybe (maybe) we could get some kind of integration job that is dedicated to parsoid [15:44:25] which would spawn parsoid and run tests of the few extensions that rely on it [15:45:31] the patch I've proposed only starts parsoid if we're running selenium or qunit, do you see that as too risky? [15:45:55] and that is where the meeting is way too early for the west coast us folks bah :D [15:46:54] kostajh: not that risky, but that add feature to Quibble which is potentially asking for trouble in the future ! [15:46:57] It's low priority for us. My motivation is to be able to use Selenium for asserting that our client-side event logging is working correctly for an extension. [15:47:08] yeah definitely a good thing [15:47:14] and VisualEditor would most probably benefit from that [15:47:18] as well as parsoid I guess [15:47:25] lets talk about it tomorrow and see what we can do [15:48:08] but most probably [15:48:13] hashar: ok. If another day/time works better, let me know. [15:48:14] I would use quibble as is [15:48:50] and have the jenkins job to spawn parsoid , use Quibble to prepare a mediawiki , inject the relevant $wgParsoidServer or whatever config then run the tests [15:49:36] kostajh: the time slot is 15:30 for me so that is nearly perfect to me. I will just pass the information to the west coast folks on our monday meeting ;) [15:59:08] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments: 1.34.0-wmf.3 deployment blockers - https://phabricator.wikimedia.org/T220728 (10Jdforrester-WMF) [16:00:05] PROBLEM - Mathoid on deployment-mathoid is CRITICAL: connect to address 172.16.5.73 and port 10042: Connection refused [16:10:03] RECOVERY - Mathoid on deployment-mathoid is OK: HTTP OK: HTTP/1.1 200 OK - 925 bytes in 0.033 second response time [16:31:06] PROBLEM - Mathoid on deployment-mathoid is CRITICAL: connect to address 172.16.5.73 and port 10042: Connection refused [16:35:11] 10Continuous-Integration-Infrastructure, 10Operations: Upload Zuul 2.5.1-wmf7 package to apt.wikimedia.org - https://phabricator.wikimedia.org/T220380 (10hashar) I have already upgraded Zuul on the production machines as well as the WMCS instances. So uploading to apt.wikimedia.org would be a noop :] [16:41:06] RECOVERY - Mathoid on deployment-mathoid is OK: HTTP OK: HTTP/1.1 200 OK - 925 bytes in 0.037 second response time [16:58:37] 10Beta-Cluster-Infrastructure: Migrate away from Debian Jessie to Debian Stretch - https://phabricator.wikimedia.org/T218729 (10Krenair) [17:02:11] 10Release-Engineering-Team, 10Operations, 10Release Pipeline, 10Wikidata, and 5 others: Introduce wikidata termbox SSR to kubernetes - https://phabricator.wikimedia.org/T220402 (10Pablo-WMDE) @mobrovac Thanks for the feedback. If it is possible at all we would really appreciate if you could link us to the... [17:13:27] 10Continuous-Integration-Config, 10MediaWiki-Core-Testing, 10MediaWiki-extensions-MultimediaViewer, 10MobileFrontend, and 9 others: Audit tests/selenium/LocalSettings.php file aiming at possibly deprecating the feature - https://phabricator.wikimedia.org/T199939 (10Jdlrobson) [17:26:08] 10Beta-Cluster-Infrastructure: Migrate away from Debian Jessie to Debian Stretch - https://phabricator.wikimedia.org/T218729 (10Krenair) >>! In T218729#5143033, @fgiunchedi wrote: > re: logstash, prod hosts are stretch so starting up a stretch instance with the same roles/hiera is expected to work. There will be... [17:26:48] 10Beta-Cluster-Infrastructure: Migrate away from Debian Jessie to Debian Stretch - https://phabricator.wikimedia.org/T218729 (10Krenair) [17:48:04] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments: 1.34.0-wmf.3 deployment blockers - https://phabricator.wikimedia.org/T220728 (10sbassett) [17:51:15] PROBLEM - Puppet errors on deployment-logstash03 is CRITICAL: CRITICAL: 6.25% of data above the critical threshold [3.0] [18:14:04] PROBLEM - Mathoid on deployment-mathoid is CRITICAL: connect to address 172.16.5.73 and port 10042: Connection refused [18:24:06] RECOVERY - Mathoid on deployment-mathoid is OK: HTTP OK: HTTP/1.1 200 OK - 925 bytes in 0.033 second response time [18:51:17] RECOVERY - Puppet errors on deployment-logstash03 is OK: OK: Less than 1.00% above the threshold [2.0] [18:57:48] 10Gerrit, 10Release-Engineering-Team, 10Operations: Gerrit Hardware Upgrade - https://phabricator.wikimedia.org/T222391 (10thcipriani) [19:08:21] 10Gerrit, 10Release-Engineering-Team, 10Operations: Gerrit Hardware Upgrade - https://phabricator.wikimedia.org/T222391 (10Paladox) Wanted to note that gerrit2001 has 64gb of ram, so this increase would match it so that we have the same ram specs in both data centres. [19:19:29] 10Gerrit, 10Wikimedia-General-or-Unknown, 10Documentation, 10Epic, and 3 others: Update Gerrit /r/p/ links to /r/ - https://phabricator.wikimedia.org/T218844 (10Paladox) Upstream are deprecating cloning over /p/, see https://bugs.chromium.org/p/gerrit/issues/detail?id=10381#c14 [19:51:58] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments: 1.34.0-wmf.4 deployment blockers - https://phabricator.wikimedia.org/T220729 (10thcipriani) [19:52:00] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments: 1.34.0-wmf.3 deployment blockers - https://phabricator.wikimedia.org/T220728 (10thcipriani) [19:53:12] 10Gerrit, 10Release-Engineering-Team, 10Operations: Gerrit Hardware Upgrade - https://phabricator.wikimedia.org/T222391 (10Dzahn) So.. cobalt is already on a list of [[ T217764 | machines will be over 5 years old during FY19-20 ]] -> T217764#5005267 which was compiled to determine the number of needed (misc)... [19:54:26] 10Gerrit, 10Release-Engineering-Team, 10Operations, 10ops-eqiad, 10serviceops: Gerrit Hardware Upgrade - https://phabricator.wikimedia.org/T222391 (10Dzahn) [19:57:37] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments: 1.34.0-wmf.3 deployment blockers - https://phabricator.wikimedia.org/T220728 (10Jdforrester-WMF) [20:04:42] 10Release-Engineering-Team (Backlog), 10Browser-Tests, 10Readers-Web-Backlog (Readers-Web-Kanbanana-Board-2018-19-Q4), 10Spike, 10User-zeljkofilipin: [Spike] Have a discussion around Minerva selenium browser test architecture - https://phabricator.wikimedia.org/T220755 (10Jdlrobson) Things we discussed t... [20:17:11] (03PS1) 10Dduvall: doc: Publish documentation for pipelinelib [integration/config] - 10https://gerrit.wikimedia.org/r/507871 (https://phabricator.wikimedia.org/T222199) [20:18:14] (03PS2) 10Dduvall: doc: Publish documentation for pipelinelib [integration/config] - 10https://gerrit.wikimedia.org/r/507871 (https://phabricator.wikimedia.org/T222199) [20:19:43] marxarelli: Don't you have to introduce the docker image in one commit and use it in a follow-up to make CI happy? [20:21:05] James_F: not strictly (in this case) if the deployer does things in the right order (deploy jjb, merge, generate docker image, reload zuul) but i could split it to make things easier [20:21:11] :) [20:21:47] Ha. [20:21:54] I'll leave it in your capable hands. [20:27:04] no, i think you're right to suggest doing them separately. i'll split em [20:34:45] (03PS3) 10Dduvall: doc: Publish documentation for pipelinelib [integration/config] - 10https://gerrit.wikimedia.org/r/507871 (https://phabricator.wikimedia.org/T222199) [20:34:46] (03PS1) 10Dduvall: dockerfiles: Create gradle image [integration/config] - 10https://gerrit.wikimedia.org/r/507872 (https://phabricator.wikimedia.org/T222199) [20:36:40] (03CR) 10Umherirrender: "The dependency is to pass phan, but I can also add a stub for that class to make phan happy" [integration/config] - 10https://gerrit.wikimedia.org/r/507707 (owner: 10Umherirrender) [20:38:53] (03PS1) 10Dduvall: doc: Link to pipelinelib documentation [integration/docroot] - 10https://gerrit.wikimedia.org/r/507873 [20:39:58] (03PS2) 10Dduvall: doc: Link to pipelinelib documentation [integration/docroot] - 10https://gerrit.wikimedia.org/r/507873 (https://phabricator.wikimedia.org/T222199) [20:45:04] PROBLEM - Mathoid on deployment-mathoid is CRITICAL: connect to address 172.16.5.73 and port 10042: Connection refused [20:55:06] RECOVERY - Mathoid on deployment-mathoid is OK: HTTP OK: HTTP/1.1 200 OK - 925 bytes in 0.026 second response time [20:57:38] 10Phabricator, 10Developer-Advocacy (Apr-Jun 2019): Re-evaluate our use of Phabricator Conpherence chat - https://phabricator.wikimedia.org/T127640 (10Aklapper) * Re T127640#2116945: "less intimidating than IRC" has become moot as we use Zulip for GSoC and Outreachy participants. * Re "messaging problematic us... [21:04:30] 10Phabricator, 10Developer-Advocacy (Apr-Jun 2019): Re-evaluate our use of Phabricator Conpherence chat - https://phabricator.wikimedia.org/T127640 (10greg) >>! In T127640#5154388, @Aklapper wrote: > Looking at the list of active channels, the only both valid and active use case currently seems to be https://p... [21:10:13] 10Gerrit: Error logging on to gerrit - https://phabricator.wikimedia.org/T222336 (10Paladox) [21:15:49] PROBLEM - Content Translation Server on deployment-sca02 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:25:40] RECOVERY - Content Translation Server on deployment-sca02 is OK: HTTP OK: HTTP/1.1 200 OK - 904 bytes in 0.029 second response time [21:40:51] (03CR) 10Jforrester: [C: 03+1] doc: Link to pipelinelib documentation [integration/docroot] - 10https://gerrit.wikimedia.org/r/507873 (https://phabricator.wikimedia.org/T222199) (owner: 10Dduvall) [22:14:00] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments: 1.34.0-wmf.3 deployment blockers - https://phabricator.wikimedia.org/T220728 (10Jdforrester-WMF) [22:14:49] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments: 1.34.0-wmf.3 deployment blockers - https://phabricator.wikimedia.org/T220728 (10thcipriani) 05Open→03Resolved [22:17:05] PROBLEM - Mathoid on deployment-mathoid is CRITICAL: connect to address 172.16.5.73 and port 10042: Connection refused [22:37:05] RECOVERY - Mathoid on deployment-mathoid is OK: HTTP OK: HTTP/1.1 200 OK - 925 bytes in 0.024 second response time [22:48:05] PROBLEM - Mathoid on deployment-mathoid is CRITICAL: connect to address 172.16.5.73 and port 10042: Connection refused [22:53:53] 10Continuous-Integration-Config, 10Release-Engineering-Team (Backlog), 10JavaScript: Switch quibble-based CI jobs from node6 to node10 - https://phabricator.wikimedia.org/T222406 (10Jdforrester-WMF) p:05Triage→03Normal [22:55:29] Such fun. [23:14:19] 10Gerrit, 10Release-Engineering-Team (Watching / External), 10Operations, 10ops-eqiad, 10serviceops: Gerrit Hardware Upgrade - https://phabricator.wikimedia.org/T222391 (10greg) [23:21:42] 10Release-Engineering-Team (Backlog), 10Browser-Tests, 10Readers-Web-Backlog (Readers-Web-Kanbanana-Board-2018-19-Q4), 10Spike, 10User-zeljkofilipin: [Spike] Have a discussion around Minerva selenium browser test architecture - https://phabricator.wikimedia.org/T220755 (10Jdlrobson) In terms of next step... [23:23:05] RECOVERY - Mathoid on deployment-mathoid is OK: HTTP OK: HTTP/1.1 200 OK - 925 bytes in 0.026 second response time [23:29:04] PROBLEM - Mathoid on deployment-mathoid is CRITICAL: connect to address 172.16.5.73 and port 10042: Connection refused [23:44:05] RECOVERY - Mathoid on deployment-mathoid is OK: HTTP OK: HTTP/1.1 200 OK - 925 bytes in 0.039 second response time