[00:01:45] 10Beta-Cluster-Infrastructure, 10VisualEditor-MediaWiki-2017WikitextEditor, 10VisualEditor (Current work): Switching to NWE is failing on the Swedish beta cluster wiki - https://phabricator.wikimedia.org/T191184 (10Krenair) Not even in Parsoid's config: ```root@deployment-parsoid09:/var/log# grep svwiki /srv... [00:54:06] RECOVERY - Free space - all mounts on integration-slave-docker-1011 is OK: OK: All targets OK [01:45:23] PROBLEM - Puppet errors on deployment-cache-text04 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [02:18:31] Project beta-scap-eqiad build #213770: 15ABORTED in 1 hr 0 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/213770/ [02:22:35] PROBLEM - Free space - all mounts on deployment-tin is CRITICAL: CRITICAL: deployment-prep.deployment-tin.diskspace._mnt.byte_percentfree (No valid datapoints found)deployment-prep.deployment-tin.diskspace._srv.byte_percentfree (<22.22%) [02:50:40] PROBLEM - Puppet errors on integration-slave-docker-1011 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [03:47:36] why is everything so slow? ^ [03:47:38] :( [03:55:40] RECOVERY - Puppet errors on integration-slave-docker-1011 is OK: OK: Less than 1.00% above the threshold [0.0] [04:57:01] (03PS1) 10Phedenskog: Add performance/perfromance repo. [integration/config] - 10https://gerrit.wikimedia.org/r/443023 (https://phabricator.wikimedia.org/T196600) [05:30:49] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10DBA, 10MediaWiki-Database, 10Quibble: Enable MariaDB/MySQL strict mode on CI db hosts - https://phabricator.wikimedia.org/T119371 (10jcrespo) `ONLY_FULL_GROUP_BY` on MariaDB (see bug https://jira.mariadb.org/browse/MDEV-11588... [07:39:18] PROBLEM - Free space - all mounts on integration-slave-jessie-1002 is CRITICAL: CRITICAL: integration.integration-slave-jessie-1002.diskspace._mnt.byte_percentfree (No valid datapoints found)integration.integration-slave-jessie-1002.diskspace._srv.byte_percentfree (<11.11%) [08:04:44] (03CR) 10Thiemo Kreuz (WMDE): [C: 04-1] Add performance/perfromance repo. (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/443023 (https://phabricator.wikimedia.org/T196600) (owner: 10Phedenskog) [08:10:22] (03CR) 10Phedenskog: "Yep, checkout the README for the background: https://gerrit.wikimedia.org/r/#/c/performance/perfromance/+/443020/4/README.md" [integration/config] - 10https://gerrit.wikimedia.org/r/443023 (https://phabricator.wikimedia.org/T196600) (owner: 10Phedenskog) [08:24:16] 10Phabricator: Cannot change tags of tasks that are "Release" tasks - https://phabricator.wikimedia.org/T198411 (10Aklapper) a:05daniel>03mmodell [08:29:46] (03CR) 10Thiemo Kreuz (WMDE): "Oh dear. Why not call it performance-metrics-browser, if this is what it is?" [integration/config] - 10https://gerrit.wikimedia.org/r/443023 (https://phabricator.wikimedia.org/T196600) (owner: 10Phedenskog) [09:31:32] (03CR) 10Phedenskog: "IMHO this is smart and creative. A good name will help us getting more known in the performance community and that will help us in the fut" [integration/config] - 10https://gerrit.wikimedia.org/r/443023 (https://phabricator.wikimedia.org/T196600) (owner: 10Phedenskog) [09:38:36] 10Phabricator (Upstream), 10Developer-Relations, 10Upstream: Complete the round of Phabricator improvements funded by Wikimedia - https://phabricator.wikimedia.org/T136213 (10Aklapper) [09:38:39] 10Phabricator (2016-11-16), 10Upstream: Phabricator project auto-complete is arbitrary (in a bad way) - https://phabricator.wikimedia.org/T99739 (10Aklapper) [09:38:47] 10Phabricator (Upstream), 10Upstream: Exact matches should always win when suggesting/auto-completing - https://phabricator.wikimedia.org/T76732 (10Aklapper) 05Open>03Resolved >>! In T76732#3040513 in February 2017, @Aklapper wrote: > Upstream task https://secure.phabricator.com/T8510#202060 is only still... [09:43:39] 10Phabricator (Upstream), 10Developer-Relations (Apr-Jun-2018), 10Upstream: Complete the round of Phabricator improvements funded by Wikimedia - https://phabricator.wikimedia.org/T136213 (10Aklapper) [09:43:50] 10Release-Engineering-Team, 10Developer-Relations (Jul-Sep-2016), 10User-greg: Write blog post highlighting recent Phabricator improvements - https://phabricator.wikimedia.org/T137727 (10Aklapper) [09:43:53] 10Phabricator (Upstream), 10Developer-Relations (Apr-Jun-2018), 10Upstream: Complete the round of Phabricator improvements funded by Wikimedia - https://phabricator.wikimedia.org/T136213 (10Aklapper) 05Open>03Resolved Regarding the two remaining open tasks, * T96464 is less important nowadays thanks to m... [09:49:12] PROBLEM - Free space - all mounts on integration-slave-jessie-1004 is CRITICAL: CRITICAL: integration.integration-slave-jessie-1004.diskspace._srv.byte_percentfree (<55.56%) [09:55:08] (03CR) 10Thiemo Kreuz (WMDE): "Creative, yes. But literally making a typo the *only* distinctive feature of a product name is not smart." [integration/config] - 10https://gerrit.wikimedia.org/r/443023 (https://phabricator.wikimedia.org/T196600) (owner: 10Phedenskog) [09:57:24] PROBLEM - Free space - all mounts on integration-slave-docker-1001 is CRITICAL: CRITICAL: integration.integration-slave-docker-1001.diskspace.root.byte_percentfree (<33.33%) [09:57:58] zeljkof: if you're around want to go through the wikibase-docker stuff 'live'? [09:58:13] tarrow: sure! [09:58:42] awesome! so I seemed to be able to get the same problem as you but clearly my fix didn't work [09:58:54] we can do screen sharing via hangouts/meet if you prefer that [09:59:19] I think the problem is that wikibase dependencies are not installed via composer, somehow [09:59:20] yeah, that would be awesome shall I call your wmf email? [09:59:31] tarrow: yes [10:10:46] 10Release-Engineering-Team (Kanban), 10MediaWiki-Core-Tests, 10User-zeljkofilipin: Refactor mediawiki-core-qunit-selenium-jessie Jenkins job so qunit/karma and webdriverio are invoked via npm script - https://phabricator.wikimedia.org/T180125 (10zeljkofilipin) [10:12:15] Build timed out (after 45 minutes). Marking the build as failed. [10:12:18] ^ [10:12:24] guys any fix for that? [10:12:29] it's blocking core tests [10:37:24] RECOVERY - Free space - all mounts on integration-slave-docker-1001 is OK: OK: All targets OK [10:38:47] Hauskatze: looks like hashar is not yet around [10:39:39] zeljkof: ok, by the way, do you edit hrwiki? [10:40:02] Hauskatze: no :/ I did a few edits, but I am not regular at all [10:40:26] zeljkof: no prob! :) [10:41:21] Hauskatze: why do you ask? do you edit hrwiki? ;) [10:43:58] zeljkof: yes I do. I post notifications from meta :D [10:45:31] :) [10:50:37] 10Phabricator, 10Release-Engineering-Team (Someday), 10Patch-For-Review, 10Technical-Debt: Replace deprecated phabricator conduit api calls in phab_epipe.py file - https://phabricator.wikimedia.org/T159043 (10Aklapper) 05Open>03declined As this is an unused code path nowadays I decline this task becaus... [10:53:11] 10Release-Engineering-Team (Kanban), 10MediaWiki-Core-Tests, 10User-zeljkofilipin: Refactor mediawiki-core-qunit-selenium-jessie Jenkins job so qunit/karma and webdriverio are invoked via npm script - https://phabricator.wikimedia.org/T180125 (10zeljkofilipin) [10:53:43] 10Release-Engineering-Team (Kanban), 10MediaWiki-Core-Tests, 10User-zeljkofilipin: Run Selenium tests in CI per-patch for extensions - https://phabricator.wikimedia.org/T164721 (10zeljkofilipin) [10:53:45] 10Release-Engineering-Team (Kanban), 10MediaWiki-Core-Tests, 10User-zeljkofilipin: Refactor mediawiki-core-qunit-selenium-jessie Jenkins job so qunit/karma and webdriverio are invoked via npm script - https://phabricator.wikimedia.org/T180125 (10zeljkofilipin) 05Open>03Invalid Resolved by #quibble. [10:54:12] 10Gerrit, 10Phabricator, 10Release-Engineering-Team (Someday), 10Technical-Debt: Replace deprecated phabricator conduit api calls in gerrit's its-phabricator plugin - https://phabricator.wikimedia.org/T159041 (10Aklapper) p:05High>03Low Stalled on Gerrit upstream review, however if upstream Phab remove... [10:54:28] 10Release-Engineering-Team, 10MediaWiki-Core-Tests, 10User-zeljkofilipin: Q4 Selenium framework improvements - https://phabricator.wikimedia.org/T190994 (10zeljkofilipin) [10:54:31] 10Release-Engineering-Team (Kanban), 10MediaWiki-Core-Tests, 10User-zeljkofilipin: Run Selenium tests in CI per-patch for extensions - https://phabricator.wikimedia.org/T164721 (10zeljkofilipin) 05Open>03Resolved [11:04:20] 10Phabricator, 10Operations, 10Operations-Software-Development, 10Technical-Debt: Update Puppet repo code that uses maniphest.update and maniphest.createtask conduit api - https://phabricator.wikimedia.org/T159045 (10Aklapper) [11:14:57] 10Release-Engineering-Team, 10MediaWiki-Core-Tests, 10User-zeljkofilipin: Selenium test job should install local dependencies before starting tests - https://phabricator.wikimedia.org/T193943 (10zeljkofilipin) >>! In T193943#4313350, @Krinkle wrote: > 1. The selenium test do not need to be run from core beca... [11:15:56] 10Release-Engineering-Team, 10MediaWiki-Core-Tests, 10Quibble, 10User-zeljkofilipin: Selenium test job should install local dependencies before starting tests - https://phabricator.wikimedia.org/T193943 (10zeljkofilipin) [11:21:39] 10Phabricator, 10Security-General: Streamline process for uploading private files to public tasks - https://phabricator.wikimedia.org/T181317 (10Aklapper) @Tgr: > when the task should be public but the file private In that case, folks should never drag and drop a (to be private) file into a public task's comme... [11:22:53] 10Release-Engineering-Team, 10MediaWiki-Core-Tests, 10Quibble, 10User-zeljkofilipin: Selenium test job should install local dependencies before starting tests - https://phabricator.wikimedia.org/T193943 (10zeljkofilipin) >>! In T193943#4313350, @Krinkle wrote: > * Update the Jenkins job for selenium to run... [11:24:12] 10Release-Engineering-Team (Kanban), 10MediaWiki-Core-Tests, 10User-zeljkofilipin: Set up CI for running Selenium tests for repositories that have Ruby tests - https://phabricator.wikimedia.org/T198409 (10zeljkofilipin) a:03zeljkofilipin [11:24:15] 10Phabricator: The Edit Task form of an existing tag should expose the 'Due Date' field - https://phabricator.wikimedia.org/T193770 (10Aklapper) [11:32:10] PROBLEM - Puppet errors on integration-slave-docker-1012 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [11:32:49] 10Release-Engineering-Team (Kanban), 10MediaWiki-Core-Tests, 10User-zeljkofilipin: Set up CI for running Node.js+Selenium tests for repositories that have Ruby+Selenium tests - https://phabricator.wikimedia.org/T198409 (10zeljkofilipin) [11:33:00] PROBLEM - Puppet errors on integration-slave-docker-1004 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [11:34:31] 10MediaWiki-Releasing, 10Phabricator, 10Project-Admins, 10Security-Team: Create "security pre-announce" group in Phab (to easier allow 3rd parties who get pre-release notifications to access Security tasks) - https://phabricator.wikimedia.org/T108360 (10Aklapper) [11:36:14] PROBLEM - Puppet errors on integration-slave-docker-1015 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [12:07:11] RECOVERY - Puppet errors on integration-slave-docker-1012 is OK: OK: Less than 1.00% above the threshold [0.0] [12:08:00] RECOVERY - Puppet errors on integration-slave-docker-1004 is OK: OK: Less than 1.00% above the threshold [0.0] [12:11:15] RECOVERY - Puppet errors on integration-slave-docker-1015 is OK: OK: Less than 1.00% above the threshold [0.0] [12:16:54] PROBLEM - Puppet errors on deployment-maps03 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [12:18:13] PROBLEM - Puppet errors on deployment-mathoid is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [12:20:24] PROBLEM - Puppet errors on deployment-tin is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [12:23:20] PROBLEM - Puppet errors on deployment-restbase02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [12:23:32] 10Release-Engineering-Team (Kanban), 10MediaWiki-Core-Tests, 10User-zeljkofilipin: Set up CI for running Node.js+Selenium tests for repositories that have Ruby+Selenium tests - https://phabricator.wikimedia.org/T198409 (10zeljkofilipin) [12:24:10] PROBLEM - Puppet errors on deployment-mediawiki06 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [12:26:42] PROBLEM - Puppet errors on deployment-restbase01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [12:27:32] PROBLEM - Puppet errors on deployment-pdfrender02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [12:28:36] 10Release-Engineering-Team (Kanban), 10MediaWiki-Core-Tests, 10User-zeljkofilipin: Set up CI for running Node.js+Selenium tests for repositories that have Ruby+Selenium tests - https://phabricator.wikimedia.org/T198409 (10zeljkofilipin) [12:30:20] 10Release-Engineering-Team (Kanban), 10MediaWiki-Core-Tests, 10User-zeljkofilipin: Set up CI for running Node.js+Selenium tests for repositories that have Ruby+Selenium tests - https://phabricator.wikimedia.org/T198409 (10zeljkofilipin) [12:30:40] PROBLEM - Puppet errors on deployment-zotero01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [12:30:57] PROBLEM - Puppet errors on deployment-logstash2 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [12:32:11] PROBLEM - Puppet errors on deployment-mcs01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [12:33:49] Hi - changes to the new extension "MintyDocs" are not getting handled by Jenkins: https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/MintyDocs/+/443017/ [12:34:05] What needs to be done to get Jenkins to know about MintyDocs? [12:34:09] PROBLEM - Puppet errors on deployment-aqs03 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [12:36:10] PROBLEM - Puppet errors on deployment-cpjobqueue is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [12:37:33] PROBLEM - Puppet errors on deployment-cassandra3-01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [12:37:45] PROBLEM - Puppet errors on deployment-sca02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [12:38:09] PROBLEM - Puppet errors on deployment-parsoid09 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [12:38:36] PROBLEM - Puppet errors on deployment-sca01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [12:38:57] 10Release-Engineering-Team (Kanban), 10MediaWiki-Core-Tests, 10User-zeljkofilipin: Set up CI for running Node.js+Selenium tests for repositories that have Ruby+Selenium tests - https://phabricator.wikimedia.org/T198409 (10zeljkofilipin) [12:39:01] PROBLEM - Puppet errors on deployment-aqs01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [12:39:13] PROBLEM - Puppet errors on deployment-changeprop is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [12:41:01] 10Continuous-Integration-Infrastructure, 10Patch-For-Review, 10Upstream, 10Wikimedia-log-errors (Shared Build Failure): Quibble CI jobs time out after 30min due to instance stalling at "npm install parse" step - https://phabricator.wikimedia.org/T198348 (10hashar) Tentatively the incident is resolved on np... [12:44:17] PROBLEM - Puppet errors on deployment-aqs02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [12:45:18] PROBLEM - Puppet errors on deployment-cassandra3-02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [12:47:07] PROBLEM - Puppet errors on deployment-mira is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [12:47:14] PROBLEM - Puppet errors on deployment-imagescaler01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [12:50:12] 10Phabricator: The Edit Task form of an existing task should expose the 'Due Date' field - https://phabricator.wikimedia.org/T193770 (10dbarratt) [12:58:18] 10Release-Engineering-Team (Kanban), 10MediaWiki-Core-Tests, 10User-zeljkofilipin: Set up CI for running Node.js+Selenium tests for repositories that have Ruby+Selenium tests - https://phabricator.wikimedia.org/T198409 (10zeljkofilipin) [13:04:53] Hi - changes to the new extension "MintyDocs" are not getting handled by Jenkins: https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/MintyDocs/+/443017/ . What needs to be done to get Jenkins to know about MintyDocs? [13:05:23] 10Release-Engineering-Team (Kanban), 10MediaWiki-Core-Tests, 10User-zeljkofilipin: Set up CI for running Node.js+Selenium tests for repositories that have Ruby+Selenium tests - https://phabricator.wikimedia.org/T198409 (10zeljkofilipin) [13:08:10] 10Release-Engineering-Team (Kanban), 10MediaWiki-Core-Tests, 10User-zeljkofilipin: Set up CI for running Node.js+Selenium tests for repositories that have Ruby+Selenium tests - https://phabricator.wikimedia.org/T198409 (10zeljkofilipin) [13:08:43] 10Release-Engineering-Team (Kanban), 10MediaWiki-Core-Tests, 10User-zeljkofilipin: Q1 Selenium framework improvements - https://phabricator.wikimedia.org/T198389 (10zeljkofilipin) [13:08:47] 10Release-Engineering-Team (Kanban), 10MediaWiki-Core-Tests, 10User-zeljkofilipin: Run Selenium tests in CI per-patch for extensions - https://phabricator.wikimedia.org/T164721 (10zeljkofilipin) [13:08:49] 10Release-Engineering-Team (Kanban), 10MediaWiki-Core-Tests, 10User-zeljkofilipin: Set up CI for running Node.js+Selenium tests for repositories that have Ruby+Selenium tests - https://phabricator.wikimedia.org/T198409 (10zeljkofilipin) [13:09:16] 10Release-Engineering-Team (Kanban), 10MediaWiki-Core-Tests, 10User-zeljkofilipin: Run Selenium tests in CI per-patch for extensions - https://phabricator.wikimedia.org/T164721 (10zeljkofilipin) [13:09:18] 10Release-Engineering-Team (Kanban), 10MediaWiki-Core-Tests, 10User-zeljkofilipin: Set up CI for running Node.js+Selenium tests for repositories that have Ruby+Selenium tests - https://phabricator.wikimedia.org/T198409 (10zeljkofilipin) 05Open>03Invalid Resolved by #quibble. [13:10:19] 10Release-Engineering-Team (Kanban), 10MediaWiki-Core-Tests, 10User-zeljkofilipin: Q1 Selenium framework improvements - https://phabricator.wikimedia.org/T198389 (10zeljkofilipin) [13:12:12] 10Beta-Cluster-Infrastructure, 10Parsoid, 10VisualEditor: VE is not loading on Beta Cluster, getting 503s - https://phabricator.wikimedia.org/T198421 (10Deskana) p:05Triage>03High [13:12:33] 10Release-Engineering-Team (Kanban), 10MediaWiki-Core-Tests, 10User-zeljkofilipin: Sample code in Node.js for repositories that still have Selenium+Ruby tests - https://phabricator.wikimedia.org/T183160 (10zeljkofilipin) a:03zeljkofilipin [13:14:25] 10Release-Engineering-Team (Kanban), 10MediaWiki-Core-Tests, 10User-zeljkofilipin: Sample code in Node.js for repositories that still have Selenium+Ruby tests - https://phabricator.wikimedia.org/T183160 (10zeljkofilipin) [13:14:47] 10Release-Engineering-Team (Kanban), 10MediaWiki-Core-Tests, 10User-zeljkofilipin: Patches in Gerrit deleting Selenium+Ruby tests for repositories that still have them - https://phabricator.wikimedia.org/T183162 (10zeljkofilipin) [13:22:34] 10Continuous-Integration-Infrastructure, 10Patch-For-Review, 10Upstream, 10Wikimedia-log-errors (Shared Build Failure): Quibble CI jobs time out after 30min due to instance stalling at "npm install parse" step - https://phabricator.wikimedia.org/T198348 (10hashar) From my investigation it seems the npm iss... [13:40:30] 10Project-Admins, 10Security-Team: Undefined #Security-General and #Security-Other - https://phabricator.wikimedia.org/T109328 (10Aklapper) * I propose to archive the tags #Security-Core, #Security-General, #Security-Other, #Security-Extensions. These four tags are from pre-2015 Bugzilla times when a task coul... [13:50:31] 10Release-Engineering-Team (Kanban), 10Fundraising-Backlog, 10MediaWiki-extensions-CentralNotice: Port CentralNotice Selenium tests from Ruby to Node.js - https://phabricator.wikimedia.org/T180223 (10zeljkofilipin) a:03zeljkofilipin [13:50:41] 10Release-Engineering-Team (Kanban), 10Fundraising-Backlog, 10MediaWiki-extensions-CentralNotice, 10User-zeljkofilipin: Port CentralNotice Selenium tests from Ruby to Node.js - https://phabricator.wikimedia.org/T180223 (10zeljkofilipin) [13:53:14] Hi - how do I get a new extension (in this case, MintyDocs) to be handled by Jenkins? [14:04:10] PROBLEM - Free space - all mounts on integration-slave-jessie-1004 is CRITICAL: CRITICAL: integration.integration-slave-jessie-1004.diskspace._srv.byte_percentfree (<44.44%) [14:09:42] 10Release-Engineering-Team, 10Performance-Team: Saving timings up 50% since 2018-06-28 20:53 - https://phabricator.wikimedia.org/T198483 (10jcrespo) [14:10:10] 10Release-Engineering-Team (Kanban), 10Fundraising-Backlog, 10MediaWiki-extensions-CentralNotice, 10User-zeljkofilipin: Port CentralNotice Selenium tests from Ruby to Node.js - https://phabricator.wikimedia.org/T180223 (10zeljkofilipin) [14:12:39] zeljkof: hi...!! :) thanks for working on those browser tests!!!! Mmmm indeed the centralnotice vagrant role is not maintained, afik in any case... I've never used it ;p [14:33:51] AndyRussG|a-whey: is there another way of setting centralnotice locally? [14:34:49] Yaron hi, you need to add it to integration/config [14:37:17] 10Release-Engineering-Team (Kanban), 10MediaWiki-Core-Tests, 10Multimedia, 10UploadWizard, and 2 others: UploadWizard: Selenium tests failing - https://phabricator.wikimedia.org/T198384 (10matmarex) p:05High>03Unbreak! This is UBN, we've just made a breaking change in MediaWiki core (https://gerrit.wik... [14:40:25] 10Release-Engineering-Team (Kanban), 10MediaWiki-Core-Tests, 10Multimedia, 10UploadWizard, and 2 others: UploadWizard: Selenium tests failing - https://phabricator.wikimedia.org/T198384 (10Paladox) You can temporarily add the right to v+2 in that repo (if your the project owner of mediawiki/extensions/Uplo... [14:40:36] 10Release-Engineering-Team (Kanban), 10Fundraising-Backlog, 10MediaWiki-extensions-CentralNotice, 10Patch-For-Review, 10User-zeljkofilipin: Port CentralNotice Selenium tests from Ruby to Node.js - https://phabricator.wikimedia.org/T180223 (10zeljkofilipin) a:05zeljkofilipin>03None [14:40:51] In 20 minutes there will be a brief nova outage. I'll be stopping nodepool for a few minutes to keep it from getting too confused. [14:41:01] we're just swapping a network cable so it should be painless. [14:42:03] 10Release-Engineering-Team (Kanban), 10Fundraising-Backlog, 10MediaWiki-extensions-CentralNotice, 10Patch-For-Review, 10User-zeljkofilipin: Port CentralNotice Selenium tests from Ruby to Node.js - https://phabricator.wikimedia.org/T180223 (10zeljkofilipin) [14:42:52] 10Release-Engineering-Team (Kanban), 10MediaWiki-Core-Tests, 10Multimedia, 10UploadWizard, and 2 others: UploadWizard: Selenium tests failing - https://phabricator.wikimedia.org/T198384 (10matmarex) So, this is the first failing test: https://integration.wikimedia.org/ci/job/quibble-vendor-mysql-hhvm-docke... [14:43:12] 10Release-Engineering-Team (Kanban), 10MediaWiki-Core-Tests, 10User-zeljkofilipin: Sample code in Node.js for repositories that still have Selenium+Ruby tests - https://phabricator.wikimedia.org/T183160 (10zeljkofilipin) [14:43:17] 10Release-Engineering-Team (Kanban), 10Fundraising-Backlog, 10MediaWiki-extensions-CentralNotice, 10Patch-For-Review, 10User-zeljkofilipin: Port CentralNotice Selenium tests from Ruby to Node.js - https://phabricator.wikimedia.org/T180223 (10zeljkofilipin) [14:44:09] 10Release-Engineering-Team (Kanban), 10MediaWiki-Core-Tests, 10User-zeljkofilipin: Sample code in Node.js for repositories that still have Selenium+Ruby tests - https://phabricator.wikimedia.org/T183160 (10zeljkofilipin) [14:44:47] zeljkof: I just install it in a locally set-up MW instance. You don't really need two wikis, a single wiki can have both as controller ("infrastructure") and "subscribing" roles [14:44:51] 10Release-Engineering-Team (Kanban), 10MediaWiki-Core-Tests, 10User-zeljkofilipin: Sample code in Node.js for repositories that still have Selenium+Ruby tests - https://phabricator.wikimedia.org/T183160 (10zeljkofilipin) [14:45:03] 10Release-Engineering-Team (Kanban), 10MediaWiki-Core-Tests, 10User-zeljkofilipin: Sample code in Node.js for repositories that still have Selenium+Ruby tests - https://phabricator.wikimedia.org/T183160 (10zeljkofilipin) [14:45:11] https://www.mediawiki.org/wiki/Extension:CentralNotice [14:45:19] 10Release-Engineering-Team (Kanban), 10MediaWiki-Core-Tests, 10User-zeljkofilipin: Sample code in Node.js for repositories that still have Selenium+Ruby tests - https://phabricator.wikimedia.org/T183160 (10zeljkofilipin) [14:45:28] zeljkof: If you're on vagrant, you could also just set it up manually there [14:45:31] 10Release-Engineering-Team (Kanban), 10MediaWiki-Core-Tests, 10User-zeljkofilipin: Sample code in Node.js for repositories that still have Selenium+Ruby tests - https://phabricator.wikimedia.org/T183160 (10zeljkofilipin) [14:45:45] 10Release-Engineering-Team (Kanban), 10MediaWiki-Core-Tests, 10User-zeljkofilipin: Sample code in Node.js for repositories that still have Selenium+Ruby tests - https://phabricator.wikimedia.org/T183160 (10zeljkofilipin) [14:46:10] AndyRussG|a-whey: is it documented somewhere? [14:46:25] zeljkof: ^ see link above [14:46:52] https://www.mediawiki.org/wiki/Extension:CentralNotice#Quick_developer_setup [14:47:38] AndyRussG|a-whey: thanks, I think I'll skip, this is just one of several extensions I need to set up, I don't really have the time to install it manually :) [14:48:17] vagrant role is pretty much all the time I have for it, if it does not work, I skip the extension :) [14:48:43] zeljkof: okok! yeah very understandable... I'll try to look at the selenium node stuff soon! [14:48:59] pls ping if I'm taking too long ;p [14:49:11] AndyRussG|a-whey: I have just pushed some sample code https://gerrit.wikimedia.org/r/c/mediawiki/extensions/CentralNotice/+/443093 [14:49:13] 10Phabricator, 10Project-Admins, 10Security-Team: Create Tag for WMDE Fundraising Security issues - https://phabricator.wikimedia.org/T194286 (10Aklapper) If there are further questions, please ask. Also please tell how to proceed - as I said, I do not see a need for the tag proposed in the task summary but... [14:49:21] 10Release-Engineering-Team (Kanban), 10MediaWiki-Core-Tests, 10Multimedia, 10UploadWizard, and 2 others: UploadWizard: Selenium tests failing - https://phabricator.wikimedia.org/T198384 (10matmarex) a:03matmarex [14:49:26] paladox: oh yeah, now I remember, thank... I'll create a Phabricator task for that. [14:49:30] (thanks) [14:49:31] zeljkof: cool!!!! thx!!!! [14:49:37] your welcome :) [14:51:08] 10Release-Engineering-Team (Kanban), 10MediaWiki-extensions-GettingStarted: Port GettingStarted Selenium tests from Ruby to Node.js - https://phabricator.wikimedia.org/T180224 (10zeljkofilipin) a:03zeljkofilipin [14:51:17] 10Release-Engineering-Team (Kanban), 10MediaWiki-extensions-GettingStarted, 10User-zeljkofilipin: Port GettingStarted Selenium tests from Ruby to Node.js - https://phabricator.wikimedia.org/T180224 (10zeljkofilipin) [14:54:11] RECOVERY - Free space - all mounts on integration-slave-jessie-1004 is OK: OK: All targets OK [14:54:37] 10Continuous-Integration-Config: New extension, MintyDocs, needs to be added to zuul/layout.yaml - https://phabricator.wikimedia.org/T198488 (10Yaron_Koren) [14:57:26] i wonder if we should add to the topic a link to a page where it describes on how to add new ext to the ci repo [15:02:27] 10MediaWiki-Releasing, 10Security, 10Security-Extensions: Backport security fixes to stable + LTS [non-bundled] extension branches - https://phabricator.wikimedia.org/T108734 (10Aklapper) [15:04:34] 10Release-Engineering-Team (Kanban), 10MediaWiki-extensions-GettingStarted, 10User-zeljkofilipin: Port GettingStarted Selenium tests from Ruby to Node.js - https://phabricator.wikimedia.org/T180224 (10zeljkofilipin) p:05Triage>03Low [15:13:36] update: the nodepool interruption is postponed two hours [15:13:48] So, happening at 10AM PDT [15:18:29] 10Release-Engineering-Team (Kanban), 10MediaWiki-extensions-GettingStarted, 10Patch-For-Review, 10User-zeljkofilipin: Port GettingStarted Selenium tests from Ruby to Node.js - https://phabricator.wikimedia.org/T180224 (10zeljkofilipin) [15:21:13] 10Release-Engineering-Team (Watching / External), 10Pywikibot-General, 10Pywikibot-core: Share Appveyor account credentials with Release Engineering - https://phabricator.wikimedia.org/T104306 (10Aklapper) [15:23:58] 10Release-Engineering-Team (Kanban), 10MediaWiki-Core-Tests, 10Multimedia, 10UploadWizard, and 3 others: UploadWizard: Selenium tests failing - https://phabricator.wikimedia.org/T198384 (10matmarex) Changes waiting to be merged after this is fixed: https://gerrit.wikimedia.org/r/q/project:mediawiki%252Fext... [15:24:10] paladox: maybe new extensions should get automatically added to that file (zuul/layout.yaml)? [15:24:11] 10Phabricator, 10Pywikibot-General, 10Pywikibot-core, 10Pywikibot-RfCs, 10RfC: RFC: Add pywikibot-core to all new pywikibot-script/module/class tasks automatically - https://phabricator.wikimedia.org/T170090 (10Aklapper) [15:24:19] 10Phabricator, 10Project-Admins, 10Pywikibot-core: Merge #Pywikibot-General to #Pywikibot-core - https://phabricator.wikimedia.org/T195856 (10Aklapper) 05Open>03Resolved * Added the #pywikibot-core tag to 137 tasks that were in #pywikibot-general * Updated the project description of #pywikibot-general ac... [15:24:25] Yaron i doin't think it's as easy as that [15:24:29] 10Release-Engineering-Team (Kanban), 10MediaWiki-extensions-MultimediaViewer, 10Multimedia: Port MultimediaViewer Selenium tests from Ruby to Node.js - https://phabricator.wikimedia.org/T180226 (10zeljkofilipin) a:03zeljkofilipin [15:24:32] Oh. [15:24:34] 10Release-Engineering-Team (Kanban), 10MediaWiki-extensions-MultimediaViewer, 10Multimedia: Port MultimediaViewer Selenium tests from Ruby to Node.js - https://phabricator.wikimedia.org/T180226 (10zeljkofilipin) p:05Triage>03Low [15:24:43] 10Release-Engineering-Team (Kanban), 10MediaWiki-extensions-MultimediaViewer, 10Multimedia, 10User-zeljkofilipin: Port MultimediaViewer Selenium tests from Ruby to Node.js - https://phabricator.wikimedia.org/T180226 (10zeljkofilipin) [15:25:12] PROBLEM - Free space - all mounts on integration-slave-jessie-1004 is CRITICAL: CRITICAL: integration.integration-slave-jessie-1004.diskspace._srv.byte_percentfree (<44.44%) [15:36:18] 10Release-Engineering-Team (Kanban), 10MediaWiki-Core-Tests, 10User-zeljkofilipin: Sample code in Node.js for repositories that still have Selenium+Ruby tests - https://phabricator.wikimedia.org/T183160 (10zeljkofilipin) [15:37:26] 10Release-Engineering-Team (Kanban), 10MediaWiki-extensions-MultimediaViewer, 10Multimedia, 10Patch-For-Review, 10User-zeljkofilipin: Port MultimediaViewer Selenium tests from Ruby to Node.js - https://phabricator.wikimedia.org/T180226 (10zeljkofilipin) [15:38:28] 10Release-Engineering-Team (Kanban), 10QuickSurveys, 10Readers-Web-Backlog (Tracking): Rewrite QuickSurveys browser tests in Node.js or remove them - https://phabricator.wikimedia.org/T177261 (10zeljkofilipin) a:03zeljkofilipin [15:38:38] 10Release-Engineering-Team (Kanban), 10QuickSurveys, 10Readers-Web-Backlog (Tracking), 10User-zeljkofilipin: Rewrite QuickSurveys browser tests in Node.js or remove them - https://phabricator.wikimedia.org/T177261 (10zeljkofilipin) [15:42:59] 10Continuous-Integration-Infrastructure, 10Wikidata, 10Jenkins: php-lint in wikibase times out - https://phabricator.wikimedia.org/T198493 (10cscott) [15:45:06] 10Continuous-Integration-Infrastructure, 10Patch-For-Review, 10Upstream, 10Wikimedia-log-errors (Shared Build Failure): Quibble CI jobs time out after 30min due to instance stalling at "npm install parse" step - https://phabricator.wikimedia.org/T198348 (10cscott) >>! In T198348#4323165, @hashar wrote: >>>... [15:51:34] 10Beta-Cluster-Infrastructure, 10Parsoid, 10VisualEditor: VE is not loading on Beta Cluster, getting 503s - https://phabricator.wikimedia.org/T198421 (10cscott) Confirmed that Parsoid is on b068bb51d29e294a4f4a875ae829cca8cf314205 in both prod and beta. beta: ``` deployment-tin$ curl http://deployment-parsoi... [15:53:35] 10Release-Engineering-Team, 10Performance-Team: Saving timings up 50% since 2018-06-28 20:53 - https://phabricator.wikimedia.org/T198483 (10jcrespo) I also saw this, not sure what it is, but in case it is related: https://grafana.wikimedia.org/dashboard/db/network-performances-global?panelId=19&fullscreen&orgI... [16:03:39] 10Release-Engineering-Team (Kanban), 10MediaWiki-Core-Tests, 10Multimedia, 10UploadWizard, and 3 others: UploadWizard: Selenium tests failing - https://phabricator.wikimedia.org/T198384 (10matmarex) [16:04:17] RECOVERY - Free space - all mounts on integration-slave-jessie-1002 is OK: OK: integration.integration-slave-jessie-1002.diskspace._mnt.byte_percentfree (No valid datapoints found) [16:09:38] PROBLEM - Long lived cherry-picks on puppetmaster on deployment-puppetmaster03 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [16:28:52] 10Release-Engineering-Team (Kanban), 10MediaWiki-extensions-MultimediaViewer, 10Multimedia, 10Patch-For-Review, 10User-zeljkofilipin: Port MultimediaViewer Selenium tests from Ruby to Node.js - https://phabricator.wikimedia.org/T180226 (10zeljkofilipin) a:05zeljkofilipin>03None [16:29:34] 10Release-Engineering-Team (Kanban), 10QuickSurveys, 10Readers-Web-Backlog (Tracking), 10User-zeljkofilipin: Rewrite QuickSurveys browser tests in Node.js or remove them - https://phabricator.wikimedia.org/T177261 (10zeljkofilipin) p:05Normal>03Low [16:41:17] 10Release-Engineering-Team (Kanban), 10QuickSurveys, 10Patch-For-Review, 10Readers-Web-Backlog (Tracking), 10User-zeljkofilipin: Rewrite QuickSurveys browser tests in Node.js or remove them - https://phabricator.wikimedia.org/T177261 (10zeljkofilipin) [16:41:41] 10Release-Engineering-Team (Kanban), 10QuickSurveys, 10Patch-For-Review, 10Readers-Web-Backlog (Tracking), 10User-zeljkofilipin: Rewrite QuickSurveys browser tests in Node.js or remove them - https://phabricator.wikimedia.org/T177261 (10zeljkofilipin) a:05zeljkofilipin>03None [16:42:49] 10Release-Engineering-Team (Kanban), 10MediaWiki-Core-Tests, 10User-zeljkofilipin: Sample code in Node.js for repositories that still have Selenium+Ruby tests - https://phabricator.wikimedia.org/T183160 (10zeljkofilipin) [16:43:34] 10Release-Engineering-Team (Kanban), 10MediaWiki-extensions-GettingStarted, 10Patch-For-Review, 10User-zeljkofilipin: Port GettingStarted Selenium tests from Ruby to Node.js - https://phabricator.wikimedia.org/T180224 (10zeljkofilipin) [16:43:37] 10Release-Engineering-Team (Kanban), 10MediaWiki-extensions-MultimediaViewer, 10Multimedia, 10Patch-For-Review, 10User-zeljkofilipin: Port MultimediaViewer Selenium tests from Ruby to Node.js - https://phabricator.wikimedia.org/T180226 (10zeljkofilipin) [16:43:40] 10Release-Engineering-Team (Kanban), 10MediaWiki-Core-Tests, 10User-zeljkofilipin: Sample code in Node.js for repositories that still have Selenium+Ruby tests - https://phabricator.wikimedia.org/T183160 (10zeljkofilipin) [16:43:42] 10Release-Engineering-Team (Kanban), 10QuickSurveys, 10Patch-For-Review, 10Readers-Web-Backlog (Tracking), 10User-zeljkofilipin: Rewrite QuickSurveys browser tests in Node.js or remove them - https://phabricator.wikimedia.org/T177261 (10zeljkofilipin) [16:43:46] 10Release-Engineering-Team (Kanban), 10Fundraising-Backlog, 10MediaWiki-extensions-CentralNotice, 10Patch-For-Review, 10User-zeljkofilipin: Port CentralNotice Selenium tests from Ruby to Node.js - https://phabricator.wikimedia.org/T180223 (10zeljkofilipin) [16:46:51] 10Release-Engineering-Team (Kanban), 10Revision-Slider, 10TCB-Team, 10WMDE volunteer workboard, and 2 others: Port RevisionSlider Selenium tests from Ruby to Node.js - https://phabricator.wikimedia.org/T180247 (10zeljkofilipin) a:03zeljkofilipin [16:47:04] 10Release-Engineering-Team (Kanban), 10Revision-Slider, 10TCB-Team, 10WMDE volunteer workboard, and 3 others: Port RevisionSlider Selenium tests from Ruby to Node.js - https://phabricator.wikimedia.org/T180247 (10zeljkofilipin) [16:50:17] 10Release-Engineering-Team (Kanban), 10Revision-Slider, 10TCB-Team, 10WMDE volunteer workboard, and 3 others: Port RevisionSlider Selenium tests from Ruby to Node.js - https://phabricator.wikimedia.org/T180247 (10zeljkofilipin) p:05Normal>03Low [16:51:44] 10Release-Engineering-Team (Kanban), 10MediaWiki-Core-Tests, 10User-zeljkofilipin: Sample code in Node.js for repositories that still have Selenium+Ruby tests - https://phabricator.wikimedia.org/T183160 (10zeljkofilipin) [16:52:02] 10Release-Engineering-Team (Kanban), 10Revision-Slider, 10TCB-Team, 10WMDE volunteer workboard, and 4 others: Port RevisionSlider Selenium tests from Ruby to Node.js - https://phabricator.wikimedia.org/T180247 (10zeljkofilipin) [16:52:54] 10Release-Engineering-Team (Kanban), 10Revision-Slider, 10TCB-Team, 10WMDE volunteer workboard, and 4 others: Port RevisionSlider Selenium tests from Ruby to Node.js - https://phabricator.wikimedia.org/T180247 (10zeljkofilipin) [16:53:13] 10Release-Engineering-Team (Kanban), 10Revision-Slider, 10TCB-Team, 10WMDE volunteer workboard, and 4 others: Port RevisionSlider Selenium tests from Ruby to Node.js - https://phabricator.wikimedia.org/T180247 (10zeljkofilipin) a:05zeljkofilipin>03None [16:53:42] what credentials should be used for https://logstash-beta.wmflabs.org/ [16:53:48] 10Release-Engineering-Team (Kanban), 10MediaWiki-Core-Tests, 10User-zeljkofilipin: Sample code in Node.js for repositories that still have Selenium+Ruby tests - https://phabricator.wikimedia.org/T183160 (10zeljkofilipin) [16:53:51] 10Release-Engineering-Team (Kanban), 10Revision-Slider, 10TCB-Team, 10WMDE volunteer workboard, and 4 others: Port RevisionSlider Selenium tests from Ruby to Node.js - https://phabricator.wikimedia.org/T180247 (10zeljkofilipin) [16:54:26] 10Release-Engineering-Team (Kanban), 10TCB-Team, 10Two-Column-Edit-Conflict-Merge, 10WMDE-mediawiki-maintenance, 10WMDE-QWERTY-XMAS-Sprint-2017-12-20: Port TwoColConflict Selenium tests from Ruby to Node.js - https://phabricator.wikimedia.org/T180248 (10zeljkofilipin) a:03zeljkofilipin [16:54:32] 10Release-Engineering-Team (Kanban), 10TCB-Team, 10Two-Column-Edit-Conflict-Merge, 10WMDE-mediawiki-maintenance, 10WMDE-QWERTY-XMAS-Sprint-2017-12-20: Port TwoColConflict Selenium tests from Ruby to Node.js - https://phabricator.wikimedia.org/T180248 (10zeljkofilipin) p:05Normal>03Low [16:54:45] 10Release-Engineering-Team (Kanban), 10TCB-Team, 10Two-Column-Edit-Conflict-Merge, 10WMDE-mediawiki-maintenance, and 2 others: Port TwoColConflict Selenium tests from Ruby to Node.js - https://phabricator.wikimedia.org/T180248 (10zeljkofilipin) [16:58:06] ok, the earlier-predicted nodepool outage is coming up. [17:05:31] that's it, nodepool is back [17:08:40] thanks andrewbogott [17:17:59] 10Release-Engineering-Team (Kanban), 10MediaWiki-Core-Tests, 10User-zeljkofilipin: Sample code in Node.js for repositories that still have Selenium+Ruby tests - https://phabricator.wikimedia.org/T183160 (10zeljkofilipin) [17:19:09] 10Release-Engineering-Team (Kanban), 10TCB-Team, 10Two-Column-Edit-Conflict-Merge, 10WMDE-mediawiki-maintenance, and 3 others: Port TwoColConflict Selenium tests from Ruby to Node.js - https://phabricator.wikimedia.org/T180248 (10zeljkofilipin) [17:19:43] 10Release-Engineering-Team (Kanban), 10TCB-Team, 10Two-Column-Edit-Conflict-Merge, 10WMDE-mediawiki-maintenance, and 3 others: Port TwoColConflict Selenium tests from Ruby to Node.js - https://phabricator.wikimedia.org/T180248 (10zeljkofilipin) a:05zeljkofilipin>03None [17:20:35] 10Release-Engineering-Team (Kanban), 10TCB-Team, 10Two-Column-Edit-Conflict-Merge, 10WMDE-mediawiki-maintenance, and 3 others: Port TwoColConflict Selenium tests from Ruby to Node.js - https://phabricator.wikimedia.org/T180248 (10zeljkofilipin) [17:20:36] 10Release-Engineering-Team (Kanban), 10MediaWiki-Core-Tests, 10User-zeljkofilipin: Sample code in Node.js for repositories that still have Selenium+Ruby tests - https://phabricator.wikimedia.org/T183160 (10zeljkofilipin) [17:21:24] 10Release-Engineering-Team (Kanban), 10MediaWiki-Core-Tests, 10User-zeljkofilipin: Q1 Selenium framework improvements - https://phabricator.wikimedia.org/T198389 (10zeljkofilipin) [17:21:27] 10Release-Engineering-Team, 10MediaWiki-Core-Tests, 10User-zeljkofilipin: Q4 Selenium framework improvements - https://phabricator.wikimedia.org/T190994 (10zeljkofilipin) [17:21:29] 10Release-Engineering-Team (Kanban), 10MediaWiki-Core-Tests, 10User-zeljkofilipin: Sample code in Node.js for repositories that still have Selenium+Ruby tests - https://phabricator.wikimedia.org/T183160 (10zeljkofilipin) 05Open>03Resolved [17:25:37] thcipriani: does https://gerrit.wikimedia.org/ load for you? [17:26:24] bearND: yes, just slowly [17:26:56] thcipriani: It [17:27:07] it's been hanging for me for over 15 minutes [17:27:11] could it be comcast? [17:27:20] related to https://phabricator.wikimedia.org/T198502 ? [17:27:23] Yeah, I'm using Comcast :( [17:27:32] comcast is currently having a outage [17:27:48] paladox: that's it. Thank you! [17:28:05] Is there a workaround? [17:28:20] i doin't think so currently [17:28:24] unless you have a vpn [17:28:34] using a different provider or in another country [17:30:35] bummer. Phab works fine, btw. Just no Gerrit. [17:32:01] phab uses the cache proxy [17:32:07] gerrit is directly to eqiad [17:38:02] 10Project-Admins, 10Developer-Relations (Jul-Sep 2018): Sort out scope/confusion between #Possible-Tech-Projects and #Outreach-Programs-Projects tags - https://phabricator.wikimedia.org/T198101 (10Aklapper) [17:38:02] translatewiki.net is also affected I think [17:38:10] 10Project-Admins, 10Developer-Relations (Jul-Sep 2018): Sort out scope/confusion between #Possible-Tech-Projects and #Outreach-Programs-Projects tags - https://phabricator.wikimedia.org/T198101 (10Aklapper) a:03Aklapper [18:05:01] 10Phabricator, 10Security-General: Streamline process for uploading private files to public tasks - https://phabricator.wikimedia.org/T181317 (10Tgr) A custom form for uploading with the access configuration preset (yourself + admins or something like that) would be nice. [18:32:03] 10Beta-Cluster-Infrastructure, 10Parsoid, 10VisualEditor: VE is not loading on Beta Cluster, getting 503s - https://phabricator.wikimedia.org/T198421 (10Arlolra) >>! In T198421#4323585, @Krenair wrote: > Tried restarting parsoid service on deployment-parsoid09 and then getting the URI above. > Based on `tail... [18:45:10] PROBLEM - Free space - all mounts on integration-slave-jessie-1004 is CRITICAL: CRITICAL: integration.integration-slave-jessie-1004.diskspace._srv.byte_percentfree (<44.44%) [18:55:05] 10Release-Engineering-Team, 10Performance-Team: Saving timings up 50% since 2018-06-28 20:53 - https://phabricator.wikimedia.org/T198483 (10greg) Those do correlate with: * [[ http://tools.wmflabs.org/sal/log/AWRH0vlioDEJc1hAakLm | 19:16 ]] "rebuilt and synchronized wikiversions files: Group1 (less commons) to... [19:05:11] PROBLEM - Free space - all mounts on integration-slave-jessie-1004 is CRITICAL: CRITICAL: integration.integration-slave-jessie-1004.diskspace._srv.byte_percentfree (<44.44%) [20:10:20] 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10Release, 10Train Deployments: 1.32.0-wmf.10 deployment blockers - https://phabricator.wikimedia.org/T191056 (10matmarex) [20:18:04] we might want to backport the patch for https://phabricator.wikimedia.org/T198449 today. (looks like another issue from the MCR stuff.) cc greg-g [20:22:20] ugh, that's no good. [20:24:34] 10Release-Engineering-Team (Kanban), 10MediaWiki-Core-Tests, 10MediaWiki-User-preferences, 10User-zeljkofilipin, 10Wikimedia-log-errors (Shared Build Failure): Selenium "User should be able to change preferences" test flaky - https://phabricator.wikimedia.org/T198137 (10matmarex) [20:24:37] 10Release-Engineering-Team (Kanban), 10MediaWiki-Core-Tests, 10Multimedia, 10UploadWizard, and 2 others: UploadWizard: Selenium tests failing - https://phabricator.wikimedia.org/T198384 (10matmarex) 05Open>03Resolved I'm able to merge changes in UploadWizard now. [20:32:56] MatmaRex: seems like a small fix that is fairly strightforward (albeit with a TODO attached which doesn't help my confidence) but it's Friday afternoon so if we can avoid a sync right before the weekend I'd feel better about that. Is this something that is going to cause major impact? [20:34:38] sorry, back around, looking [20:36:40] stuff not getting auto-patrolled when it should be means more work for RC patrollers [20:37:08] question is how much more work [20:37:28] yeah, I'm not 100% on how much chance there is for duplicate work due to this [20:37:36] 100% clear* [20:37:50] not sure I'd describe it as 'duplicate' [20:38:25] basically autopatrollers are people who are classed as safe for anti-vandalism purposes [20:38:37] I was just assuming someone reviewing the edit before the first person has also marked it as patrolled... but I guess this is only for rollbacks, so right [20:38:38] their edits are less likely to be reviewed by anti-vandalism people [20:39:21] actually, I had that wrong, but now I get it and yes, thanks :) [20:39:30] now if it's only reverts being wrongly marked for patrolling... [20:39:41] my guess is patrollers can easily spot that and mentally filter out [20:39:46] right [20:39:50] thcipriani: i am not sure how much impact it has, tbh. but it has like 5 duplicate reports already [20:40:23] thcipriani: the code is actually copy-pasted from another place in the same file, along with the TODO, if that helps your confidence ;) [20:40:36] duplicating a TODO, even better! [20:40:36] if it's huge volume then it might be a problem [20:40:38] heh, oh good [20:42:19] so, I see the fix, but what was the cause? [20:42:45] oh, "MCR" [20:44:19] I'm sure we'll get more reports over the weekend [20:45:14] the patch to me doesn't look hugely risky if it's been tested properly [20:45:59] big if :) [20:46:08] MatmaRex: how tested is this patch? [20:47:10] greg-g: i would imagine as well as any other. ask daniel :) [20:47:58] I presume he is not at his computer at 10:50pm on a Friday [20:48:18] just want to get some confirmation that folks have run this patch locally and verified it before we put it into production on Friday [20:50:50] MatmaRex: ^ [20:52:04] i can rollback some changes locally, if you wish, and see what happens. i wasn't really involved in the patch, i just noticed a task describing a regression with half a dozen duplicates [20:52:35] ack [20:53:00] (honestly, doing deployments on thursday is too late if we want issues to be reported and fixed before friday :/) [20:56:46] when we're lucky, most stuff is caught on group1 rollout wednesday, but that's not always the case :( [20:59:53] 10Release-Engineering-Team: Selenium test failure blocking merge of core patch - https://phabricator.wikimedia.org/T198514 (10SBisson) [21:01:27] 10Release-Engineering-Team: Selenium test failure blocking merge of core patch - https://phabricator.wikimedia.org/T198514 (10matmarex) [21:01:27] 10Release-Engineering-Team (Kanban), 10MediaWiki-Core-Tests, 10MediaWiki-User-preferences, 10User-zeljkofilipin, 10Wikimedia-log-errors (Shared Build Failure): Selenium "User should be able to change preferences" test flaky - https://phabricator.wikimedia.org/T198137 (10matmarex) [21:01:50] MatmaRex: gah, you beat me to do that, why aren't you testing that patch! ;) [21:02:04] * greg-g got the "page updated" popup while merging them [21:02:34] greg-g: in the middle of something else, but also i just fixed another Selenium bug today [21:02:40] hah, thanks :) [21:04:17] MatmaRex: if you can test the change, I can get it out the door [21:08:22] i am not sure why you have to wait for me to do that, but sure, i'll do it [21:08:30] 10Release-Engineering-Team (Kanban), 10MediaWiki-Core-Tests, 10MediaWiki-User-preferences, 10User-zeljkofilipin, 10Wikimedia-log-errors (Shared Build Failure): Selenium "User should be able to change preferences" test flaky - https://phabricator.wikimedia.org/T198137 (10Catrope) [21:09:06] 10Release-Engineering-Team (Kanban), 10MediaWiki-Core-Tests, 10MediaWiki-User-preferences, 10User-zeljkofilipin, 10Wikimedia-log-errors (Shared Build Failure): Selenium "User should be able to change preferences" test flaky - https://phabricator.wikimedia.org/T198137 (10Catrope) Added #4, which involves... [21:09:29] MatmaRex: I mean on mwdebug1002 [21:10:46] if you've got time I'd appreciate someone making sure I'm not missing something during the deploy [21:11:29] thcipriani: hm, sure [21:11:33] thanks :) [21:13:26] patch is merging now, I'll keep an eye on zuul and ping you when it lands [21:23:38] hey anyone knows about the selenium failures that prevent us from merging commits? [21:23:47] it's happening both in core and PageTriage right now :\ [21:24:15] The failures look unrelated completely to the submitted code, both similar failures, and both are not fixed with constant attempts at re-running with 'recheck'. We're stuck [21:25:10] example 1: https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/PageTriage/+/443127/ [21:25:11] example 2 (that ended up passing with +2 checkes, randmly) https://gerrit.wikimedia.org/r/#/c/mediawiki/core/+/443110/ [21:28:03] thcipriani: tagging you for help since you seem to be around -- not sure who to poke on this one [21:28:28] mooeypoo: I was just looking to see if there was a task, I have not heard anything about this today [21:28:44] thcipriani: so there's a related thing that we THOUGHT was the cause, hang on I'll find it [21:29:06] I've seen similar failures in the past and I don't remember what the resolution was :( [21:29:20] thcipriani: we've had a bunch of problems in the past couple of days [21:29:31] https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/PageTriage/+/440018/ <-- another example (ended up merging eventually) [21:29:42] hashar commented on that one [21:29:46] https://phabricator.wikimedia.org/T198137 [21:29:48] there was a merge of a change to the "Iñtërnâtiônàlizætiøn" random name generation today [21:29:56] https://phabricator.wikimedia.org/T198201#4320467 <-- there's this one too [21:30:07] that was https://gerrit.wikimedia.org/r/#/c/mediawiki/core/+/443095 [21:30:37] This failure started happening yesterday and the day before, too... we thought it was something with the job runner on CI if I understand hashar's comment right [21:30:57] but today it just prevents the merge on PAgeTriage patch that we need to have in so we can SWAT on monday [21:32:16] So T198137 ? [21:32:18] T198137: Selenium "User should be able to change preferences" test flaky - https://phabricator.wikimedia.org/T198137 [21:32:42] :\ [21:33:41] here's the best summary https://phabricator.wikimedia.org/T198137#4321464 [21:33:51] of what the status/next steps are, that is [21:33:59] Well, the PageTriage one fails on "Special:RecentChanges shows page creation" [21:34:21] yeah, seems unrelated [21:35:12] Do we wait, do we v+2, do we keep trying to +2 or 'recheck' ? [21:35:22] seems the task is not "UBN" but "high" but we are not getting it rarely [21:35:24] mooeypoo: you can override Jenkins by removing its V-2 vote and voting V+2 yourself (unless someone idealistically disables this functionality for a given repo, but it's not disabled for PageTriage) [21:35:51] yeah I wanted to avoid that, but if that I need to do, I'll do it [21:37:09] are there two separate issues? It seems like what zeljko.f is describing is a flaky test and hashar is saying something is preventing recentChangesUpdate from happening? [21:37:10] I would feel better if y'all think that's the best way to go :P I wanted to see if this is being resolved and we should wait first, but judging from the task attached, it might take a bit to resolve this one [21:37:20] well, that test is only as old as June 12th... [21:37:26] * greg-g is still looking [21:37:55] thcipriani: i think the flaky test is the one we're experiencing, but it's not just one test (?) I have another example: https://integration.wikimedia.org/ci/job/quibble-vendor-mysql-hhvm-docker/6434/consoleFull That's a different test than the one we looked at [21:37:57] https://phabricator.wikimedia.org/source/mediawiki/history/master/tests/selenium/specs/specialrecentchanges.js [21:38:02] but similar reason, seems like? internationalization? [21:38:30] There's the 'user can't change pref' and 'special recent changes new page' tests, as far as I can see [21:38:56] yeah, one had a recent refactor, and one is new (june 12th) [21:39:03] so much green text for a failure [21:39:16] hehe it makes you feel good, and then it gets ya [21:43:11] if this test is flaky I wonder if it could be related to resource contention on the ci boxen somehow? [21:47:34] 10Release-Engineering-Team, 10Performance-Team: Saving timings up 50% since 2018-06-28 20:53 - https://phabricator.wikimedia.org/T198483 (10daniel) A slight rise in save timing was to be expected dues to the fact that we are now writing to five tables instead of three, but 50% is a LOT. I don't see how tgr's... [21:52:49] hrm, but there's only 1 executor on the machine where it last failed, so that's not it [21:55:29] 10Release-Engineering-Team (Kanban), 10ORES, 10Quibble, 10Browser-Tests, and 4 others: Quibble must include tests/selenium/LocalSettings.php (was ORES webdriver.io selenium test fail on CI due to lack of ORES server) - https://phabricator.wikimedia.org/T198201 (10Niharika) p:05Triage>03High Triaging th... [21:58:23] mooeypoo: unfortunately the folks most familiar with the flaky test aren't around at the moment and I'm not sure I have a lot of confidence that I'm going to be able to solve the cause over the short term. I'll continue to look, but if you're blocked on this and the issue is unrelated (which it seems like it's ongoing and completely unrealated) feel free to force merge. [22:00:39] * mooeypoo nods [22:01:09] thcipriani: alright, I'll have to probably force it. Thanks! [22:01:39] sure, sorry I don't have any good answers :( [22:05:29] Project beta-scap-eqiad build #213886: 15ABORTED in 1 hr 0 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/213886/ [22:05:59] thcipriani: thanks for the help though, I was not happy v+2'ing casually, wanted to make sure that if I have to, I at least explored all options beforehand, and got a second opinion to verify I'm not missing something. It is unrelated, though, so I may have to force merge after all. [22:06:24] * thcipriani nods [22:07:35] Finished l10n-update (duration: 56m 15s) from that beta timeout [22:08:33] happening a couple-ish times/day, but is faster the next time (so far) [22:12:34] PROBLEM - Free space - all mounts on deployment-tin is CRITICAL: CRITICAL: deployment-prep.deployment-tin.diskspace._mnt.byte_percentfree (No valid datapoints found)deployment-prep.deployment-tin.diskspace._srv.byte_percentfree (<11.11%) [22:22:35] PROBLEM - Free space - all mounts on deployment-tin is CRITICAL: CRITICAL: deployment-prep.deployment-tin.diskspace._mnt.byte_percentfree (No valid datapoints found)deployment-prep.deployment-tin.diskspace._srv.byte_percentfree (<11.11%) [22:36:37] 10Release-Engineering-Team (Kanban), 10Quibble: Quibble docker instance running on CI instance for 6 hours - https://phabricator.wikimedia.org/T198517 (10thcipriani) [22:43:33] 10Beta-Cluster-Infrastructure, 10VisualEditor-MediaWiki-2017WikitextEditor, 10VisualEditor (Current work): Switching to NWE is failing on the Swedish beta cluster wiki - https://phabricator.wikimedia.org/T191184 (10Krenair) Actually that file doesn't match up with `deployment-tin:/srv/deployment/parsoid/depl... [22:51:26] where do I find the spec for the tox-docker container? [22:52:34] * bd808 finds some clues in past phab tasks [22:53:28] the CI one you're looking for? [22:53:45] yeah. looks like integration/config/dockerfiles/tox/Dockerfile.template is the place [22:53:58] https://github.com/wikimedia/integration-config/tree/master/dockerfiles/tox [22:54:01] yep [22:54:23] I need to fix "build/temp.linux-x86_64-3.4/_openssl.c:429:30: fatal error: openssl/opensslv.h: No such file or directory" [23:01:09] needs libssl-dev, I guess. I think the basic steps are: update dockerfile.template, bump changelog file, +2, use fab to rebuild the graph of images, update the job with the tag for the latest image, +2, deploy to jenkins [23:06:40] (03PS1) 10BryanDavis: tox-docker: Add libssl-dev [integration/config] - 10https://gerrit.wikimedia.org/r/443205 [23:07:05] thcipriani: ^ a hack attempt at the part I could figure out :) [23:08:01] that maybe needs to be split though? I'll add hashar as a reviewer [23:09:28] probably doesn't need to be able to. Merger can figure out the order to make it all live. [23:09:32] (03CR) 10BryanDavis: "Patterned after https://gerrit.wikimedia.org/r/#/c/integration/config/+/387728/, but I'm unsure if in the modern era it works to do this a" [integration/config] - 10https://gerrit.wikimedia.org/r/443205 (owner: 10BryanDavis) [23:10:22] * bd808 wonders what the next missing dev package will be for his project [23:16:33] (03PS2) 10BryanDavis: tox-docker: Add libssl-dev [integration/config] - 10https://gerrit.wikimedia.org/r/443205 (https://phabricator.wikimedia.org/T198076)