[00:02:11] 10Deployment-Systems, 06Operations, 10Stashbot: [[wikitech:Server_admin_log]] should not rely on freenode irc for logmsgbot entries - https://phabricator.wikimedia.org/T46791#3023722 (10demon) >>! In T46791#3023711, @bd808 wrote: > Some related thoughts/explanations on {T156079}. > > wm-bot already does too... [00:10:28] 10Continuous-Integration-Config, 10Parsoid, 07HHVM, 05MW-1.29-release-notes, and 2 others: parsoidsvc-hhvm-parsertests-jessie doesn't run tidy tests - https://phabricator.wikimedia.org/T157730#3023757 (10Arlolra) 05Open>03Resolved a:03Arlolra 00:03:16 This is MediaWiki version 1.29.0-alpha (41ad223).... [00:11:07] 10Continuous-Integration-Infrastructure, 10Monitoring: Alert when Zuul/Gearman queue is stalled - https://phabricator.wikimedia.org/T70113#3023761 (10greg) >>! In T70113#2772415, @hashar wrote: >The gearman queue and gearman workers on https://grafana.wikimedia.org/dashboard/db/releng-zuul are good candidates... [00:17:31] Project selenium-Flow » chrome,beta,Linux,contintLabsSlave && UbuntuTrusty build #300: 04FAILURE in 1 min 30 sec: https://integration.wikimedia.org/ci/job/selenium-Flow/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/300/ [00:38:53] 10Continuous-Integration-Config, 06Release-Engineering-Team, 06Operations, 06Wikipedia-Android-App-Backlog, and 2 others: Investigate how to improve Android CI performance and stability - https://phabricator.wikimedia.org/T158014#3023806 (10Niedzielski) I made an image locally with `android create avd -f -... [00:39:58] thcipriani: I did a thing https://phabricator.wikimedia.org/D567. Came up as a result of https://phabricator.wikimedia.org/T46791 [00:40:09] * thcipriani looks at thing [00:40:47] neat :) [00:41:16] 10Deployment-Systems, 06Release-Engineering-Team, 10Scap, 10scap2, and 2 others: Error after "Finished deploy": xrange() arg 3 must not be zero - https://phabricator.wikimedia.org/T157136#3023807 (10dduvall) 05Open>03Resolved [00:45:35] tested: works! Accepted. [00:47:32] Landed [00:47:34] Test Plan: Tyler tested it, not me [00:47:39] Hehe [00:47:42] "So blame him if it breaks!" [00:49:30] 10Deployment-Systems, 06Operations, 10Stashbot: [[wikitech:Server_admin_log]] should not rely on freenode irc for logmsgbot entries - https://phabricator.wikimedia.org/T46791#3023858 (10demon) >>! In T46791#3023722, @demon wrote: > ** My `scap [log|sal]` suggestion as a fallback scenario also easily done as... [00:50:43] oh boy :) [00:55:06] Granted scap doesn't live everywhere but where it does hehe easy logging now [00:57:18] PROBLEM - Puppet run on deployment-eventlogging03 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [01:32:18] RECOVERY - Puppet run on deployment-eventlogging03 is OK: OK: Less than 1.00% above the threshold [0.0] [04:00:12] hi, cannot connect to bastion.wmflabs.org [04:00:43] i think someone was overactive cleaning up my security perms [04:01:08] ssh key is still added to the wikitech account [04:01:12] cc: thcipriani|afk ? [04:11:07] yurik, #-labs? [04:11:34] oh, oops, wrong channel :) [06:48:10] 10Continuous-Integration-Config, 10ContentTranslation, 03Language-2017 Sprint 2, 05Language-team January-March 2017, and 2 others: mwext-qunit-jessie test fails on unrelated change - https://phabricator.wikimedia.org/T153038#3024166 (10Nikerabbit) 05Resolved>03Open This is happening again. The symptoms... [06:53:05] PROBLEM - Puppet run on deployment-phab02 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [07:00:55] 10Continuous-Integration-Config, 10ContentTranslation, 03Language-2017 Sprint 2, 05Language-team January-March 2017, and 3 others: mwext-qunit-jessie test fails on unrelated change - https://phabricator.wikimedia.org/T153038#3024178 (10Arrbee) a:05Arrbee>03None [07:25:59] PROBLEM - Puppet run on deployment-mira is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [07:26:25] 10Continuous-Integration-Config, 10Cite, 10Parsoid, 13Patch-For-Review: Parsoid should run tests against the Cite parser tests, not just MediaWiki core's - https://phabricator.wikimedia.org/T114256#3024181 (10Arlolra) 05Open>03Resolved [07:28:09] RECOVERY - Puppet run on deployment-phab02 is OK: OK: Less than 1.00% above the threshold [0.0] [08:01:00] RECOVERY - Puppet run on deployment-mira is OK: OK: Less than 1.00% above the threshold [0.0] [08:34:29] PROBLEM - Long lived cherry-picks on puppetmaster on deployment-puppetmaster02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [09:36:50] 10Continuous-Integration-Infrastructure, 10Monitoring: Alert when Zuul/Gearman queue is stalled - https://phabricator.wikimedia.org/T70113#3024336 (10hashar) @chasemp has added monitoring of Nodepool instances state age https://gerrit.wikimedia.org/r/#/c/335373/ Eg alarm when an instance has been in a delete... [09:38:59] 10Continuous-Integration-Infrastructure, 10Monitoring: Alert when Zuul/Gearman queue is stalled - https://phabricator.wikimedia.org/T70113#3024338 (10hashar) --- >>! In T70113#2772415, @hashar wrote: > An alarm against `sumSeries(zuul.pipeline.*.resident_time.median)` might work, though the metric is repor... [09:42:28] Yippee, build fixed! [09:42:29] Project selenium-Core » firefox,beta,Linux,contintLabsSlave && UbuntuTrusty build #315: 09FIXED in 6 min 42 sec: https://integration.wikimedia.org/ci/job/selenium-Core/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/315/ [10:19:22] 10Gerrit, 06Operations, 13Patch-For-Review: Investigate seemingly random Gerrit slow-downs - https://phabricator.wikimedia.org/T148478#3024451 (10ArielGlenn) The 28G heap was for something that @demon was planning IIRC. Perhaps he can weigh in. [10:23:14] 10Gerrit, 06Operations, 13Patch-For-Review: Investigate seemingly random Gerrit slow-downs - https://phabricator.wikimedia.org/T148478#3024468 (10hashar) Happening again right 39.1 G of VIRT and 27.4G of RES. `ps -eLf` shows that Gerrit has ~ 200 threads. [10:47:43] 10Gerrit, 10Differential: Unpublished Gerrit commits show up in Differential - https://phabricator.wikimedia.org/T158043#3024557 (10TTO) [10:52:16] 10Continuous-Integration-Config, 10ContentTranslation, 03Language-2017 Sprint 2, 05Language-team January-March 2017, and 3 others: mwext-qunit-jessie test fails on unrelated change - https://phabricator.wikimedia.org/T153038#3024592 (10hashar) From a random build for ContentTranslation https://integration.... [11:03:18] 10Gerrit, 06Operations, 13Patch-For-Review: Investigate seemingly random Gerrit slow-downs - https://phabricator.wikimedia.org/T148478#3024632 (10Gehel) Some thread dumps were collected: {F5628818} Note: [[ http://fastthread.io/ | fastThread.io ]] can help with the analysis. Preliminary findings: * most t... [11:15:05] 10Gerrit, 06Operations, 13Patch-For-Review: Investigate seemingly random Gerrit slow-downs - https://phabricator.wikimedia.org/T148478#3024661 (10hashar) Another potential issue is the memory usage [[ https://grafana.wikimedia.org/dashboard/file/server-board.json?var-server=cobalt&var-network=eth0&panelId=14... [11:33:58] 10Gerrit, 06Operations, 13Patch-For-Review: Investigate seemingly random Gerrit slow-downs - https://phabricator.wikimedia.org/T148478#3024690 (10hashar) `gerrit show-caches` takes roughly 45 seconds to complete: ``` Gerrit Code Review 2.13.4-13-gc0c5cc4742 now 11:26:37 UTC... [12:13:53] 10Continuous-Integration-Infrastructure: Cannot access the database: Access denied for user 'jenkins_u0'@'127.0.0.1' to database 'closedwikis' - https://phabricator.wikimedia.org/T157815#3017501 (10hashar) On CI we provision a single database that is named with the prefix `jenkins_u` + the executor slot on the... [12:18:03] 06Release-Engineering-Team, 10DBA, 06Operations, 10Phabricator: Phabricator master and slave crashed - https://phabricator.wikimedia.org/T156905#3024779 (10Marostegui) 05Open>03Resolved a:03Marostegui I am going to close this for now as the short-term solution was to move search to ES and this hasn't... [12:20:34] 06Release-Engineering-Team, 10DBA, 06Operations, 10Phabricator: Phabricator master and slave crashed - https://phabricator.wikimedia.org/T156905#3024783 (10Paladox) Elasticsearch is actually a long term fix. I think we have permanently moved to it. We were preparing to move to it before. :) [12:20:41] 10Continuous-Integration-Infrastructure: Cannot access the database: Access denied for user 'jenkins_u0'@'127.0.0.1' to database 'closedwikis' - https://phabricator.wikimedia.org/T157815#3024784 (10hashar) The call chain is roughly: ``` MediaWikiTestCase->addCoreDBData() WikiPage->doEditContent(..) CloseWikis::u... [12:22:10] 06Release-Engineering-Team, 10DBA, 06Operations, 10Phabricator: Phabricator master and slave crashed - https://phabricator.wikimedia.org/T156905#3024788 (10Marostegui) >>! In T156905#3024783, @Paladox wrote: > Elasticsearch is actually a long term fix. I think we have permanently moved to it. We were prepa... [12:23:07] 06Release-Engineering-Team, 10DBA, 06Operations, 10Phabricator: Phabricator master and slave crashed - https://phabricator.wikimedia.org/T156905#3024790 (10Paladox) Yep, @mmodell has improved searching by a lot :) [12:25:14] 10Gerrit, 06Operations, 13Patch-For-Review: Investigate seemingly random Gerrit slow-downs - https://phabricator.wikimedia.org/T148478#3024792 (10Paladox) @hashar and @Gehel thanks for all your traces :). Looks like the i/o issue we had on friday. Though we also found gc was the problem on friday. Does gc ca... [12:30:05] 10Gerrit, 10Differential: Unpublished Gerrit commits show up in Differential - https://phabricator.wikimedia.org/T158043#3024557 (10Paladox) Upstream gerrit are removing support for drafts any ways. But also in differential we import refs/drafts and refs/changes. In gerrit refs/drafts when cloning it, doesn't... [13:01:20] Project beta-code-update-eqiad build #143182: 04FAILURE in 8 min 18 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/143182/ [13:03:26] ^ that'll be due to gerrit issues [13:06:59] 10Browser-Tests-Infrastructure, 06Reading-Web-Backlog, 07Jenkins, 07Ruby, 15User-zeljkofilipin: MEDIAWIKI_URL may be set to incorrect value in mwext-mw-selenium job - https://phabricator.wikimedia.org/T144912#3024899 (10zeljkofilipin) The state is that I did not have the time to work on this yet :( This... [13:11:13] Yippee, build fixed! [13:11:13] Project beta-code-update-eqiad build #143183: 09FIXED in 7 min 52 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/143183/ [13:11:45] 06Release-Engineering-Team, 06Operations, 07HHVM, 07Wikimedia-Incident: 2016-10-17 API cluster overload - https://phabricator.wikimedia.org/T148652#3024909 (10fgiunchedi) 05Open>03Resolved a:03fgiunchedi @greg I think it is safe to close, we've mitigated the issue by having a separate cluster for asy... [13:23:50] 10Browser-Tests-Infrastructure, 10RelatedArticles, 07JavaScript, 15User-zeljkofilipin: Set up Selenium tests in Node.js for RelatedArticles extension - https://phabricator.wikimedia.org/T158052#3024960 (10zeljkofilipin) [13:39:27] 10Continuous-Integration-Infrastructure, 10Monitoring, 13Patch-For-Review: Alert when Zuul/Gearman queue is stalled - https://phabricator.wikimedia.org/T70113#3025000 (10chasemp) > ([[ https://tools.wmflabs.org/sal/log/AVnyP3YQQMK9DA-FJ8Vd | SAL Entry ]]): This was an OpenStack / WMF Labs issue. @chasemp wr... [13:49:08] PROBLEM - Puppet run on buildlog is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [14:02:41] 10Deployment-Systems: Deploy MW+Extensions by percentage of users (instead of by domain/wiki) - https://phabricator.wikimedia.org/T104398#3025066 (10mmodell) [14:02:44] 10Deployment-Systems, 06Release-Engineering-Team (Long-Lived-Branches), 07WorkType-NewFunctionality: Use subrepos instead of git submodules for deployed MediaWiki extensions - https://phabricator.wikimedia.org/T98834#3025067 (10mmodell) [14:02:47] 10Deployment-Systems, 06Release-Engineering-Team, 07Epic: [EPIC] The future of MediaWiki deployment: Tooling - https://phabricator.wikimedia.org/T94620#3025068 (10mmodell) [14:02:49] 10Scap, 13Patch-For-Review: Automatically clean up unused wmfXX versions - https://phabricator.wikimedia.org/T73313#3025069 (10mmodell) [14:02:52] 10Deployment-Systems, 06Release-Engineering-Team (Long-Lived-Branches), 10releng-201617-q1, 07Epic: Merge to deployed branches instead of cutting a new deployment branch every week. - https://phabricator.wikimedia.org/T89945#3025064 (10mmodell) 05Open>03declined Looks like this simply isn't going to ha... [14:05:47] 06Release-Engineering-Team, 10Elasticsearch, 10Phabricator (Search): Phabricator: Support elasticsearch 5.x - https://phabricator.wikimedia.org/T155299#3025089 (10mmodell) 05Open>03Resolved [14:08:50] 10Deployment-Systems: Deploy MW+Extensions by percentage of users (instead of by domain/wiki) - https://phabricator.wikimedia.org/T104398#3025097 (10mmodell) [14:15:12] 10Browser-Tests-Infrastructure, 10RelatedArticles, 07JavaScript, 15User-zeljkofilipin: Set up Selenium tests in Node.js for RelatedArticles extension - https://phabricator.wikimedia.org/T158052#3025124 (10zeljkofilipin) p:05Triage>03Normal [14:21:56] (03PS1) 10Hashar: Add castor to operations-mw-config-composer-hhvm-jessie [integration/config] - 10https://gerrit.wikimedia.org/r/337583 (https://phabricator.wikimedia.org/T156359) [14:22:27] (03PS2) 10Hashar: Add castor to operations-mw-config-composer-hhvm-jessie [integration/config] - 10https://gerrit.wikimedia.org/r/337583 (https://phabricator.wikimedia.org/T156359) [14:34:55] 10Continuous-Integration-Config, 13Patch-For-Review: operations-mw-config-composer-hhvm-jessie job lacks composer cache - https://phabricator.wikimedia.org/T156359#3025206 (10hashar) On a Jessie Nodepool instance: ``` $ composer config --global --list|grep cache [cache-dir] /home/jenkins/.composer/cache [cache... [14:37:02] (03PS1) 10Hashar: Castor: also capture .composer/cache [integration/config] - 10https://gerrit.wikimedia.org/r/337588 (https://phabricator.wikimedia.org/T156359) [14:38:36] !log Updating castor-save publish job to properly capture composer cache on Jessie ( it is in ~/.composer/cache for some reason) T156359 [14:38:40] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [14:38:40] T156359: operations-mw-config-composer-hhvm-jessie job lacks composer cache - https://phabricator.wikimedia.org/T156359 [14:46:00] (03PS1) 10Zfilipin: Run mediawiki-core-selenium-jessie Jenkins job in experimental pipeline for RelatedArticles extension [integration/config] - 10https://gerrit.wikimedia.org/r/337590 (https://phabricator.wikimedia.org/T158052) [14:48:09] (03CR) 10Zfilipin: [C: 032] Run mediawiki-core-selenium-jessie Jenkins job in experimental pipeline for RelatedArticles extension [integration/config] - 10https://gerrit.wikimedia.org/r/337590 (https://phabricator.wikimedia.org/T158052) (owner: 10Zfilipin) [14:49:03] (03Merged) 10jenkins-bot: Run mediawiki-core-selenium-jessie Jenkins job in experimental pipeline for RelatedArticles extension [integration/config] - 10https://gerrit.wikimedia.org/r/337590 (https://phabricator.wikimedia.org/T158052) (owner: 10Zfilipin) [14:49:24] (03CR) 10Hashar: [C: 032] Add castor to operations-mw-config-composer-hhvm-jessie [integration/config] - 10https://gerrit.wikimedia.org/r/337583 (https://phabricator.wikimedia.org/T156359) (owner: 10Hashar) [14:49:30] (03PS3) 10Hashar: Add castor to operations-mw-config-composer-hhvm-jessie [integration/config] - 10https://gerrit.wikimedia.org/r/337583 (https://phabricator.wikimedia.org/T156359) [14:49:38] (03CR) 10Hashar: Add castor to operations-mw-config-composer-hhvm-jessie [integration/config] - 10https://gerrit.wikimedia.org/r/337583 (https://phabricator.wikimedia.org/T156359) (owner: 10Hashar) [14:49:45] (03CR) 10Hashar: [C: 032] Add castor to operations-mw-config-composer-hhvm-jessie [integration/config] - 10https://gerrit.wikimedia.org/r/337583 (https://phabricator.wikimedia.org/T156359) (owner: 10Hashar) [14:49:59] (03PS2) 10Hashar: Castor: also capture .composer/cache [integration/config] - 10https://gerrit.wikimedia.org/r/337588 (https://phabricator.wikimedia.org/T156359) [14:50:05] (03CR) 10Hashar: [C: 032] Castor: also capture .composer/cache [integration/config] - 10https://gerrit.wikimedia.org/r/337588 (https://phabricator.wikimedia.org/T156359) (owner: 10Hashar) [14:51:46] I forgot to bring it up at the syncup yesterday, can I get a review of https://gerrit.wikimedia.org/r/#/c/335624/ ? [14:52:27] (03Merged) 10jenkins-bot: Add castor to operations-mw-config-composer-hhvm-jessie [integration/config] - 10https://gerrit.wikimedia.org/r/337583 (https://phabricator.wikimedia.org/T156359) (owner: 10Hashar) [14:52:29] (03Merged) 10jenkins-bot: Castor: also capture .composer/cache [integration/config] - 10https://gerrit.wikimedia.org/r/337588 (https://phabricator.wikimedia.org/T156359) (owner: 10Hashar) [14:55:27] 10Continuous-Integration-Config, 13Patch-For-Review: operations-mw-config-composer-hhvm-jessie job lacks composer cache - https://phabricator.wikimedia.org/T156359#3025248 (10hashar) 05Open>03Resolved a:03hashar [14:55:51] (03CR) 10Zfilipin: "check experimental" [integration/config] - 10https://gerrit.wikimedia.org/r/337590 (https://phabricator.wikimedia.org/T158052) (owner: 10Zfilipin) [14:59:06] thcipriani|afk: twentyafterfour: also godog has a patch above so scap no more send to fluorine but to some new host \O/ [14:59:50] I saw that, I just need to move computers to +1 :) [15:02:25] 10Continuous-Integration-Infrastructure: Create a controlled and ongoing CI pipeline test job that we can alert on - https://phabricator.wikimedia.org/T158054#3025285 (10Paladox) [15:03:58] 10Deployment-Systems, 10Scap, 13Patch-For-Review: Update Debian Package for Scap3 - https://phabricator.wikimedia.org/T127762#3025292 (10mmodell) @godog: [[ /source/scap/browse/release/;debian/3.5.2-1 | debian/3.5.2-1 ]] {icon tag} is ready for packaging. [15:08:49] 10Deployment-Systems, 10Scap, 13Patch-For-Review: Update Debian Package for Scap3 - https://phabricator.wikimedia.org/T127762#3025335 (10mmodell) @mobrovac ^ fyi, new scap package is tagged [15:12:53] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team: Use ccache for CI compilation? - https://phabricator.wikimedia.org/T152688#3025369 (10hashar) 05Open>03declined Was a random idea. No intent to implement it for now. [15:14:26] 06Release-Engineering-Team, 10DBA, 10Datasets-General-or-Unknown, 13Patch-For-Review, and 2 others: Automatize the check and fix of object, schema and data drifts between mediawiki HEAD, production masters and slaves - https://phabricator.wikimedia.org/T104459#3025377 (10jcrespo) a:05jcrespo>03None Unc... [15:21:27] 10Browser-Tests-Infrastructure, 10RelatedArticles, 07JavaScript, 13Patch-For-Review, 15User-zeljkofilipin: Set up Selenium tests in Node.js for RelatedArticles extension - https://phabricator.wikimedia.org/T158052#3025400 (10zeljkofilipin) Jenkins job is failing: https://integration.wikimedia.org/ci/job... [15:45:23] Project selenium-MobileFrontend » chrome,beta,Linux,contintLabsSlave && UbuntuTrusty build #325: 04FAILURE in 23 min: https://integration.wikimedia.org/ci/job/selenium-MobileFrontend/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/325/ [15:48:22] (03Abandoned) 10Aleksey Bekh-Ivanov (WMDE): Fix absence of dev dependencies for Wikibase in jenkins job [integration/config] - 10https://gerrit.wikimedia.org/r/335215 (owner: 10Aleksey Bekh-Ivanov (WMDE)) [15:52:26] 10Continuous-Integration-Config, 10ContentTranslation, 03Language-2017 Sprint 2, 05Language-team January-March 2017, and 3 others: mwext-qunit-jessie test fails on unrelated change - https://phabricator.wikimedia.org/T153038#3025485 (10Nikerabbit) It's not happening to me locally, although I am missing few... [16:06:00] bah beta cluster jobs are stall again :( [16:29:00] 10Continuous-Integration-Infrastructure, 10Release Pipeline, 15User-mobrovac: Mathoid CI Container Image POC - https://phabricator.wikimedia.org/T157469#3025668 (10thcipriani) p:05Triage>03Normal So now that `service-runner` has the `generate` subcommand you can run something like: ``` $ cd services/mat... [16:30:23] 10Browser-Tests-Infrastructure, 07Ruby, 15User-zeljkofilipin: Update Ruby tests to Selenium 3 - https://phabricator.wikimedia.org/T158074#3025671 (10zeljkofilipin) [16:40:11] 10Gerrit, 06Operations, 13Patch-For-Review: Investigate seemingly random Gerrit slow-downs - https://phabricator.wikimedia.org/T148478#3025706 (10Gehel) @Paladox looking at the GC logs from last Friday (Feb 10) I don't really see a GC issue, but more a memory allocation issue. That is, I see the multiple ful... [16:41:26] 10Gerrit, 06Operations, 13Patch-For-Review: Investigate seemingly random Gerrit slow-downs - https://phabricator.wikimedia.org/T148478#3025707 (10Paladox) >>! In T148478#3025706, @Gehel wrote: > @Paladox looking at the GC logs from last Friday (Feb 10) I don't really see a GC issue, but more a memory allocat... [16:49:16] 10Gerrit, 06Operations, 13Patch-For-Review: Investigate seemingly random Gerrit slow-downs - https://phabricator.wikimedia.org/T148478#3025720 (10demon) >>! In T148478#3025707, @Paladox wrote: > We didn't have this problem with gerrit's old server ytterbium. I believe ytterbium had an ssd, but I'm not really... [16:49:34] 10Browser-Tests-Infrastructure, 07Ruby, 15User-zeljkofilipin: Update Ruby tests to Selenium 3 - https://phabricator.wikimedia.org/T158074#3025721 (10zeljkofilipin) p:05Triage>03Normal [16:50:59] 10Browser-Tests-Infrastructure, 10RelatedArticles, 07JavaScript, 13Patch-For-Review, 15User-zeljkofilipin: Set up Selenium tests in Node.js for RelatedArticles extension - https://phabricator.wikimedia.org/T158052#3025725 (10zeljkofilipin) 05Open>03Resolved The setup is finished. The next step is to... [16:51:48] 10Gerrit, 06Operations, 13Patch-For-Review: Investigate seemingly random Gerrit slow-downs - https://phabricator.wikimedia.org/T148478#3025744 (10Paladox) Oh, I wonder why ytterbium didn't have this problem. Could this be a bug in gerrit that is not really noticeable because upstream use google servers so th... [16:53:48] 10Gerrit, 06Operations, 13Patch-For-Review: Investigate seemingly random Gerrit slow-downs - https://phabricator.wikimedia.org/T148478#3025747 (10Gehel) >>! In T148478#3025707, @Paladox wrote: > Oh, would using ssd alleviate these problems as ssd are faster then hdd? Faster / better IO never hurts! Especial... [16:58:01] 10Browser-Tests-Infrastructure, 10Wikidata, 13Patch-For-Review, 15User-Tobi_WMDE_SW, 15User-zeljkofilipin: Increase in failures caused by Saucelabs - https://phabricator.wikimedia.org/T152963#3025773 (10zeljkofilipin) The results: Linux: [[ https://integration.wikimedia.org/ci/view/Selenium/job/selenium... [17:07:21] 10Browser-Tests-Infrastructure, 10Wikidata, 13Patch-For-Review, 15User-Tobi_WMDE_SW, 15User-zeljkofilipin: Increase in failures caused by Saucelabs - https://phabricator.wikimedia.org/T152963#3025816 (10zeljkofilipin) || Linux [[ https://integration.wikimedia.org/ci/view/Selenium/job/selenium-Wikibase/ |... [17:13:26] 10Gerrit, 06Operations, 13Patch-For-Review: Investigate seemingly random Gerrit slow-downs - https://phabricator.wikimedia.org/T148478#3025855 (10demon) >>! In T148478#3025747, @Gehel wrote: > Faster / better IO never hurts! Especially since we seem to have [[ https://grafana.wikimedia.org/dashboard/file/ser... [17:27:32] 10Browser-Tests-Infrastructure, 10Wikidata, 13Patch-For-Review, 15User-Tobi_WMDE_SW, 15User-zeljkofilipin: Increase in failures caused by Saucelabs - https://phabricator.wikimedia.org/T152963#3025896 (10zeljkofilipin) Number of failures in the last 10 runs | Linux | Windows | Mac | | | | | | | | |... [17:43:41] 10Browser-Tests-Infrastructure, 10Wikidata, 13Patch-For-Review, 15User-Tobi_WMDE_SW, 15User-zeljkofilipin: Increase in failures caused by Saucelabs - https://phabricator.wikimedia.org/T152963#3026037 (10zeljkofilipin) `MediawikiApi::ApiError` is unrelated problem (T157665) and should be ignored, so only... [17:50:33] mobrovac: thcipriani : wouldn't it be nice to have nvm packaged for debian ? :} [17:51:52] uh uh hashar, I don't think any such package manager has ever made it into debian [17:52:03] but yes, it would [17:52:30] php-pear did, but pear is hardly an example to follow :P [17:53:09] well it is just a bash script [17:53:31] probably has little dependencies, so maybe it is easy to get it packaged for us :) [18:32:28] Upstream gerrit are adding support to reply to changes through email :) [18:32:45] RainbowSprinkles ^^ [18:33:38] https://gerrit-review.googlesource.com/#/c/96854/ [18:33:55] https://gerrit-review.googlesource.com/#/c/96693/ [18:57:02] 10Gerrit, 10Diffusion: Unpublished Gerrit commits show up in Diffusion - https://phabricator.wikimedia.org/T158043#3026453 (10greg) [19:21:11] Is there a way to search in phabricator for tasks that are in one project (like say dc-ops) but NOT in other projects (like the actual onsite projects). I want to make sure we don't have onsite tasks somehow falling through cracks. [19:21:21] (figured in here is the place to ask about phab search ;) [19:22:24] twentyafterfour ^^ [19:22:29] oh, found it by just tinkering, nm. [19:22:34] not(project) [19:22:55] just doesnt autocomplete for me so manual all the way [19:23:05] im spoiled by autocomplete. [19:24:05] and there are indeed tasks that have likely been neglected in my resultset, success. [19:24:25] robh: If you type "Not foo..." it should start autocompleting [19:24:46] maybe its just my firefox is overloaded and i need to restart it but i didnt get autocompletes meh [19:24:52] but otherwise works fine. [19:24:57] https://usercontent.irccloud-cdn.com/file/d2GSDrRi/not_phab.png [19:25:28] ohhhh, i did not(project0 which worked but yeah, that is different method same result and has auto [19:25:49] cool [19:27:46] 10Gerrit, 10Diffusion: Unpublished Gerrit commits show up in Diffusion - https://phabricator.wikimedia.org/T158043#3026642 (10demon) 05Open>03declined > What's more, I thought draft changes were mostly "hidden", so I'm not even sure how the responsible service found out about this patch. You would think,... [19:29:31] 10Continuous-Integration-Infrastructure, 10Monitoring, 13Patch-For-Review: Alert when Zuul/Gearman queue is stalled - https://phabricator.wikimedia.org/T70113#3026646 (10greg) >>! In T70113#3025000, @chasemp wrote: > One thing I want to note is when making the nodepool alert actionable it seemed no one is pa... [19:36:30] the correct syntax is not(project) but the autocomplete doesn't like the ( [19:43:19] 06Release-Engineering-Team (Deployment-Blockers), 05Release: MW-1.29.0-wmf.10 deployment blockers - https://phabricator.wikimedia.org/T155525#3026699 (10mmodell) [19:49:21] 10Gerrit, 10Diffusion: Unpublished Gerrit commits show up in Diffusion - https://phabricator.wikimedia.org/T158043#3026712 (10Paladox) >>! In T158043#3026642, @demon wrote: >> What's more, I thought draft changes were mostly "hidden", so I'm not even sure how the responsible service found out about this patch.... [19:59:34] 10Continuous-Integration-Infrastructure, 10Monitoring, 13Patch-For-Review: Alert when Zuul/Gearman queue is stalled - https://phabricator.wikimedia.org/T70113#3026756 (10chasemp) I am in no way trying to push over a first domino here, I swear. The logistics of becoming an on-call team would be pretty big I... [20:01:25] 10Continuous-Integration-Infrastructure, 10Monitoring, 13Patch-For-Review: Alert when Zuul/Gearman queue is stalled - https://phabricator.wikimedia.org/T70113#3026789 (10greg) Right right :) For us I think a good first pass would be email+IRC alerts and a best effort tier SLA. [20:02:10] zeeee greg-g I think it's hard to ask this without stepping on a hive of bees but it's not my intention to be shitty here https://phabricator.wikimedia.org/T70113#3026756 [20:02:25] :) [20:02:34] chasemp: replied already :P [20:03:09] you are some kind of cyborg, but cool [20:52:53] 10Continuous-Integration-Config: composer cache-dir configuration vary between Trusty and Jessie - https://phabricator.wikimedia.org/T158117#3027046 (10hashar) [20:53:18] 10Continuous-Integration-Config, 13Patch-For-Review: operations-mw-config-composer-hhvm-jessie job lacks composer cache - https://phabricator.wikimedia.org/T156359#2972076 (10hashar) .composer/cache vs .cache/composer is the sub task T158117 [20:55:22] 10Continuous-Integration-Config, 10MediaWiki-extensions-NavigationTiming, 06Performance-Team: Create QUnit tests for NavigationTiming extension - https://phabricator.wikimedia.org/T157835#3027082 (10Jdforrester-WMF) Now just needs the last part (extension-gate) doing. [21:21:16] 10Continuous-Integration-Config, 06Release-Engineering-Team, 06Discovery, 06Discovery-Analysis: Add CI to all wikimedia/discovery repositories that are active - https://phabricator.wikimedia.org/T153856#3027171 (10debt) 05Open>03declined Declining this ticket as there isn't much we can do with it in Di... [21:29:19] thcipriani: hey is there any chance i could get a labs swat done before the 4pm swat window? https://gerrit.wikimedia.org/r/#/c/337636/ is blocking us doing some important QA and the additional hour or so would really help here (especially given im not 100% sure this is the actual cause) [21:30:17] jdlrobson: sure, give me a few though, I'm deploying the train right at the moment, I'm hoping to be done within the next 20 minutes or so [21:30:37] of course. thanks thcipriani <3 [21:33:49] 10Continuous-Integration-Config, 10Page-Previews, 06Reading-Web-Backlog, 07Browser-Tests, and 2 others: add rake entrypoint and rubocop to ext:Popups - https://phabricator.wikimedia.org/T136285#3027221 (10Jdlrobson) [21:57:14] (03PS1) 10Reedy: Stop branching PageForms [tools/release] - 10https://gerrit.wikimedia.org/r/337746 [21:58:11] 10Gerrit, 07Upstream: Gerrit should feature customizable message on Login page (No 'Forgot password' link in the gerrit login page.) - https://phabricator.wikimedia.org/T60205#3027375 (10demon) p:05Low>03Normal This can't be too hard. Let's see if we can get something upstreamed. Making it generally-config... [21:58:30] Reedy: Why can't we just move to PageForms itself yet? [21:58:40] RainbowSprinkles: Because we don't need/want to? [21:58:55] Well, we still install it/use it right? [21:59:08] The 3.7 branch of SF [21:59:08] https://github.com/wikimedia/mediawiki-tools-release/blob/master/make-wmf-branch/config.json#L175 [21:59:37] Yeah I know we install the legacy version. But why that? Why not upgrade? [21:59:41] Are there breaking changes? [22:00:14] We pin SMW and Validator for example because of weird incompat with composer & installation. But is that the case with PageForms? [22:01:11] Not sure [22:01:12] "php": ">=5.3.0", [22:01:12] "composer/installers": "1.*,>=1.0.1" [22:01:32] Didn't we try to switch over, what happened again? [22:01:58] I'm not sure we get much for tracking master [22:06:15] Well, point is that extensions/skins not tracking master are the exception, not the rule [22:06:42] And generally a pattern to avoid, IMHO. It means we avoid normal bugfixes and such that we'd actually like to get. [22:11:51] RainbowSprinkles Reed all we did was rename the extension, nothing was changed code side (I mean in the extension code source). Now it has changes that may have fixed it the last time we tryed to do it. We may want to test on beta to make sure the extension dosent cause a php error. Then Migrate other wiki's that use SemanticForms to PageForms. [22:11:57] If there is a bug, we should report it, so that it get's fixed :) [22:14:36] 06Release-Engineering-Team (Deployment-Blockers), 05Release: MW-1.29.0-wmf.12 deployment blockers - https://phabricator.wikimedia.org/T155527#3027469 (10thcipriani) [22:14:40] paladox: But we don't track master anyway [22:15:05] Yep, but didn't we try to switch to PageForms? What was the error? [22:15:05] Reedy: you're back finally o/ [22:16:12] jdlrobson: around? I can merge your labs change real quick. [22:16:16] yup [22:17:29] jdlrobson: okie doke, +2'd, will fetch and sync for house keeping on tin, should be on labs whenever then next code-update/sync runs [22:17:49] "labs" == "deployment-prep/beta" [22:17:53] * thcipriani feels shame [22:18:03] thcipriani: great. Are you able to check the value of wgPopupsBetaFeature ? [22:18:06] :D thcipriani [22:18:20] i just want to confirm its true.. before it was false [22:18:34] jdlrobson: you mean in prod? [22:18:45] in bc [22:19:55] chasemp: o/ [22:20:12] JustBerry: hi [22:21:16] jdlrobson: current value in beta is bool(true) [22:21:25] sweet that's promising [22:21:29] :) [22:21:42] just have to wait and see if that gets reflected on beta cluster js code [22:21:44] mwrepl on deployment-tin for future reference [22:23:27] is this what im waiting on ? https://integration.wikimedia.org/ci/view/Beta/job/beta-code-update-eqiad/ [22:24:29] seems to be going now [22:24:32] runs on a timer [22:24:42] * * * * */10 [22:24:54] 10Continuous-Integration-Config, 06Release-Engineering-Team, 06Discovery, 06Discovery-Analysis: Add CI to all wikimedia/discovery repositories that are active - https://phabricator.wikimedia.org/T153856#2893206 (10greg) >>! In T153856#2914722, @mpopov wrote: > The dashboard repositories are R/Shiny applica... [22:44:13] 10Gerrit, 07Upstream: Gerrit should feature customizable message on Login page (No 'Forgot password' link in the gerrit login page.) - https://phabricator.wikimedia.org/T60205#3027589 (10Paladox) You may need to add the support for both polygerrit and gwt ui. Upstream are more in favour of changes for polyger... [22:55:19] thcipriani: looks like that did the trick! Just verified :) [22:55:22] thank you for your help [22:55:35] jdlrobson: yw, glad everything's working :)