[00:06:08] 10Deployment-Systems, 06Release-Engineering-Team, 10Scap, 06Operations, 15User-Addshore: cannot delete non-empty directory: php-1.29.0-wmf.3 messages on 'scap sync' on mwdebug1002 - https://phabricator.wikimedia.org/T157030#3027833 (10thcipriani) 05Open>03Resolved a:05mmodell>03thcipriani What's... [00:17:03] Yippee, build fixed! [00:17:03] Project selenium-Flow » chrome,beta,Linux,contintLabsSlave && UbuntuTrusty build #301: 09FIXED in 1 min 1 sec: https://integration.wikimedia.org/ci/job/selenium-Flow/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/301/ [01:46:36] Yippee, build fixed! [01:46:37] Project selenium-MobileFrontend » chrome,beta,Linux,contintLabsSlave && UbuntuTrusty build #326: 09FIXED in 25 min: https://integration.wikimedia.org/ci/job/selenium-MobileFrontend/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/326/ [01:56:00] Project selenium-MobileFrontend » firefox,beta,Linux,contintLabsSlave && UbuntuTrusty build #326: 04FAILURE in 34 min: https://integration.wikimedia.org/ci/job/selenium-MobileFrontend/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/326/ [02:55:59] 06Release-Engineering-Team, 10MediaWiki-Vagrant, 06Operations, 07Epic: [EPIC] Migrate base image to Debian Jessie - https://phabricator.wikimedia.org/T136429#3028263 (10bd808) [04:19:39] Yippee, build fixed! [04:19:40] Project selenium-MultimediaViewer » firefox,beta,Linux,contintLabsSlave && UbuntuTrusty build #298: 09FIXED in 23 min: https://integration.wikimedia.org/ci/job/selenium-MultimediaViewer/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/298/ [07:26:09] RECOVERY - Puppet run on deployment-salt02 is OK: OK: Less than 1.00% above the threshold [0.0] [07:26:13] RECOVERY - Puppet run on deployment-zotero01 is OK: OK: Less than 1.00% above the threshold [0.0] [07:31:55] RECOVERY - Puppet run on deployment-phab01 is OK: OK: Less than 1.00% above the threshold [0.0] [07:41:56] 10Browser-Tests-Infrastructure, 10Wikidata, 13Patch-For-Review, 15User-Tobi_WMDE_SW, 15User-zeljkofilipin: Increase in failures caused by Saucelabs - https://phabricator.wikimedia.org/T152963#3028442 (10Tobi_WMDE_SW) @zeljkofilipin is it possible to run the [[ https://integration.wikimedia.org/ci/view/Ex... [10:01:21] 06Release-Engineering-Team, 10Icinga, 10Monitoring: Rename Icinga contact 'amusso' to 'hashar' - https://phabricator.wikimedia.org/T158167#3028668 (10hashar) [10:01:40] 06Release-Engineering-Team, 10Icinga, 10Monitoring, 06Operations, 10Ops-Access-Requests: Rename Icinga contact 'amusso' to 'hashar' - https://phabricator.wikimedia.org/T158167#3028702 (10hashar) [13:49:54] Project selenium-VisualEditor » firefox,beta,Linux,contintLabsSlave && UbuntuTrusty build #307: 04FAILURE in 5 min 52 sec: https://integration.wikimedia.org/ci/job/selenium-VisualEditor/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/307/ [14:14:52] 10Gerrit, 10Developer-Wishlist (2017), 07Documentation: Enable and document "WIP" workflow status in Gerrit - https://phabricator.wikimedia.org/T135245#3029248 (10Paladox) Upstream gerrit have this https://gerrit-review.googlesource.com/#/c/97245/ proposal for wip status in gerrit. [15:11:44] hashar: 12 hours of data https://graphite.wikimedia.org/render/?width=800&height=500&_salt=1486132582.09&lineMode=connected&from=-12h&target=cactiStyle(servers.labnet1002.nova.verify.creation)&target=cactiStyle(servers.labnet1002.nova.verify.dns)&target=cactiStyle(servers.labnet1002.nova.verify.puppet)&&target=cactiStyle(servers.labnet1002.nova.verify.ssh)&&target=cactiStyle(servers.labnet1002.nova.verify.deletion)&f [15:11:44] rom=-12h [15:11:47] gah [15:12:05] https://graphite.wikimedia.org/render/?width=800&height=500&lineMode=connected&from=-12h&target=cactiStyle(servers.labnet1002.nova.verify.creation)&target=cactiStyle(servers.labnet1002.nova.verify.dns)&target=cactiStyle(servers.labnet1002.nova.verify.puppet)&&target=cactiStyle(servers.labnet1002.nova.verify.ssh)&&target=cactiStyle(servers.labnet1002.nova.verify.deletion)&from=-12h [15:22:50] *twiddles thumbs while jenkins struggles with the number of jobs* [15:40:22] 10Browser-Tests-Infrastructure, 10Wikidata, 13Patch-For-Review, 15User-Tobi_WMDE_SW, 15User-zeljkofilipin: Increase in failures caused by Saucelabs - https://phabricator.wikimedia.org/T152963#3029555 (10zeljkofilipin) The trouble with recent data is that we had a lot of trouble with unrelated `MediawikiA... [15:44:00] (03PS1) 10Hashar: zuul: fix window-floor which is in changes not jobs [integration/config] - 10https://gerrit.wikimedia.org/r/337865 [15:46:08] (03CR) 10Hashar: [C: 032] zuul: fix window-floor which is in changes not jobs [integration/config] - 10https://gerrit.wikimedia.org/r/337865 (owner: 10Hashar) [15:46:59] (03Merged) 10jenkins-bot: zuul: fix window-floor which is in changes not jobs [integration/config] - 10https://gerrit.wikimedia.org/r/337865 (owner: 10Hashar) [15:47:32] !log Zuul reducing gate-and-submit minimum amount of changes to process from the wrong 12 down to 2. In case of repeating failures it would end up running jobs for only two jobs which would prevent cancelling jobs for up to 11 changes! [15:47:35] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [15:49:25] zeljkof: I cancelled your selenium job builds [15:49:46] zeljkof: the matrix job ended up occupying most of the jenkins executor slots for trusty :( [15:50:04] and CI is a bit busy right now https://integration.wikimedia.org/zuul/ :( [15:50:04] hashar: ouch [15:50:18] I really gotta move those Wikibase jobs to nodepool [15:50:21] Project mediawiki-core-code-coverage build #2578: 15ABORTED in 49 min: https://integration.wikimedia.org/ci/job/mediawiki-core-code-coverage/2578/ [15:50:28] oh, they are not on nodepool? [15:50:37] nop :( [15:50:41] not enough capacity [15:50:44] hm, completely forgot that [15:50:46] sorry [15:50:48] and most probably I tried and something failed [15:50:51] no problems :} [15:50:54] related to that [15:51:02] is there a way to run that job serially? https://phabricator.wikimedia.org/T152963#3028442 [15:51:03] it is just that patches kept pilling up in gate and submit [15:51:04] some failed [15:51:12] which causes Zuul to cancel all the jobs and try again [15:51:18] so there is a large backlog to process [15:51:20] so it does not take 6 slots, but one by one [15:51:28] in a matrix job yeah [15:51:39] I was looking, could not find it [15:51:57] zeljkof: https://docs.openstack.org/infra/jenkins-job-builder/project_matrix.html?highlight=sequential [15:52:02] execution-strategy: [15:52:06] sequential: true [15:52:16] default is to run them in parallel [15:52:30] the touchstone: one is to add a selection to only run some jobs before the rest [15:52:34] kind of a smoke run [15:52:41] if it pass, the rest of jobs are run [15:52:46] else they are skip [15:53:35] zeljkof: an example: https://gerrit.wikimedia.org/r/#/c/333280/6/jjb/mediawiki.yaml [15:53:48] sequential: true => run them one after the other [15:53:51] 10Browser-Tests-Infrastructure, 10Wikidata, 13Patch-For-Review, 15User-Tobi_WMDE_SW, 15User-zeljkofilipin: Increase in failures caused by Saucelabs - https://phabricator.wikimedia.org/T152963#3029667 (10zeljkofilipin) >>! In T152963#3028442, @Tobi_WMDE_SW wrote: > @zeljkofilipin is it possible to run the... [15:53:59] touchstone: is a groovy selection [15:54:11] I made that job to first run mediawiki @master with Zend php5 [15:54:20] if that fails, the job stop [15:54:24] else it runs the rest of the mess [15:56:56] chasemp: your graphs are neat :} [15:57:04] chasemp: time to load them in grafana ? :} [15:57:37] chasemp: also I was looking at the collecting script yesterday, you are using different values for the sleep() calls [15:57:46] so the metrics would jump differentily [15:58:10] yes it's true, and up for debate, I am thinking I will adjust w/ more context for the underlying operation [15:58:16] and I think the verify ssh has a sleep(60) so the time would usually be 1 minute, 2 minutes, 3 minutes etc [15:58:18] i.e. I can shorten that loop if I enforce an ssh timeout [15:58:26] that's been reduced since [15:58:29] nodepool has some similar code for ssh [15:58:34] but the basic thing is still there [15:58:37] yeah [15:58:38] iirc they just while loop and try to ssh every five seconds [15:58:49] so nodepool manages to ssh asap [15:59:00] but yeah agree the basic is there and it is very nice! [15:59:00] Yippee, build fixed! [15:59:01] Project selenium-MobileFrontend » firefox,beta,Linux,contintLabsSlave && UbuntuTrusty build #327: 09FIXED in 36 min: https://integration.wikimedia.org/ci/job/selenium-MobileFrontend/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/327/ [15:59:18] hashar: looks like running sequentially is supported only for executing strategy classic, but we use yaml :/ [15:59:22] sure, something like that I'm thinking. But I want to take the loop and make sure the ssh call timeouts inside of it etc and do some basic sanity checking there since it's configurable [15:59:32] chasemp: I was also wondering if we have some openstack logs we could eventually process to find the time it takes to spin an instance [15:59:33] but yeah, it's inflating the ssh number by a bit (and will no matter what) [15:59:46] potentially with and without contintcloud [16:00:39] you can piece it together across nova-api, teh scheduler, and labvirt nodes etc [16:00:53] but it's honestly cleaner and more controlled via some mechanism like this [16:01:05] but yeah we could in theory aim the host logs at the elastic logging cluster or something [16:01:15] I'm not sure what the thinking on that is generally [16:01:21] hmm [16:01:33] we have a few tasks to send production logs to logstash [16:01:46] but no real effort. An attempt was made for Gerrit but eventually failed afaik [16:01:59] for python software, I think there is a logging module to send logs to logstash [16:02:01] nodepool's internals and this combined are nearly the same outcome data wise [16:02:10] I haven't played with yet but will surely do it for zuul/nodepool [16:03:01] 10Browser-Tests-Infrastructure, 10Wikidata, 13Patch-For-Review, 15User-Tobi_WMDE_SW, 15User-zeljkofilipin: Increase in failures caused by Saucelabs - https://phabricator.wikimedia.org/T152963#3029702 (10zeljkofilipin) Unfortunately, looks like running sequentially is supported only for executing strategy... [16:03:22] also for deletion, Nodepool emit metrics for the time it takes for a deletion https://grafana.wikimedia.org/dashboard/db/nodepool-tasks?panelId=27&fullscreen [16:04:01] though its time resolution depends on the processing rate (rate: 6) and # of queries to openstack api that are in the queue [16:04:39] that is essentially the same that you wrote, but only for deletion (and with all the nodepool cruft overhead) [16:05:08] sure but absolute values are not really all that worrysome for 10-30s variance in polling [16:05:14] it's teh relative changes we almost always care about [16:05:36] not that it isn't good data but for these purposes I'm more interesting in tracking long term heuristics than I am in knowing within a second whta VM creation is for some SLA [16:05:49] we can do that better now but it's not the primary goal for me atm [16:07:46] yeah no complain [16:24:33] (03PS1) 10Hashar: decouple puppet modules with more specific jobs [integration/config] - 10https://gerrit.wikimedia.org/r/337870 [16:26:37] (03CR) 10Hashar: [C: 032] decouple puppet modules with more specific jobs [integration/config] - 10https://gerrit.wikimedia.org/r/337870 (owner: 10Hashar) [16:30:26] (03Merged) 10jenkins-bot: decouple puppet modules with more specific jobs [integration/config] - 10https://gerrit.wikimedia.org/r/337870 (owner: 10Hashar) [16:47:47] PROBLEM - Puppet run on deployment-cache-text04 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [16:56:26] 06Release-Engineering-Team (Deployment-Blockers), 05Release: MW-1.29.0-wmf.12 deployment blockers - https://phabricator.wikimedia.org/T155527#3029895 (10thcipriani) Last week I called out some of the logspam that lowered my confidence about deployments to great effect! All of the tasks I pointed to saw some ki... [17:22:47] RECOVERY - Puppet run on deployment-cache-text04 is OK: OK: Less than 1.00% above the threshold [0.0] [18:05:44] 06Release-Engineering-Team (Deployment-Blockers), 13Patch-For-Review, 05Release: MW-1.29.0-wmf.12 deployment blockers - https://phabricator.wikimedia.org/T155527#3030133 (10thcipriani) [18:16:23] RainbowSprinkles looks like we can now reply to gerrit comments through email, https://groups.google.com/forum/#!topic/repo-discuss/47Ss-Ix0oWc :) [18:43:34] (03PS1) 10Chad: Add Dashiki to branch config [tools/release] - 10https://gerrit.wikimedia.org/r/337891 [18:52:02] (03CR) 10Chad: [C: 032] Add Dashiki to branch config [tools/release] - 10https://gerrit.wikimedia.org/r/337891 (owner: 10Chad) [18:53:43] (03Merged) 10jenkins-bot: Add Dashiki to branch config [tools/release] - 10https://gerrit.wikimedia.org/r/337891 (owner: 10Chad) [19:03:21] (sorry, didn't know about branch config) [19:42:08] 10Gerrit: Grant +2 rights to MediaWiki extensions for Piotr - https://phabricator.wikimedia.org/T158233#3030611 (10Jdlrobson) [19:42:17] 10Gerrit: Grant +2 rights to MediaWiki extensions for @pmiazga - https://phabricator.wikimedia.org/T158233#3030623 (10Jdlrobson) [19:51:29] the Zuul Gearman queue now has an Icinga alarm https://gerrit.wikimedia.org/r/#/c/337552/ !!! :] [20:20:40] PROBLEM - Work requests waiting in Zuul Gearman server https://grafana.wikimedia.org/dashboard/db/zuul-gearman on contint1001 is CRITICAL: CRITICAL: Anomaly detected: 7 data above and 20 below the confidence bounds [20:21:40] PROBLEM - Work requests waiting in Zuul Gearman server https://grafana.wikimedia.org/dashboard/db/zuul-gearman on contint2001 is CRITICAL: CRITICAL: Anomaly detected: 6 data above and 22 below the confidence bounds [20:23:22] hashar ^^ [20:23:25] this "X data above and Y data below" thing [20:24:01] i'll give it a chance since it just got added and needs to get some data [20:24:40] but if it doesnt go away.. then it's the same problem we had with other check_graphite checks and was my concern about adding it. people will wonder "what should i do"/"is it actionable" [20:24:49] ok [20:25:08] first let's just wait an hour [20:25:23] it was literally just added and started to get data... [20:26:53] but if it doesn't go away or keeps happening, let's not just silence it [20:26:59] rather remove it again [20:27:40] okk [20:27:45] woops, i meant ok [20:32:49] 10Gerrit: Grant +2 rights to MediaWiki extensions for @pmiazga - https://phabricator.wikimedia.org/T158233#3030611 (10dr0ptp4kt) I approve the access request. [21:01:41] 06Release-Engineering-Team, 10Icinga, 10Monitoring, 06Operations, 10Ops-Access-Requests: Rename Icinga contact 'amusso' to 'hashar' - https://phabricator.wikimedia.org/T158167#3030863 (10RobH) a:03RobH I don't mind making this change, I'll just restrict myself to doing so tomorrow in the AM. (This has... [21:02:36] 10Gerrit: Grant +2 rights to MediaWiki extensions for @pmiazga - https://phabricator.wikimedia.org/T158233#3030611 (10Legoktm) This should probably be a #ldap-access-requests for the 'wmf' group, not via Gerrit. [21:12:38] PROBLEM - Work requests waiting in Zuul Gearman server https://grafana.wikimedia.org/dashboard/db/zuul-gearman on contint1001 is CRITICAL: CRITICAL: Anomaly detected: 3 data above and 12 below the confidence bounds [21:14:38] PROBLEM - Work requests waiting in Zuul Gearman server https://grafana.wikimedia.org/dashboard/db/zuul-gearman on contint1001 is CRITICAL: CRITICAL: Anomaly detected: 3 data above and 12 below the confidence bounds [21:15:38] RECOVERY - Work requests waiting in Zuul Gearman server https://grafana.wikimedia.org/dashboard/db/zuul-gearman on contint1001 is OK: OK: No anomaly detected [21:15:39] RECOVERY - Work requests waiting in Zuul Gearman server https://grafana.wikimedia.org/dashboard/db/zuul-gearman on contint2001 is OK: OK: No anomaly detected [21:20:28] 10Gerrit: Gerrit needs a very wide screen or has big horizontal scroll - https://phabricator.wikimedia.org/T38471#3030927 (10Paladox) Upstream are also doing this https://gerrit-review.googlesource.com/#/c/94671/ proposal for the ux. Which doesn't mention the long scroll bar's, but does mention the need to have... [21:20:39] (03PS1) 10Hashar: zuul: sync with upstream @4c2053d [integration/docroot] - 10https://gerrit.wikimedia.org/r/337947 [21:22:04] (03CR) 10Hashar: [C: 032] zuul: sync with upstream @4c2053d [integration/docroot] - 10https://gerrit.wikimedia.org/r/337947 (owner: 10Hashar) [21:23:19] (03Merged) 10jenkins-bot: zuul: sync with upstream @4c2053d [integration/docroot] - 10https://gerrit.wikimedia.org/r/337947 (owner: 10Hashar) [21:23:26] (03CR) 10jenkins-bot: zuul: sync with upstream @4c2053d [integration/docroot] - 10https://gerrit.wikimedia.org/r/337947 (owner: 10Hashar) [21:24:00] 10Gerrit, 06Release-Engineering-Team, 07Zuul: Update zuul to upstream master - https://phabricator.wikimedia.org/T158243#3030929 (10Paladox) [21:24:43] paladox: you can make it a blocker :] [21:24:46] paladox: well spotted! [21:24:49] 10Gerrit, 06Release-Engineering-Team, 07Zuul: Update zuul to upstream master - https://phabricator.wikimedia.org/T158243#3030959 (10Paladox) [21:24:51] 10Gerrit, 06Release-Engineering-Team: Update gerrit to 2.14 - https://phabricator.wikimedia.org/T156120#2964656 (10Paladox) [21:25:04] paladox: I guess I will have to clean up the integration/zuul.git branch madness [21:25:18] and the patch is easy to cherry pick [21:25:22] hashar thanks :), i also know the change that broke it. It broke its-base too. [21:25:23] but that is fixed now. :) [21:25:24] thank you!!! [21:25:24] yep [21:25:28] Your welcome :) [21:25:41] Should i set it as high pri? Or normal? [21:26:27] 10Gerrit, 06Release-Engineering-Team, 07Zuul: Update zuul to upstream master - https://phabricator.wikimedia.org/T158243#3030929 (10Paladox) This task blocks T156120 [21:26:35] paladox: leave it untriaged or normal [21:26:36] hashar ^^ :) [21:26:38] ok [21:26:46] Chad and I are in the same team [21:26:54] ok :) [21:27:12] and the team has weekly meetings every monday that is when we typically talk about Gerrit upgrade / blockers/ work to be done [21:27:22] so when times come to upgrade Gerrit, I will be tasked in updating zuul [21:27:29] now that we have a blocker task \o/ [21:27:32] oh :). [21:27:47] meanwhile I guess it is going to stay idle until it is actually required [21:27:59] I haven't actually tested zuul with 2.14 yet as i've been testing alot of changes. [21:28:06] one sure thing, you monitoring the various upstream is very helpful [21:28:31] Your welcome :), i've actually updated a ton of gerrit plugins for bazel. Since buck was removed. [21:28:48] neat [21:28:54] maven -> buck -> bazel [21:28:58] it never stops :] [21:29:09] Yep, maven is deprecated in gerrit [21:29:20] have you seen facebook rewrote npm ? :] [21:29:22] https://gerrit-review.googlesource.com/#/q/owner:%22Paladox+none+%253Cthomasmulhall410%2540yahoo.com%253E%22 [21:29:25] nope [21:29:27] it is called yarn. And is arguably wayyy faster [21:29:31] oh [21:29:42] i guess they built it on hhvm. [21:29:46] https://code.facebook.com/posts/1840075619545360 [21:29:50] hashar when i can play farmville and it loads faster then i will believe that [21:29:50] https://github.com/yarnpkg/yarn [21:29:53] oh [21:30:02] paladox: no it is a javascript package [21:30:09] so you can just npm -g install yarn [21:30:13] Oh. [21:30:14] :) [21:30:16] and it is more or less a dropin replacement for npm [21:30:22] at least for the basic commands I am using [21:30:29] I've seen a task in phab about migrating to yarn. [21:30:33] hashar do i see a future for yarn in the WMF code? [21:30:34] the facebook post is a good read [21:30:54] oh [21:31:02] Zppix: maybe yarn has a way to grant extra cash/credits in Farmville ? :] [21:31:09] for WMF development, I dont know [21:31:23] hashar i dont know let me ask my kitten [21:31:25] it is up to the developers I guess. Some people experimented with it but I am not aware of any effort to switch [21:31:49] hashar you can now delete abandoned changes and merged changes in gerrit. [21:31:58] hashar i've not done any research into yarn really, npm I don't use a whole lot [21:32:05] paladox: that is scary :] [21:32:08] As long as your an admin, i've created this https://gerrit-review.googlesource.com/#/c/97062/ [21:32:18] to allow owners too [21:32:26] paladox :O that should help clear up some DB getting rid of the uneeded changes on gerrit [21:32:38] hashar it is safe i think. As google added it. [21:33:18] hashar and even if it had issues its not like someone could go in and possible revert the damage i mean we do backups no? [21:33:22] Zppix, yep but users should only do it on changes that are mistake patches as in merging in the wrong branch or a cherry pic [21:33:23] pick [21:33:34] paladox hence "uneeded changes" [21:34:10] Yep, They also migrated external id's to a git repo now. [21:34:15] it just that editing history sounds scary overall [21:34:22] but maybe it has some use case here and there [21:34:35] anyway I am off to sleep. good night [21:34:42] hashar they are also supporting deleting comments by rewriting history. (not implemented yet) [21:34:44] ok [21:35:32] 10Gerrit, 06Release-Engineering-Team, 07Zuul: Update zuul to upstream master - https://phabricator.wikimedia.org/T158243#3030985 (10Paladox) p:05Triage>03Normal Changing to normal as we need it to have a higher priority then the gerrit 2.14 upgrade task. As this will need to be done before then :) [21:35:41] have a good sleep hashar [21:37:01] 06Release-Engineering-Team (Deployment-Blockers), 13Patch-For-Review, 05Release: MW-1.29.0-wmf.12 deployment blockers - https://phabricator.wikimedia.org/T155527#3030988 (10aaron) [21:37:14] thank you Zppix! [21:37:28] Polygerrit will be exposed in the 2.14 update :) [21:37:47] though it will be broken for us, as polygerrit currently dosent work with prefix like /r/ [21:43:06] paladox don't take polygerrit's digity away [21:43:20] ^ sorry i couldn't resist [21:43:48] Zppix im not, i've been trying to get it to work on https://gerrit-new.wmflabs.org/q/ but i carn't + this is a known bug. [21:44:23] i was talking about it being exposed paladox :P [21:44:30] polygerrit shouldn't assume it operates out of the docroot [21:44:36] Stupid gerrit [21:44:40] oh. Yeh. [21:44:40] Dumb regression [21:44:50] RainSprinkles that's because it uses a html file now [21:44:53] and uses imports [21:45:06] ... whose bright idea was that [21:45:44] it's also because of some internal servet as the bower_components can be accessed at /r/ but it just dosent do that in polygerrit's ui. [21:45:51] Zppix googles [21:46:07] Gerrit is owned by google [21:46:48] paladox no wonder the default UI is quite crap and has no decent .css style [21:47:26] (i would redesign but I dont generally do design i'm no artist :P [21:47:28] ) [21:47:34] lol, Zppix that's why they are creating polygerrit. Though it isent even complete. Has no inline edit screen. You carn't access projects. [21:50:04] polygerrit sounds pretty functional, lets just push to prod now :P [21:52:51] Zppix now that will break everyones url. [21:53:31] paladox good point [21:53:44] As i said it has a bug in that it dosent support prefix in the url such as /r/. + you will have to wait till the 2.14 release as in there you can have both ui's. Such as old ui and polygerrit :) [21:58:32] paladox you realise i was just joking right. [21:58:38] yep [22:01:38] PROBLEM - Work requests waiting in Zuul Gearman server https://grafana.wikimedia.org/dashboard/db/zuul-gearman on contint1001 is CRITICAL: CRITICAL: Anomaly detected: 11 data above and 0 below the confidence bounds [22:02:16] oh hey, look at that! [22:02:38] PROBLEM - Work requests waiting in Zuul Gearman server https://grafana.wikimedia.org/dashboard/db/zuul-gearman on contint2001 is CRITICAL: CRITICAL: Anomaly detected: 10 data above and 0 below the confidence bounds [22:04:17] paladox discussion about schema fyi in #wikimedia-office [22:04:34] What about schema? [22:04:38] Zppix ^^ [22:08:04] paladox https://www.mediawiki.org/wiki/User:Brion_VIBBER/Compacting_the_revision_table_round_2 [22:08:26] oh [22:09:10] PROBLEM - Work requests waiting in Zuul Gearman server https://grafana.wikimedia.org/dashboard/db/zuul-gearman-panelId-20-fullscreen-var-check_window-30-from-now-30m-to-now on contint1001 is CRITICAL: CRITICAL: Anomaly detected: 15 data above and 0 below the confidence bounds [22:13:50] greg-g: hey, do you guys have a releng mailing list? I'd like to send you the container reqs doc services is working on so that you can chime in [22:14:09] ACKNOWLEDGEMENT - Work requests waiting in Zuul Gearman server https://grafana.wikimedia.org/dashboard/db/zuul-gearman-panelId-20-fullscreen-var-check_window-30-from-now-30m-to-now on contint1001 is CRITICAL: CRITICAL: Anomaly detected: 14 data above and 1 below the confidence bounds daniel_zahn just added - https://phabricator.wikimedia.org/T70113 [22:15:20] mobrovac qa@lists.wikimedia.org [22:15:26] qa != releng [22:15:55] Reedy according to the releng mw page thats the email mailing list [22:15:59] so i guess it will need updated [22:16:17] or i need to reread the page... [22:16:35] mobrovac sorry heres the correct email releng@lists.wikimedia.org [22:17:19] mobrovac: ^ that list goes to just releng team [22:17:34] kk gracias muchachos! [22:18:05] thcipriani someone should create a seprate mailing list to include a list with releng wmf staff as well as releng volunteers such as paladox and myself [22:18:35] Why not just use wikitech-l? [22:19:54] also, most use of the releng list is about meetings :) [22:21:04] individual topic lists or the engineering lists are what we use for broader communication. Either qa or wikitech-l depending on the content. [22:21:33] ah [22:21:40] wasnt aware [22:23:08] Is there a way to kick jenkins to merge something? [22:23:22] After a recheck and he gave a CR+2? [22:23:29] Without reapplying a CR+2 from a third party [22:23:36] Ask nicely [22:23:40] Or with a hammer [22:23:46] DO THE NEEDFUL JENKINS [22:25:02] Reedy you got to datacenter you go to the server jenkins files are on and you kick the server? [22:25:06] go* [22:25:23] Why go over there when I can get someone to do that for me? [22:25:36] Reedy because its more fun to do it yourself? [22:25:42] Or you could log in as jenkins. [22:25:45] It requires crossing the atlantic [22:25:49] Dealing with lol cbp [22:26:04] what is cbp? [22:26:10] border patrol [22:26:21] us customs and border patrol [22:26:26] they're making the news a lot atm [22:26:33] lol [22:26:38] Reedy its not really even there fault they are... [22:26:42] blame the oval office [22:26:58] Well, they weren't ever always the nicest people [22:27:21] Reedy but they are closing our borders too. [22:27:25] Reedy yeah but to be fair they are worried about screwing up and 9/11 happening again [22:27:34] By banning people from countries that haven't done anything? [22:27:36] Yeah, good job. [22:27:46] i mean our borders not the us. [22:27:52] Reedy thats not their decision thats the oval office guy [22:28:04] i would say president but thats defaces that title atm [22:28:05] Doesn't make it any less stupid [22:28:21] paladox: Our gov are fscking useless too [22:28:30] honestly if border patrol had a choice im sure they would just open up [22:28:34] lol [22:28:44] lol [22:28:57] The Americans have made it difficult for people for years [22:29:11] you think they like telling people whom have been raped threaten from death they cant come in [22:29:13] ? [22:29:22] Reedy all the govement can do is cut the money [22:29:38] "Hey, you want to come here, with your degrees, and get a high paid job and pay taxes? You're gonna have to go through 6+ months of visa application stuff where we may or may not let you come in" [22:29:50] Some people get off on some weird things [22:29:52] yea, it's usually the liberal open-borders kind of guy who picks a career in law enforcement [22:30:04] heh [22:30:07] Reedy get an irish passport. You get in everywhere with that. [22:30:16] But I'm not Irish? [22:30:26] Reedy trust me tsa and border patrol i'm sure they are all drunks right now they are all probably depressed because they see these mothers from other countries being barred by the president, because they come from a country thats not ours [22:30:27] mutante: Amusingly, it's often staffed by recent immigrants anyway... [22:30:28] Oh, do you have any irish grandparents? [22:30:36] Northern, not ROI [22:30:46] oh. /me does. [22:30:57] just go to germany get an EU passport problem solved [22:30:59] Zppix: They're giving Americans citizens, green card holders etc a hard enough time [22:31:00] or work at the UN [22:31:05] but i can already get in the us as i have a us passport. [22:31:36] Reedy why not take it up with boris johnson lol [22:31:43] Zppix: "just go to Germany, get an EU passport" what are you talking about [22:31:51] He knows how to do deals. [22:31:59] paladox: My MP is the brexit secretary [22:32:00] european union passports are allowed into the us [22:32:03] it's much harder to become German citizen than US citizen [22:32:17] Reedy my mp is the women that said theresa may never had children. [22:32:45] She has to deal with the farming stuff. [22:33:09] Reedy yours is dave daves (I think it's davis) [22:33:20] Aye [22:33:51] lolololol [22:33:55] well, "much" maybe wasnt right. but normally 8 years waiting [22:34:00] and German test [22:34:10] RECOVERY - Work requests waiting in Zuul Gearman server https://grafana.wikimedia.org/dashboard/db/zuul-gearman-panelId-20-fullscreen-var-check_window-30-from-now-30m-to-now on contint1001 is OK: OK: No anomaly detected [22:34:20] ^ yes :) [22:34:30] Ich bin ein Berliner [22:34:45] Reedy all they want is a hard brexit. But no one actually agreed to a hard brexit. [22:34:57] No one agreed on pretty much anything [22:35:09] It's questionable whether the country even agreed [22:35:20] hard calexit [22:35:38] mutante: Isn't that just an earthquake away? [22:35:46] lol [22:36:09] Reedy what's so funny is they managed to get the law through without needing any changes. Though it is still being processed in the lords chambers. [22:36:14] Lol [22:39:22] Reedy so dave davis is a mp for yorkshire. Well isen't there alot of farming there? Well she is going to have alot of work to do with farming. [22:39:57] He doesn't even live in a village [22:40:13] Hmm. apparently it is a village [22:40:18] I'm not sure I believe wikipedia [22:40:22] It has no amenities [22:40:25] lol [22:40:41] Wait theres mp's for villiges [22:43:48] Reedy i also live in a villege outside of my main town, though our mp covers towcester [22:46:19] when is the next Gerrit maintenance ?:) [22:47:24] RainbowSprinkles: ^ i ask because i see the "enable logstash" change again :) [22:47:41] We need a window [22:47:46] For that + ldap change [22:47:52] yep [22:48:00] ok, what does the ldap change do [22:48:38] jouncebot: find window [22:49:40] mutante it converts everyones username to lowercase. [22:50:07] Which would be a big change. But allows users to log in either with for example Mutante or mutante. Just and example. [22:50:38] Currently if your username is Mutante, then doing mutante wont work. But if your username is mutante you can then do Mutante. [22:50:39] :) [22:51:04] It's not a big change, should only be a minute or two of downtime for it :) [22:51:47] oh [22:51:49] :) [22:53:26] ok.. hmm [22:54:58] if downtime < 5 min then .. skip scheduling windows? dunno [22:56:11] Eh, probably give ourselves a 30 min window to be safe [22:56:20] Just in case things break and we gotta rollback [23:01:54] ok [23:20:06] Project beta-update-databases-eqiad build #15016: 04FAILURE in 5.1 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/15016/ [23:46:22] 10Gerrit, 07Jenkins: Jenkins complains about wrong version of oojs-ui - https://phabricator.wikimedia.org/T158257#3031546 (10Huji) [23:55:18] 10Continuous-Integration-Infrastructure, 10OOjs-UI: Jenkins complains about wrong version of oojs-ui - https://phabricator.wikimedia.org/T158257#3031632 (10Paladox) [23:55:25] 10Continuous-Integration-Infrastructure, 10OOjs-UI: Jenkins complains about wrong version of oojs-ui - https://phabricator.wikimedia.org/T158257#3031546 (10Reedy) Known issue. Upgrade to 0.19.2 isn't going well [23:57:14] 10Continuous-Integration-Infrastructure, 10OOjs-UI: Jenkins complains about wrong version of oojs-ui - https://phabricator.wikimedia.org/T158257#3031664 (10Paladox) @Reedy should priority be unbreak if it is breaking tests? [23:58:25] 10Continuous-Integration-Infrastructure, 10OOjs-UI: Jenkins complains about wrong version of oojs-ui - https://phabricator.wikimedia.org/T158257#3031546 (10Jdforrester-WMF) I'll merge this into T158141. [23:58:34] 10Continuous-Integration-Infrastructure, 10OOjs-UI: Jenkins complains about wrong version of oojs-ui - https://phabricator.wikimedia.org/T158257#3031671 (10Jdforrester-WMF)