[00:00:35] Project beta-scap-eqiad build #242760: 04FAILURE in 10 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/242760/ [00:11:46] Project beta-scap-eqiad build #242761: 04STILL FAILING in 9 min 49 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/242761/ [00:14:03] Krenair: Hmm, did your fiddle break beta-scap-eqiad? [00:14:59] very possibly [00:15:05] File "/usr/lib/python2.7/dist-packages/scap/opcache_manager.py", line 47, in invalidate_all [00:15:08] that stack trace does involve opcache_manager [00:18:58] think I missed a setting [00:22:58] Project beta-scap-eqiad build #242762: 04STILL FAILING in 9 min 40 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/242762/ [00:32:13] so hopefully the next one should be ok... [00:35:18] Project beta-scap-eqiad build #242763: 04STILL FAILING in 9 min 33 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/242763/ [00:38:23] Project beta-scap-eqiad build #242764: 04STILL FAILING in 1 min 38 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/242764/ [00:42:54] 10Continuous-Integration-Config, 10MediaWiki-Core-Testing, 10MediaWiki-ResourceLoader, 10Performance-Team: Run `maintenance/resources/manageForeignResources.php verify` as a test on MediaWiki core - https://phabricator.wikimedia.org/T203694 (10Krinkle) [00:43:00] 10Continuous-Integration-Config, 10MediaWiki-Core-Testing, 10MediaWiki-ResourceLoader, 10Performance-Team: Run `maintenance/resources/manageForeignResources.php verify` as a test on MediaWiki core - https://phabricator.wikimedia.org/T203694 (10Krinkle) Actual Jenkins integration still to be done later. [00:43:37] 10Beta-Cluster-Infrastructure, 10Performance-Team: Move XHGui from tungsten to webperf-002 - https://phabricator.wikimedia.org/T180761 (10Krinkle) [00:45:40] stat("/etc/dsh/group/api_appserver", 0x7fffa0afc400) = -1 ENOENT (No such file or directory) [00:46:54] root@deployment-deploy01:/srv/mediawiki-staging# ls /etc/dsh/group [00:46:54] all librenms mediawiki-api-canaries mediawiki-appserver-canaries mediawiki-installation restbase-betacluster scap-masters scap-proxies webperf [00:47:59] looks like mediawiki-installation is what we want [00:48:02] probably [00:48:38] * Krenair can't help but feel there is some lurking prod-beta difference here [00:49:17] Project beta-scap-eqiad build #242765: 04STILL FAILING in 4 min 33 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/242765/ [00:51:30] well it should be slightly less upset now I think [00:51:33] probably [00:52:16] but in the installation list is deploy01, deploy02, jobrunner03, mediawiki-07, mediawiki-09, mwmaint01, snapshot01 [00:52:36] it failed on deploy01, deploy02, mwmaint01 and snapshot01 [00:52:59] so basically only worked on jobrunners and the main appservers [00:53:28] which, looking at the patch, may be logical [00:53:43] wonder what those dsh groups are supposed to be created by [00:59:13] looks like there's conftool in the process [00:59:26] IIRC I got that sort of working in beta at one point, I wonder if it still works today [01:03:06] (03PS1) 10Krinkle: Allow +2er to amend a patch set before merging. [core] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/499030 [01:03:50] greg-g: I have to proposed config changes for MW core that restore previously available abilities I think are stagnating productivity. Would appreciate feedback :) [01:03:53] two* [01:04:04] Krinkle: we could just change that to project owner [01:04:11] And do it in All-projects [01:04:14] paladox: Is already, read commit message. [01:04:17] Yippee, build fixed! [01:04:17] Project beta-scap-eqiad build #242766: 09FIXED in 9 min 56 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/242766/ [01:04:26] Oh ok, sorry. [01:05:03] Ah I see [01:09:32] James_F, it is succeeding now [01:32:18] (03PS3) 10Krinkle: Allow +2'ers to set assignee and amend patch sets [core] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/498739 [01:32:28] (03Abandoned) 10Krinkle: Allow +2er to amend a patch set before merging. [core] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/499030 (owner: 10Krinkle) [01:40:24] (03Abandoned) 10Krinkle: Allow +2'ers to set assignee and amend patch sets [core] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/498739 (owner: 10Krinkle) [02:07:14] Krinkle: I'm entirely too sick to reason right now, sorry [03:21:08] PROBLEM - Puppet staleness on deployment-elastic06 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [43200.0] [03:38:53] greg-g: Tyler's made the magic happen. All good now :) [05:37:19] Krinkle: can you file a ticket for the ops/puppet thing? [05:37:30] in codesearch [05:37:39] it has a special setup that totally could have broken [07:28:42] 10Phabricator-Bot-Requests, 10WMSE-Bug-Reporting-and-Translation-2019: Show active projects first on subproject page, then archived projects - https://phabricator.wikimedia.org/T218041 (10Sebastian_Berlin-WMSE) That was quick. Thanks, @mmodell. [07:53:27] PROBLEM - Host integration-slave-docker-1046 is DOWN: CRITICAL - Host Unreachable (172.16.1.115) [09:23:28] 10Project-Admins: Create project ISA - https://phabricator.wikimedia.org/T219167 (10SandraF_WMF) I can vouch for this! It's a piece of software which will be developed as a #structureddataoncommons #glam pilot project. We have just kicked off development and will also document it on wiki soon. [09:35:03] (03CR) 10Hashar: [C: 03+2] "I have updated the job. It would be great to figure out whether the tests can be run in parallel or whether we might need more power to r" [integration/config] - 10https://gerrit.wikimedia.org/r/496843 (owner: 10Smalyshev) [09:37:18] (03Merged) 10jenkins-bot: Bump Blazegraph time limit again [integration/config] - 10https://gerrit.wikimedia.org/r/496843 (owner: 10Smalyshev) [09:47:28] 10Continuous-Integration-Config, 10Operations, 10Operations-Software-Development, 10Patch-For-Review: Puppet tox: properly lint both Py2 and Py3 files - https://phabricator.wikimedia.org/T184435 (10fgiunchedi) >>! In T184435#5051826, @Volans wrote: > I think we could do some test of the real impact of migr... [09:53:10] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10releng-201718-q3, 10Epic, and 2 others: [EPIC] Migrate Mediawiki jobs from Nodepool to Docker - https://phabricator.wikimedia.org/T183512 (10hashar) [10:13:24] 10Release-Engineering-Team (Kanban), 10Quibble, 10Patch-For-Review: Quibble space separated options shallow arguments - https://phabricator.wikimedia.org/T218357 (10hashar) The root cause is we have multiple arguments with `args='*'`: `--run`, `--skip` ============= Used to specify stages, since we know th... [10:21:42] 10Project-Admins: Create project ISA - https://phabricator.wikimedia.org/T219167 (10Peachey88) Is ISA the name of the tool or Information Structured Additions? My main concern would be that ISA wouldn't be a [[ https://www.mediawiki.org/wiki/Phabricator/Creating_and_renaming_projects#Good_practices_for_name_and_... [10:46:07] RECOVERY - Puppet staleness on deployment-elastic06 is OK: OK: Less than 1.00% above the threshold [3600.0] [10:46:34] 10Gerrit, 10Phabricator, 10Wikimedia-Logstash, 10wikitech.wikimedia.org, 10Security: Include Gerrit apache logs in Logstash - https://phabricator.wikimedia.org/T219271 (10fgiunchedi) [10:50:05] hello folks [10:50:34] I added a new scap repository via https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/499146/ [10:50:47] everything went fine on deploy1001 and stat1007 (the target) [10:50:52] but on deploy2001 I get [10:51:02] Error: Execution of '/usr/bin/scap deploy --init' returned 70: 10:47:19 deploy failed: Failed to acquire lock "/var/lock/scap-global-lock"; owner is "root"; reason is "Not the active deployment server, use deploy1001.eqiad.wmnet [10:51:13] that makes sense, but puppet fails :) [10:51:21] is there a way to circumvent this? [10:51:33] Cc: thcipriani|afk --^ [10:58:51] elukey: remove the global lock? generally we don't allow deploying except from the active deployment server [10:59:04] I think puppet will put the lock file back if you remove it [10:59:46] twentyafterfour: hello! Yeah I didn't want to take actions without knowing what I was doing, this is why I am asking :) [11:00:57] # Lock the passive servers, leave untouched the active one. [11:00:57] file { '/var/lock/scap-global-lock': [11:00:57] ensure => 'present', [11:00:57] owner => 'root', [11:00:57] group => 'root', [11:00:59] content => "Not the active deployment server, use ${main_deployment_server}", [11:01:02] } [11:01:05] yep :) [11:01:07] twentyafterfour: --^ [11:01:38] right, so if you just needed to do a quick deploy I suppose you could remove the lock and deploy before puppet replaces it [11:01:49] but why do you need to deploy from there? [11:02:16] nono I just need to make a puppet run happy [11:02:36] the Error above is emitted by puppet, I think while trying to do scap init of the new repo [11:02:39] on deploy2001 [11:02:52] so I only need to remove the lock and run puppet IIUC [11:03:10] ohhh [11:03:27] yeah that sounds right. and also that sounds like a bug [11:03:42] I'll make a task [11:05:09] wait ... shouldn't deploying from deploy1001 sync the repo to deploy2001 automatically? I think that's what's supposed to happen [11:05:10] confirmed that now puppet runs fine and the lock is restored :) [11:05:12] thanks for the task! [11:05:32] no idea about the sync [11:05:55] maybe I didn't follow the correct procedure [11:06:14] but I have ran puppet on deploy1001, and it took a couple of runs to work for some failures [11:06:34] then deploy2001 kept erroring (puppet I mean) [11:07:53] 10Scap: scap global lock on secondary deployment server prevents puppet from initializing a deployment - https://phabricator.wikimedia.org/T219272 (10mmodell) p:05Triage→03Normal [11:08:07] 10Scap: scap global lock on secondary deployment server prevents puppet from initializing a deployment - https://phabricator.wikimedia.org/T219272 (10mmodell) @thcipriani: any idea what we're missing? [11:08:11] https://phabricator.wikimedia.org/T219272 [11:09:04] elukey: it seems like you did everything right, but maybe I'm confused. [11:09:16] confused / forgetting something [11:11:49] 10Scap: scap global lock on secondary deployment server prevents puppet from initializing a deployment - https://phabricator.wikimedia.org/T219272 (10elukey) To add more info, this was caused by a new scap repo that I have added via: * https://gerrit.wikimedia.org/r/#/c/499146/ * https://gerrit.wikimedia.org/r/... [11:11:58] added info to --^ [11:12:03] thanks for the help twentyafterfour ! [11:12:29] elukey: you're welcome [11:25:57] 10Project-Admins: Create project ISA - https://phabricator.wikimedia.org/T219167 (10SandraF_WMF) Isa (or ISA) is its name indeed. It's going to be a tagging tool, and as @Eugene233 said, the word isa means 'to put' in the Shona language. If it is too short, perhaps the project's name can be #Isa-Tool or something? [11:26:59] 10Phabricator-Bot-Requests, 10WMSE-Bug-Reporting-and-Translation-2019: Show active projects first on subproject page, then archived projects - https://phabricator.wikimedia.org/T218041 (10Aklapper) How is this implemented (link?), and would this be something to propose to upstream Phabricator developers? [11:43:57] https://phabricator.wikimedia.org/T219087 - would appreciate some input on doing the master swap [12:01:51] maybe twentyafterfour ? [12:37:26] 10Gerrit, 10GitHub-Mirrors: Puppet repo not being updated on github - https://phabricator.wikimedia.org/T219264 (10MarcoAurelio) p:05Triage→03High Can someone with access check the `replication` gerrit plugin logs (if any) to see if there's any trace? Puppet is not the only repo that stopped being mirrored... [12:38:10] hashar / thcipriani|afk -> https://phabricator.wikimedia.org/T219264#5057389 ? [12:46:19] (03PS3) 10Hashar: Better arg handling [integration/quibble] - 10https://gerrit.wikimedia.org/r/496125 (https://phabricator.wikimedia.org/T218357) [12:46:55] (03CR) 10jerkins-bot: [V: 04-1] Better arg handling [integration/quibble] - 10https://gerrit.wikimedia.org/r/496125 (https://phabricator.wikimedia.org/T218357) (owner: 10Hashar) [12:46:57] 10Release-Engineering-Team (Kanban), 10Quibble, 10Patch-For-Review: Quibble space separated options shallow arguments - https://phabricator.wikimedia.org/T218357 (10hashar) a:03hashar [13:05:59] 10Gerrit, 10Phabricator, 10Wikimedia-Logstash, 10wikitech.wikimedia.org, and 2 others: Include Gerrit apache logs in Logstash - https://phabricator.wikimedia.org/T219271 (10fgiunchedi) Confirmed apache logs are making their way in logstash, searching for `program:"input-file-gerrit-apache2-*"` yields the e... [13:12:32] 10Gerrit, 10Phabricator, 10Wikimedia-Logstash, 10wikitech.wikimedia.org, and 2 others: Include Gerrit apache logs in Logstash - https://phabricator.wikimedia.org/T219271 (10fgiunchedi) 05Open→03Resolved a:03fgiunchedi [13:20:27] 10Gerrit, 10Phabricator, 10Wikimedia-Logstash, 10wikitech.wikimedia.org, 10Security: Include Phabricator apache access logs in Logstash - https://phabricator.wikimedia.org/T219278 (10fgiunchedi) [13:28:53] (03PS4) 10Hashar: Better arg handling [integration/quibble] - 10https://gerrit.wikimedia.org/r/496125 (https://phabricator.wikimedia.org/T218357) [13:36:03] (03PS5) 10Hashar: Better arg handling [integration/quibble] - 10https://gerrit.wikimedia.org/r/496125 (https://phabricator.wikimedia.org/T218357) [13:45:40] (03CR) 10Hashar: "This is ready for review. I have added some guidances/hints for reviewing :]" (035 comments) [integration/quibble] - 10https://gerrit.wikimedia.org/r/496125 (https://phabricator.wikimedia.org/T218357) (owner: 10Hashar) [13:51:43] 10Beta-Cluster-Infrastructure, 10cloud-services-team, 10Patch-For-Review: Ensure we are unlikely to have both deployment-prep DB instances hosted together again in future - https://phabricator.wikimedia.org/T219088 (10Krenair) 05Open→03Resolved [13:51:49] (03CR) 10Lars Wirzenius: [C: 03+1] "good work. I had a couple of nitpicks, but they're not worth preventing a merge. if you don't care to change the code to pacify my nitpick" (032 comments) [integration/quibble] - 10https://gerrit.wikimedia.org/r/496125 (https://phabricator.wikimedia.org/T218357) (owner: 10Hashar) [13:54:28] 10Gerrit, 10Upstream: Gerrit Change-Id should not be copied with "Change-Id: " prefix - https://phabricator.wikimedia.org/T195277 (10Ammarpad) 05Open→03Resolved a:03Ammarpad I think this has been resolved. Now copying change-id does exactly what it should. [13:55:06] 10Gerrit, 10Upstream: Gerrit Change-Id should not be copied with "Change-Id: " prefix - https://phabricator.wikimedia.org/T195277 (10Ammarpad) a:05Ammarpad→03None [13:56:58] 10Gerrit, 10Upstream: Gerrit Change-Id should not be copied with "Change-Id: " prefix - https://phabricator.wikimedia.org/T195277 (10Dvorapa) 05Resolved→03Open Not for me [14:04:20] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: 5 of the 15 prioritized repositories have at least 1 end-to-end test - https://phabricator.wikimedia.org/T206621 (10zeljkofilipin) [14:05:13] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: 5 of the 15 prioritized repositories have at least 1 end-to-end test - https://phabricator.wikimedia.org/T206621 (10zeljkofilipin) [14:05:15] 10Release-Engineering-Team (Kanban), 10TimedMediaHandler, 10Patch-For-Review, 10User-zeljkofilipin: The first Selenium test for TimedMediaHandler - https://phabricator.wikimedia.org/T214480 (10zeljkofilipin) 05Open→03Stalled No activity in over a month. [14:05:23] 10Release-Engineering-Team (Kanban), 10TimedMediaHandler, 10Patch-For-Review: The first Selenium test for TimedMediaHandler - https://phabricator.wikimedia.org/T214480 (10zeljkofilipin) [14:07:00] 10Gerrit, 10GitHub-Mirrors: Puppet repo not being updated on github - https://phabricator.wikimedia.org/T219264 (10hashar) From `gerrit show-queue --wide --by-queue` ` Queue: ReplicateTo-slaves 88880f41 13:49:15.987 [a88b933a] push gerrit2@gerrit2001.wikimedia.org:/srv/gerrit/git/operations/... [14:18:31] PROBLEM - Work requests waiting in Zuul Gearman server on contint1001 is CRITICAL: CRITICAL: 35.71% of data above the critical threshold [140.0] https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1 [14:30:17] 10Gerrit, 10GitHub-Mirrors: Puppet repo not being updated on github - https://phabricator.wikimedia.org/T219264 (10hashar) I tried to manually force the replication for a repository at 14:04:22 UTC with: ssh -p 29418 gerrit.wikimedia.org replication start operations/puppet --now --wait But nothing shows... [14:35:33] o/ been waiting a bit for a service tag to trigger a build [14:35:37] usually it happens within 5 minutes [14:36:23] RECOVERY - Work requests waiting in Zuul Gearman server on contint1001 is OK: OK: Less than 30.00% above the threshold [90.0] https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1 [14:36:54] actually...where is my tag? does gerrit have a limit to the number of tags it can hold? ?? [14:37:08] a fresh clone of the repo doesn't have the tag...but there surely were more tags before [14:37:31] e.g. 1.0.8-wmf0 [14:37:36] triggered https://integration.wikimedia.org/ci/blue/organizations/jenkins/trigger-service-pipeline-test-and-publish/detail/trigger-service-pipeline-test-and-publish/89/pipeline [14:37:50] but gerrit doesn't show any tags past v1.0.4 here? [14:37:51] https://gerrit.wikimedia.org/r/#/admin/projects/eventgate-ci,tags [14:38:19] Nope [14:38:26] Gerrit can have unlimited tags [14:38:40] i would expect so, where'd my tags go? [14:38:43] :) [14:39:15] I pusehd a bunch last week that triggered builds: [14:39:17] https://integration.wikimedia.org/ci/job/trigger-service-pipeline-test-and-publish/ [14:39:25] you can see them in the job history [14:43:00] So you have tags missing? [14:45:48] it seems so [14:45:56] and the ones i just pushed don't show up either [14:46:01] so they are not triggering a build [14:46:16] (03Abandoned) 10Zfilipin: Do not run composer in WikibaseLexeme's node selenium job [integration/config] - 10https://gerrit.wikimedia.org/r/434011 (https://phabricator.wikimedia.org/T191537) (owner: 10WMDE-leszek) [14:46:27] i just cloned the repo fresh and fetch --all, but only tags up to v1.0.4-wmf1 [14:46:31] nothing more cent [14:46:39] i should see up to v1.0.9-wmf1 [14:47:11] I think I may know why [14:47:25] Not 100% sure but may explain it [14:48:05] (03CR) 10Zfilipin: [C: 03+1] "Is this still valid? Should it be abandoned?" [integration/config] - 10https://gerrit.wikimedia.org/r/441178 (owner: 10WMDE-leszek) [14:49:25] 10Continuous-Integration-Config, 10Code-Health-Metrics, 10User-zeljkofilipin: Report results from SonarCloud to Gerrit - https://phabricator.wikimedia.org/T217008 (10kostajh) Real-world POST from the SonarCloud webhook: ` json { "serverUrl": "https://sonarcloud.io", "taskId": "AWm6bF2mtK8xldclsL0c", "... [14:50:32] gehel: dcausse hello java folks :] We got some issue with gerrit and trying to figure out which threads are eating the cpu :] [14:50:52] but htop / ps don't give any meaningful thread name, nor does java melody has any instance cpu usage per thread [14:50:56] so we are hmm kind of lost [14:51:13] paladox: oh? [14:51:46] get a few thread dumps at regular intervals and upload to https://fastthread.io/ [14:51:51] hashar: index? [14:52:00] *reindex [14:52:03] gehel: ;) [14:52:07] ottomata: yup [14:52:22] paladox: yeah there is an index going on then the index queue only has 1 worker and all 16 cpus are burning .. [14:56:10] paladox: is it a secret? :p [14:56:14] hashar: let me know if you have the thread dumps somewhere, I can have a look [14:56:45] PROBLEM - Work requests waiting in Zuul Gearman server on contint1001 is CRITICAL: CRITICAL: 57.14% of data above the critical threshold [140.0] https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1 [14:56:53] gehel: taking traces :] [14:57:29] https://gerrit.wikimedia.org/r/monitoring?part=threadsDump is great for dummies [14:58:49] thcipriani: https://fastthread.io/ft-thread-report.jsp?dumpId=1&oTxnId_value=fd61da60-e5d5-4402-8c71-d5757052db48 ;D [14:58:52] gehel: https://fastthread.io/ft-thread-report.jsp?dumpId=1&oTxnId_value=fd61da60-e5d5-4402-8c71-d5757052db48 ;) [14:59:17] hrm [14:59:19] hashar: I get an error on that link [14:59:40] but! [14:59:42] it works for me ;] [14:59:57] I also get an error on that link [15:00:04] grmblblblbl [15:01:08] ah i gotta share it [15:01:09] thcipriani: gehel https://fastthread.io/my-thread-report.jsp?p=c2hhcmVkLzIwMTkvMDMvMjYvLS1nZXJyaXRkdW1wLnR4dC0tMTQtNTctMzI= [15:02:38] and a second report https://fastthread.io/my-thread-report.jsp?p=c2hhcmVkLzIwMTkvMDMvMjYvLS1nZXJyaXRkdW1wMi50eHQtLTE1LTEtNTQ= [15:06:18] hashar: can you upload all the dumps to the same report? [15:06:52] yeah will do a new one [15:07:56] (03CR) 10Lucas Werkmeister (WMDE): "> Patch Set 1:" [integration/config] - 10https://gerrit.wikimedia.org/r/494466 (owner: 10Lucas Werkmeister (WMDE)) [15:08:02] 10Gerrit, 10Release-Engineering-Team, 10Release Pipeline: Missing Gerrit tags / service-pipeline-test-and-publish not triggering - https://phabricator.wikimedia.org/T219286 (10Ottomata) [15:08:30] 10Gerrit, 10Release-Engineering-Team, 10Release Pipeline: Missing Gerrit tags / service-pipeline-test-and-publish not triggering - https://phabricator.wikimedia.org/T219286 (10Ottomata) [15:08:32] gehel: with 3 thread dumps https://fastthread.io/my-thread-report.jsp?p=c2hhcmVkLzIwMTkvMDMvMjYvLS1kdW1wcy56aXAtLTE1LTYtNDA= [15:11:41] hashar: looks like most threads are waiting or parked [15:12:02] and the once actually consuming are somewhere in jgit [15:12:18] looks like mostly uploads [15:12:33] maybe someone is uploading huge files? [15:13:01] yup that is what I am thinking about [15:13:35] gehel: thank you for fastthread ;) [15:13:58] the "Most used methods" is probably what's interesting in this case [15:14:05] or the "CPU consuming threads" [15:14:49] in both cases, it points to jgit, with one lucene thread as well [15:15:30] now I have no idea if that's the normal profile for gerrit or not :) [15:16:15] hashar: not sure if that's enough to point you on the right track or if you want me to dig further [15:16:49] I noticed the consuming threads in the reports [15:17:10] and yeaht that sounds like http queries that cause some magic t o happen related to jgit and git pack files [15:17:14] will dig that a bit more [15:17:18] gehel: thank you very much [15:17:23] (and indexing is crazy as usual) [15:18:04] looks like a git push to me [15:20:16] 10Continuous-Integration-Config, 10MediaWiki-Core-Testing, 10Patch-For-Review: Update selenium-daily-beta-* jobs to node10/npm6 - https://phabricator.wikimedia.org/T217545 (10zeljkofilipin) [15:20:21] hrm, the jgit packfile being the most used method aside from sleep and wait is interesting. [15:21:30] we've disabled git gc due to a jgit problem https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/493963/ [15:21:59] I wonder if a lot of this is due to jGit juggling various packfiles? [15:22:24] since gc hasn't consolidated in a bit [15:22:25] so progress [15:22:58] sudo -u gerrit2 jstack 11240 [15:23:01] to get a threaddump [15:23:09] each thread has a nid=0xABCD [15:23:13] where ABCD is the thread id [15:23:26] that matches the linux identifier :] [15:24:09] anything new and interesting from that? [15:25:50] so [15:25:56] Index-Batch-1 takes cpu [15:25:59] yeah that is obvious [15:26:43] yep, should be limited to a single thread using up a ton of cpu [15:27:59] now I cant see them anymore bah [15:28:27] hashar: yeh, I thought we disabled auto reindeer [15:28:32] *reindex [15:29:14] ReplicateTo-slaves-1 is another [15:29:34] that is for the long running ones [15:30:55] and all the ones I have seen are HTTP requests [15:31:02] hitting jgit [15:32:23] auto reindeer... bash-worthy [15:33:45] :D [15:34:05] gehel: thank you very much. That comes frmo bunch of http queries apparently [15:35:47] Also apergos lol [15:35:58] Ignore the also [15:36:12] :-D [15:36:43] ottomata: atm yes I think [15:37:55] thcipriani: you could be into something [15:38:30] ah k [15:38:32] danke [15:54:41] Well actually if it’s reindexing, it would be reading the repos which are notedb thcipriani hashar [16:10:30] PROBLEM - Host deployment-db03 is DOWN: CRITICAL - Host Unreachable (172.16.5.23) [16:18:00] PROBLEM - Work requests waiting in Zuul Gearman server on contint1001 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [140.0] https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1 [16:18:23] anyone willing to help me with a beta DB master change later? [16:19:08] 10Phabricator (Upstream), 10Upstream: Bots should not subscribe themselves to tasks - https://phabricator.wikimedia.org/T137508 (10epriestley) After the 2019 Week 12 upstream release, bots no longer subscribe themselves to objects automatically as a side effect of interaction. They can still subscribe themselv... [16:22:43] 10Phabricator, 10Phabricator (Upstream), 10Upstream: Feature request: filter task's bot CCs to the end of the ellipsis - https://phabricator.wikimedia.org/T115742 (10epriestley) After the 2019 Week 12 release, bots no longer automatically subscribe themselves to objects as a side effect of interaction (see T... [16:23:57] 10Phabricator (Upstream), 10Upstream: Task rename notifications are hard to understand due to missing markup for old and new title - https://phabricator.wikimedia.org/T166358 (10epriestley) I believe this will be resolved by the 2019 Week 13 upstream release. See for par... [16:27:58] 10Phabricator (Upstream), 10Upstream: Inactive diffusion repos should display differently - https://phabricator.wikimedia.org/T182232 (10epriestley) I believe this will be resolved by the 2019 Week 13 upstream release, which renders repository references like `rXYZ` as "closed" (with a strikethrough) if the re... [16:28:06] Do web have some problems with Gerrit to GitHub replication? [16:28:09] *we [16:29:04] known, sadly [16:29:16] still be investigated [16:29:18] being [16:30:01] thx greg-g [16:30:07] is there a Phab task I can watch? [16:30:22] not sure, honestly [16:30:53] 10Phabricator (Upstream), 10Legalpad, 10Upstream: Legalpad access controls are confusing and seemingly broken - https://phabricator.wikimedia.org/T208254 (10epriestley) I've applied a substantially similar patch to the upstream in the upcoming 2019 Week 13 release, see ... [16:31:39] Urbanecm: https://phabricator.wikimedia.org/T219264 [16:33:42] 10Phabricator (Upstream), 10Mobile, 10Upstream: Task dependency tree is unusable in mobile devices - https://phabricator.wikimedia.org/T210482 (10epriestley) After the upcoming 2019 Week 13 release, the task graph itself (the colored line drawing) is hidden by default on devices with smaller screens, so the... [16:37:51] RECOVERY - Work requests waiting in Zuul Gearman server on contint1001 is OK: OK: Less than 30.00% above the threshold [90.0] https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1 [16:40:09] 10Phabricator (Upstream), 10Upstream: Display of subtasks in graphs for large tree is unusable as it cuts off task names or task IDs - https://phabricator.wikimedia.org/T171648 (10epriestley) Upstream changes with some impact here: - After 2019 Week 13, the actual graph (the colored line drawing) is hidden... [16:40:14] thanks greg-g [17:05:59] (03PS1) 10Paladox: Modify access rules [All-Users] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/499253 [17:06:06] twentyafterfour thcipriani ^^ [17:06:55] (03CR) 1020after4: "Maybe just on external-ids?" [All-Users] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/499253 (owner: 10Paladox) [17:07:52] huh https://gerrit.wikimedia.org/r/changes/All-Users~499253/edit/project.config 500 error [17:08:19] twentyafterfour would you be able to check the logs for the 500 error please? [17:12:56] (03PS1) 10WMDE-Fisch: Remove BetaFeatures req from AdvancedSearch [integration/config] - 10https://gerrit.wikimedia.org/r/499257 [17:14:09] (03PS1) 10Paladox: Modify access rules [All-Users] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/499258 [17:14:16] (03Abandoned) 10Paladox: Modify access rules [All-Users] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/499253 (owner: 10Paladox) [17:14:22] twentyafterfour https://gerrit.wikimedia.org/r/#/c/All-Users/+/499258/ [17:25:19] (03PS1) 10Krinkle: Revert Flow js documentation back from node10 to jsduck [integration/config] - 10https://gerrit.wikimedia.org/r/499262 (https://phabricator.wikimedia.org/T215562) [17:25:59] (03CR) 10Krinkle: "Still pending word from upstream npm. Ruled out a bunch of things that /didn't/ cause it, but no closer it feels like to solving it, so le" [integration/config] - 10https://gerrit.wikimedia.org/r/499262 (https://phabricator.wikimedia.org/T215562) (owner: 10Krinkle) [17:31:04] 10Gerrit, 10Release-Engineering-Team, 10Release Pipeline: Missing Gerrit tags / service-pipeline-test-and-publish not triggering - https://phabricator.wikimedia.org/T219286 (10Ottomata) There apparently was a recent gerrit ACL change that affected the way tags were displayed. I was able to see tags after pu... [17:31:05] (03CR) 10Jforrester: "Also TemplateData. :-(" [integration/config] - 10https://gerrit.wikimedia.org/r/499262 (https://phabricator.wikimedia.org/T215562) (owner: 10Krinkle) [17:31:18] 10Gerrit, 10Release-Engineering-Team, 10Release Pipeline: Missing Gerrit tags / service-pipeline-test-and-publish not triggering - https://phabricator.wikimedia.org/T219286 (10Ottomata) 05Open→03Resolved a:03Ottomata [17:36:58] 10Continuous-Integration-Config, 10Code-Health-Metrics, 10User-zeljkofilipin: Report results from SonarCloud to Gerrit - https://phabricator.wikimedia.org/T217008 (10kostajh) a:03kostajh [17:37:20] (03CR) 10Krinkle: "Link?" [integration/config] - 10https://gerrit.wikimedia.org/r/499262 (https://phabricator.wikimedia.org/T215562) (owner: 10Krinkle) [17:37:22] (03CR) 10Krinkle: [C: 03+2] Revert Flow js documentation back from node10 to jsduck [integration/config] - 10https://gerrit.wikimedia.org/r/499262 (https://phabricator.wikimedia.org/T215562) (owner: 10Krinkle) [17:40:46] (03Merged) 10jenkins-bot: Revert Flow js documentation back from node10 to jsduck [integration/config] - 10https://gerrit.wikimedia.org/r/499262 (https://phabricator.wikimedia.org/T215562) (owner: 10Krinkle) [17:42:42] !log Reloading Zuul to deploy merged-but-not-deployed patches https://gerrit.wikimedia.org/r/#/c/integration/config/+/497800/ and https://gerrit.wikimedia.org/r/#/c/498684/ [17:42:43] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [17:43:04] !log Reloading Zuul to deploy https://gerrit.wikimedia.org/r/499262 [17:43:04] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [17:52:08] 10Project-Admins, 10Wikimedia-Technical-Conference-2018: Create "MediaWiki-Decoupling" phabricator project - https://phabricator.wikimedia.org/T207976 (10Aklapper) Superseded by #cpt-decoupling [17:53:24] (03CR) 10Legoktm: "This will (I think) make older branches fail tests, which is probably fine. But I'd recommend waiting until the current wmf/ branch is fin" [integration/config] - 10https://gerrit.wikimedia.org/r/499257 (owner: 10WMDE-Fisch) [17:53:35] (03CR) 10Jforrester: "> Patch Set 1:" [integration/config] - 10https://gerrit.wikimedia.org/r/499262 (https://phabricator.wikimedia.org/T215562) (owner: 10Krinkle) [18:05:27] Project mediawiki-core-doxygen-docker build #5711: 04FAILURE in 1 min 22 sec: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen-docker/5711/ [18:05:33] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: 5 of the 15 prioritized repositories have at least 1 end-to-end test - https://phabricator.wikimedia.org/T206621 (10hashar) [18:15:40] 10Gerrit, 10Release-Engineering-Team, 10Release Pipeline: Missing Gerrit tags / service-pipeline-test-and-publish not triggering - https://phabricator.wikimedia.org/T219286 (10hashar) 05Resolved→03Open a:05Ottomata→03None That looks like a #gerrit server side error! There are three errors looking lik... [18:19:50] 10Continuous-Integration-Config, 10Growth-Team, 10StructuredDiscussions, 10Wikimedia-production-error (Shared Build Failure): Repeat mwext-node10-rundoc-docker failures for Flow - https://phabricator.wikimedia.org/T218978 (10Jdforrester-WMF) 05Open→03Resolved a:03Krinkle "Resolved" for now via https:... [18:23:43] 10Gerrit, 10Release-Engineering-Team, 10Release Pipeline: Missing Gerrit tags / service-pipeline-test-and-publish not triggering - https://phabricator.wikimedia.org/T219286 (10hashar) 05Open→03Resolved So there have been some kind of lock failure when receiving the tag for `refs/tags/v1.0.9-wmf0`. `phabr... [18:51:34] 10Continuous-Integration-Config, 10MediaWiki-General-or-Unknown, 10User-zeljkofilipin: `npm install` fails for mediawiki/core with EPEERINVALID when running on Node 11 - https://phabricator.wikimedia.org/T210506 (10Jdforrester-WMF) [18:55:09] (03CR) 1020after4: [C: 03+1] "thcipriani: do you object?" [All-Users] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/499258 (owner: 10Paladox) [18:57:54] 10Phabricator-Bot-Requests, 10WMSE-Bug-Reporting-and-Translation-2019: Show active projects first on subproject page, then archived projects - https://phabricator.wikimedia.org/T218041 (10mmodell) [18:58:57] 10Phabricator-Bot-Requests, 10WMSE-Bug-Reporting-and-Translation-2019: Show active projects first on subproject page, then archived projects - https://phabricator.wikimedia.org/T218041 (10mmodell) @aklapper: see {rPHAB2b3156af9a94} [19:02:58] (03CR) 10Thcipriani: [C: 04-1] "I don't think this is necessary/will solve your ability to push." [All-Users] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/499258 (owner: 10Paladox) [19:13:43] 10Phabricator (Upstream), 10Mobile, 10Upstream: Cannot type into assigned to, tags and subscribers fields when using Maxthon browser on mobile - https://phabricator.wikimedia.org/T209045 (10Aklapper) >>! In T209045#4737305, @Dvorapa wrote: > Filled, thank you for help! @Dvorapa: Did you receive any feedback? [19:15:33] Yippee, build fixed! [19:15:34] Project mediawiki-core-doxygen-docker build #5712: 09FIXED in 11 min: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen-docker/5712/ [19:21:08] 10Phabricator, 10Developer-Advocacy, 10Developer-Wishlist (2017), 10Goal: Consolidate the many tech events calendars in Phabricator's calendar - https://phabricator.wikimedia.org/T1035 (10Aklapper) [19:25:45] 10Gerrit, 10Release-Engineering-Team (Kanban), 10Scap, 10Patch-For-Review, 10User-zeljkofilipin: `scap clean` failure - https://phabricator.wikimedia.org/T218783 (10thcipriani) I think what's happening is that scap overwrites the environment `$SSH_AUTH_SOCK` with whatever is in `/etc/scap.cfg:ssh_auth_so... [19:50:58] 10Continuous-Integration-Infrastructure, 10MediaWiki-Documentation, 10Regression, 10Upstream: Doxygen search.php no longer works on doc.wikimedia.org - https://phabricator.wikimedia.org/T218233 (10epriestley) [19:53:54] 10Phabricator, 10Wikimedia-Logstash, 10Patch-For-Review, 10Security: Include Phabricator apache access logs in Logstash - https://phabricator.wikimedia.org/T219278 (10hashar) [19:54:35] 10Continuous-Integration-Config, 10Patch-For-Review, 10Upstream: npm 6 consistently fails with "Z_DATA_ERROR: invalid distance too far back" on some repos - https://phabricator.wikimedia.org/T215562 (10Krinkle) [19:54:45] 10Continuous-Integration-Config, 10Patch-For-Review, 10Upstream: npm 6 consistently fails with "Z_DATA_ERROR: invalid distance too far back" on some repos - https://phabricator.wikimedia.org/T215562 (10Krinkle) [19:57:20] (03PS1) 10Krinkle: Revert use of npm-run-doc on node10 for TemplateData and Wikibase [integration/config] - 10https://gerrit.wikimedia.org/r/499312 (https://phabricator.wikimedia.org/T215562) [19:57:27] (03CR) 10Krinkle: [C: 03+2] Revert use of npm-run-doc on node10 for TemplateData and Wikibase [integration/config] - 10https://gerrit.wikimedia.org/r/499312 (https://phabricator.wikimedia.org/T215562) (owner: 10Krinkle) [19:58:04] * James_F sobs in Krinkle's general direction. [19:59:59] (03Merged) 10jenkins-bot: Revert use of npm-run-doc on node10 for TemplateData and Wikibase [integration/config] - 10https://gerrit.wikimedia.org/r/499312 (https://phabricator.wikimedia.org/T215562) (owner: 10Krinkle) [20:00:57] !log Reloading Zuul to deploy https://gerrit.wikimedia.org/r/499312 [20:00:58] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [20:06:21] 10Gerrit, 10Release-Engineering-Team, 10Release Pipeline: Missing Gerrit tags / service-pipeline-test-and-publish not triggering - https://phabricator.wikimedia.org/T219286 (10Ottomata) I think @thcipriani has more context. [20:09:19] 10Gerrit, 10GitHub-Mirrors: Puppet repo not being updated on github - https://phabricator.wikimedia.org/T219264 (10thcipriani) >>! In T219264#5058049, @hashar wrote: > I tried to manually force the replication for a repository at 14:04:22 UTC with: > > ssh -p 29418 gerrit.wikimedia.org replication start o... [20:13:16] 10Phabricator (Upstream), 10Mobile, 10Upstream: Cannot type into assigned to, tags and subscribers fields when using Maxthon browser on mobile - https://phabricator.wikimedia.org/T209045 (10Dvorapa) Nope, I can try now. [20:19:48] 10Phabricator (Upstream), 10Mobile, 10Upstream: Cannot type into assigned to, tags and subscribers fields when using Maxthon browser on mobile - https://phabricator.wikimedia.org/T209045 (10Dvorapa) It is still not working, but the issue is only with the fields with magnifier icon/placeholder, everything els... [20:22:05] 10Phabricator (Upstream), 10Mobile, 10Upstream: Cannot type into assigned to, tags and subscribers fields when using Maxthon browser on mobile - https://phabricator.wikimedia.org/T209045 (10Dvorapa) Everytime I fill something into the fields, it is replaced by the placeholder. Isn't there some issue with the... [20:23:59] 10Phabricator (Upstream), 10Mobile, 10Upstream: Cannot type into assigned to, tags and subscribers fields when using Maxthon browser on mobile - https://phabricator.wikimedia.org/T209045 (10Aklapper) I did not write Phabricator. Maybe someone in their [forum](https://discourse.phabricator-community.org/) mig... [20:29:39] 10Phabricator (Upstream), 10Mobile, 10Upstream: Cannot type into assigned to, tags and subscribers fields when using Maxthon browser on mobile - https://phabricator.wikimedia.org/T209045 (10Dvorapa) I know. I filled also a task on Phab community (https://discourse.phabricator-community.org/t/can-not-type-int... [20:30:32] 10Gerrit, 10GitHub-Mirrors: Puppet repo not being updated on github - https://phabricator.wikimedia.org/T219264 (10Urbanecm) Rather just a random idea, but can somebody check the SSH auth between Gerrit machine and git@github.com still works? I encouradged an error a few days ago, when I was suddently unable t... [20:36:43] !log create gerrit repo operations/software/gerrit/plugins/WikimediaBlocks T219300 [20:36:44] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [21:05:24] PROBLEM - Host deployment-sessionstore01 is DOWN: CRITICAL - Host Unreachable (172.16.3.4) [21:20:21] 10Gerrit, 10GitHub-Mirrors: Puppet repo not being updated on github - https://phabricator.wikimedia.org/T219264 (10thcipriani) >>! In T219264#5059889, @Urbanecm wrote: > Rather just a random idea, but can somebody check the SSH auth between Gerrit machine and git@github.com still works? I encouradged an error... [21:21:40] 10Gerrit, 10GitHub-Mirrors: Puppet repo not being updated on github - https://phabricator.wikimedia.org/T219264 (10MarcoAurelio) https://github.com/wikimedia shows activity for some repos so if it were an authentication issue I think no repo would be updating, right? [21:24:13] 10Continuous-Integration-Config, 10Code-Health-Metrics, 10User-zeljkofilipin: Report results from SonarCloud to Gerrit - https://phabricator.wikimedia.org/T217008 (10kostajh) [21:25:33] 10Continuous-Integration-Config, 10Code-Health-Metrics, 10User-zeljkofilipin: Report results from SonarCloud to Gerrit - https://phabricator.wikimedia.org/T217008 (10kostajh) @mmodell could you possibly help with "Create a SonarQubeBot user in gerrit" and "Add SonarQubeBot user to stream-events group" from t... [21:26:18] thcipriani: wrt github not updating, could it be gerrit stuck in a task? [21:26:39] I'm reading https://wikitech.wikimedia.org/wiki/Gerrit#Tasks_management [21:28:13] hauskatze: there is a ReplicateTo-slaves queue in there, but I haven't seen it in there, so it's not stuck. I have seen it a bit in replication logs, I'm digging deeper into those now. [21:28:44] I dug into the github audit log and the puppet repo config at github but saw nothing of interest [21:34:01] hi folks! I'm trying to swap the fundraising (donationinterface) tests over to run on mediawiki 1.31 and php7 [21:34:31] (03PS1) 10Ejegg: DonationInterface tests run on PHP7 and MediaWiki 1.31 [integration/config] - 10https://gerrit.wikimedia.org/r/499334 [21:34:54] Does that look like it's heading in the right direction? ^^^ [21:36:43] (03CR) 10jerkins-bot: [V: 04-1] DonationInterface tests run on PHP7 and MediaWiki 1.31 [integration/config] - 10https://gerrit.wikimedia.org/r/499334 (owner: 10Ejegg) [21:36:50] ejegg: jerkins says no :P [21:36:57] heh [21:37:14] the replication plugin certainly takes a lot of time thcipriani [21:37:30] (03CR) 10Reedy: DonationInterface tests run on PHP7 and MediaWiki 1.31 (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/499334 (owner: 10Ejegg) [21:37:43] I did `$ ssh -p 29418 gerrit.wikimedia.org replication start operations/puppet --wait` but still no answer after a couple of minutes, when it normally takes seconds to say something [21:39:59] hauskatze: Replicate operations/puppet ref ..all.. to github.com, Succeeded! (OK) [21:40:40] wow, I'm still waiting for bash to say me something ;) [21:40:52] process killed [21:40:53] I think I figured it out [21:46:20] aha! -- now last puppet commit is from 15 hours ago [21:47:12] 10Gerrit, 10GitHub-Mirrors: Puppet repo not being updated on github - https://phabricator.wikimedia.org/T219264 (10thcipriani) 05Open→03Resolved a:03thcipriani Seems to be working now (manually at least) `(/^ヮ^)/*:・ﾟ✧ ssh -p 29418 gerrit.wikimedia.org -- replication start operations/puppet --wait 2019-0... [21:49:12] oh lol thcipriani, didn't realise the plugin uses that group [21:49:54] so it was mediawiki-replication after all [21:50:57] I have updated that group's description to something that does not begin with "[ARCHIVED]" so hopefully we can avoid making the same mistakes in the future. [21:51:23] so replication should work just fine again [21:52:00] that's my hope. It wasn't even attempting to replicate to git@github according to replication logs. [21:52:20] now it does so when you kick off replication [21:53:10] strange that some repos were updating? [21:53:37] that is strange, I may suspect phabricator was doing something? [21:53:59] according to the replication_log gerrit was not doing anything with github [21:54:05] oh yes [21:54:13] some repos on phab replicate to github [21:54:20] MF is one of them i think [21:54:35] 10Project-Admins: Create project ISA - https://phabricator.wikimedia.org/T219167 (10Aklapper) 05Open→03Resolved a:03Aklapper Requested public project #isa has been created: https://phabricator.wikimedia.org/project/view/3981/ Please encourage interested people to visit the project and to join the project... [21:54:46] https://phabricator.wikimedia.org/diffusion/EMFR/manage/uris/ [21:55:11] Some of them were set to replicate to github yep [21:55:26] although they all from gerrit replicate to github [21:55:37] that shouldn't be necessary anymore imho [21:59:32] (03PS1) 10Krinkle: Use php72 instead of php70 for more mw jobs where possible (1) [integration/config] - 10https://gerrit.wikimedia.org/r/499338 [22:12:11] thcipriani: last q hopefully - there are some repos that seems to be stalled. I guess they'll all get to be updated once a commit is merged on gerrit right? What about if we want to refresh them all so the mirror is actually a good mirror? I saw --all is NOT to be used though. [22:18:10] (03CR) 10Krinkle: [C: 03+2] "Reviewed the diff to confirm no conditionals or excludes or skips had unintended side-effects (e.g. the job no longer running or running m" [integration/config] - 10https://gerrit.wikimedia.org/r/499338 (owner: 10Krinkle) [22:19:56] (03PS1) 10Krinkle: Use php72 instead of php70 for more mw jobs where possible (2) [integration/config] - 10https://gerrit.wikimedia.org/r/499343 [22:20:07] (03Merged) 10jenkins-bot: Use php72 instead of php70 for more mw jobs where possible (1) [integration/config] - 10https://gerrit.wikimedia.org/r/499338 (owner: 10Krinkle) [22:27:06] (03CR) 10Krinkle: [C: 03+2] "Same. No removals or additions due to interaction with filters. All good." [integration/config] - 10https://gerrit.wikimedia.org/r/499343 (owner: 10Krinkle) [22:29:08] (03Merged) 10jenkins-bot: Use php72 instead of php70 for more mw jobs where possible (2) [integration/config] - 10https://gerrit.wikimedia.org/r/499343 (owner: 10Krinkle) [22:29:19] (03PS1) 10Krinkle: Use php72 instead of php70 for more mw jobs where possible (3) [integration/config] - 10https://gerrit.wikimedia.org/r/499344 [22:48:16] 10Gerrit, 10Release-Engineering-Team (Kanban): Bless Brennen with Gerrit administrator rights - https://phabricator.wikimedia.org/T218858 (10hashar) We have been doing some brain dumping. More discussions to happen :] [22:49:07] (03CR) 10Krinkle: [C: 03+2] "Same as others. No rms/adds, just change, except for mw/skin/Empty now having the job not skipped." [integration/config] - 10https://gerrit.wikimedia.org/r/499344 (owner: 10Krinkle) [22:50:27] paladox: still around? :] [22:50:31] yup [22:50:34] eventually I found out https://wikitech.wikimedia.org/wiki/User_talk:Paladox [22:50:44] which state you got shell access on WMCS on April 24th 2014 [22:50:57] so it is almost your five years anniversary! We should arrange some party of some sort :] [22:51:04] lol [22:51:06] (03Merged) 10jenkins-bot: Use php72 instead of php70 for more mw jobs where possible (3) [integration/config] - 10https://gerrit.wikimedia.org/r/499344 (owner: 10Krinkle) [22:51:57] congratulations paladox [22:52:15] thanks :) [22:52:23] paladox: and refresh that talk page :] [22:52:42] thanks hashar!! [22:53:08] ;]]]] [22:53:11] well deserved [22:53:30] :) [22:55:28] * hauskatze night [23:04:33] !log Reloading Zuul to deploy https://gerrit.wikimedia.org/r/499344 (part 1, 2 and 3) [23:04:34] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [23:15:00] 10Phabricator, 10Wikipedia-iOS-App-Backlog: Update Herald (H187) to include project 3890 - https://phabricator.wikimedia.org/T219236 (10MBinder_WMF) 05Open→03Resolved {meme, src="seal-of-approval", above=Lobsters, below="I enjoy them"} [23:17:36] 10Gerrit, 10Release-Engineering-Team, 10Patch-For-Review: Upgrade Gerrit to 2.15.12 - https://phabricator.wikimedia.org/T218515 (10Paladox) 05Open→03Resolved a:03thcipriani [23:27:48] 10Gerrit, 10GitHub-Mirrors: Puppet repo not being updated on github - https://phabricator.wikimedia.org/T219264 (10hashar) Seems that is working again indeed: ` [2019-03-26 21:37:14,820] [] scheduling replication operations/puppet:..all.. => git@github.com:wikimedia/operations-puppet ` @thcipriani can you po... [23:39:45] PROBLEM - Free space - all mounts on deployment-fluorine02 is CRITICAL: CRITICAL: deployment-prep.deployment-fluorine02.diskspace._srv.byte_percentfree (<20.00%) [23:48:12] paladox: fun thing, I had my WMCS access granted on my birthday :] [23:48:19] heh [23:48:33] hashar fun fact, the first gerrit release was on my birthday :) [23:48:41] ahah [23:49:44] gerrit 2.16 was the anniversary release too hashar