[00:01:28] running puppet, mysql client [00:02:13] maybe it is just cumin logging in [00:02:35] .bash_history has timestamps now? [00:02:51] meeting time [00:03:31] I run puppet across everything through cumin a lot so it might be that [00:03:35] in fact I have a cron that does that [00:03:47] 10Release-Engineering-Team (Next), 10MediaWiki-General-or-Unknown, 10MW-1.29-release: Formalise and Announce REL1_29 EOL - https://phabricator.wikimedia.org/T197669 (10Krinkle) For your attention: * * haven't been doing mysql clients from there though [00:26:08] TimStarling: I don't know [00:35:52] 10Continuous-Integration-Infrastructure: castor02 integration node /srv disk is full - https://phabricator.wikimedia.org/T202341 (10Legoktm) It's just a cache, we could just wipe it and let it naturally fill up again. [00:36:32] so interpreting the numeric timestamps in .bash_history, I think the answer is that nobody has touched them this year [00:37:10] 10Continuous-Integration-Infrastructure, 10Jenkins, 10Patch-For-Review: Jenkins should auto-depool nodes if they run out of disk space on specific partitions - https://phabricator.wikimedia.org/T201224 (10Legoktm) Awesome, will it alert us anywhere (e.g. IRC) that it depooled a node? Or will we just rely on... [01:13:13] !log triggering seccheck jobs manually on contint1001 [01:13:18] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [04:03:14] (03CR) 10Legoktm: [C: 032] Run seccheck for Score [integration/config] - 10https://gerrit.wikimedia.org/r/453975 (owner: 10Umherirrender) [04:04:45] (03Merged) 10jenkins-bot: Run seccheck for Score [integration/config] - 10https://gerrit.wikimedia.org/r/453975 (owner: 10Umherirrender) [04:05:18] !log deploying https://gerrit.wikimedia.org/r/453975 [04:05:23] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [04:18:51] (03PS1) 10Legoktm: Run seccheck for ReplaceText [integration/config] - 10https://gerrit.wikimedia.org/r/454192 [04:19:02] (03CR) 10Legoktm: [C: 032] Run seccheck for ReplaceText [integration/config] - 10https://gerrit.wikimedia.org/r/454192 (owner: 10Legoktm) [04:20:26] (03Merged) 10jenkins-bot: Run seccheck for ReplaceText [integration/config] - 10https://gerrit.wikimedia.org/r/454192 (owner: 10Legoktm) [04:20:45] !log deployed https://gerrit.wikimedia.org/r/454192 [04:20:49] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [04:47:21] sigh [04:47:26] lost power right after I deployed that [07:20:22] legoktm: lame :P [07:42:47] (03PS1) 10Legoktm: Jobs for MusicBrainzAuth [integration/config] - 10https://gerrit.wikimedia.org/r/454201 [07:43:25] (03CR) 10Legoktm: [C: 032] Jobs for MusicBrainzAuth [integration/config] - 10https://gerrit.wikimedia.org/r/454201 (owner: 10Legoktm) [07:45:53] (03Merged) 10jenkins-bot: Jobs for MusicBrainzAuth [integration/config] - 10https://gerrit.wikimedia.org/r/454201 (owner: 10Legoktm) [07:46:13] !log deployed https://gerrit.wikimedia.org/r/454201 [07:46:18] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [07:52:28] (03PS1) 10Umherirrender: Make seccheck voting for AbuseFilter [integration/config] - 10https://gerrit.wikimedia.org/r/454202 (https://phabricator.wikimedia.org/T197002) [08:06:05] (03CR) 10Hashar: [C: 032] Remove unused mediawiki-phpunit-* jobs [integration/config] - 10https://gerrit.wikimedia.org/r/454154 (https://phabricator.wikimedia.org/T183512) (owner: 10Hashar) [08:07:56] https://usercontent.irccloud-cdn.com/file/eXTGpwr1/image.png [08:08:01] hashar: ^^ migration nearly complete ;) [08:08:58] (03Merged) 10jenkins-bot: Remove unused mediawiki-phpunit-* jobs [integration/config] - 10https://gerrit.wikimedia.org/r/454154 (https://phabricator.wikimedia.org/T183512) (owner: 10Hashar) [08:09:17] !log castor: removing cache for mediawiki-extensions-* jobs (no more used) rm -fR /srv/jenkins-workspace/caches/*/*/mediawiki-extensions* | T202341 [08:09:23] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [08:09:23] T202341: castor02 integration node /srv disk is full - https://phabricator.wikimedia.org/T202341 [08:09:35] addshore: yeah somehow magic :] [08:09:40] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10releng-201718-q3, 10Epic, 10Patch-For-Review: [EPIC] Migrate Mediawiki jobs from Nodepool to Docker - https://phabricator.wikimedia.org/T183512 (10Addshore) Just an update to the status 1 year on: {F25197044} [08:09:46] addshore: though the graphite metrics are deleted after 90 days iirc :] [08:10:13] hmmmmmm? are they? I dont think so [08:10:28] that graph is a 2 year graph :p [08:10:50] Alos hashar we have some Math quibble issues https://phabricator.wikimedia.org/T202266 :( [08:11:12] graphite::whisper_cleanup { 'graphite-zuul': [08:11:12] directory => "${storage_dir}/whisper/zuul", [08:11:12] } [08:11:13] :D [08:11:45] which deletes the metrics that have not been touched for 30 days [08:11:47] but yeah [08:12:46] addshore: ahh yeah [08:13:03] addshore: so when running install.php , the extension configuration from extension.json IS NOT LOADED [08:13:10] so all variables default to null iirc [08:13:35] hashar: aaaaaah, some magic zuul only metric cleanup, i see! [08:13:47] hashar: how was that solved for other extensions? :/ [08:13:53] https://phabricator.wikimedia.org/T196173#4252779 [08:13:59] for kunal explanation [08:14:20] and a few other extensions got hit by that ( https://phabricator.wikimedia.org/T198331 https://phabricator.wikimedia.org/T198330 ) [08:14:29] the workaround is to have the extension to set sane defaults internally [09:13:37] 10Project-Admins: Create WMF-Engineering-Administration team project - https://phabricator.wikimedia.org/T102307 (10Aklapper) As discussed via emails, project archived as it is not used by that team. [09:47:33] (03PS1) 10Hashar: Move npm assets builders to perm slaves [integration/config] - 10https://gerrit.wikimedia.org/r/454218 [09:59:34] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10releng-201718-q4, 10Discovery, 10Wikimedia-Portals: Migrate Jenkins jobs wikimedia-portals-build and wikidata-query-gui-build to Docker and to use an entry point (eg: npm b... - https://phabricator.wikimedia.org/T190073 [09:59:37] 10Release-Engineering-Team, 10Epic: Migrate all CI jobs from Nodepool, deprecate its use - https://phabricator.wikimedia.org/T190097 (10hashar) [09:59:39] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10releng-201718-q4, 10Discovery, 10Wikimedia-Portals: Migrate Jenkins jobs wikimedia-portals-build and wikidata-query-gui-build to Docker and to use an entry point (eg: npm b... - https://phabricator.wikimedia.org/T190073 [09:59:48] (03PS2) 10Hashar: Move npm assets builders to perm slaves [integration/config] - 10https://gerrit.wikimedia.org/r/454218 (https://phabricator.wikimedia.org/T190073) [10:00:16] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10releng-201718-q4, 10Discovery, and 2 others: Migrate Jenkins jobs wikimedia-portals-build and wikidata-query-gui-build to Docker and to use an entry point (eg: npm builddeplo... - https://phabricator.wikimedia.org/T190073 [10:07:51] (03CR) 10Daimona Eaytoy: [C: 031] Make seccheck voting for AbuseFilter [integration/config] - 10https://gerrit.wikimedia.org/r/454202 (https://phabricator.wikimedia.org/T197002) (owner: 10Umherirrender) [10:12:28] I'm pushing some lage files to git lfs of gerrit. Tell me if anything is not working [10:17:55] (03CR) 10Alexandros Kosiaris: [C: 031] Blubber: Move config paths for pipeline jobs [integration/config] - 10https://gerrit.wikimedia.org/r/453535 (https://phabricator.wikimedia.org/T202068) (owner: 10Thcipriani) [10:37:31] !log ladsgroup@deployment-deploy01:~$ mwscript extensions/Wikibase/repo/maintenance/rebuildTermSqlIndex.php --wiki=wikidatawiki --rebuild-all-terms (T202260) [10:37:38] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [10:37:38] T202260: Some property labels don't appear on beta - https://phabricator.wikimedia.org/T202260 [10:39:58] (03CR) 10Hashar: [C: 032] "They work fine!" [integration/config] - 10https://gerrit.wikimedia.org/r/454218 (https://phabricator.wikimedia.org/T190073) (owner: 10Hashar) [10:41:57] (03Merged) 10jenkins-bot: Move npm assets builders to perm slaves [integration/config] - 10https://gerrit.wikimedia.org/r/454218 (https://phabricator.wikimedia.org/T190073) (owner: 10Hashar) [10:52:49] (03PS1) 10Hashar: Migrate NoCat to Quibble [integration/config] - 10https://gerrit.wikimedia.org/r/454231 [10:52:51] (03PS1) 10Hashar: Migrate AutoCategoriseUploads to Quibble [integration/config] - 10https://gerrit.wikimedia.org/r/454232 [10:52:53] (03PS1) 10Hashar: Migrate CategoryTests to Quibble [integration/config] - 10https://gerrit.wikimedia.org/r/454233 [10:52:55] (03PS1) 10Hashar: Migrate DataTransfer to Quibble [integration/config] - 10https://gerrit.wikimedia.org/r/454234 [10:53:02] (03CR) 10Hashar: [C: 032] Migrate NoCat to Quibble [integration/config] - 10https://gerrit.wikimedia.org/r/454231 (owner: 10Hashar) [10:53:11] (03CR) 10Hashar: [C: 032] Migrate AutoCategoriseUploads to Quibble [integration/config] - 10https://gerrit.wikimedia.org/r/454232 (owner: 10Hashar) [10:53:14] (03CR) 10Hashar: [C: 032] Migrate CategoryTests to Quibble [integration/config] - 10https://gerrit.wikimedia.org/r/454233 (owner: 10Hashar) [10:53:18] (03CR) 10Hashar: [C: 032] Migrate DataTransfer to Quibble [integration/config] - 10https://gerrit.wikimedia.org/r/454234 (owner: 10Hashar) [10:57:04] uh oh! [10:57:10] that's not good [10:58:47] Nikerabbit: op yourself and /msg Sigyn unkline wikibugs_ [10:59:07] I'm not an op :o [10:59:39] sorry, I though this was -i18n [10:59:50] we never talked outside that channel xD [12:03:07] Nikerabbit, Hauskatze: wikibugs should be fixed now [12:03:18] so for next time, trick is [12:03:46] aha [12:03:49] * First, identify which channel something got killed *for*. They would have sent a message to a channel at the same second as getting killed [12:04:38] * Then, identify an op for that channel - look for people with active +o and also "/msg ChanServ access #wikimedia-releng list" - and in this case also "/msg ChanServ access #mediawiki-core list" because those ops are included here too [12:04:52] (Annoying but makes sense otherwise any channel op anywhere could unkline, I guess.) [12:04:57] * Ask one of them to "/msg Sigyn unkline wikibugs" [12:05:35] ack Krenair , thanks for taking care :) [12:05:53] * Failing that, ask in #freenode - as this one has a wikimedia/bot/* cloak it's unlikely to be a real problem [12:05:58] * Failing that, ask a GC to talk to them [12:06:47] it's on many channels, how would one know which one is the cause [12:06:56] Hopefully wikibugs will get exempted in Sigyn so this shouldn't happen again to that particular bot. [12:07:07] Nikerabbit, with great difficulty. [12:07:29] Oh by the way, the person running "/msg Sigyn unkline wikibugs" will need to op up in the relevant channel first [12:08:47] (03PS1) 10Umherirrender: Make seccheck non-voting for Echo [integration/config] - 10https://gerrit.wikimedia.org/r/454258 [12:10:03] In this case I can see [12:10:29] 2018-08-21 10:53:17,748 - irc3.wikibugs - DEBUG - > PRIVMSG #wikimedia-releng :(03CR) 10Hashar: [C: 032] Migrate DataTransfer to Quibble [integration/config] - 10https://gerrit.wikimedia.org/r/454234 (owner: 10Hashar) [12:10:30] 2018-08-21 10:53:18,734 - irc3.wikibugs - CRITICAL - connection lost (47189715600384): None [12:11:01] In tools-bastion-03:/data/project/wikibugs/redis2irc.log [12:11:53] was hoping for an example output from sigyn with a match for the unkline command but it looks like that's gone now :/ [12:14:16] Looks like we need to give our GCs a list of bots so they can be exempted across all wikimedia channels [12:38:49] so you need to have +o before the kline or you can op yourself afterwards to ask sigyn to unban the bot? [12:39:05] if the first, that means you need to be perma-opped (@) [12:40:44] 10Continuous-Integration-Config, 10phan-taint-check-plugin: phan-taint-check-plugin should ignore /extensions/ folder - https://phabricator.wikimedia.org/T201794 (10Umherirrender) [12:40:56] 10Continuous-Integration-Config, 10Wikimedia-General-or-Unknown, 10phan-taint-check-plugin, 10MW-1.32-release-notes (WMF-deploy-2018-08-21 (1.32.0-wmf.18)), 10Patch-For-Review: Enable phan-taint-check-plugin on all Wikimedia-deployed repositories where it is curr... - https://phabricator.wikimedia.org/T201219 [12:49:16] They've added some exemptions in some channels [12:49:22] I'm putting a very large warning in channels.yaml [12:49:36] Hauskatze, you can op yourself afterwards to ask Sigyn to unban [12:50:11] https://gerrit.wikimedia.org/r/#/c/labs/tools/wikibugs2/+/454263/ [12:50:48] Hauskatze, when you run the "unkline x" command it just checks for recent klines coming from channels where you are currently opped [12:52:19] So if you "/msg ChanServ op #wikimedia-releng" then "/msg Sigyn unkline wikibugs" you'll get something like "No matches wikibugs in recent bans from #wikimedia-releng" - unless there was something recent there ofc [12:52:20] ah, good; I though you had to go channel by channel == headshot [12:52:55] Obviously this requires you to know which channel the problem occurred in and be op there [12:52:57] curiously I did that on -rename (wikibugs is there) but sigyn didn't replied to me (I was opped) [12:53:06] uh [12:53:08] maybe /query Sigyn [12:53:11] however sigyn was not there [12:53:17] oh [12:53:25] yeah if Sigyn isn't there I doubt it will do anything [12:53:48] so you need to do the /msg from a channel you both are an op and sigyn idles [12:53:57] there are 9 channels in which both Sigyn and wikibugs are present (or configured to join if need be) [12:54:03] I can do it for -es and -ops [13:07:39] * Hauskatze attempts to restart wikibugs [13:12:36] done [13:19:34] 10Beta-Cluster-Infrastructure, 10Citoid: On beta cluster citoid should self update and reload after change is merged - https://phabricator.wikimedia.org/T95652 (10Deskana) [13:47:59] (03PS1) 10Hashar: Phase out mwext-qunit-jessie [integration/config] - 10https://gerrit.wikimedia.org/r/454280 [13:49:02] (03CR) 10Hashar: [C: 032] Phase out mwext-qunit-jessie [integration/config] - 10https://gerrit.wikimedia.org/r/454280 (owner: 10Hashar) [13:50:57] (03Merged) 10jenkins-bot: Phase out mwext-qunit-jessie [integration/config] - 10https://gerrit.wikimedia.org/r/454280 (owner: 10Hashar) [14:23:01] 10Phabricator: Unhandled Exception ("AphrontSchemaQueryException") - https://phabricator.wikimedia.org/T202409 (10MarcoAurelio) [14:23:42] 10Phabricator: Unhandled Exception ("AphrontSchemaQueryException") - https://phabricator.wikimedia.org/T202409 (10MarcoAurelio) [14:42:59] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Jenkins, 10Patch-For-Review: Jenkins should auto-depool nodes if they run out of disk space on specific partitions - https://phabricator.wikimedia.org/T201224 (10thcipriani) 05Resolved>03Open >>! In T201224#4517385, @Legokt... [14:43:23] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Jenkins, 10Patch-For-Review: Jenkins should auto-depool nodes if they run out of disk space on specific partitions - https://phabricator.wikimedia.org/T201224 (10thcipriani) p:05High>03Normal [14:48:34] 10Phabricator: Unhandled Exception ("AphrontSchemaQueryException") - https://phabricator.wikimedia.org/T202409 (10Aklapper) (I wonder how to trick people into searching for "ft_doc.epochModified" first... ) [14:48:41] 10Phabricator: Unhandled Exception ("AphrontSchemaQueryException") - https://phabricator.wikimedia.org/T202409 (10Aklapper) [14:48:46] 10Phabricator (Upstream), 10Upstream: #1054: Unknown column 'ft_doc.epochModified' in 'order clause' - https://phabricator.wikimedia.org/T187604 (10Aklapper) [14:49:09] (03CR) 10Thcipriani: [C: 032] Blubber: Move config paths for pipeline jobs [integration/config] - 10https://gerrit.wikimedia.org/r/453535 (https://phabricator.wikimedia.org/T202068) (owner: 10Thcipriani) [14:50:55] (03Merged) 10jenkins-bot: Blubber: Move config paths for pipeline jobs [integration/config] - 10https://gerrit.wikimedia.org/r/453535 (https://phabricator.wikimedia.org/T202068) (owner: 10Thcipriani) [14:53:34] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure (shipyard), 10BlueSpice, 10Patch-For-Review: Enable unit tests on BlueSpice* repos - https://phabricator.wikimedia.org/T130811 (10hashar) [14:56:22] (03PS1) 10Hashar: WikiLexicalData is broken [integration/config] - 10https://gerrit.wikimedia.org/r/454296 (https://phabricator.wikimedia.org/T198089) [14:56:36] (03PS2) 10Hashar: WikiLexicalData is broken [integration/config] - 10https://gerrit.wikimedia.org/r/454296 (https://phabricator.wikimedia.org/T198089) [14:57:34] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10releng-201718-q3, 10Epic, 10Patch-For-Review: [EPIC] Migrate Mediawiki jobs from Nodepool to Docker - https://phabricator.wikimedia.org/T183512 (10hashar) [14:57:37] (03CR) 10Hashar: [C: 032] WikiLexicalData is broken [integration/config] - 10https://gerrit.wikimedia.org/r/454296 (https://phabricator.wikimedia.org/T198089) (owner: 10Hashar) [14:59:29] (03Merged) 10jenkins-bot: WikiLexicalData is broken [integration/config] - 10https://gerrit.wikimedia.org/r/454296 (https://phabricator.wikimedia.org/T198089) (owner: 10Hashar) [14:59:38] (03PS1) 10Hashar: SiteSettings extension is broken [integration/config] - 10https://gerrit.wikimedia.org/r/454298 (https://phabricator.wikimedia.org/T196504) [15:00:06] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10releng-201718-q3, 10Epic, 10Patch-For-Review: [EPIC] Migrate Mediawiki jobs from Nodepool to Docker - https://phabricator.wikimedia.org/T183512 (10hashar) [15:01:41] (03CR) 10Hashar: [C: 032] SiteSettings extension is broken [integration/config] - 10https://gerrit.wikimedia.org/r/454298 (https://phabricator.wikimedia.org/T196504) (owner: 10Hashar) [15:03:37] (03Merged) 10jenkins-bot: SiteSettings extension is broken [integration/config] - 10https://gerrit.wikimedia.org/r/454298 (https://phabricator.wikimedia.org/T196504) (owner: 10Hashar) [15:05:02] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10releng-201718-q3, 10Epic, 10Patch-For-Review: [EPIC] Migrate Mediawiki jobs from Nodepool to Docker - https://phabricator.wikimedia.org/T183512 (10hashar) [15:07:54] (03PS1) 10Hashar: LinkedWiki extension is broken [integration/config] - 10https://gerrit.wikimedia.org/r/454300 (https://phabricator.wikimedia.org/T198919) [15:08:18] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10releng-201718-q3, 10Epic, 10Patch-For-Review: [EPIC] Migrate Mediawiki jobs from Nodepool to Docker - https://phabricator.wikimedia.org/T183512 (10hashar) [15:08:22] (03CR) 10Hashar: [C: 032] LinkedWiki extension is broken [integration/config] - 10https://gerrit.wikimedia.org/r/454300 (https://phabricator.wikimedia.org/T198919) (owner: 10Hashar) [15:10:01] (03Merged) 10jenkins-bot: LinkedWiki extension is broken [integration/config] - 10https://gerrit.wikimedia.org/r/454300 (https://phabricator.wikimedia.org/T198919) (owner: 10Hashar) [15:18:47] (03PS2) 10Zfilipin: ffmpeg is needed for recording videos of Selenium tests [integration/config] - 10https://gerrit.wikimedia.org/r/452703 (https://phabricator.wikimedia.org/T193157) [15:20:07] (03CR) 10Zfilipin: "PS2 updates changelog files. I'm not sure if what I did makes sense. :/" [integration/config] - 10https://gerrit.wikimedia.org/r/452703 (https://phabricator.wikimedia.org/T193157) (owner: 10Zfilipin) [15:21:06] (03CR) 10Zfilipin: ffmpeg is needed for recording videos of Selenium tests (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/452703 (https://phabricator.wikimedia.org/T193157) (owner: 10Zfilipin) [15:26:27] (03CR) 10Zfilipin: ffmpeg is needed for recording videos of Selenium tests (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/452703 (https://phabricator.wikimedia.org/T193157) (owner: 10Zfilipin) [15:27:11] (03CR) 10Zfilipin: "> Patch Set 1: Code-Review-1" [integration/config] - 10https://gerrit.wikimedia.org/r/452703 (https://phabricator.wikimedia.org/T193157) (owner: 10Zfilipin) [15:28:12] (03CR) 10Zfilipin: "> Patch Set 1: Code-Review-1" [integration/config] - 10https://gerrit.wikimedia.org/r/452703 (https://phabricator.wikimedia.org/T193157) (owner: 10Zfilipin) [15:29:52] (03PS1) 10Thcipriani: Notify if node disconnected due to disk space [integration/config] - 10https://gerrit.wikimedia.org/r/454306 (https://phabricator.wikimedia.org/T201224) [15:33:32] (03PS1) 10Hashar: Migrate Lingo to Quibble [integration/config] - 10https://gerrit.wikimedia.org/r/454307 (https://phabricator.wikimedia.org/T196456) [15:33:46] (03CR) 10Hashar: [C: 032] Migrate Lingo to Quibble [integration/config] - 10https://gerrit.wikimedia.org/r/454307 (https://phabricator.wikimedia.org/T196456) (owner: 10Hashar) [15:35:08] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10releng-201718-q3, 10Epic, 10Patch-For-Review: [EPIC] Migrate Mediawiki jobs from Nodepool to Docker - https://phabricator.wikimedia.org/T183512 (10hashar) [15:36:08] (03Merged) 10jenkins-bot: Migrate Lingo to Quibble [integration/config] - 10https://gerrit.wikimedia.org/r/454307 (https://phabricator.wikimedia.org/T196456) (owner: 10Hashar) [15:47:51] 10Gerrit, 10User-zeljkofilipin: Can not download a specific patch from Gerrit using git-review - https://phabricator.wikimedia.org/T194520 (10thcipriani) 05Open>03Resolved This looks to be fixed in upstream master: https://review.openstack.org/#/c/541791/ Closing this task. [16:13:11] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10releng-201718-q3, 10Epic, 10Patch-For-Review: [EPIC] Migrate Mediawiki jobs from Nodepool to Docker - https://phabricator.wikimedia.org/T183512 (10hashar) [16:20:33] (03PS1) 10Hashar: Add FileSystemImageServer extension [integration/config] - 10https://gerrit.wikimedia.org/r/454312 [16:20:35] (03PS1) 10Hashar: Add UILangCode extension [integration/config] - 10https://gerrit.wikimedia.org/r/454313 [16:20:46] (03CR) 10Hashar: [C: 032] Add FileSystemImageServer extension [integration/config] - 10https://gerrit.wikimedia.org/r/454312 (owner: 10Hashar) [16:20:49] (03CR) 10Hashar: [C: 032] Add UILangCode extension [integration/config] - 10https://gerrit.wikimedia.org/r/454313 (owner: 10Hashar) [16:21:44] (03PS1) 10Hashar: Add Wigo3 extension [integration/config] - 10https://gerrit.wikimedia.org/r/454314 [16:21:54] (03CR) 10Hashar: [C: 032] Add Wigo3 extension [integration/config] - 10https://gerrit.wikimedia.org/r/454314 (owner: 10Hashar) [16:22:15] (03Merged) 10jenkins-bot: Add FileSystemImageServer extension [integration/config] - 10https://gerrit.wikimedia.org/r/454312 (owner: 10Hashar) [16:22:39] (03Merged) 10jenkins-bot: Add UILangCode extension [integration/config] - 10https://gerrit.wikimedia.org/r/454313 (owner: 10Hashar) [16:23:30] (03Merged) 10jenkins-bot: Add Wigo3 extension [integration/config] - 10https://gerrit.wikimedia.org/r/454314 (owner: 10Hashar) [16:25:09] (03PS1) 10Hashar: QA report allows extension-quibble-composer-noselenium [integration/config] - 10https://gerrit.wikimedia.org/r/454315 [16:26:03] (03CR) 10Hashar: [C: 032] QA report allows extension-quibble-composer-noselenium [integration/config] - 10https://gerrit.wikimedia.org/r/454315 (owner: 10Hashar) [16:27:28] (03Merged) 10jenkins-bot: QA report allows extension-quibble-composer-noselenium [integration/config] - 10https://gerrit.wikimedia.org/r/454315 (owner: 10Hashar) [17:04:08] 10Phabricator: CURLE_COULDNT_CONNECT when trying to use Conduit - https://phabricator.wikimedia.org/T201746 (10mmodell) Is this running on toolforge? If so then could it be that the cron jobs are running in an environment without outbound network access? [17:24:38] re: search in Phabricator. if i go to advanced search and "include users" and enter more than one user, it does not tell me whether this is a logical AND or a logical OR. It looks like it's an "OR" but i want to find something by narrowing it down and what i know is the users who were involved [17:26:36] hey releng, any chance someone can unblock my Phabricator account (ldap user jdrewniak)? [17:31:24] jan_drewniak: any idea why it's disabled? [17:37:44] mutante: I think it became disabled after I wrote this task this morning. https://phabricator.wikimedia.org/T202374 [17:38:41] or this comment. https://phabricator.wikimedia.org/T197499 that's the only Phab activity on my account today [17:38:43] jan_drewniak: i see.. looks like you got hit by false positive from antivandalism features. i enabled you [17:38:51] you should be good again [17:39:08] mutante: great! thanks! [17:39:18] welcome [17:39:22] jan_drewniak: it looks enabled and I added you to https://phabricator.wikimedia.org/project/members/3104/ [17:39:31] which I imagine is what happened, the auto spam stuff caught you and now should not [17:39:54] :) [17:40:20] aha, the "trusted" group, cool [17:40:40] that's of course even better to prevent it from happening again [17:42:40] yeah gotta find a way to advertise that more widely [17:44:38] alrighty.. task #1, make myself a more clever profile title https://phabricator.wikimedia.org/project/members/3104/ [17:45:33] is the group membership in trusted manual so far? or does membership in certain other LDAP groups already make you "trusted" as well.. that would probably be nice [17:51:17] 10Phabricator: Embedded Commons videos are broken - https://phabricator.wikimedia.org/T200757 (10Quiddity) This seems to be fixed, although (minor issue) it displays a broken-image graphic in the preview https://commons.wikimedia.org/wiki/File:Dr._MacIntyre%27s_X-Ray_film.webm Screenshot of broken-image graphi... [17:51:41] 10Phabricator: CURLE_COULDNT_CONNECT when trying to use Conduit - https://phabricator.wikimedia.org/T201746 (10Tgr) No, stat1005 (it needs Hive access). It was working until Aug. 8 or so, was something reconfigured then? [17:55:59] mutante, no it's manual [17:56:04] phab doesn't have such a mechanism afaik [17:57:19] *nod*, thanks [18:06:41] there is an upstream task for it that went silent but on that task is a "sync" script iirc to do one time operations [18:06:43] but nothing ongoing [18:06:46] we just never pursued it [18:08:29] gotcha, well upstream task is something [18:19:08] (03CR) 10Krinkle: [C: 032] Minor cleanups to the UnusedUseStatement sniff [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/453935 (owner: 10Thiemo Kreuz (WMDE)) [18:20:07] (03Merged) 10jenkins-bot: Minor cleanups to the UnusedUseStatement sniff [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/453935 (owner: 10Thiemo Kreuz (WMDE)) [18:20:40] (03CR) 10jenkins-bot: Minor cleanups to the UnusedUseStatement sniff [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/453935 (owner: 10Thiemo Kreuz (WMDE)) [20:29:13] 10Phabricator: Phabricator account with existing email address - https://phabricator.wikimedia.org/T201960 (10Nettrom) 05Open>03Resolved a:03Nettrom Fixed it by getting an email alias set up, so I'm marking this as resolved. [20:58:39] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10Wikimedia-log-errors (Shared Build Failure): mediawiki-quibble jobs fails due to disk full (sql insert failed) - https://phabricator.wikimedia.org/T202457 (10Krinkle) [21:01:40] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10Wikimedia-log-errors (Shared Build Failure): mediawiki-quibble jobs fails due to disk full (sql insert failed) - https://phabricator.wikimedia.org/T202457 (10hashar) [21:04:29] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10Wikimedia-log-errors (Shared Build Failure): mediawiki-quibble jobs fails due to disk full (sql insert failed) - https://phabricator.wikimedia.org/T202457 (10hashar) integration-slave-docker-1026 is a new Docker host that has several Jenk... [21:04:48] marxarelli: the new docker slave cant hold so many executors due to disk space constraint ( https://phabricator.wikimedia.org/T202457 ) [21:05:10] marxarelli: the root cause is Quibble points mysql to use a datadir under /tmp which is the / partition [21:05:21] and that partition already has the system + the docker images [21:05:33] yikes [21:05:37] so when too many mysql are filled , there is no more disk space on / :\\\ [21:05:53] seems like /tmp should be on a different partition [21:06:07] possibly [21:06:39] and the docker images as well. But we don't have a good way to partition the instnaces [21:06:51] anyway, I havent touched anything. But I guess you want to reduce the # of executors [21:06:57] jobs should probably not be able to write to the partition that / shares in general [21:07:35] yeah I guess Quibble can be made to use a datadir of some sort instead of /tmp [21:08:34] is the lvm mount available to quibble/containers? [21:08:46] that seems like a more appropriate location [21:09:03] yup the extended disk space is mounted to /srv [21:09:06] i.e. /srv [21:09:08] and that is where the jenkins workspaces are [21:09:21] but quibble uses: [21:09:21] quibble/backend.py: self.tmpdir = tempfile.TemporaryDirectory(prefix='quibble-mysql-') [21:09:21] quibble/backend.py: self.tmpdir = tempfile.TemporaryDirectory(prefix='quibble-sqlite-') [21:09:23] ;D [21:09:37] which ends up creating it somewhere under /tmp [21:09:49] and i'm guessing the workspace is mounted to quibbles container? [21:10:11] yup more or less [21:10:18] cool [21:10:24] yeah, i'll lower the executors for now [21:10:59] it would be great to have quibble set up mysql's datadir under the workspace though [21:11:20] yeah [21:12:07] anyway I was just lurking. I should sleep :] [21:12:20] hashar: go to sleep! :) [21:12:28] roger!! [21:12:34] and thanks! [21:15:00] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Wikimedia-log-errors (Shared Build Failure): mediawiki-quibble jobs fails due to disk full (sql insert failed) - https://phabricator.wikimedia.org/T202457 (10dduvall) p:05Triage>03High a:03dduvall I've reduced the number of... [21:16:47] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments: 1.32.0-wmf.18 deployment blockers - https://phabricator.wikimedia.org/T191064 (10Catrope) [21:36:56] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Wikimedia-log-errors (Shared Build Failure): mediawiki-quibble jobs fails due to disk full (sql insert failed) - https://phabricator.wikimedia.org/T202457 (10Umherirrender) It is not alone mysql: ``` 20:41:00 error: unable to wr... [23:53:05] thcipriani: so I think we need the auto-offliner to also look at the root partition in addition to /srv [23:53:14] given https://phabricator.wikimedia.org/T202457 today [23:53:42] k, I can have it look at that as well [23:53:48] :) [23:54:26] * thcipriani fiddles [23:59:01] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments: 1.32.0-wmf.18 deployment blockers - https://phabricator.wikimedia.org/T191064 (10Legoktm)