[00:10:31] 6Labs, 10Tool-Labs, 10Mail, 6Operations: remove toolserver mail aliases - https://phabricator.wikimedia.org/T127543#2105335 (10Dzahn) removed ``` -# Not actually an OTRS queue -ts-admins: ts-admins@toolserver.org -zedler-admins: ts-admins@toolserver.org ``` [00:12:42] 6Labs, 10Tool-Labs, 10Mail, 6Operations: remove toolserver mail aliases - https://phabricator.wikimedia.org/T127543#2105340 (10Dzahn) 5Open>3Resolved a:3Dzahn [01:15:37] Hi, the replication for zhwiki_p seems to be broken: https://zh.wikipedia.org/?curid=5259556 exists but select * from page where page_id = 5259556; returns empty [01:20:24] 6Labs, 10Labs-Infrastructure: Replication broken for zhwiki_p - https://phabricator.wikimedia.org/T129432#2105526 (10jimmyxu) [02:29:46] yuvipanda, where does invisible-unicorn log to? [02:35:41] 6Labs: Can't delete security groups (in horizon or OSM) - https://phabricator.wikimedia.org/T129437#2105635 (10AlexMonk-WMF) [02:36:35] 6Labs: Can't create security rules via OSM - https://phabricator.wikimedia.org/T129438#2105636 (10AlexMonk-WMF) [03:09:26] 6Labs, 10Labs-Infrastructure, 10MediaWiki-extensions-OpenStackManager, 10labs-sprint-118: Can't delete NovaProxy instance with malformed DNS hostname - https://phabricator.wikimedia.org/T69927#2105660 (10Krenair) Yep. Someone with access to the project-proxy project I think. [04:22:01] 6Labs, 6Operations, 10wikitech.wikimedia.org: Update wikitech-static OS/PHP version - https://phabricator.wikimedia.org/T126385#2105742 (10Krenair) Also: * Updated the hostname on the new host from wikitech-static-jessie to wikitech-static - the previous name no longer resolves and sudo was not happy about t... [05:58:16] Krenair: default upstart location, I think [05:58:17] 6Labs, 10Tool-Labs, 10pywikibot-core: Tool Labs Pywikibot does not work with new shared Pywikibot config files - https://phabricator.wikimedia.org/T129406#2105875 (10Ato_01) [06:37:22] 6Labs, 10Tool-Labs, 10pywikibot-core: Tool Labs Pywikibot does not work with new shared Pywikibot config files - https://phabricator.wikimedia.org/T129406#2105910 (10Ato_01) [07:26:11] 6Labs, 10Tool-Labs: Linux Error: libgcc_s.so.1 - https://phabricator.wikimedia.org/T129361#2105937 (10doctaxon) 5Open>3Resolved a:3doctaxon Thank you ... [07:33:42] 10PAWS: Shared code area - https://phabricator.wikimedia.org/T128163#2105950 (10jayvdb) It appears a temporary solution is being used by WMF staff. E.g. http://paws-public.wmflabs.org/paws-public/EpochFail/projects/headings/extract_headings.ipynb [07:53:59] 10PAWS: Shared code area - https://phabricator.wikimedia.org/T128163#2065815 (10yuvipanda) Indeed, but I want to note that this is very temporary - and more importantly, these URLs *will* break in the near future (and hence shouldn't be spread around too much :D ) [08:06:37] 6Labs, 10Tool-Labs, 10pywikibot-core: Tool Labs Pywikibot does not work with new shared Pywikibot config files - https://phabricator.wikimedia.org/T129406#2106001 (10Ato_01) [10:36:33] (03CR) 10MarcoAurelio: [C: 032] General overhaul of CSS/JS content [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/276206 (owner: 10MarcoAurelio) [10:38:15] (03Merged) 10jenkins-bot: General overhaul of CSS/JS content [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/276206 (owner: 10MarcoAurelio) [10:48:50] (03PS1) 10MarcoAurelio: Fix links [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/276433 [10:50:07] (03CR) 10MarcoAurelio: [C: 032] Fix links [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/276433 (owner: 10MarcoAurelio) [10:57:32] (03Merged) 10jenkins-bot: Fix links [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/276433 (owner: 10MarcoAurelio) [11:07:13] !log tools.stewardbots Merged https://gerrit.wikimedia.org/r/276206 and https://gerrit.wikimedia.org/r/276433 [11:07:17] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.stewardbots/SAL, Master [13:22:32] 6Labs, 10MediaWiki-extensions-SemanticForms, 10wikitech.wikimedia.org: "Edit with form" missing on a Tools access request page - https://phabricator.wikimedia.org/T118136#2106665 (10Yaron_Koren) 5Open>3Resolved That seems to be the obvious explanation; I'm marking this as "Resolved". [13:32:13] (03PS1) 10Youni Verciti: Fix the link generator absent [labs/tools/vocabulary-index] - 10https://gerrit.wikimedia.org/r/276454 [13:34:02] (03Abandoned) 10Youni Verciti: First commit on branch dev [labs/tools/vocabulary-index] - 10https://gerrit.wikimedia.org/r/275802 (owner: 10Youni Verciti) [13:34:27] 6Labs, 10Labs-Infrastructure, 10DBA: Replication broken for zhwiki_p - https://phabricator.wikimedia.org/T129432#2106677 (10Peachey88) [13:41:01] 6Labs, 10DBA: write irc bot to report high replag of s{1,2,3}.labsdb on #wikimedia-labsdb - https://phabricator.wikimedia.org/T106151#2106705 (10Peachey88) I believe @Krinkle already has a bot that does this for the production DB lags on IRC, I wonder if it could be modify to accommodate these instead of writi... [13:54:50] 6Labs, 10DBA: write irc bot to report high replag of s{1,2,3}.labsdb on #wikimedia-labsdb - https://phabricator.wikimedia.org/T106151#1460476 (10jcrespo) "it is unkown by labs admins/operators until sb. reports replag problems on irc." That is untrue- lag is monitored by several admin-only tools, however those... [14:00:14] 6Labs, 10Labs-Infrastructure, 10DBA: Data missing in zhwiki on labs replicas - https://phabricator.wikimedia.org/T129432#2106743 (10jcrespo) [14:00:29] 6Labs, 10Labs-Infrastructure, 10DBA: Data missing in zhwiki on labs replicas - https://phabricator.wikimedia.org/T129432#2105526 (10jcrespo) p:5Triage>3High a:3jcrespo [14:02:55] 6Labs, 10Labs-Infrastructure, 10DBA: Data missing in zhwiki on labs replicas - https://phabricator.wikimedia.org/T129432#2106749 (10jcrespo) It seems that the page creation and the first revision of that page failed to be inserted (revision id 5259556 has 2 revisions, not one, and obviously 1 page in product... [14:15:14] 6Labs, 10Labs-Infrastructure, 10DBA: Data missing in zhwiki on labs replicas - https://phabricator.wikimedia.org/T129432#2106816 (10jcrespo) Strangely enough, the number of records are the same than in production, so maybe it was inserted with a different id? ``` db1069 MariaDB db1069 zhwiki > SELECT count(... [14:23:31] 6Labs, 10Labs-Infrastructure, 10DBA: Data missing in zhwiki on labs replicas - https://phabricator.wikimedia.org/T129432#2106854 (10jcrespo) I made a mistake and searched for the record that was there. There is indeed, a few records missing on that time band: $ mysql -h dbstore1002 zhwiki -e "SELECT count(*... [14:32:24] 6Labs, 10DBA: write irc bot to report high replag of s{1,2,3}.labsdb on #wikimedia-labsdb - https://phabricator.wikimedia.org/T106151#1460476 (10valhallasw) The best option is probably to make the heartbeat tables available through a Diamond collector. That allows us to aggregate the data in Graphite, and to c... [14:36:15] 6Labs, 10Labs-Infrastructure, 10DBA: Data missing in zhwiki on labs replicas - https://phabricator.wikimedia.org/T129432#2106904 (10jcrespo) Same for page: ``` $ mysql -h db1069 -P3312 zhwiki -e "SELECT count(*) FROM page WHERE page_id BETWEEN 5260000 AND 5270000" +----------+ | count(*) | +----------+ |... [14:39:55] 6Labs, 10DBA: write irc bot to report high replag of s{1,2,3}.labsdb on #wikimedia-labsdb - https://phabricator.wikimedia.org/T106151#2106912 (10jcrespo) We are doing that for production (directly from the database), so no need for a separate ticket for labs. I have been working last 2 weeks to prepare the pr... [15:12:41] 6Labs, 10Labs-Infrastructure, 10MediaWiki-extensions-OpenStackManager, 10labs-sprint-118: Can't delete NovaProxy instance with malformed DNS hostname - https://phabricator.wikimedia.org/T69927#2107112 (10scfc) 5Open>3Resolved a:3scfc I logged into `novaproxy-01`, backed up `/etc/dynamicproxy-api/data... [15:12:58] 6Labs, 10Labs-Infrastructure, 10MediaWiki-extensions-OpenStackManager, 10labs-sprint-118: Can't delete NovaProxy instance with malformed DNS hostname - https://phabricator.wikimedia.org/T69927#2107115 (10scfc) a:5scfc>3AlexMonk-WMF [16:10:29] 6Labs, 10Labs-Infrastructure, 10MediaWiki-extensions-OpenStackManager, 10labs-sprint-118: Can't delete NovaProxy instance with malformed DNS hostname - https://phabricator.wikimedia.org/T69927#2107338 (10AlexMonk-WMF) a:5AlexMonk-WMF>3Krenair [16:42:53] (03PS1) 10Youni Verciti: Explicit names for variables - step 2 [labs/tools/vocabulary-index] - 10https://gerrit.wikimedia.org/r/276496 [16:44:32] (03CR) 10Youni Verciti: "tested" [labs/tools/vocabulary-index] - 10https://gerrit.wikimedia.org/r/276496 (owner: 10Youni Verciti) [16:45:33] (03CR) 10Youni Verciti: [C: 032 V: 032] "gerrit" [labs/tools/vocabulary-index] - 10https://gerrit.wikimedia.org/r/276496 (owner: 10Youni Verciti) [18:25:16] 6Labs, 10Labs-Infrastructure, 10DBA: Data missing in zhwiki on labs replicas - https://phabricator.wikimedia.org/T129432#2108182 (10jcrespo) I have reimported all the missing records from production to labs. The original query is fixed now for me: ``` mysql -h labsdb1001 zhwiki_p -e "select * from page whe... [18:29:54] 6Labs, 10Labs-Infrastructure, 10DBA: Data missing in zhwiki on labs replicas - https://phabricator.wikimedia.org/T129432#2108213 (10jimmyxu) 5Open>3Resolved Looks resolved to me, thanks! [18:32:00] 6Labs, 10Labs-Infrastructure, 10DBA: Data missing in zhwiki on labs replicas - https://phabricator.wikimedia.org/T129432#2108248 (10jcrespo) Thank you very much for reporting, if you find more issues, please continue reporting them. These reports will help making sure both labs and production are healthy. [18:42:17] 6Labs, 10Labs-Infrastructure, 10DBA: Data missing in zhwiki on labs replicas - https://phabricator.wikimedia.org/T129432#2108308 (10jimmyxu) 5Resolved>3Open Hmm, seems the user table is also missing rows: ``` MariaDB [zhwiki_p]> select * from user where user_name = '长几'; Empty set (0.01 sec) ``` [19:05:38] andrewbogott, just reading the new public dns / proxy email. Not sure how affected the project would be. Currently warper.wmflabs.org is used. Its in the Maps project. the instance is maps-warper. Would be the proposed future URL be maps-warper.maps.wmflabs.org ? [19:07:33] chippy: yes, that’s right, unless maps-warper.wmflabs.org is currently using our automatic proxy [19:07:35] * andrewbogott checks [19:07:59] hmm, okay, I shall reply to the email with lots of "hurt" :) [19:08:58] ok, looks like the maps project doesn’t have any public access other than via the proxy [19:09:03] so most likely nothing would change in your case [19:09:40] andrewbogott: only instances w/ a public address woudl be affected yeah? [19:09:47] right [19:09:54] maybe we can say, hey if you don't have a public ip then don't worry [19:10:17] yeah, I’ll send another followup [19:11:13] I'd imagine that if the instance had a public IP, it would be more suitable to have a custom domain for it, rather than use the proxy? [19:11:41] chippy: right :) or even possible [19:12:17] okay, thanks for clearing it up. I'll hold off on the "hurt" email then! :) [19:12:53] chasemp: ‘the proxy’ is literally just two VMs with one address each — almost all tools web traffic runs through one and almost all labs web traffic through the other. [19:13:05] So it’s pretty easy for us to administrate those wholesale. [19:13:23] understood :) [19:13:39] um… sorry, that was supposed to be chippy: [19:13:53] you have similar autocomplete profiles [19:14:15] :-) [19:14:19] heh I wondered [19:14:45] andrewbogott, I think I see yeah [19:19:20] 10Tool-Labs-tools-stewardbots, 13Patch-For-Review: Make elections.php work again - https://phabricator.wikimedia.org/T128742#2108518 (10MarcoAurelio) Now working again and counting votes correctly. Some CSS fixes to avoid overlapping of icons and text in the table would be good though. [19:20:41] 10Tool-Labs-tools-stewardbots: hat-web-tools import for stewardbots - https://phabricator.wikimedia.org/T128743#2084536 (10MarcoAurelio) 5Open>3Resolved a:3MarcoAurelio [19:35:18] 6Labs, 10Tool-Labs: Cluebot writes massive logs that are making labstore run out of space and surge in load making toollabs unavailable - https://phabricator.wikimedia.org/T127222#2108577 (10yuvipanda) p:5Unbreak!>3Normal [19:35:41] 6Labs, 10Tool-Labs: Cluebot writes massive logs that are making labstore run out of space and surge in load making toollabs unavailable - https://phabricator.wikimedia.org/T127222#2036169 (10yuvipanda) Not really UBN anymore. [19:49:46] 6Labs, 10Labs-Infrastructure, 10Tool-Labs, 10ores: watroles/ores returns 500 Internal Server Error, works for other projects - https://phabricator.wikimedia.org/T128871#2108735 (10Dzahn) Yuvi merged the request. [19:50:39] 6Labs, 10Labs-Infrastructure, 10Tool-Labs, 10ores: watroles/ores returns 500 Internal Server Error, works for other projects - https://phabricator.wikimedia.org/T128871#2108740 (10Dzahn) thank you @scfc [19:55:25] 10PAWS, 7Documentation: Documentation for PAWS - https://phabricator.wikimedia.org/T129548#2108813 (10Quiddity) [19:55:37] 10PAWS, 7Documentation: Documentation for PAWS - https://phabricator.wikimedia.org/T129548#2108826 (10Quiddity) p:5Triage>3Normal [19:57:04] 10PAWS, 7Documentation: Write inline documentation - https://phabricator.wikimedia.org/T122545#2108841 (10Quiddity) [19:57:18] 10PAWS, 7Documentation: Documentation for PAWS - https://phabricator.wikimedia.org/T129548#2108813 (10Quiddity) [20:35:55] 6Labs, 6Operations, 10ops-eqiad: disk failure on labsdb1002 - https://phabricator.wikimedia.org/T126946#2109051 (10Cmjohnson) @chasemp any updates on this disk? [21:40:16] 6Labs, 6Operations, 10ops-eqiad: disk failure on labsdb1002 - https://phabricator.wikimedia.org/T126946#2109414 (10chasemp) >>! In T126946#2109051, @Cmjohnson wrote: > @chasemp any updates on this disk? I didn't have this on my radar Is the status: >>! In T126946#2035452, @Cmjohnson wrote: > Problem I a... [21:58:26] 6Labs, 10Tool-Labs, 6Developer-Relations, 7Documentation: Run a documentation sprint for Labs - https://phabricator.wikimedia.org/T101659#1344508 (10Qgil) [22:06:48] 6Labs, 10Tool-Labs, 10pywikibot-core: Tool Labs Pywikibot does not work with new shared Pywikibot config files - https://phabricator.wikimedia.org/T129406#2104686 (10Incola) Also my bot that use the shared Pywikibot code is affected by the issue: see http://tools.wmflabs.org/incolabot/bar.php [22:39:38] andrewbogott: hi, i get a daily email about puppet failing on graphite-labs [22:40:07] can you please fix/update me what to do ? i can't even ssh into this machine [22:40:19] matanya: yep, looking [22:40:25] what’s the fqdn? [22:40:53] graphite-labs.graphite.eqiad.wmflabs [22:41:18] puppet was disabled on that box by godog [22:41:22] godog, your thoughts? [22:41:59] he is away [22:42:12] i think i'll just remove myself from the project [22:42:21] i didn't contribute there for months [22:43:06] 6Labs, 7Graphite: Puppet disabled on graphite-labs - https://phabricator.wikimedia.org/T129579#2109803 (10Andrew) [22:43:10] matanya: ^ [22:43:16] what happened to wikitech logo ?? [22:43:19] yeah, removing yourself as admin is fine [22:43:43] thanks andrewbogott [22:43:49] matanya: that /is/ the wikitech logo :) Previously it was the labs logo, which never really made sense since wikitech does other stuff too. [22:44:04] labs logo is now the logo of https://horizon.wikimedia.org [22:44:13] which will soon be renamed to https://labsdashboard.wikimedia.org [22:44:40] and the user/pass with do 0auth ? [22:44:57] horizon should use the same creds as wikitech [22:45:09] I mean, does use :) [22:45:54] what about the Totp Token ? [22:46:07] same as the token you use when logging in to wikitec [22:46:14] I don't [22:46:17] oh... [22:46:33] You can’t do any projectadmin stuff on wikitech without 2fa... [22:46:47] my phone didn't support it when i regestered to wikitech [22:47:02] and didn't bother with it ever since [22:47:16] Maybe there’s some use case I’m missing? My thought is that without 2fa you already can’t do any of the things on wikitech that you can now do on horizon [22:47:48] I do a lot of project adminship without 2fa [22:47:55] like what? [22:48:05] create instances, add/remove users [22:48:25] you can create instances without 2fa? [22:48:26] edit security groups [22:48:30] yess [22:48:35] * andrewbogott ’s eyebrows shoot up [22:48:47] that’s for sure a bug! [22:48:48] basically, i can do anything [22:48:53] But maybe one that’s been that way for years :( [22:49:12] well, i am quite happy with this specific bug [22:50:04] andrewbogott: i have just removed myself from a project without 2fa [22:51:22] yeah, so can I. Something has regressed [22:52:22] I was not blocked ever, as far as i can tell [22:52:45] andrewbogott: do i get extra disk space for pointing this out ? :) [22:52:54] maybe! [22:53:13] oh, btw — do your video processing images use nfs? Like, at all? [22:53:23] I think we were seeing nfs traffic but I don’t recall [22:53:26] chasemp do you remember? [22:53:35] It shouldn't [22:53:53] I never looked to see if it was, but if it isn't then yeah def not an issue [22:53:55] on that front [22:54:11] just dieing for more storage [22:54:19] the demand is amazingly high [22:55:02] andrewbogott: i don't know what you did, but my ability to manage instances just disapeared [22:55:24] as far as I know I didn’t do anything... [22:55:35] Are you seeing "Two-factor authentication is required. Please enable it and try again.“ now? [22:55:50] no [22:55:56] just a blank list [22:56:09] i.e no list inquote [22:56:13] no instance list [22:56:16] ah, ok, that’s… a different bug :( [22:56:31] the 2fa check works properly on my test wiki but not on the live one, that’s inconvenient [22:57:03] * andrewbogott does a sync-common on labtestweb2001 [22:57:06] well, this bug does bother me [23:00:18] try turning it off and back on again [23:01:08] hello IT, that worked [23:01:37] wikitech and keystone are in perpetual disagreement about session length. One more reason to move to horizon :) [23:02:27] andrewbogott: can there be an instance type of 16 cpu, 16 G ram and 160 G HDD ? [23:03:02] or better, just change gigantic to 160 G HDD ? [23:03:20] Yeah, I can probably do that. Right now I have to drop everything and fix 2fa, can you make a ticket? [23:03:30] sure [23:05:33] 6Labs, 10Labs-Infrastructure: change m1.gigantic type to 160 G HDD space - https://phabricator.wikimedia.org/T129581#2109941 (10Matanya) [23:05:43] this one ^ [23:13:41] bd808: You here? [23:13:50] 10PAWS, 6Research-and-Data: Create a mailing list for PAWS - https://phabricator.wikimedia.org/T129297#2110003 (10ggellerman) [23:14:07] * Luke081515 made his wiki throw 503 by only running 'vagrant provision' [23:14:21] 10PAWS, 6Research-and-Data: Create a mailing list for PAWS - https://phabricator.wikimedia.org/T129297#2101483 (10ggellerman) @yuvipanda @DarTar would like to know if you can own this [23:19:56] andrewbogott: just so you get the numbers, in the month and a half the tool is in the air more than 2000 videos were uploaded using it [23:21:30] bd808: An update solved it (update of vagrant and composer) [23:22:36] Luke081515: that's the mw-vagrant version of "have you tried turning it off and on again?" ;) [23:23:10] no, I thing this was another error: https://phabricator.wikimedia.org/P2741 [23:23:16] more than 2000 lines error output [23:23:26] but the most are failed depencies [23:24:35] Luke081515: that [23:24:39] that's https://phabricator.wikimedia.org/T129343 [23:25:08] ah, solved yesterday, ok [23:41:43] matanya: long story short: you’re right, that was always permitted, and I was confused. [23:41:57] I still don’t think it /should/ be permitted, but that’s for another day [23:42:07] andrewbogott: glad to here :) [23:42:10] *hear [23:42:42] 2fa is enforced for /me/ because I have super-duper privs but that’s a special case [23:43:04] andrewbogott: can I share with you some of my pain points of the video project ? [23:43:09] sure [23:43:36] I think https://tools.wmflabs.org/nagf/?project=video and https://grafana.wikimedia.org/dashboard/db/labs-project-board?var-project=video&var-server=All [23:43:43] tell most of the sotry [23:43:48] *story [23:45:07] hm… it’s easier for me to give you RAM than it is to give you disk space [23:45:20] but we can probably throw one more instance at the problem as long as it’s on the right host [23:45:35] disk space and cpu are the main issues [23:45:52] yeah, there’s no glut of those unfortunately [23:45:53] RAM is mostly consumed by redis (as it should) [23:47:31] So what is do you suggest andrewbogott ? rate limiting ? :) [23:48:27] also, i can create it, and you can move it to the right one [23:58:23] 10PAWS, 6Research-and-Data: Create a mailing list for PAWS - https://phabricator.wikimedia.org/T129297#2110206 (10yuvipanda) I'm definitely not the best person to own this, because I don't actually know who all were at CSCW, even at the workshop. [23:58:32] matanya: ok, encoding03 [23:58:39] thanks andrewbogott [23:58:53] I may or may have to back that out if labvirt1010 crumbles under the weight :) should be ok though [23:59:14] will give it a week and see :) [23:59:32] :D [23:59:47] * andrewbogott -> dinner [23:59:59] thank you so much