[00:12:07] PROBLEM - Puppet run on tools-exec-1438 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [00:14:01] chasemp: you jinxed the puppet runs by talking about them :/ [00:14:09] yep [00:29:18] RECOVERY - Puppet run on tools-exec-1433 is OK: OK: Less than 1.00% above the threshold [0.0] [00:47:08] RECOVERY - Puppet run on tools-exec-1438 is OK: OK: Less than 1.00% above the threshold [0.0] [00:50:17] PROBLEM - Puppet run on tools-exec-1433 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [00:59:26] 10Tool-Labs-tools-Xtools, 03Community-Tech-Sprint: Fix caching problems with XTools - https://phabricator.wikimedia.org/T162753#3196335 (10Samwilson) a:03Samwilson It's something weird about the redis extension being installed and so it's ignorning the \Predis\Client that a 5.5-compatible library is providin... [01:00:54] PROBLEM - Puppet run on tools-exec-1441 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [01:25:26] PROBLEM - Puppet run on tools-exec-1437 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [01:30:19] RECOVERY - Puppet run on tools-exec-1433 is OK: OK: Less than 1.00% above the threshold [0.0] [01:34:45] 10Tool-Labs-tools-Xtools, 03Community-Tech-Sprint: Fix caching problems with XTools - https://phabricator.wikimedia.org/T162753#3196369 (10Samwilson) So the error being thrown when purging prod cache was this: PHP Fatal error: Call to undefined method Redis::scan() in vendor/symfony/symfony/src/Symfony/Co... [01:40:51] RECOVERY - Puppet run on tools-exec-1441 is OK: OK: Less than 1.00% above the threshold [0.0] [02:00:29] RECOVERY - Puppet run on tools-exec-1437 is OK: OK: Less than 1.00% above the threshold [0.0] [02:14:45] 10Tool-Labs-tools-Xtools, 06Community-Tech: Epic: Rewriting XTools - https://phabricator.wikimedia.org/T153112#3196395 (10MusikAnimal) [02:14:48] 10Tool-Labs-tools-Xtools, 03Community-Tech-Sprint: Fix caching problems with XTools - https://phabricator.wikimedia.org/T162753#3196392 (10MusikAnimal) 05Open>03Resolved >>! In T162753#3196369, @Samwilson wrote: > So, to fix, manually nuke the cache directories How frustrating, could have sworn I tried th... [02:23:04] PROBLEM - Puppet run on tools-exec-1432 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [02:58:05] RECOVERY - Puppet run on tools-exec-1432 is OK: OK: Less than 1.00% above the threshold [0.0] [03:20:35] PROBLEM - Puppet run on tools-exec-1442 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [03:55:36] RECOVERY - Puppet run on tools-exec-1442 is OK: OK: Less than 1.00% above the threshold [0.0] [04:20:05] 06Labs, 10Labs-Infrastructure, 07artificial-intelligence: Provide large disk space to WikiBrain for memory-mapped file - https://phabricator.wikimedia.org/T161554#3196506 (10Shilad) Just to follow up on this. Aaron's estimates are pretty accurate. The disk cached data structures require about 200GB for large... [04:25:45] PROBLEM - Host tools-docker-builder-04 is DOWN: CRITICAL - Host Unreachable (10.68.22.217) [05:12:02] 06Labs, 10Labs-Infrastructure, 07artificial-intelligence: Provide large disk space to WikiBrain for memory-mapped file - https://phabricator.wikimedia.org/T161554#3196520 (10bd808) >>! In T161554#3153726, @Halfak wrote: > It seems that, right now, at least 200GB is essential. I expect that might double over... [05:19:05] PROBLEM - Puppet run on tools-exec-1432 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [05:21:08] 10Tool-Labs-tools-Xtools, 03Community-Tech-Sprint: Have Edit Counter use same architecture and front-end as the other pieces that have been re-written - https://phabricator.wikimedia.org/T160481#3100408 (10kaldari) Just one nit-pick from me and a suggestion from Matthew. [05:54:04] RECOVERY - Puppet run on tools-exec-1432 is OK: OK: Less than 1.00% above the threshold [0.0] [06:01:53] PROBLEM - Puppet run on tools-exec-1441 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [06:36:53] RECOVERY - Puppet run on tools-exec-1441 is OK: OK: Less than 1.00% above the threshold [0.0] [06:46:13] PROBLEM - Puppet run on tools-webgrid-lighttpd-1412 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [07:16:34] 06Labs, 10Tool-Labs, 10Tool-Labs-tools-Other: Shut down "cewbot" - https://phabricator.wikimedia.org/T160264#3196617 (10Kanashimi) OK. It is done. [07:21:11] RECOVERY - Puppet run on tools-webgrid-lighttpd-1412 is OK: OK: Less than 1.00% above the threshold [0.0] [07:27:26] PROBLEM - Puppet run on tools-exec-1437 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [08:32:29] RECOVERY - Puppet run on tools-exec-1437 is OK: OK: Less than 1.00% above the threshold [0.0] [08:45:31] PROBLEM - Puppet run on tools-exec-1430 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [08:58:47] (03PS3) 10Lokal Profil: Prepare monument_tables for Wikidata and add tests [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/342038 [09:07:13] 10Tool-Labs-tools-Xtools, 06Community-Tech: Epic: Rewriting XTools - https://phabricator.wikimedia.org/T153112#3196882 (10Matthewrbowker) [09:08:33] 10Tool-Labs-tools-Xtools, 06Community-Tech: Epic: Rewriting XTools - https://phabricator.wikimedia.org/T153112#2870106 (10Matthewrbowker) [09:10:50] PROBLEM - Puppet run on tools-exec-1440 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [09:14:39] (03PS1) 10Lokal Profil: Replace homemade assertion with assertLessEqual [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/349178 [09:20:31] RECOVERY - Puppet run on tools-exec-1430 is OK: OK: Less than 1.00% above the threshold [0.0] [09:45:48] RECOVERY - Puppet run on tools-exec-1440 is OK: OK: Less than 1.00% above the threshold [0.0] [10:17:35] PROBLEM - Puppet run on tools-exec-1434 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [10:41:49] PROBLEM - Puppet run on tools-exec-1440 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [10:57:33] RECOVERY - Puppet run on tools-exec-1434 is OK: OK: Less than 1.00% above the threshold [0.0] [10:58:30] hi, why isn't it possible to read the actual contents of de:Portal_Diskussion:Lebewesen by API? [11:16:48] RECOVERY - Puppet run on tools-exec-1440 is OK: OK: Less than 1.00% above the threshold [0.0] [12:23:29] PROBLEM - Puppet run on tools-exec-1437 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [12:36:26] <_joe_> silver (wikitech) spits this every second or so: [12:36:27] <_joe_> [Thu Apr 20 12:35:22.496170 2017] [:error] [pid 21544] [client 10.68.23.199:56256] PHP Warning: Memcached::touch(): touch is only supported with binary protocol in /srv/mediawiki/php-1.29.0-wmf.20/includes/libs/objectcache/MemcachedPeclBagOStuff.php on line 253, referer: https://wikitech.wikimedia.org [12:36:48] <_joe_> andrewbogott, chasemp ^^ [12:37:42] thanks _joe_ [12:38:15] 06Labs, 10wikitech.wikimedia.org: VisualEditor is broken on wikitech - https://phabricator.wikimedia.org/T163438#3197386 (10Joe) [12:38:58] 06Labs, 06Operations: wikitech logging constant errors from /MemcachedPeclBagOStuff.php - https://phabricator.wikimedia.org/T163439#3197399 (10chasemp) [12:39:00] 06Labs, 10wikitech.wikimedia.org: VisualEditor is broken on wikitech - https://phabricator.wikimedia.org/T163438#3197411 (10Joe) p:05Triage>03High [12:39:03] <_joe_> chasemp: thank me later! ^^ [12:39:04] 06Labs, 06Operations: wikitech logging constant errors from /MemcachedPeclBagOStuff.php - https://phabricator.wikimedia.org/T163439#3197413 (10chasemp) p:05Triage>03Normal [12:39:05] <_joe_> :P [12:39:18] <_joe_> VE is broken on wikitech [12:39:45] <_joe_> I discovered by pressing "edit" instead of "edit source", because I'll admit it, I find VE nice [12:40:04] huh, iirc andrew and bryan were doing something there within the last week or so. I'm throwing a task their way on the memcache thing but I'm not sure what the deal with VE is [12:40:32] 06Labs, 06Operations: wikitech logging constant errors from /MemcachedPeclBagOStuff.php - https://phabricator.wikimedia.org/T163439#3197399 (10chasemp) @bd808 and @andrew I think possibly you guys were doing something related here recently? [13:08:29] RECOVERY - Puppet run on tools-exec-1437 is OK: OK: Less than 1.00% above the threshold [0.0] [13:11:08] !log tools.cobot set crons to schedule on tools-exec-1422 only for testing. This tool is launching jobs that write and then read hundreds of megs in a few minutes. I caught it tripping up puppet on tools-exec-1437 [13:11:10] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cobot/SAL [13:12:48] 06Labs: IO issues for Tools instances flapping with iowait and puppet failure - https://phabricator.wikimedia.org/T161898#3197517 (10chasemp) !log tools.cobot set crons to schedule on tools-exec-1422 only for testing. This tool is launching jobs that write and then read hundreds of megs in a few minutes. I cau... [13:27:06] 06Labs: IO issues for Tools instances flapping with iowait and puppet failure - https://phabricator.wikimedia.org/T161898#3197570 (10chasemp) I see one maintainer for cobot and I can't find a Phab account for them https://wikitech.wikimedia.org/wiki/Shell_Request/MistrX [14:19:43] 10Tool-Labs-tools-Xtools, 06Collaboration-Team-Triage, 06Community-Tech, 10Flow: Add Flow contributions to Xtools - https://phabricator.wikimedia.org/T136950#3197711 (10Matthewrbowker) a:05Matthewrbowker>03Samwilson [14:20:26] 10Tool-Labs-tools-Xtools, 06Community-Tech, 07TestMe: Everything is displayed as 0 - https://phabricator.wikimedia.org/T97153#3197713 (10Matthewrbowker) a:05Matthewrbowker>03Samwilson [14:22:32] 10Tool-Labs-tools-Xtools: RfX Analysis for oversight and checkuser elections on dewiki - https://phabricator.wikimedia.org/T163444#3197717 (10Freddy2001) [14:25:39] 10Tool-Labs-tools-Xtools: RfX Analysis for oversight and checkuser elections on dewiki - https://phabricator.wikimedia.org/T163444#3197732 (10Freddy2001) The prefixes for oversight is https://de.wikipedia.org/wiki/Wikipedia:Oversightkandidaturen and for checkuser is https://de.wikipedia.org/wiki/Wikipedia:Checku... [14:42:27] 06Labs, 06Operations: Ensure we can survive a loss of labservices1001 - https://phabricator.wikimedia.org/T163402#3197808 (10Paladox) @chasemp im not sure if you already thought of this but what about switching labs services to the second labservice if there is a secondary one? I am unsure if nodepool will be... [14:59:26] 10Tool-Labs-tools-Other, 06Commons: CropTool broken: Empty input file, JPEGErrorHandler - https://phabricator.wikimedia.org/T163397#3197840 (10zhuyifei1999) a:03Danmichaelo (Assigning to [[https://tools.wmflabs.org/?tool=croptool|tool maintainer]]) [15:02:51] 06Labs: IO issues for Tools instances flapping with iowait and puppet failure - https://phabricator.wikimedia.org/T161898#3197879 (10bd808) >>! In T161898#3197570, @chasemp wrote: > I see one maintainer for cobot and I can't find a Phab account for them https://wikitech.wikimedia.org/wiki/Shell_Request/MistrX A... [15:03:09] chasemp: ^ I'm going to shutdown the cobot jobs and leave a talk page note [15:03:36] ok [15:03:44] good w/ me [15:04:30] !log tools.cobot Disabled cron jobs [15:04:32] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cobot/SAL [15:14:28] things like that cobot tool are why I'd really like some sort of proof of life check for tool mantainers [15:24:46] !log tools.heritage Deploy latest from Git master: ae1e775 (T138517), 2bd9781, 04a19e0 (T158911), e8dbe35, 4de8898, ea942d8, f956ab5, b63ebac, dd8c0c8, 63b3bc8, 8b15472 [15:24:53] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.heritage/SAL [15:24:55] T158911: Docker setup is broken because of InnoDB and FULLTEXT - https://phabricator.wikimedia.org/T158911 [15:24:56] T138517: mysqldump is timing out preventing all tables from being included in the dump - https://phabricator.wikimedia.org/T138517 [15:29:44] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/1-Byte was created, changed by 1-Byte link https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Access_Request/1-Byte edit summary: Created page with "{{Tools Access Request |Justification=To develop applications to ease maintaining items on Wikidata (mainly for items related to Germany/Switzerland/Austria) |Completed=false..." [15:42:53] 06Labs, 06Operations: wikitech logging constant errors from /MemcachedPeclBagOStuff.php - https://phabricator.wikimedia.org/T163439#3198029 (10bd808) [15:42:54] 06Labs, 10wikitech.wikimedia.org, 07Wikimedia-log-errors: PHP Warning: Memcached::touch(): touch is only supported with binary protocol - https://phabricator.wikimedia.org/T158613#3198027 (10bd808) [15:43:51] 06Labs, 06Operations: Update documentation for Tools Proxy failover - https://phabricator.wikimedia.org/T163390#3198035 (10chasemp) first pass https://wikitech.wikimedia.org/w/index.php?title=Nova_Resource%3ATools%2FAdmin&type=revision&diff=1757014&oldid=1756658 [15:49:16] 06Labs: IO issues for Tools instances flapping with iowait and puppet failure - https://phabricator.wikimedia.org/T161898#3198062 (10bd808) [15:49:18] 06Labs, 10Tool-Labs, 10Tool-Labs-tools-Other: perfectbot runs ListSpeedyDeletions job once per minute - https://phabricator.wikimedia.org/T162978#3198061 (10bd808) [15:49:31] 06Labs: IO issues for Tools instances flapping with iowait and puppet failure - https://phabricator.wikimedia.org/T161898#3146683 (10bd808) [15:49:33] 06Labs, 10Tool-Labs, 10Tool-Labs-tools-Other: request tool runs secWatch job once per minute - https://phabricator.wikimedia.org/T162979#3198063 (10bd808) [15:50:12] 06Labs, 10Tool-Labs, 10Tool-Labs-tools-Other: ruarbcom tool runs count.py job once per minute - https://phabricator.wikimedia.org/T163075#3198069 (10bd808) [15:58:48] 06Labs, 10Labs-Infrastructure, 13Patch-For-Review: Shorter token life for novaobserver/novaadmin - https://phabricator.wikimedia.org/T163259#3198106 (10Andrew) 05Open>03Resolved root@MISC m5[keystone]> SELECT COUNT(*) FROM token; +----------+ | COUNT(*) | +----------+ | 49197 | +----------+ 1 row in s... [16:15:46] 10PAWS, 10Jupyter-Hub: I can't login my bot in JUPYTER - https://phabricator.wikimedia.org/T135306#3198206 (10Aklapper) Is this still a problem? [16:17:30] (03CR) 10ArthurPSmith: [C: 031] "Hi - sorry to be slowing checking on this. Your change looks fine. The only issue is Hydrogen is now its own color (well same as nitrogen " [labs/tools/ptable] - 10https://gerrit.wikimedia.org/r/348696 (owner: 10Ricordisamoa) [17:09:12] 06Labs, 10Tool-Labs, 10Tool-Labs-tools-Other, 15User-bd808: bigbrother not trying to start missing iabot job - https://phabricator.wikimedia.org/T163265#3198493 (10bd808) 05Open>03Resolved The bigbrother watchdog script had been broken since I ported it from perl to python. I know that I tested it, but... [17:11:47] !log tools kill -INT 19897 on tools-proxy-02 to stop a hung nginx child process left from the last graceful restart of nginx [17:11:50] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [17:15:49] !log tools Deleted shutdown VM tools-docker-builder-04; tools-docker-builder-05 is the new hotness [17:15:52] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [17:24:58] 10Tool-Labs-tools-Xtools, 06Community-Tech: XTools: Cleanup i18n messages - https://phabricator.wikimedia.org/T163468#3198596 (10MusikAnimal) [17:26:11] 10Tool-Labs-tools-Xtools: If revisions are revdel'd, articleinfo compares the surrounding edits as if it were one edit - https://phabricator.wikimedia.org/T148857#3198612 (10MusikAnimal) [17:26:16] 10Tool-Labs-tools-Xtools, 06Community-Tech: [Epic] Rewrite XTools: Articleinfo - https://phabricator.wikimedia.org/T157602#3198613 (10MusikAnimal) [17:26:32] 10Tool-Labs-tools-Xtools, 06Community-Tech: If revisions are revdel'd, articleinfo compares the surrounding edits as if it were one edit - https://phabricator.wikimedia.org/T148857#2734900 (10MusikAnimal) [17:28:26] 10Tool-Labs-tools-Xtools, 06Community-Tech, 07I18n: XTools: Cleanup i18n messages - https://phabricator.wikimedia.org/T163468#3198596 (10MusikAnimal) [17:29:14] 10Tool-Labs-tools-Xtools, 06Community-Tech, 07I18n: XTools: Cleanup i18n messages - https://phabricator.wikimedia.org/T163468#3198596 (10MusikAnimal) [17:29:15] 10Tool-Labs-tools-Xtools, 07I18n: [[Intuition:X'stools-time ago/ksh]] i18n issue - https://phabricator.wikimedia.org/T125296#3198643 (10MusikAnimal) [17:30:15] Hello, I was just granted labs access to access replica databases. When I try to run "sql enwiki" from a shell after sshing in, I get a permission denied error. What's wrong? [17:32:09] 10Tool-Labs-tools-Xtools: Populate labs/tools/wikiviewstats git repo - https://phabricator.wikimedia.org/T108058#3198651 (10MusikAnimal) 05Open>03Resolved a:03MusikAnimal Boldly closing as I don't think wikiviewstats will be brought back to life. R.I.P. 🛌 ☮ [17:32:11] codeofdusk: it sounds like your account did not get proper db credentials. [17:32:28] I'll take a look at the files and see if the problem is obvious [17:32:37] what is your shell user name? [17:32:57] bd808: codeofdusk [17:34:00] codeofdusk: ok. checking things out now [17:34:16] bd808: Thanks. [17:36:12] hmm.. you have a replica.my.cnf file that looks reasonable, but apparently the db servers don't have matching data for your credentials. [17:36:23] bd808: i can check [17:36:38] madhuvishy: awesome. u16865 [17:37:29] 10Tool-Labs-tools-Xtools, 07JavaScript: Adminstats table doesn't sort properly due to space thousand separator - https://phabricator.wikimedia.org/T137636#2373846 (10MusikAnimal) This has been fixed in the rewrite, which you should expect in the coming months :) I will not close this but in my opinion "Decline... [17:37:47] madhuvishy: can you walk me through how to check? I think I have access to all the right servers these days, but I don't know where to actually look [17:38:16] ya once sec, i'll run these commands, and then share them with you [17:38:41] perfect [17:39:30] 10Tool-Labs-tools-Xtools: Update Goan Konkani namespace names in X!'s Tools edit counter - https://phabricator.wikimedia.org/T156641#2982134 (10MusikAnimal) This has been fixed in the rewrite. [17:44:50] 10Tool-Labs-tools-Xtools, 03Community-Tech-Sprint: Remove references to "Range Contributions" and "Autoblock" within xTools code - https://phabricator.wikimedia.org/T163374#3198724 (10MusikAnimal) a:03MusikAnimal [17:46:49] 10Tool-Labs-tools-Xtools, 06Community-Tech, 07I18n: XTools: Cleanup i18n messages - https://phabricator.wikimedia.org/T163468#3198740 (10MusikAnimal) [17:54:15] TBolliger: that was too late for me, please pm me if you are around, if not, i'll send an email [17:54:49] codeofdusk: madhuvishy is still trying to figure out why your account isn't setup properly everywhere. [17:56:37] bd808: madhuvishy: Alright. Please keep me up to date. [17:57:47] !log tools.openstack-browser Deployed 7abd6ee to show LDAP user count now that T162745 is fixed [17:57:50] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.openstack-browser/SAL [17:57:50] T162745: allow paging to work properly in ldap - https://phabricator.wikimedia.org/T162745 [17:58:40] codeofdusk: current status is that your account exists on the new db servers that are not in common use but not on the main servers. [17:59:16] bd808: How do I set up my account for use on the main DB servers? [17:59:50] you should not have to do anything special. this is some bug in our account creation process [18:00:18] bd808: Alright so when can I expect my account to be provisioned on all servers? [18:00:46] codeofdusk: as soon as we figure out why the user creation is failing :) [18:01:12] codeofdusk: in the mean time you could try using https://quarry.wmflabs.org/ to run queries [18:02:49] bd808: Yep, I knew about that. But it has fairly short timeouts and doesn't do large queries. Can you leave a message on my talk page when everything is working? [18:03:04] codeofdusk: sure [18:03:23] bd808: Great, thanks! [18:04:44] 06Labs, 06Operations: Update documentation for Tools Proxy failover - https://phabricator.wikimedia.org/T163390#3198863 (10chasemp) a:05chasemp>03Andrew https://wikitech.wikimedia.org/w/index.php?title=Nova_Resource%3ATools%2FAdmin&type=revision&diff=1757036&oldid=1756658 [18:11:18] codeofdusk: check now? [18:12:36] madhuvishy: Yep, working! [18:12:38] madhuvishy: Thanks. [18:12:54] codeofdusk: cool :) np! [18:26:28] 06Labs, 10Striker, 10wikitech.wikimedia.org: Allow viewing/searching LDAP account creations including date - https://phabricator.wikimedia.org/T163478#3198972 (10bd808) [18:40:20] 10Tool-Labs-tools-Xtools, 03Community-Tech-Sprint: Remove references to "Range Contributions" and "Autoblock" within xTools code - https://phabricator.wikimedia.org/T163374#3199076 (10MusikAnimal) Done with https://github.com/x-tools/xtools-rebirth/commit/2c51e601bb371f00e1bb6a9b0cd44f0a6f58e749 This commit a... [18:42:09] 10PAWS, 10Jupyter-Hub: I can't login my bot in JUPYTER - https://phabricator.wikimedia.org/T135306#2294878 (10Framawiki) @Aklapper As no commits was pushed on [[ https://github.com/yuvipanda/ipynb-paws | on ]] [[ https://github.com/yuvipanda/paws | Github ]] since many months, I suppose that this issue is not... [18:56:15] !log wikilabels deactivates campains (37, 43, 47) on enwiki https://en.wikipedia.org/wiki/Wikipedia_talk:Labels#Deactivated_campaigns [18:56:18] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikilabels/SAL [19:05:37] 06Labs, 06Operations: Initial OpenStack Neutron PoC deployment in Labtest - https://phabricator.wikimedia.org/T153099#3199148 (10chasemp) some key points I have taken ss of: {F7671560} {F7671561} {F7671562} {F7671563} [19:56:33] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/1-Byte was modified, changed by BryanDavis link https://wikitech.wikimedia.org/w/index.php?diff=1757048 edit summary: [20:04:16] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Angela was modified, changed by BryanDavis link https://wikitech.wikimedia.org/w/index.php?diff=1757050 edit summary: [20:05:48] 06Labs, 06Operations, 13Patch-For-Review: Ensure kubelet is stopped on Tools Proxy hosts - https://phabricator.wikimedia.org/T163391#3199496 (10madhuvishy) 05Open>03Resolved [20:06:57] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Fisherinformation was modified, changed by BryanDavis link https://wikitech.wikimedia.org/w/index.php?diff=1757052 edit summary: [20:21:11] Anyone having an idea how to investigate / proceed in https://phabricator.wikimedia.org/T135306 ? It's been stuck for a while hence asking here :-/ [20:21:22] ("I can't login my bot in JUPYTER") [20:21:47] andre__: no promises but I will look into it if i have the ability to do so [20:22:05] thanks [20:22:09] no problem [20:24:27] andre__: sorry i know nothing about paws, and when i try logging in it works fine for me on chrome. [20:24:53] yeah, I wonder if someone with shell access could investigate [20:44:47] andre__: as far as I know, yuvipanda and madhuvishy are the only folks who know how to debug and fix PAWS backend problems [20:45:01] I was afraid of something like that :-/ [20:45:04] madhuvishy: context is T135306 [20:45:05] T135306: I can't login my bot in JUPYTER - https://phabricator.wikimedia.org/T135306 [20:45:10] * madhuvishy looks [20:45:23] * andre__ CC's [20:45:40] thanks in advance in case you find time... [20:46:20] ha ha and comments in tamil :) [20:47:29] 10Tool-Labs-tools-Xtools, 03Community-Tech-Sprint: XTools: Clean up "Pages created" tool - https://phabricator.wikimedia.org/T162752#3199700 (10MusikAnimal) PR: https://github.com/x-tools/xtools-rebirth/pull/20 All the changes to i18n messages were just changing one key name, and then sorting the list. Live d... [20:47:38] 10Tool-Labs-tools-Xtools, 03Community-Tech-Sprint: XTools: Clean up "Pages created" tool - https://phabricator.wikimedia.org/T162752#3199702 (10MusikAnimal) [20:47:46] I think this was someone that Yuvi worked with at a hackathon event at some point [20:48:02] right [20:49:58] 10Tool-Labs-tools-Xtools, 03Community-Tech-Sprint: XTools: Clean up "Pages created" tool - https://phabricator.wikimedia.org/T162752#3199712 (10MusikAnimal) [20:53:36] andre__: I don't see a running notebook for this user, and not really sure what's up - especially if it wasn't trivial enough for Yuvi to fix it - He's back in May and said he'll work on this kinda stuff then. The best I've got is, lets wait for him to be back? [20:54:32] madhuvishy: Uh yay, he's back in May? I didn't know that. Sounds good! :D Thanks a lot! :) [20:54:59] no problem :) [21:43:58] 10Tool-Labs-tools-Xtools, 06Community-Tech: XTools: Trying to get info about a talk page gives 500 internal server error - https://phabricator.wikimedia.org/T163508#3199887 (10kaldari) [22:08:56] 06Labs, 10wikitech.wikimedia.org, 05codfw-rollout: VisualEditor broken on wikitech: "Error loading data from server: apierror-visualeditor-docserver-http: HTTP 500." - https://phabricator.wikimedia.org/T163438#3199966 (10Aklapper) [22:10:55] 06Labs, 10wikitech.wikimedia.org, 05codfw-rollout: VisualEditor broken on wikitech: "Error loading data from server: apierror-visualeditor-docserver-http: HTTP 500." - https://phabricator.wikimedia.org/T163438#3197386 (10bd808) I saw this too right after the DC switch. I imagine it has something to do with t... [22:28:26] 10Tool-Labs-tools-Xtools, 03Community-Tech-Sprint: XTools: Clean up "Pages created" tool - https://phabricator.wikimedia.org/T162752#3200038 (10kaldari) 05Open>03Resolved Looks great! [22:28:28] 10Tool-Labs-tools-Xtools, 06Community-Tech: Epic: Rewriting XTools - https://phabricator.wikimedia.org/T153112#3200040 (10kaldari) [23:12:03] 06Labs, 10Labs-Infrastructure, 10MediaWiki-extensions-GettingStarted: Labs undefined index due to data center switchover - https://phabricator.wikimedia.org/T163514#3200135 (10Mattflaschen-WMF) [23:21:06] 06Labs, 10Labs-Infrastructure, 10MediaWiki-extensions-GettingStarted: Labs undefined index due to data center switchover - https://phabricator.wikimedia.org/T163514#3200173 (10Mattflaschen-WMF)