[00:32:43] PROBLEM - Work requests waiting in Zuul Gearman server on contint1001 is CRITICAL: CRITICAL: 35.71% of data above the critical threshold [140.0] https://www.mediawiki.org/wiki/Continuous_integration/Zuul https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1 [00:40:41] RECOVERY - Work requests waiting in Zuul Gearman server on contint1001 is OK: OK: Less than 30.00% above the threshold [90.0] https://www.mediawiki.org/wiki/Continuous_integration/Zuul https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1 [01:07:55] PROBLEM - Work requests waiting in Zuul Gearman server on contint1001 is CRITICAL: CRITICAL: 42.86% of data above the critical threshold [140.0] https://www.mediawiki.org/wiki/Continuous_integration/Zuul https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1 [01:14:19] RECOVERY - Work requests waiting in Zuul Gearman server on contint1001 is OK: OK: Less than 30.00% above the threshold [90.0] https://www.mediawiki.org/wiki/Continuous_integration/Zuul https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1 [03:10:24] 10Phabricator, 10serviceops-radar, 10Patch-For-Review: Video embeds, even of OGV files, no longer work - https://phabricator.wikimedia.org/T228225 (10Dzahn) [10:40:48] 10Phabricator, 10serviceops-radar: Video embeds, even of OGV files, no longer work - https://phabricator.wikimedia.org/T228225 (10matmarex) 05Open→03Resolved Thank you, this indeed works! [12:08:45] legoktm: hmm the generatefancycaptcha mwscript fails to even start on beta cluster [12:09:02] hauskatze, what does it say? [12:09:18] Generating 10000 new captchas..Could not run generation script. [12:09:28] and nothing else [12:09:58] trying to see if T230245 was reproducible there [12:09:59] T230245: Mediawiki maintenance job "generate-fancycaptcha" - fatal error when trying to copy new captchas to storage - https://phabricator.wikimedia.org/T230245 [12:15:46] nice [12:18:13] hauskatze, what was the full command you ran to get that? [12:19:14] mwscript extensions/ConfirmEdit/maintenance/GenerateFancyCaptchas.php --wiki=aawiki --wordlist=/etc/fancycaptcha/words --font=/usr/share/fonts/truetype/freefont/FreeMonoBoldOblique.ttf --blacklist=/etc/fancycaptcha/badwords --fill=10000 --oldcaptcha --delete --threads=4 [12:19:27] and [12:19:40] PHP=php7.2 sudo -u www-data /usr/local/bin/mwscript extensions/ConfirmEdit/maintenance/GenerateFancyCaptchas.php enwiki --wordlist=/etc/fancycaptcha/words --font=/usr/share/fonts/truetype/freefont/FreeMonoBoldOblique.ttf --blacklist=/etc/fancycaptcha/badwords --fill=10000 --oldcaptcha --delete --threads=4 [12:19:53] both of them gave same "Could not run generation script." [12:20:35] Thinking about it, I think there was a task for confirm edit @ beta [12:21:44] Right there was T204611 which you did Krenair [12:21:45] T204611: Generate beta captchas - https://phabricator.wikimedia.org/T204611 [12:22:40] that time it let you generate 952 words only, this time it refuses to even start [12:44:09] interesting. different error [12:48:33] great [12:48:46] so I made it var_dump $cmd [12:49:10] "python '/srv/mediawiki-staging/php-master/extensions/ConfirmEdit/captcha-old.py' --key 'redacted, obviously' --output '/tmp/mw-fancycaptcha-1565441275-7c1e3b' --count '10000' --dirs '3' --wordlist '/etc/fancycaptcha/words' --font '/usr/share/fonts/truetype/freefont/FreeMonoBoldOblique.ttf' --blacklist '/etc/fancycaptcha/badwords' --threads '4'" [12:49:19] and if I run that directly, IOError: [Errno 2] No such file or directory: '/etc/fancycaptcha/badwords' [12:51:32] gonna guess that comes from puppet-private in prod [12:54:11] used /home/krenair/get-words-lists.sh to get the word files [12:55:02] (I had to sudo mkdir /etc/fancycaptcha first) [12:55:40] It's now theoretically generating 10k new captcha images [12:56:52] by the looks of htop, it's actually doing it [13:00:09] Oh dear. [13:00:21] Generating 10000 new captchas.. Done. [13:00:21] Generated 10000 captchas in 212.2 seconds [13:00:21] Getting a list of old captchas to delete... Done. [13:00:21] Copying the new captchas to storage...[Sat Aug 10 12:58:18 2019] [hphp] [26463:7f0cefb683c0:0:000001] [] [13:00:21] Fatal error: require(/srv/mediawiki-staging/php-master/includes/json/FormatJson.php): File not found in /srv/mediawiki-staging/php-master/includes/AutoLoader.php on line 109 [18:38:20] 10Phabricator, 10Mobile: [MOBILE] Search option is still on screen after click on profile option - https://phabricator.wikimedia.org/T230277 (10Zoranzoki21) [18:39:36] 10Phabricator, 10Mobile: [MOBILE] Search option is still on screen after click on profile option - https://phabricator.wikimedia.org/T230277 (10Zoranzoki21) Will test in Mozilla too. [18:42:53] 10Phabricator, 10Mobile: [MOBILE] Search option is still on screen after click on profile option - https://phabricator.wikimedia.org/T230277 (10Zoranzoki21) >>! In T230277#5407292, @Zoranzoki21 wrote: > Will test in Mozilla too. I can also reproduce this in Mozilla 68.0 [18:48:14] 10Phabricator: [MOBILE] Search option is still on screen after click on profile option - https://phabricator.wikimedia.org/T230277 (10Zoranzoki21) [20:35:23] 10Phabricator: [MOBILE] Search option is still on screen after click on profile option - https://phabricator.wikimedia.org/T230277 (10Aklapper) 05Open→03Invalid I don't see a bug [23:00:49] (03PS1) 10Urbanecm: Whitelist Viztor [integration/config] - 10https://gerrit.wikimedia.org/r/529584 [23:06:35] 10Gerrit, 10MediaWiki-extensions-LinkedWiki, 10Software-Licensing: Extension:LinkedWiki is licensed under a non-free CC-NC-BY-SA - https://phabricator.wikimedia.org/T230287 (10MaxSem) [23:07:33] 10Gerrit, 10MediaWiki-extensions-LinkedWiki, 10Software-Licensing: Extension:LinkedWiki is licensed under a non-free CC-NC-BY-SA - https://phabricator.wikimedia.org/T230287 (10MaxSem) [23:15:17] 10Gerrit, 10MediaWiki-extensions-LinkedWiki, 10Software-Licensing: Extension:LinkedWiki is licensed under a non-free CC-NC-BY-SA - https://phabricator.wikimedia.org/T230287 (10ashley) Duplicate of T189083? [23:43:08] 10Gerrit, 10MediaWiki-extensions-LinkedWiki, 10Software-Licensing: Extension:LinkedWiki is licensed under a non-free CC-NC-BY-SA - https://phabricator.wikimedia.org/T230287 (10Aklapper) Yes, good catch. I've reopened and updated T189083. [23:43:15] 10Gerrit, 10MediaWiki-extensions-LinkedWiki, 10Software-Licensing: Extension:LinkedWiki is licensed under a non-free CC-NC-BY-SA - https://phabricator.wikimedia.org/T230287 (10Aklapper)