[00:08:16] RECOVERY - English Wikipedia Mobile Main page on beta-cluster is OK: HTTP OK: HTTP/1.1 200 OK - 33396 bytes in 0.715 second response time [00:20:03] Project beta-update-databases-eqiad build #17299: 04STILL FAILING in 3.7 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/17299/ [00:32:29] howdy! was there a change deployed today that might affect imageinfo API queries on Commons? [00:44:24] nevermind, tracked down my problem. [00:53:57] PROBLEM - Puppet errors on deployment-ores-redis-01 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [00:59:36] * AndyRussG waves at ragesoss ;) [01:07:04] * ragesoss waves back at AndyRussG [01:20:04] Project beta-update-databases-eqiad build #17300: 04STILL FAILING in 3.7 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/17300/ [01:33:59] RECOVERY - Puppet errors on deployment-ores-redis-01 is OK: OK: Less than 1.00% above the threshold [0.0] [01:41:55] PROBLEM - Puppet errors on deployment-mathoid is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [01:44:48] PROBLEM - Puppet errors on deployment-logstash2 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [02:05:21] PROBLEM - Puppet errors on deployment-restbase02 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [02:06:26] PROBLEM - Puppet errors on deployment-aqs01 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [02:19:49] RECOVERY - Puppet errors on deployment-logstash2 is OK: OK: Less than 1.00% above the threshold [0.0] [02:20:03] Project beta-update-databases-eqiad build #17301: 04STILL FAILING in 3.4 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/17301/ [02:21:55] RECOVERY - Puppet errors on deployment-mathoid is OK: OK: Less than 1.00% above the threshold [0.0] [02:31:48] PROBLEM - Puppet errors on deployment-conf03 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [02:40:21] RECOVERY - Puppet errors on deployment-restbase02 is OK: OK: Less than 1.00% above the threshold [0.0] [02:41:29] RECOVERY - Puppet errors on deployment-aqs01 is OK: OK: Less than 1.00% above the threshold [0.0] [02:45:46] PROBLEM - Puppet errors on deployment-elastic05 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [02:45:48] PROBLEM - Puppet errors on deployment-logstash2 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [02:55:45] PROBLEM - Puppet errors on deployment-urldownloader is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [02:56:07] PROBLEM - Puppet errors on deployment-kafka01 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [03:11:47] RECOVERY - Puppet errors on deployment-conf03 is OK: OK: Less than 1.00% above the threshold [0.0] [03:20:04] Project beta-update-databases-eqiad build #17302: 04STILL FAILING in 4.2 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/17302/ [03:20:44] RECOVERY - Puppet errors on deployment-elastic05 is OK: OK: Less than 1.00% above the threshold [0.0] [03:20:50] RECOVERY - Puppet errors on deployment-logstash2 is OK: OK: Less than 1.00% above the threshold [0.0] [03:25:58] PROBLEM - Puppet errors on deployment-mira is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [03:30:43] RECOVERY - Puppet errors on deployment-urldownloader is OK: OK: Less than 1.00% above the threshold [0.0] [03:31:05] RECOVERY - Puppet errors on deployment-kafka01 is OK: OK: Less than 1.00% above the threshold [0.0] [03:43:08] PROBLEM - Puppet errors on deployment-mcs01 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [04:06:00] RECOVERY - Puppet errors on deployment-mira is OK: OK: Less than 1.00% above the threshold [0.0] [04:10:34] Project selenium-MultimediaViewer » firefox,beta,Linux,BrowserTests build #401: 04FAILURE in 14 min: https://integration.wikimedia.org/ci/job/selenium-MultimediaViewer/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/401/ [04:18:13] RECOVERY - Puppet errors on deployment-mcs01 is OK: OK: Less than 1.00% above the threshold [0.0] [04:20:04] Project beta-update-databases-eqiad build #17303: 04STILL FAILING in 3.8 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/17303/ [05:11:46] PROBLEM - Puppet errors on deployment-elastic05 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [05:20:03] Project beta-update-databases-eqiad build #17304: 04STILL FAILING in 3.5 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/17304/ [05:36:08] PROBLEM - Puppet errors on deployment-elastic06 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [05:46:49] PROBLEM - Puppet errors on deployment-logstash2 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [05:51:45] RECOVERY - Puppet errors on deployment-elastic05 is OK: OK: Less than 1.00% above the threshold [0.0] [06:11:08] RECOVERY - Puppet errors on deployment-elastic06 is OK: OK: Less than 1.00% above the threshold [0.0] [06:12:46] PROBLEM - Puppet errors on deployment-elastic05 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [06:20:04] Project beta-update-databases-eqiad build #17305: 04STILL FAILING in 3.8 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/17305/ [06:21:49] RECOVERY - Puppet errors on deployment-logstash2 is OK: OK: Less than 1.00% above the threshold [0.0] [07:12:35] RECOVERY - Free space - all mounts on deployment-fluorine02 is OK: OK: All targets OK [07:20:04] Project beta-update-databases-eqiad build #17306: 04STILL FAILING in 3.9 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/17306/ [07:22:44] PROBLEM - Puppet errors on deployment-urldownloader is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [07:22:46] RECOVERY - Puppet errors on deployment-elastic05 is OK: OK: Less than 1.00% above the threshold [0.0] [07:42:51] PROBLEM - Puppet errors on deployment-logstash2 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [08:02:43] RECOVERY - Puppet errors on deployment-urldownloader is OK: OK: Less than 1.00% above the threshold [0.0] [08:20:04] Project beta-update-databases-eqiad build #17307: 04STILL FAILING in 4.1 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/17307/ [08:22:54] RECOVERY - Puppet errors on deployment-logstash2 is OK: OK: Less than 1.00% above the threshold [0.0] [08:38:49] !log updated puppet on deployment-puppetmaster02 to 3.8.5-2~bpo8+2 [08:38:53] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [09:20:04] Project beta-update-databases-eqiad build #17308: 04STILL FAILING in 4.2 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/17308/ [09:22:56] PROBLEM - Puppet staleness on deployment-prometheus01 is CRITICAL: CRITICAL: 11.11% of data above the critical threshold [43200.0] [09:54:20] !help [09:54:36] I am running http://meta.wikimedia.org/wiki/WM-Bot version wikimedia bot v. 2.8.0.0 [libirc v. 1.0.3] my source code is licensed under GPL and located at https://github.com/benapetr/wikimedia-bot I will be very happy if you fix my bugs or implement new features [09:54:36] @help [09:55:12] No results were found, remember, the bot is searching through content of keys and their names [09:55:12] @search prometheus [09:56:01] @key prometheus http://beta-prometheus.wmflabs.org/beta/alerts [09:56:06] Unknown command type @commands for a list of all commands I know [09:56:06] @help key [09:56:10] I know: add, changepass, channel-info, channellist, commands, configure, drop, github-, github+, github-off, github-on, grant, grantrole, help, info, instance, join, language, notify, optools-off, optools-on, optools-permanent-off, optools-permanent-on, part, rc-ping, rc-restart, reauth, recentchanges-bot-off, recentchanges-bot-on, recentchanges-minor-off, recentchanges-minor-on, recentchanges-off, recentchanges-on, reload, restart, revoke, revokerole, seen, seen-host, seen-off, seen-on, seenrx, suppress-off, suppress-on, systeminfo, system-rm, time, traffic-off, traffic-on, translate, trustadd, trustdel, trusted, uptime, verbosity--, verbosity++, wd, whoami [09:56:10] @commands [09:56:57] !prometheus is http://beta-prometheus.wmflabs.org/beta/alerts [09:56:57] Key was added [09:57:02] !prometheus [09:57:02] http://beta-prometheus.wmflabs.org/beta/alerts [10:09:03] !log deployment-etcd-01: fixed puppet run [10:09:06] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [10:10:43] !log deployment-prep : resetting puppet master to last known snapshot snapshot-20170523T0010 . All cherry picks got deleted [10:10:46] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [10:19:43] RECOVERY - Puppet errors on deployment-etcd-01 is OK: OK: Less than 1.00% above the threshold [0.0] [10:19:45] !log deployment-prep rebased puppet repo with: git rebase -X theirs [10:19:48] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [10:20:04] Project beta-update-databases-eqiad build #17309: 04STILL FAILING in 4.1 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/17309/ [10:20:27] RECOVERY - Long lived cherry-picks on puppetmaster on deployment-puppetmaster02 is OK: OK: Less than 100.00% above the threshold [0.0] [10:32:48] PROBLEM - Puppet errors on deployment-conf03 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [10:45:55] PROBLEM - Puppet errors on deployment-imagescaler01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [10:59:33] PROBLEM - Puppet errors on deployment-pdf01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [11:12:49] RECOVERY - Puppet errors on deployment-conf03 is OK: OK: Less than 1.00% above the threshold [0.0] [11:20:04] Project beta-update-databases-eqiad build #17310: 04STILL FAILING in 3.8 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/17310/ [12:13:49] PROBLEM - Puppet errors on deployment-logstash2 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [12:14:33] PROBLEM - Puppet errors on deployment-restbase01 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [12:15:17] PROBLEM - Puppet errors on deployment-mediawiki04 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [12:18:27] PROBLEM - Puppet errors on deployment-fluorine02 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [12:20:04] Project beta-update-databases-eqiad build #17311: 04STILL FAILING in 3.6 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/17311/ [12:22:19] !log deployment-prep: finished rebase of puppet.git [12:22:22] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [12:22:27] and of course that broke puppet on other hosts [12:22:31] !prometheus [12:22:32] http://beta-prometheus.wmflabs.org/beta/alerts [12:33:51] RECOVERY - Puppet errors on deployment-logstash2 is OK: OK: Less than 1.00% above the threshold [0.0] [12:34:33] RECOVERY - Puppet errors on deployment-restbase01 is OK: OK: Less than 1.00% above the threshold [0.0] [12:35:17] RECOVERY - Puppet errors on deployment-mediawiki04 is OK: OK: Less than 1.00% above the threshold [0.0] [12:38:27] RECOVERY - Puppet errors on deployment-fluorine02 is OK: OK: Less than 1.00% above the threshold [0.0] [13:11:45] (03CR) 10Hashar: "The mw-tools-codesniffer-mwcore-testrun job seems fixed. It is now running on Jessie with HHVM and that closes T157750." [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/355067 (https://phabricator.wikimedia.org/T142474) (owner: 10Legoktm) [13:12:56] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 10MediaWiki-Codesniffer, 06Release-Engineering-Team (Kanban), 13Patch-For-Review: mw-tools-codesniffer-mwcore-testrun test fails with php is not hhvm - https://phabricator.wikimedia.org/T157750#3288989 (10hashar) 05Open>03Resolv... [13:20:04] Project beta-update-databases-eqiad build #17312: 04STILL FAILING in 4.2 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/17312/ [13:24:39] PROBLEM - Puppet errors on integration-slave-trusty-1006 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [13:38:53] PROBLEM - Puppet errors on integration-slave-trusty-1004 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [13:40:09] PROBLEM - Puppet errors on integration-slave-trusty-1001 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [13:44:52] PROBLEM - Puppet errors on integration-slave-trusty-1003 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [13:46:51] Project selenium-VisualEditor » firefox,beta,Linux,BrowserTests build #407: 04FAILURE in 2 min 51 sec: https://integration.wikimedia.org/ci/job/selenium-VisualEditor/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/407/ [13:59:50] RECOVERY - Puppet errors on integration-slave-trusty-1003 is OK: OK: Less than 1.00% above the threshold [0.0] [14:04:37] RECOVERY - Puppet errors on integration-slave-trusty-1006 is OK: OK: Less than 1.00% above the threshold [0.0] [14:15:57] 06Release-Engineering-Team (Kanban), 06Performance-Team, 10Phabricator: Enable storage and display of OGV and WEBM videos on Phabricator - https://phabricator.wikimedia.org/T166112#3289086 (10Aklapper) (I got reminded of T75990#2384736. Did this regress or do I misunderstand?) [14:18:50] RECOVERY - Puppet errors on integration-slave-trusty-1004 is OK: OK: Less than 1.00% above the threshold [0.0] [14:20:05] Project beta-update-databases-eqiad build #17313: 04STILL FAILING in 5.1 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/17313/ [14:20:11] RECOVERY - Puppet errors on integration-slave-trusty-1001 is OK: OK: Less than 1.00% above the threshold [0.0] [14:30:22] 06Release-Engineering-Team (Kanban), 06Performance-Team, 10Phabricator: Enable storage and display of OGV and WEBM videos on Phabricator - https://phabricator.wikimedia.org/T166112#3289124 (10Gilles) Possibly a regression, then, both .ogv and .webm files I try show an error when the upload reaches 100%: {F8... [14:31:17] nError: 1091 Can't DROP 'flow_ext_ref_idx_v2'; check that column/key exists (10.68.23.30)\n\nBacktrace:\n#0 /srv/mediawiki-staging/php-master/includes/libs/rdbms/database/Database.php(967): Wikimedia\\Rdbms\\Database->reportQueryError(string, integer, string, string, boolean)\n#1 /srv/mediawiki-staging/php- [14:31:37] Needs a drop if exists type of sql query. [14:32:28] 06Release-Engineering-Team, 06Language-Team, 06MediaWiki-Platform-Team, 10MediaWiki-extensions-WikimediaIncubator, and 2 others: Make creating a new Language project easier - https://phabricator.wikimedia.org/T165585#3289130 (10Aklapper) [offtopic] @Verdy_p: In the future, please follow my recommendation t... [14:33:41] Yippee, build fixed! [14:33:42] Project selenium-WikiLove » firefox,beta,Linux,BrowserTests build #402: 09FIXED in 1 min 41 sec: https://integration.wikimedia.org/ci/job/selenium-WikiLove/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/402/ [14:53:47] PROBLEM - Puppet errors on deployment-urldownloader is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [15:20:03] Project beta-update-databases-eqiad build #17314: 04STILL FAILING in 3.5 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/17314/ [15:28:43] RECOVERY - Puppet errors on deployment-urldownloader is OK: OK: Less than 1.00% above the threshold [0.0] [15:30:13] hi! we had another build failure on fr-tech, this time for the crm project: https://integration.wikimedia.org/ci/job/wikimedia-fundraising-civicrm/4385/console [15:30:47] the error is occurring where it tries to remove privileges for a user [15:31:54] Is it that, or the git clean just before? [15:32:01] 14:21:12 git.exc.GitCommandError: 'git clean -x -f -d' returned with exit code 1 [15:33:46] RainbowSprinkles oh interesting it seems to be erroring both places--could the failure of git clean cause the db errors? [15:36:15] Possibly? I dunno the specifics of this script/test. [15:38:35] i think hashar would perhaps be a better person to ask [15:39:50] i wonder would --force work? [15:39:51] stderr: 'warning: failed to remove sites/default/files/ [15:39:54] woops [15:39:59] https://stackoverflow.com/questions/10064801/git-failing-to-remove-with-clean [15:40:33] That's what the -f is [15:42:23] oh [15:43:02] is there a way to know if there's been recent changes to the build script? [15:43:03] -f, --force [15:43:03] If the Git configuration variable clean.requireForce is not set to false, git clean will refuse to delete files or directories unless given -f, -n or -i. Git will refuse to delete directories with .git sub directory or file unless a second -f is given. [15:44:26] mepps: Last obvious change I see was a month ago [15:44:30] And prior to that in February [15:46:03] hmm and the build succeeded 17 hours ago [15:46:31] mepps: could it possibly be a code error and not CI error? [15:51:46] O/ [15:51:47] Zppix certainly possible [15:52:12] mepps: the jenkins machine is in a dirty state. Gotta clean it up manually [15:54:00] rm: cannot remove ‘sites/default/settings.php’: Permission denied [15:54:02] bah [15:54:17] I see that ejegg is currently doing a recheck [15:55:17] hashar is it weird that this would have just cropped up? [15:55:39] went with sudo rm /srv/jenkins-workspace/workspace/wikimedia-fundraising-civicrm/src/wikimedia/fundraising/crm/sites/default/settings.php [15:56:05] which will probably cause the build to fail bah [15:56:18] ah no it ran on an other jenkins instance [15:58:23] mepps: ejegg|away did a recheck of the change https://gerrit.wikimedia.org/r/#/c/355252/ [15:58:26] hashar: doesnt jenkins do workspace clean ups automatically? [15:58:34] it fails differently this time https://integration.wikimedia.org/ci/job/wikimedia-fundraising-civicrm/4386/console [15:58:49] PHP Fatal error: Class 'UtcDate' not found in /srv/jenkins-workspace/workspace/wikimedia-fundraising-civicrm/src/wikimedia/fundraising/crm/sites/all/modules/wmf_civicrm/wmf_civicrm.module on line 360 [15:59:00] ah! that's helpful [15:59:05] Zppix: git clean supposedly does that. But for some reason the file could not be deleted [15:59:14] mepps: so might be fixed [15:59:23] potentially integration-slave-trusty-1003 has some file system issue [15:59:42] hashar: maybe running git clean with sudo instead? [16:00:40] hashar so that would mean the git clean fail issue is fixed but there's an issue with the UtcDate class not being found? [16:00:49] mepps: yes [16:01:03] mepps: that build ran on a different jenkins instance [16:01:35] and I have no clue what provides UtcDate [16:03:04] hashar yeah i'm looking into that and will confer with ejegg, but that's a more helpful error to work with :) [16:04:32] !log rebooting integration-slave-trusty-1003 to catch up with kernel upgrade [16:04:35] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:04:49] mepps: maybe UtcDate is only available in a more recent version [16:06:22] hashar ejegg just updated the patch to include the utc date class [16:07:47] hashar, i'm going to lunch but thanks for the help! hopefully the build succeeds now! [16:09:06] mepps: ejegg|away is the best :-} [16:20:05] Project beta-update-databases-eqiad build #17315: 04STILL FAILING in 3.8 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/17315/ [16:22:45] Whats up with db update build always failing ^ [16:24:58] PROBLEM - Puppet errors on deployment-ores-redis-01 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [16:26:26] Zppix: Because there's a bad schema change in Flow. [16:26:31] It's clear from the log [16:29:27] It's caused by https://github.com/wikimedia/mediawiki-extensions-Flow/blob/17e2695a85a0d8edd12260b5283cb16522d00b0e/db_patches/patch-ref_target_not_null.sql#L1 [16:29:33] from september last year [16:30:30] It should be moved to it's own file [16:30:37] the drop key sql query [16:31:20] 06Release-Engineering-Team (Kanban), 06Performance-Team, 10Phabricator: Enable storage and display of OGV and WEBM videos on Phabricator - https://phabricator.wikimedia.org/T166112#3289372 (10mmodell) Strange. It could be that it is exceeding the filesize limit but that isn't a very friendly error message. [16:31:36] RainbowSprinkles is there a mediawiki function to drop a key [16:31:49] ALTER TABLE /*_*/flow_ext_ref DROP KEY flow_ext_ref_idx_v2; [16:32:01] Doesn't need a function, just do it in the SQL file [16:32:08] Should be DROP KEY IF EXISTS [16:32:26] Ah, but what version does DROP KEY IF EXISTS support? [16:32:26] Errr [16:32:27] Or not [16:32:29] mysql 5.1? [16:32:31] That's not a thing [16:32:36] ok [16:32:53] Curious as to why this started breaking *now* if it was changed in Sept. [16:33:18] * paladox too [16:34:16] Well good thing i said something... [16:35:31] I guess they make it a key [16:35:45] RainbowSprinkles we could just make it a try and catch [16:39:14] 06Release-Engineering-Team (Kanban), 06Performance-Team, 10Phabricator: Enable storage and display of OGV and WEBM videos on Phabricator - https://phabricator.wikimedia.org/T166112#3289376 (10mmodell) Ok I can confirm that it's storage size limited. It's configured to a max of 4MB. I made that change last d... [16:42:25] 06Release-Engineering-Team (Kanban), 06Performance-Team, 10Phabricator: Enable storage and display of OGV and WEBM videos on Phabricator - https://phabricator.wikimedia.org/T166112#3289379 (10mmodell) This is annoying - I don't see any way to set the limit larger than 8mb without making it effectively unlimi... [16:43:07] RainbowSprinkles heh https://github.com/wikimedia/mediawiki-extensions-Flow/blob/1a5e4802a6eff1dcc1fe019a5ffaba59483ab49f/flow.sql is out of date. [16:44:23] I tried dropping the whole table and recreating yesterday, but that didn't work either [16:44:45] We ended up with an un-migratable schema still. [16:44:59] Oh, apparently that key never existed [16:45:05] I spent the better bart of yesterday trying to fix this :\ [16:45:08] not from the flow.sql file im loking at [16:45:21] I wonder if I should just make that key manually? [16:45:24] Then the DROP will work [16:45:29] ah [16:45:53] RainbowSprinkles i belive it was a spelling mistake. The patch description says index and nothing about a key [16:45:57] https://github.com/wikimedia/mediawiki-extensions-Flow/commit/5bf7057a1a01967aa05f6296cf850587ffedb419 [16:46:23] i carn't find the key in the flow.sql file which was last updated in 2015 [16:46:29] 06Release-Engineering-Team (Kanban), 06Performance-Team, 10Phabricator: Enable storage and display of OGV and WEBM videos on Phabricator - https://phabricator.wikimedia.org/T166112#3289395 (10mmodell) @gilles: If you can make your video just slightly smaller than 8mb then we can accommodate that. If I increa... [16:46:31] * paladox submits patch [16:48:17] 06Release-Engineering-Team (Kanban), 06Performance-Team, 10Phabricator: Enable storage and display of OGV and WEBM videos on Phabricator - https://phabricator.wikimedia.org/T166112#3289398 (10Gilles) Well, a higher limit (or unlimited in that case) could require special rights. But I imagine that requires up... [16:50:23] RainbowSprinkles https://gerrit.wikimedia.org/r/#/c/355448/ :) [16:50:24] not sure if i should use DROP IF EXISTS because that's supported in mysql 5.5+ [16:50:30] but we support mysql 5.1+ [16:50:39] Yeah no :p [16:51:27] ok :) [16:53:28] 06Release-Engineering-Team (Kanban), 06Performance-Team, 10Phabricator, 15User-greg: Create Performance phame blog - https://phabricator.wikimedia.org/T166110#3285627 (10Framawiki) Note: It's currently impossible to add it to the homepage: {T166236} [16:58:52] 06Release-Engineering-Team (Kanban), 06Performance-Team, 10Phabricator: Enable storage and display of OGV and WEBM videos on Phabricator - https://phabricator.wikimedia.org/T166112#3289451 (10Gilles) Another option is allowing youtube embedding... ``` remarkup.enable-embedded-youtube ``` [16:59:57] RECOVERY - Puppet errors on deployment-ores-redis-01 is OK: OK: Less than 1.00% above the threshold [0.0] [17:05:38] PROBLEM - Puppet errors on buildlog is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [17:05:44] PROBLEM - Puppet errors on swift is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [17:07:19] PROBLEM - Puppet errors on swift-storage-01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [17:20:08] Project beta-update-databases-eqiad build #17316: 04STILL FAILING in 7.8 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/17316/ [17:21:59] PROBLEM - Puppet errors on deployment-mira is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [17:29:10] 10Gerrit, 06Release-Engineering-Team (Backlog), 06Operations, 07LDAP: Remove user gerrit2 from ldap - https://phabricator.wikimedia.org/T160122#3289546 (10demon) Just did this with @MoritzMuehlenhoff. Shouldn't have any issues, but please reopen if anyone sees something broken. [17:29:18] 10Gerrit, 06Release-Engineering-Team (Kanban), 06Operations, 07LDAP: Remove user gerrit2 from ldap - https://phabricator.wikimedia.org/T160122#3289547 (10demon) 05Open>03Resolved a:03demon [17:30:32] 10Gerrit, 06Release-Engineering-Team (Kanban), 06Operations, 07LDAP: Remove user gerrit2 from ldap - https://phabricator.wikimedia.org/T160122#3289554 (10Paladox) Ah thanks :) [17:40:34] 06Release-Engineering-Team (Kanban), 10Phabricator: Switch phabricator production to codfw - https://phabricator.wikimedia.org/T164810#3289567 (10mmodell) So we need a proxy set up for `phab2001-vcs`, I'm not sure how to test it currently. [17:46:35] 06Release-Engineering-Team (Kanban), 10Phabricator: Switch phabricator production to codfw - https://phabricator.wikimedia.org/T164810#3289593 (10mmodell) [17:47:28] 10Gerrit, 06Developer-Relations, 10GitHub-Mirrors, 06Repository-Admins: Add CODE_OF_CONDUCT.md to Wikimedia projects - https://phabricator.wikimedia.org/T165540#3269385 (10demon) >>! In T165540#3273805, @Legoktm wrote: > Automated. I know ^d has a script he's used in the past when making batch changes to .... [17:50:18] 06Release-Engineering-Team (Kanban), 06Performance-Team, 10Phabricator: Enable storage and display of OGV and WEBM videos on Phabricator - https://phabricator.wikimedia.org/T166112#3289611 (10mmodell) Indeed, I would not be apposed to that though others might not like it. [17:52:34] 10Gerrit, 10MediaWiki-extensions-Other, 06Repository-Admins, 07Technical-Debt: Archive PageLanguageApi extension - https://phabricator.wikimedia.org/T160371#3096576 (10demon) 1.29 branches are already in place, this can proceed. [17:54:43] PROBLEM - Puppet errors on deployment-urldownloader is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [17:55:14] (03PS1) 10Chad: PageLanguageApi has been archived, no need to test [integration/config] - 10https://gerrit.wikimedia.org/r/355460 (https://phabricator.wikimedia.org/T160371) [17:55:45] 10Gerrit, 10MediaWiki-extensions-Other, 06Repository-Admins, 13Patch-For-Review, 07Technical-Debt: Archive PageLanguageApi extension - https://phabricator.wikimedia.org/T160371#3289658 (10demon) [18:01:59] RECOVERY - Puppet errors on deployment-mira is OK: OK: Less than 1.00% above the threshold [0.0] [18:03:08] 06Release-Engineering-Team (Kanban), 06Performance-Team, 10Phabricator, 15User-greg: Create Performance phame blog - https://phabricator.wikimedia.org/T166110#3289693 (10mmodell) @gilles: You should be able to create the blog now. [18:04:59] 10Gerrit, 10MediaWiki-extensions-Other, 06Repository-Admins, 13Patch-For-Review, 07Technical-Debt: Archive PageLanguageApi extension - https://phabricator.wikimedia.org/T160371#3289703 (10demon) [18:07:27] (03PS1) 10Chad: AccountAudit: Archived extension [integration/config] - 10https://gerrit.wikimedia.org/r/355465 (https://phabricator.wikimedia.org/T163791) [18:10:54] 10Continuous-Integration-Config, 10Gerrit, 10MediaWiki-extensions-AccountAudit, 06Project-Admins, 13Patch-For-Review: Archive the AccountAudit extension - https://phabricator.wikimedia.org/T163791#3210080 (10demon) Archived in gerrit, removed on github, dropped from extension meta-repo. Just needs CI rem... [18:11:26] 10Gerrit, 10MediaWiki-extensions-Other, 06Repository-Admins, 13Patch-For-Review, 07Technical-Debt: Archive PageLanguageApi extension - https://phabricator.wikimedia.org/T160371#3289734 (10demon) [18:20:08] Project beta-update-databases-eqiad build #17317: 04STILL FAILING in 8.3 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/17317/ [18:29:42] RECOVERY - Puppet errors on deployment-urldownloader is OK: OK: Less than 1.00% above the threshold [0.0] [18:48:33] 10Beta-Cluster-Infrastructure, 06Performance-Team, 07Availability (Multiple-active-datacenters): Install and use mcrouter in deployment-prep - https://phabricator.wikimedia.org/T151466#3289909 (10aaron) a:03aaron [18:49:43] 10Scap (Scap3-MediaWiki-MVP), 10Fundraising-Backlog, 10MediaWiki-extensions-ContributionTracking, 03Fundraising Sprint Judgement Suspenders, 13Patch-For-Review: Clean up Contribution Tracking settings in main wmf config repo - https://phabricator.wikimedia.org/T147479#3289911 (10Ejegg) [19:08:40] 06Release-Engineering-Team (Kanban), 10Scap (Scap3-MediaWiki-MVP), 10Fundraising-Backlog, 10MediaWiki-extensions-ContributionTracking, and 2 others: Clean up Contribution Tracking settings in main wmf config repo - https://phabricator.wikimedia.org/T147479#3289986 (10demon) 05Open>03Resolved a:03demon... [19:10:20] 06Release-Engineering-Team (Kanban), 06Performance-Team, 10Phabricator: Enable storage and display of OGV and WEBM videos on Phabricator - https://phabricator.wikimedia.org/T166112#3289993 (10Gilles) We do host all our recorded meetings on youtube in addition to commons, there would some convenience in being... [19:11:38] 06Release-Engineering-Team (Kanban), 06Performance-Team, 10Phabricator, 15User-greg: Create Performance phame blog - https://phabricator.wikimedia.org/T166110#3289996 (10Gilles) It still states: > You do not have permission to create these objects. When I press the "create blog" button. [19:18:23] yo dudes [19:18:23] ah! [19:18:29] yesterday, scap deploy was nice to me [19:18:31] today [19:18:32] it says [19:18:39] Could not chdir to home directory /nonexistent: No such file or directory [19:18:45] when trying to deploy local on the target [19:20:05] Project beta-update-databases-eqiad build #17318: 04STILL FAILING in 4.3 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/17318/ [19:20:23] hm, weird, i just fixed it by running a scap deploy-local manually on the target [19:20:28] now regular scap deploy works too [19:20:36] ¯\_(ツ)_/¯ [19:20:40] Weird [19:30:16] 06Release-Engineering-Team (Kanban), 06Performance-Team, 10Phabricator, 15User-greg: Create Performance phame blog - https://phabricator.wikimedia.org/T166110#3290083 (10mmodell) @gilles: How about now? I missed one permission before.... [19:32:35] 06Release-Engineering-Team (Kanban), 10Phabricator (Upstream), 07Upstream: Clicking "Add Existing Panel" and entering `Wxxx` in the field shows no results - https://phabricator.wikimedia.org/T166236#3290101 (10mmodell) a:03mmodell Needs more testing, this may be fixed by pulling upstream or it may need to... [19:33:47] PROBLEM - Puppet errors on deployment-conf03 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [19:35:04] 06Release-Engineering-Team (Kanban), 06Performance-Team, 10Phabricator, 06Project-Admins: Enable storage and display of OGV and WEBM videos on Phabricator - https://phabricator.wikimedia.org/T166112#3290107 (10mmodell) Indeed I think aklapper would be the person to ask. I've added project-admins just to ra... [19:42:58] (03CR) 10Hashar: [C: 032] AccountAudit: Archived extension [integration/config] - 10https://gerrit.wikimedia.org/r/355465 (https://phabricator.wikimedia.org/T163791) (owner: 10Chad) [19:46:29] (03CR) 10Hashar: [C: 032] AccountAudit: Archived extension [integration/config] - 10https://gerrit.wikimedia.org/r/355465 (https://phabricator.wikimedia.org/T163791) (owner: 10Chad) [19:46:32] (03CR) 10Hashar: [C: 032] PageLanguageApi has been archived, no need to test [integration/config] - 10https://gerrit.wikimedia.org/r/355460 (https://phabricator.wikimedia.org/T160371) (owner: 10Chad) [19:47:41] (03Merged) 10jenkins-bot: PageLanguageApi has been archived, no need to test [integration/config] - 10https://gerrit.wikimedia.org/r/355460 (https://phabricator.wikimedia.org/T160371) (owner: 10Chad) [19:47:44] (03Merged) 10jenkins-bot: AccountAudit: Archived extension [integration/config] - 10https://gerrit.wikimedia.org/r/355465 (https://phabricator.wikimedia.org/T163791) (owner: 10Chad) [19:50:47] 06Release-Engineering-Team (Kanban), 06Reading-Admin, 05Deployment Blockers, 05Release: MW-1.30.0-wmf.2 deployment blockers - https://phabricator.wikimedia.org/T163512#3290172 (10dr0ptp4kt) [19:52:39] hashar: ty [20:02:00] RainbowSprinkles: you are welcome :] [20:02:30] 10Continuous-Integration-Config, 10Gerrit, 06Release-Engineering-Team (Kanban), 10MediaWiki-extensions-AccountAudit, and 2 others: Archive the AccountAudit extension - https://phabricator.wikimedia.org/T163791#3290182 (10demon) 05Open>03Resolved a:03demon [20:02:38] I am already fed up with Docker RUN command [20:03:03] I am really tempted to just: [20:03:03] COPY . provision.sh [20:03:05] RUN provision.sh [20:03:10] {done} [20:08:40] hah [20:11:49] 10Gerrit, 10MediaWiki-extensions-Other, 06Repository-Admins, 13Patch-For-Review, 07Technical-Debt: Archive PageLanguageApi extension - https://phabricator.wikimedia.org/T160371#3290215 (10demon) [20:12:02] 10Gerrit, 06Release-Engineering-Team (Kanban), 10MediaWiki-extensions-Other, 06Repository-Admins, and 2 others: Archive PageLanguageApi extension - https://phabricator.wikimedia.org/T160371#3290218 (10demon) 05Open>03Resolved a:03demon [20:13:27] 06Release-Engineering-Team (Kanban), 06Performance-Team, 10Phabricator, 06Project-Admins: Enable storage and display of OGV and WEBM videos on Phabricator - https://phabricator.wikimedia.org/T166112#3285669 (10Legoktm) Embedding youtube videos in Phabricator would be a privacy policy violation. [20:13:45] RECOVERY - Puppet errors on deployment-conf03 is OK: OK: Less than 1.00% above the threshold [0.0] [20:20:04] Project beta-update-databases-eqiad build #17319: 04STILL FAILING in 3.6 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/17319/ [20:33:16] (03CR) 10Hashar: [C: 032] Reorder commands in Dockerfile (032 comments) [integration/quibble] - 10https://gerrit.wikimedia.org/r/355241 (owner: 10Hashar) [20:33:21] (03CR) 10Hashar: [C: 04-2] Reorder commands in Dockerfile [integration/quibble] - 10https://gerrit.wikimedia.org/r/355241 (owner: 10Hashar) [20:36:53] (03PS2) 10Hashar: Reorder commands in Dockerfile [integration/quibble] - 10https://gerrit.wikimedia.org/r/355241 [20:41:51] Yippee, build fixed! [20:41:51] Project selenium-Echo » firefox,beta,Linux,BrowserTests build #404: 09FIXED in 51 sec: https://integration.wikimedia.org/ci/job/selenium-Echo/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/404/ [20:41:52] Yippee, build fixed! [20:41:52] Project selenium-Echo » chrome,beta,Linux,BrowserTests build #404: 09FIXED in 51 sec: https://integration.wikimedia.org/ci/job/selenium-Echo/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/404/ [20:55:43] PROBLEM - Puppet errors on deployment-urldownloader is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [21:14:15] 06Release-Engineering-Team (Kanban), 06Performance-Team, 10Phabricator, 06Project-Admins: Enable storage and display of OGV and WEBM videos on Phabricator - https://phabricator.wikimedia.org/T166112#3290374 (10Aklapper) I'm technically neutral; hard to foresee whether allowing embedding videos would increa... [21:17:06] 06Release-Engineering-Team (Kanban), 06Performance-Team, 10Phabricator, 06Project-Admins: Enable storage and display of OGV and WEBM videos on Phabricator - https://phabricator.wikimedia.org/T166112#3285669 (10demon) Embedding videos & images from commons/mw.org could be useful... Would sidestep both the "... [21:20:04] Project beta-update-databases-eqiad build #17320: 04STILL FAILING in 4 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/17320/ [21:21:08] (03PS1) 10Hashar: quibble: docker build it from CI [integration/config] - 10https://gerrit.wikimedia.org/r/355548 [21:21:27] I have a patch to fix that ^^ [21:21:28] https://gerrit.wikimedia.org/r/#/c/355448/ [21:22:01] I want someone who actually understands that dang schema to look at it :\ [21:25:05] (03CR) 10Hashar: "Example full run https://integration.wikimedia.org/ci/job/integration-quibble-docker-test/4/console" [integration/quibble] - 10https://gerrit.wikimedia.org/r/355236 (owner: 10Hashar) [21:25:27] https://stackoverflow.com/questions/1401572/what-are-differences-between-index-v-s-key-in-mysql [21:25:32] lol RainbowSprinkles ^^ [21:25:35] key is index [21:26:55] (03CR) 10Hashar: "Example full run https://integration.wikimedia.org/ci/job/integration-quibble-docker-test/4/console" (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/355548 (owner: 10Hashar) [21:28:20] paladox: They're sorta the same [21:28:23] (for most purposes) [21:28:27] Not in DDL statements! [21:28:34] lol [21:28:56] drop index if exists should have been in mysql 4 [21:29:01] not 5.5 [21:29:03] :) [21:29:44] Maybe a chage in core broke this? [21:30:44] RECOVERY - Puppet errors on deployment-urldownloader is OK: OK: Less than 1.00% above the threshold [0.0] [21:31:09] i did fix the updater a few days ago [21:31:10] https://gerrit.wikimedia.org/r/#/c/354585/ [21:32:16] 06Release-Engineering-Team, 10Wikimedia-Site-requests: Consider to switch frrwiki from group2 to group1 per Proofreadpage - https://phabricator.wikimedia.org/T166263#3290416 (10Dereckson) [21:32:31] Could revert from master [21:32:37] If it fixes things.... [21:32:48] But it then breaks [21:32:51] the web updater [21:33:32] Maybe https://github.com/wikimedia/mediawiki/commit/6519c42d248a78d2d42edee1beb21f926d227044 should be reverted then my change. [21:33:48] 06Release-Engineering-Team, 10Wikimedia-Site-requests: Consider to switch frrwiki from group2 to group1 per Proofreadpage - https://phabricator.wikimedia.org/T166263#3290314 (10Dereckson) [21:35:56] 10Beta-Cluster-Infrastructure, 06Collaboration-Team-Triage, 10Flow: Beta update.php fails with Can't DROP 'flow_ext_ref_idx_v2'; - https://phabricator.wikimedia.org/T166266#3290426 (10Paladox) [21:35:58] RainbowSprinkles created ^^ [21:35:59] Eh, I'm a little more concerned about update.php being broken in beta right this minute [21:36:26] 10Beta-Cluster-Infrastructure, 06Collaboration-Team-Triage, 10Flow: Beta update.php fails with Can't DROP 'flow_ext_ref_idx_v2'; - https://phabricator.wikimedia.org/T166266#3290439 (10Paladox) p:05Triage>03Unbreak! [21:36:35] I'm wondering if you created another problem by fixing your issue [21:36:41] * RainbowSprinkles hmmmms [21:37:06] RainbowSprinkles if i did, im sorry. We should revert my change [21:37:08] and https://github.com/wikimedia/mediawiki/commit/6519c42d248a78d2d42edee1beb21f926d227044 [21:37:18] ^^ that one seems to be breaking more then what it fixes [21:37:54] Well, that hadn't been causing problems in beta until now [21:38:41] Yep, but it caused the web updater to break which needed my change to fix it. And then my change could be breaking beta [21:39:23] Was it just the web updater, or did update.php break too? [21:39:37] web updater [21:40:31] Then that makes me think it wasn't so deep down that was broken [21:40:34] PROBLEM - Puppet errors on deployment-cache-text04 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [21:40:34] Vs. like output [21:40:56] I wonder is there a way we can test this in beta [21:41:07] ie apply the revert to see? [21:41:17] I mean, just revert in master [21:41:20] And it'll go out [21:41:20] :) [21:42:04] Ok [21:42:41] I will do the change now [21:43:39] RainbowSprinkles https://gerrit.wikimedia.org/r/#/c/355554/ [21:44:22] :) [21:44:23] thnks [21:44:25] thanks [21:46:22] The change was backported to 1.29 so will need to be reverted there if it fixes beta. [21:46:39] I will also revert 5d56bcb925af653fed427e90a1cfb982de386720 if it fixes beta [21:46:40] :) [21:49:54] I don't know if we want to revert that [21:49:59] Or find a new fix for your change [21:50:04] (03PS2) 10Hashar: quibble: docker build it from CI [integration/config] - 10https://gerrit.wikimedia.org/r/355548 [21:50:51] RainbowSprinkles but then it will block 1.29 release due to breakage of web updater. [21:50:59] Ok. [21:50:59] (03CR) 10Hashar: "Use date/time in the image tag. Eg:" [integration/config] - 10https://gerrit.wikimedia.org/r/355548 (owner: 10Hashar) [21:51:06] I said we'll find a new fix [21:51:12] Ok [21:51:17] I will reopen my task [21:55:31] RECOVERY - Puppet errors on deployment-cache-text04 is OK: OK: Less than 1.00% above the threshold [0.0] [21:56:56] revert merged. /me waits for beta updater to run [22:04:22] RainbowSprinkles, wondering could you press the button that starts the beta update job? [22:04:27] Or do we need to wait for beta to pull my change in? [22:04:34] Just gotta be patient [22:04:40] * RainbowSprinkles goes to grab a coffee [22:04:40] Ok [22:09:34] Last failure was 50m ago, so just 10 more minutes :) [22:09:40] lol :) [22:10:32] dbtree fixed :p literally an extra "." [22:11:00] (03PS1) 10Hashar: quibble: test running the container [integration/config] - 10https://gerrit.wikimedia.org/r/355560 [22:11:28] (03CR) 10Hashar: "Example run: https://integration.wikimedia.org/ci/job/integration-quibble-docker-test/12/console" [integration/config] - 10https://gerrit.wikimedia.org/r/355560 (owner: 10Hashar) [22:11:38] lol [22:17:17] 3 mins :) [22:19:04] 06Release-Engineering-Team (Kanban), 06Performance-Team, 10Phabricator, 06Project-Admins: Enable storage and display of OGV and WEBM videos on Phabricator - https://phabricator.wikimedia.org/T166112#3290586 (10Aklapper) That's {T116515} already ;) [22:20:04] Project beta-update-databases-eqiad build #17321: 04STILL FAILING in 4.5 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/17321/ [22:20:32] Fails still [22:20:32] https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/17321/console [22:20:39] dammit [22:20:41] I give up [22:20:49] * RainbowSprinkles cries [22:20:59] Eh, revert your revert. That's clearly not it [22:21:02] ok [22:21:06] https://gerrit.wikimedia.org/r/#/c/355562/ [22:21:32] thanks [22:22:53] 10Beta-Cluster-Infrastructure, 06Collaboration-Team-Triage, 10Flow: Beta update.php fails with Can't DROP 'flow_ext_ref_idx_v2'; - https://phabricator.wikimedia.org/T166266#3290594 (10Aklapper) @Paladox: Could you elaborate how this task is related to T110446#3289439 (//if// it is)? Thanks! [22:23:49] 10Beta-Cluster-Infrastructure, 06Collaboration-Team-Triage, 10Flow: Beta update.php fails with Can't DROP 'flow_ext_ref_idx_v2'; - https://phabricator.wikimedia.org/T166266#3290597 (10Paladox) @Aklapper @demon thought my change could have broke beta. So we reverted it. turns out my change did not break beta.... [22:24:22] It last passed on https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/17267/ [22:24:23] So between then it broke [22:24:28] 10Beta-Cluster-Infrastructure, 06Collaboration-Team-Triage, 10Flow: Beta update.php fails with Can't DROP 'flow_ext_ref_idx_v2'; - https://phabricator.wikimedia.org/T166266#3290605 (10demon) This `flow_ext_ref` table is woefully inconsistent in beta. A migration went poorly. [22:25:21] The changes in core show only three on the day of the breakage [22:29:15] 10Beta-Cluster-Infrastructure, 06Collaboration-Team-Triage, 10Flow, 10MediaWiki-Database: Beta update.php fails with Can't DROP 'flow_ext_ref_idx_v2'; - https://phabricator.wikimedia.org/T166266#3290612 (10Paladox) [22:29:38] I see nothing suspect in Flow [22:29:38] Only changes in last ~2d are unrelated [22:30:06] (03CR) 10Krinkle: "This should probably add an example to one of the test cases to verify that it is indeed allowed now." [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/355368 (https://phabricator.wikimedia.org/T154789) (owner: 10MtDu) [22:30:33] Yep, i see nothing in core [22:30:36] from may 22 [22:30:52] (nothing related to are problem in core on that date) [22:32:22] Did someone play with beta db on that date without logging? [22:33:44] I don't know [22:35:22] ok [23:20:05] Project beta-update-databases-eqiad build #17322: 04STILL FAILING in 3.7 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/17322/ [23:26:08] Project beta-update-databases-eqiad build #17323: 04STILL FAILING in 3.3 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/17323/ [23:30:41] Project beta-update-databases-eqiad build #17324: 04STILL FAILING in 3.3 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/17324/ [23:31:33] Ok, well manually trying to get the schema into a place where the updater fixes it seems...not easy [23:33:14] lolol. [23:33:23] There's *zero* flow data on Commonswiki in beta. [23:33:29] Blowing away all the tables should work [23:33:33] Then it'll create fresh [23:34:39] Project beta-update-databases-eqiad build #17325: 04STILL FAILING in 11 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/17325/ [23:35:14] Dammit [23:35:15] Still no good [23:35:18] How did this ever work? [23:47:52] Project beta-update-databases-eqiad build #17326: 04STILL FAILING in 3.2 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/17326/ [23:48:06] Ok, now that's /super/ bizzare [23:48:16] Because update.php works just fine from running it manually on deployment-tin [23:49:11] Now that makes me wonder if something in the script changed... [23:50:05] ...ref_target in table flow_ext_ref already modified by patch /srv/mediawiki-staging/php-master/extensions/Flow/db_patches/patch-ref_target_not_null.sql. [23:50:15] So why does the script think it hasn't? [23:53:30] RoanKattouw: So this is f'ing bizarre [23:53:43] The schema is *correct* in beta, update.php confirms it [23:53:53] But running update.php from the jenkins job explodes on *that* particular patch [23:55:06] Strange that it even runs it [23:55:13] If the schema is correct, why would it decide to run the patch [23:55:22] Yeah [23:55:37] I mean, the weird part is that it disagrees. [23:55:43] Modifying ref_target field of table flow_ext_ref ...[ [23:55:45] In theory: shouldn't it be exploding on the command line too? [23:55:51] (if it's really broken?) [23:55:57] Yes [23:56:01] Are you saying it runs correctly there?! [23:56:04] Yep [23:56:12] ...ref_target in table flow_ext_ref already modified by patch /srv/mediawiki-staging/php-master/extensions/Flow/db_patches/patch-ref_target_not_null.sql. [23:56:39] :O [23:56:45] I wonder how the detection works exactly [23:56:47] Confirmed it's in updatelog on master & replica, so it's not a stale data issue [23:56:57] Checks updatelog [23:56:58] Is it just a LoggedUpdateMaintenance? [23:57:10] Or, I guess not technically that but updatelog table [23:57:13] Very strange [23:57:16] And assumes it's written in a way that can consistently re-apply in case updatelog disappears (as all schema changes should be) [23:57:35] /usr/local/bin/mwscript update.php --wiki=aawiki --quick [23:57:46] It's possible that that might not be true [23:57:53] Bam [23:57:54] Hmm [23:57:56] We have had bug reports from 3rd parties too, and we've changed that in the past [23:57:58] But ahm [23:57:59] aawiki? [23:58:13] Is there an aawiki database? And what is in its updatelog? [23:58:18] Yes [23:58:34] (I mostly checked the CLI args to verify --force wasn't present, I admit) [23:59:15] Weird that aawiki fails from command line, but says nothing in the web updater. [23:59:24] Oh wait. [23:59:27] It's aawiki failing [23:59:28] Not commons [23:59:32] Silly threads [23:59:35] Made it hard to read [23:59:44] * RainbowSprinkles has been messing with the wrong database this whole time