| *** menesis has quit IRC | 00:27 | |
| *** menesis has joined #schooltool | 00:28 | |
| *** menesis has quit IRC | 03:35 | |
| *** yvl has joined #schooltool | 10:36 | |
| *** menesis has joined #schooltool | 11:21 | |
| *** menesis has quit IRC | 12:37 | |
| *** menesis has joined #schooltool | 12:46 | |
| *** menesis has quit IRC | 12:56 | |
| *** khildin has joined #schooltool | 13:46 | |
| *** menesis has joined #schooltool | 14:51 | |
| *** replaceafill has joined #schooltool | 15:23 | |
| *** th1a has joined #schooltool | 15:30 | |
| th1a | hi replaceafill. | 15:31 |
|---|---|---|
| replaceafill | hey th1a | 15:31 |
| th1a | OK... where are we replaceafill? | 15:34 |
| replaceafill | with the social skills profiling? | 15:35 |
| replaceafill | i was able to cut from ~900 seconds to ~200 seconds | 15:35 |
| replaceafill | (which is still a lot) | 15:35 |
| th1a | How'd you do that? | 15:36 |
| th1a | Yeah... three minutes! | 15:36 |
| replaceafill | there were two unnecessary loops | 15:37 |
| replaceafill | executed too many times | 15:37 |
| replaceafill | i just moved them out | 15:38 |
| replaceafill | one of them was in the cando gradebook | 15:38 |
| replaceafill | the main problem in the social skills case | 15:38 |
| replaceafill | is that they're using 4 terms | 15:38 |
| replaceafill | and the gradebook usually lookup previous sections | 15:39 |
| replaceafill | i now understand what yvl says about performance in relationships | 15:39 |
| th1a | Which is? | 15:40 |
| replaceafill | the lookup i mention uses .findAllEquivalent() | 15:40 |
| replaceafill | which in turn uses relationship to check equivalent skills | 15:40 |
| replaceafill | those line show higher cumtime in the profile output | 15:41 |
| replaceafill | it's also worth mentioning that this single view gathers ~470 skills | 15:42 |
| replaceafill | just for a single student | 15:42 |
| th1a | So yvl is working on indexing for relationships, right? | 15:43 |
| yvl | right | 15:43 |
| th1a | Hi yvl. ;-) | 15:43 |
| replaceafill | hey yvl! | 15:43 |
| yvl | hey :) | 15:43 |
| th1a | Is there a way we can try this with the view replaceafill is working on, soonish? | 15:45 |
| replaceafill | here's my latest profile results: | 15:45 |
| replaceafill | http://pastebin.com/xsf7X0Zs | 15:45 |
| yvl | soonish is probably the right word :) | 15:45 |
| replaceafill | maybe yvl can spot something obvious :) | 15:45 |
| replaceafill | th1a, at this point i'd like to try using a table formatter | 15:47 |
| th1a | Well, I guess the question is whether we should switch to some small tasks for the bugfix release. | 15:48 |
| th1a | For a couple days. | 15:48 |
| replaceafill | but i guess that won't improve the previous score lookup anyway | 15:48 |
| replaceafill | trying yvl's change sounds like a good idea | 15:49 |
| yvl | 586907 calls to getRelationshipLinks :) | 15:49 |
| replaceafill | yvl, yes :( | 15:49 |
| th1a | What are these looking for? | 15:51 |
| replaceafill | http://bazaar.launchpad.net/~schooltool-owners/schooltool.cando/trunk/view/head:/src/schooltool/cando/gradebook.py#L157 | 15:51 |
| replaceafill | getPreviousScore's | 15:51 |
| th1a | A higher previous score? | 15:51 |
| replaceafill | any previous score != None | 15:52 |
| yvl | hmm, cando gradebook getScore is taking a long time | 15:52 |
| th1a | What does it do with these? | 15:52 |
| * th1a needs a reminder... | 15:53 | |
| yvl | finds skills that are equivalent to this skill | 15:53 |
| th1a | Yeah... why do we need them in this view? | 15:54 |
| replaceafill | sorry, back | 15:54 |
| replaceafill | th1a, equivalents are used to get the "same" skill in the previous section | 15:55 |
| th1a | Yeah... why? | 15:55 |
| th1a | Is it doing some kind of calculation? | 15:56 |
| replaceafill | no, it's just for presenting it in the gradebook | 15:56 |
| * th1a is trying to remember this... | 15:57 | |
| th1a | I guess I should be looking at it. | 15:57 |
| th1a | Does it always show previous terms? | 15:57 |
| replaceafill | yes | 15:57 |
| replaceafill | if the score is None | 15:57 |
| replaceafill | and there are linked previous sections | 15:57 |
| th1a | OK, it is only looking if the current score is none. | 15:58 |
| th1a | ? | 15:58 |
| replaceafill | yes | 15:58 |
| th1a | OK, that's all I was trying to get at. | 15:58 |
| th1a | I didn't get your pseudocode, replaceafill. | 15:59 |
| replaceafill | ah sorry :D | 15:59 |
| replaceafill | hhmm... | 16:00 |
| replaceafill | since skillsets are global | 16:00 |
| replaceafill | and they're deployed using the original __name__ | 16:00 |
| replaceafill | maybe we could use __name__'s for the section skillset and skill | 16:00 |
| replaceafill | to get the equivalent in the previous sections | 16:01 |
| replaceafill | instead of the relationship lookup | 16:01 |
| replaceafill | luckily, cando has a stest for this "feature" | 16:02 |
| replaceafill | so i'd probably can try this quickly | 16:02 |
| yvl | btw, if you disable get previous score (return None) | 16:04 |
| yvl | everything works fast enough? | 16:04 |
| th1a | That's a good question. | 16:05 |
| replaceafill | on it! | 16:06 |
| replaceafill | 626447 function calls (602976 primitive calls) in 0.759 seconds | 16:07 |
| replaceafill | :D | 16:07 |
| th1a | What's that? | 16:07 |
| replaceafill | full output: http://pastebin.com/g6Psw0Kk | 16:07 |
| replaceafill | th1a, time with get previous score disabled | 16:07 |
| th1a | It goes from .8 seconds to 200? | 16:08 |
| replaceafill | yes | 16:08 |
| th1a | Oy. | 16:08 |
| yvl | good. | 16:08 |
| yvl | what happens if all grades are filled in? | 16:08 |
| yvl | and maybe even have score history? | 16:09 |
| yvl | (edited twice, at least some of them) | 16:09 |
| replaceafill | ah ok, will grade some | 16:09 |
| yvl | thanks man | 16:09 |
| replaceafill | (this is the profiling art yvl talked about) :D | 16:09 |
| yvl | :D | 16:09 |
| th1a | Oh, .8 is just finding nothing? | 16:10 |
| yvl | not searching | 16:11 |
| yvl | there's many equivalent skills there it seems | 16:11 |
| th1a | Yeah, ok. | 16:11 |
| yvl | we have 1000+ score history requests that result in 550000+ relationship requests | 16:12 |
| th1a | 500 each? | 16:12 |
| yvl | roughly | 16:13 |
| th1a | Do we know why so many? | 16:14 |
| yvl | no | 16:15 |
| yvl | but if for some reason we are interested, we can use pdb in a single findAllEquivalent call and see | 16:15 |
| th1a | You'd think we could cut that down A LOT. | 16:16 |
| yvl | maybe | 16:16 |
| yvl | depends | 16:16 |
| replaceafill | 442136 function calls (419167 primitive calls) in 0.397 seconds | 16:17 |
| replaceafill | with most of the skills graded | 16:17 |
| replaceafill | for that section | 16:17 |
| replaceafill | (and search disabled) | 16:17 |
| replaceafill | going to enable search | 16:17 |
| replaceafill | 45323163 function calls (44878668 primitive calls) in 54.711 seconds | 16:19 |
| replaceafill | (that's with search enabled) | 16:19 |
| replaceafill | and ~80% of the skills graded | 16:19 |
| yvl | and previous was 200 secs, right? | 16:19 |
| replaceafill | yes | 16:19 |
| replaceafill | because ALL the grades are searched | 16:20 |
| th1a | ALL of what exactly? | 16:20 |
| replaceafill | th1a, i mean, this forth term section doesn't have any grades of its own | 16:21 |
| yvl | how many equivalent skills on average are we talking here? | 16:21 |
| th1a | Shouldn't it be, like, four? | 16:23 |
| replaceafill | i don't think so | 16:23 |
| replaceafill | iirc | 16:23 |
| replaceafill | findAllEquivalent gets skills from any section | 16:23 |
| replaceafill | i mean, not only linked one | 16:23 |
| replaceafill | s | 16:23 |
| th1a | Is this a section view? | 16:24 |
| replaceafill | a student gradebook view | 16:24 |
| replaceafill | th1a, remember that view with checkmarks and X's | 16:25 |
| replaceafill | the autism center uses | 16:25 |
| th1a | Is this the report view or the entry view? | 16:25 |
| replaceafill | the entry | 16:25 |
| replaceafill | report does basically the same | 16:26 |
| th1a | So... there is really no reason we're looking at all sections in this case? | 16:26 |
| replaceafill | right | 16:26 |
| th1a | So let's not do that! | 16:26 |
| replaceafill | that's the reason of filterEquivalent | 16:26 |
| replaceafill | http://bazaar.launchpad.net/~schooltool-owners/schooltool.cando/trunk/view/head:/src/schooltool/cando/gradebook.py#L184 | 16:26 |
| replaceafill | it filters the skills | 16:27 |
| replaceafill | but it acts AFTER the lookup | 16:27 |
| th1a | OK, lets NOT do that? | 16:27 |
| replaceafill | yvl, do the __name__ lookup seems fragile to you? | 16:27 |
| th1a | Is this where an index helps? | 16:27 |
| yvl | a little bit yes, replaceafill | 16:29 |
| replaceafill | yvl, kk | 16:29 |
| yvl | not saying that would not work though | 16:29 |
| * yvl didn't think about it yet :) | 16:30 | |
| yvl | in any case, the right thing to do here is not to use findallequivelent obviously | 16:32 |
| yvl | you need findEquivalent(activity, section) | 16:33 |
| yvl | and need it to work fast :) | 16:33 |
| yvl | if you'll end up using __name__s, so be it :) | 16:33 |
| yvl | but probably you'll end up simply checking | 16:35 |
| yvl | for skill in section something | 16:35 |
| yvl | if skill in activity.equivalent | 16:35 |
| yvl | result.add(skill) | 16:35 |
| yvl | of equivalent = set(activity.equivalent) | 16:36 |
| yvl | and then the for | 16:36 |
| yvl | and then if skill in equivalent | 16:36 |
| yvl | something along the lines | 16:36 |
| replaceafill | so basically redefining findAllEquivalent to filter by section? | 16:37 |
| replaceafill | btw, i just pdb'd getPreviousScore | 16:37 |
| replaceafill | it finds 475 raw equivalent for a single skill | 16:37 |
| th1a | If we can find the score for the current term in essentially no time at all, why can't we do the same for the previous linked terms? | 16:38 |
| replaceafill | and after filtering only gets a single skill back | 16:38 |
| yvl | ok, so if it makes sense to ALWAYS look only in previous sections | 16:39 |
| yvl | (not projects, not antything else) | 16:39 |
| yvl | you should override getPreviousScore | 16:39 |
| yvl | and instead of using .findAllEquivalent and the .filterEquivalent | 16:40 |
| yvl | use your new .finEquivalentInSection(activity, section) | 16:40 |
| replaceafill | got it | 16:40 |
| th1a | Projects in the relevant sections? | 16:41 |
| yvl | th1a, we'd need to profile further | 16:41 |
| yvl | but my bet is that loading all 475 skills is slowing things a bit | 16:41 |
| yvl | well 1025 * 475 | 16:41 |
| yvl | technically, a student could have completed the equivalent skill not just in previous section | 16:42 |
| yvl | but in a "previous" section taught by another instructor | 16:42 |
| th1a | That's not relevant to this view. | 16:43 |
| yvl | so it's still an equivalent skill, still a section in previous term, but not a directly previous section with this instructor | 16:43 |
| yvl | that's good for us then :) | 16:44 |
| replaceafill | :D | 16:44 |
| th1a | We're really only interested in linked terms for this view. | 16:48 |
| th1a | Especially since it is the edit view. | 16:50 |
| th1a | OK, I guess you should keep working on this replaceafill... although I'd like that extra leading space bug to be gone in 2.6.1 | 16:51 |
| replaceafill | th1a, oh | 16:51 |
| replaceafill | i tested it yesterday | 16:51 |
| replaceafill | i didn't see the "extra leading" part | 16:51 |
| th1a | Maybe it is a Chrome thing. | 16:51 |
| replaceafill | i know how to do the value.strip() | 16:51 |
| th1a | Yes, indeed. | 16:52 |
| th1a | Do that. | 16:52 |
| replaceafill | oh | 16:52 |
| replaceafill | and i had a question for yvl | 16:52 |
| replaceafill | if you're still around :) | 16:52 |
| yvl | I am :) | 16:52 |
| replaceafill | https://bugs.launchpad.net/schooltool/+bug/1241022 | 16:53 |
| replaceafill | it's about the entity part | 16:53 |
| replaceafill | i remember in old times we did: | 16:53 |
| replaceafill | <!ENTITY nbsp " "> | 16:53 |
| replaceafill | at the top of the rml | 16:53 |
| replaceafill | i haven't checked yet, but do you think this problem could be related to that tag missing in the current templates | 16:54 |
| replaceafill | i'm not sure what's the role of <!entity ...> in the rml | 16:54 |
| yvl | It's a good question if we want non breaking spaces at all here | 16:57 |
| yvl | I'd rather have them stripped | 16:57 |
| replaceafill | ah | 16:57 |
| replaceafill | kk | 16:57 |
| replaceafill | but that's discrimination against old old people :P | 16:57 |
| yvl | oh, right | 16:58 |
| yvl | also me | 16:58 |
| yvl | so. | 16:58 |
| replaceafill | :| | 16:58 |
| yvl | Stripping - yes | 16:58 |
| th1a | I don't mind losing double spaces, if that's the question. | 16:58 |
| replaceafill | ah cool | 16:58 |
| replaceafill | i could strip and remove multiple spaces in the same data converter | 16:59 |
| replaceafill | for the htmlfragmentfield widget | 16:59 |
| replaceafill | i guess i need to use a new problem... i mean, a regular expression | 17:00 |
| replaceafill | unless there's an easier way to remove multiple spaces in strings :) | 17:00 |
| th1a | I'd be happy to just strip the leading spaces and move onto the next issue. | 17:02 |
| th1a | OK, I have to go get the girls. | 17:03 |
| th1a | Later guys. | 17:03 |
| replaceafill | kk | 17:03 |
| replaceafill | thanks th1a | 17:03 |
| th1a | Thanks! | 17:03 |
| yvl | bye th1a | 17:06 |
| yvl | replaceafill, it may be just a missing entity in rml | 17:06 |
| replaceafill | yvl, right | 17:06 |
| replaceafill | i'll check | 17:06 |
| yvl | pound is probably missing too | 17:07 |
| replaceafill | oh | 17:07 |
| replaceafill | i noticed pound | 17:07 |
| yvl | I'd say we need that in all templates | 17:07 |
| replaceafill | why do we need it? | 17:07 |
| replaceafill | i thought it was pound, as in british pound? | 17:07 |
| yvl | yes, I think fckn editor escapes it | 17:08 |
| yvl | if you could check, that would be great :) | 17:08 |
| yvl | oh, and entity should be somewhere in schooltool/skin/flourish/rml/ | 17:09 |
| replaceafill | right | 17:09 |
| replaceafill | it was in rml_macros | 17:09 |
| yvl | pdf.pt probably | 17:09 |
| replaceafill | yvl, correct, the fck editor saves it as   | 17:18 |
| replaceafill | u'<p> Foo bar.  Another quick.</p>' | 17:18 |
| yvl | that editor! | 17:18 |
| replaceafill | lol | 17:24 |
| replaceafill | ok | 17:24 |
| replaceafill | this is a side effect | 17:24 |
| replaceafill | of the FCKConfig.ProcessHTMLEntities = false change | 17:24 |
| replaceafill | the change was needed because the editor saves ó or ñ characters as entities | 17:25 |
| replaceafill | which in turn are displayed like that in the pdfs | 17:25 |
| replaceafill | but in that case the extra spaces are kept | 17:25 |
| replaceafill | u'<p> Foo bar. Another quick fix. This is niño, comunicación.</p>' | 17:25 |
| * replaceafill goes look up the fckeditor docs | 17:26 | |
| *** khildin has quit IRC | 17:34 | |
| *** khildin has joined #schooltool | 17:49 | |
| replaceafill | turns out that just setting: | 18:24 |
| replaceafill | -FCKConfig.ProcessHTMLEntities = false ; | 18:24 |
| replaceafill | +FCKConfig.IncludeLatinEntities = false ; | 18:24 |
| replaceafill | +FCKConfig.IncludeGreekEntities = false ; | 18:24 |
| replaceafill | fixes all the problems with the editor :| | 18:24 |
| replaceafill | áéí... characters work | 18:24 |
| replaceafill | and spaces are not escaped anymore | 18:25 |
| yvl | \o/ | 18:35 |
| replaceafill | :D | 18:35 |
| *** menesis has quit IRC | 18:59 | |
| *** menesis has joined #schooltool | 19:10 | |
| *** menesis has quit IRC | 19:59 | |
| *** khildin has quit IRC | 20:43 | |
| *** khildin has joined #schooltool | 20:51 | |
| *** menesis has joined #schooltool | 20:52 | |
| *** khildin has quit IRC | 20:59 | |
| *** khildin has joined #schooltool | 21:25 | |
| *** khildin has quit IRC | 21:46 | |
| *** khildin has joined #schooltool | 21:49 | |
| replaceafill | from 900 to 10 seconds :) | 23:44 |
| th1a | Pft. | 23:45 |
| th1a | So...? | 23:45 |
| replaceafill | hopefully same output :P | 23:45 |
| th1a | What did you do? | 23:45 |
| replaceafill | running cando tests now | 23:45 |
| th1a | That's still a long time, ofc... | 23:45 |
| replaceafill | used recursion properly i think | 23:45 |
| replaceafill | yes | 23:45 |
| th1a | So were we just looping 1000x what we needed to? | 23:46 |
| replaceafill | there are still some numbers that seem high in the profile output | 23:46 |
| replaceafill | correct | 23:46 |
| th1a | Yeah... sometimes these things seem more like bugs than performance constraints. | 23:46 |
| replaceafill | i stopped lookup after finding the first equivalent too | 23:46 |
| th1a | Oh... so? I'm not sure what that means. | 23:47 |
| replaceafill | findAllEquivalent was finding... well... all! | 23:47 |
| replaceafill | and after filtering | 23:47 |
| replaceafill | you always got 1 skill | 23:47 |
| replaceafill | matching | 23:47 |
| replaceafill | i'm still using for loops to traverse as yvl suggested | 23:47 |
| replaceafill | but they stop after the match is found | 23:48 |
| replaceafill | also | 23:48 |
| replaceafill | the previous sections calculation was done over and over | 23:48 |
| replaceafill | finding the same 4 sections for each worksheet iteration | 23:49 |
| replaceafill | and the method didn't needed the 4 | 23:49 |
| replaceafill | just the previous one | 23:49 |
| th1a | OK. | 23:54 |
| th1a | Well, I'm glad this is more a bug than just hitting the limits of the ZODB or something like that. | 23:54 |
| replaceafill | indeed | 23:54 |
Generated by irclog2html.py 2.15.1 by Marius Gedminas - find it at mg.pov.lt!