IRC log of #schooltool for Wednesday, 2013-10-09

*** menesis has quit IRC00:04
*** menesis has joined #schooltool00:04
th1areplaceafill:  Are we copying the text of the skill into every section now?00:05
replaceafilllet me check00:05
replaceafillth1a, yes i think so00:06
replaceafillsee line 35300:08
replaceafillin the social skills case ALL those text are the same00:09
th1aWell, if we could just get rid of that redundancy isn't that probably a lot of the bulk?00:15
th1aJust look up the path if the attributes are missing toward the original?00:15
* replaceafill puts his eyes on SpecificationDecoratorBase and wonders if it could help with that00:19
th1aThis would be more to prevent the problem in the future.00:20
*** menesis has quit IRC01:07
th1aI logged onto our tektonic server.01:54
th1aI guess I should tell you about it.  ;-)01:54
replaceafilldo we have a tektonic server?!?!01:54
th1aWe got it for Telly!01:54
th1aYou must have forgotten about it.01:54
th1aOr maybe I forgot to tell you.01:55
th1aAnyhow, I'm going to need to set up a social skills demo with new skills for this meeting Friday.01:55
replaceafillah ok01:55
th1aWe can do that tomorrow.01:55
replaceafillwhy don't we do it in the dev vps?01:56
th1aWe could...01:56
th1aBut we also have this other one!01:56
replaceafilli mean, i know everything works on that one01:56
th1aGood point.01:56
replaceafilland we should start using the other one!01:56
th1aLuckily I knew the password and everything.01:56
replaceafillyou should create a user for me ;)01:57
replaceafillor send me the login details for root01:57
replaceafilli have an email from you 22/01/1301:57
replaceafillyour tektonic vps has been created01:58
replaceafillit's me then, i forgot :(01:58
replaceafillth1a, you there?02:08
replaceafilljelkner has this skills list02:08
replaceafillwhich include skill titles such as:02:08
replaceafillUse <strong></strong>, <b></b>, <em></em>, <i></i>, <q></q>, <code></code>,02:08
replaceafill<kbd></kbd>, <sup></sup>, and <sub></sub> elements appropriately.02:08
replaceafillit's an html class02:09
replaceafillbut the markup breaks some parts of the gradebook02:09
replaceafilli think he raised the issue last summer02:09
replaceafilland i failed to file a bug02:09
th1aWell, escape it.02:09
replaceafillcan i tell him i'll fix it after we're done with the social skills?02:10
replaceafillkk, thanks02:13
*** th1a has quit IRC02:14
*** replaceafill has quit IRC02:38
*** alex____ has joined #schooltool04:56
alex____speak spanish?04:56
alex____alguien sabe como me puedo conectar en red a school tool lo instale en una maquina virtual05:02
alex____tengo ping desde afuera a la maquina virtual05:02
alex____pero no puedo entrar con la ip a schooltool05:02
*** yvl has quit IRC07:42
*** alex____ has quit IRC08:47
*** yvl has joined #schooltool09:03
*** menesis has joined #schooltool11:37
*** mibofra has quit IRC14:22
*** mibofra has joined #schooltool14:22
*** mibofra has quit IRC14:22
*** mibofra has joined #schooltool14:23
*** th1a has joined #schooltool15:13
*** replaceafill has joined #schooltool15:17
th1ahi replaceafill, yvl, menesis.15:31
replaceafillgood afternoon15:31
yvlgood morning / afternoon15:31
th1aOK, where are we menesis?15:32
menesisnot yet released15:33
menesisI am done with translations15:33
menesishave found some untranslated strings15:33
menesisdefault score systems are not translated15:33
menesishave fixed that already15:34
menesisafter a little more testing15:35
menesiswill tag the release and upload everywhere15:36
menesistoday, really15:36
th1aCan we slip in replaceafill's gradebook export sort fix?15:36
th1aIt is encouraging in a way that they are so worried about it.15:37
menesisI see the fix is already in trunk15:37
th1aThey must be really using SchoolTool! ;-)15:37
th1aIn Thailand, yvl.15:37
yvloh :)15:38
replaceafill"Please add it in the coming release as we need it on urgent basis."15:38
menesisI have also fixed journal evolution issue when upgrading from 2.1.015:38
menesisalthough I could not reproduce it15:38
menesisbut it seems there is no journal data, so nothing to evolve15:39
menesissimulated the situation and evolution passed15:39
menesisalso fixed evolution order15:39
menesisjournal needs scoresystems container to be initialized15:40
menesisan error about this also appeared in tests15:40
menesisthat's all I've done15:42
th1aOK.  Sounds good.15:42
menesisnow I am updating changelogs15:43
menesistag, upload, package15:43
menesiswill be done tonight15:43
menesisone thing I haven't looked into15:43
menesisis the update of jqueryui15:44
menesisso, the older included version will be used15:44
menesisthere were styling issues, and tweaking css tends to take a lot of time15:44
th1aOh, We'll just include the old one.15:45
th1aSo it is ok, just slightly kludgy from a packaging point of view?15:45
menesisit would be better to use system-installed library15:46
menesisbut if not, so be it15:46
menesisother packages do the same sometimes, e.g. firefox15:46
* menesis done15:47
th1aYes, no problem.15:47
th1aThanks menesis!15:47
th1ayvl: Have you gained enlightenment?15:47
yvl3 strategically placed savepoints make everything somewhat better15:48
yvlit keeps the total memory bloat around 2.5GB during evolution15:49
yvland with other parts (celery, zeo, redis) it peaks to ~4GB15:50
yvlRAM + swap15:50
th1aIs there any way to profile what's taking up all the RAM?15:50
yvlor to be more accurate - changes to data15:51
yvlfirst, there's a LOT of skills deployed15:51
yvlthen, scores are added15:51
yvlthere is just a lot of entries15:52
yvlbtw, if need be, I can make it eat less memory15:52
yvlI don't want to put too many savepoints15:54
yvlI don't want to poke garbage collector manually15:54
yvland not sure if it's worth poking connection.cacheMinimize()15:54
yvland it saves some 800MB15:55
th1aBecause?  Just too much of a slowdown?15:55
th1aThat is, for something we're going to put in SchoolTool proper, not just a workaround in this crazy case.15:55
th1aI'm just saying,15:55
th1aI've kind of been thinking about this in terms of what changes we might put in just for this extreme case.15:56
th1aThen again, this is an evolution script anyhow...15:57
* th1a is just muttering to himself.15:57
th1aYou don't want to add more savepoints, poke the garbage collector, etc. because you think it would slow things down more than would be useful?15:57
yvlit would take time to check15:58
th1aIt is just helpful for me to know in broad terms what the tradeoffs are.15:58
th1aI guess it also might just be a waste of your time for limited gain.15:58
yvlit would somewhat slow down15:58
yvlhow much?15:58
yvlwho knows.15:58
yvlneed to check15:58
yvlnext time, it will be different15:58
yvlwe need to profile each time anyway15:59
th1aWell, replaceafill has pruned I think half the sections out of their instance which weren't actually used.15:59
yvlso there's just no generic knowledge to be gained here IMHO15:59
th1aSo the two together will hopefully make this work.15:59
yvlrule of thumb:15:59
yvluse savepoints15:59
yvlnot too many of them15:59
yvlor you can go crazy with many savepoints and see what happens :)16:00
replaceafilli inserted one after each call to evolveCourse and saw no difference in memory comsuption16:00
menesissafe to include in the release?16:00
yvlyes replaceafill16:00
replaceafilland an increase in time taken16:01
yvlthe thing is, that there are like 4 courses or something16:01
yvlsavepoint per course skillset makes more sense there16:01
yvlboth when deploying and when populating them16:02
yvlcaching machinery seems to be pretty decent, and cleans itself up nicely16:02
yvlpython's garbage collector also seems to know what he's doing16:02
replaceafillhow much RAM do you have?16:03
yvllemme check :)16:04
yvl3.7 GB16:04
replaceafillah ok16:04
replaceafilli'll try your change with my 50% less sections db16:04
yvlhmm, you know what16:04
replaceafillsteven said that it was ok to remove sections with no instructors or students16:05
yvlI'll add one more savepoint16:05
replaceafillah ok16:05
replaceafillyvl, i have two concerns in all this16:05
replaceafillwell two *main* concerns :)16:05
replaceafill1. they use multi term sections, right?16:05
replaceafillso the evolve's try to detect where to move scores16:06
replaceafillshouldn't the datetime of the score be the main aspect to take into account16:07
replaceafillhere's why i'm saying this16:07
replaceafilli have this score for example:16:07
replaceafill1 / Pass16:07
replaceafillNancy Caruso 16-01-2013 - 19:0416:07
replaceafillwhich is landing in a Term 1 section16:07
replaceafillterm 1 goes from:16:07
replaceafill05-09-2012 05-11-201216:08
replaceafillso imho that score should have landed in term 316:08
replaceafilloh, sorry, term 216:08
replaceafill07-11-2012 18-01-201316:08
th1aWhat is happening now?16:09
yvlwait, doesn't pick_section do that?16:10
replaceafilli thought it did16:10
yvlI thought so too16:10
yvlmaybe the student did not attend that section?16:11
yvlit would be great if you could check why that happens16:11
yvlor maybe the teacher is different there16:12
replaceafillbut we all agree that's a mistake?16:12
replaceafilland the score should have landed in term 2?16:12
replaceafillbased on the datetime?16:12
replaceafillsome other scores seem fine16:12
yvlif the student is in that section and the evaluator is teaching that section, then yes16:13
yvlmy guess is that instructors of that section changed since the evaluation16:13
replaceafilli'll check16:14
replaceafillmy second concern is about performance16:15
replaceafillafter evolving the app *feels* very slow16:15
replaceafilleven after packing the db16:15
replaceafilli started blaming their custom grading view16:16
replaceafill(which i'll try to optimize)16:16
replaceafillbut clicking around in other "standard" parts also takes time16:16
replaceafillis there any chance to unplug zeo, celery and redis?16:17
replaceafillmay those be the cause of the slowdown16:17
yvlthe reports will break, but it should run16:17
yvlmaybe those16:18
yvlperson annotations are fully loaded for each student16:18
yvlor not16:18
yvlwhen it comes to perfomance16:19
yvldon't guess16:19
yvlprofile, profile, profile.16:19
replaceafillhow, how, how? :D16:19
yvlwith python!16:19
replaceafillprint end-start16:19
replaceafillor ncalls in the profile module?16:20
yvl has interesting posts on profiling16:20
replaceafillprofilehooks is my goto option always :D16:20
replaceafillsimple decorator on the thing you want to profile16:20
yvlI used Marius' branch of Dozer last time IIRC16:22
* replaceafill googles dozer :)16:22
yvlI'm sorry, I really don't know how to answer questions like this16:23
yvl"how to profile" falls under category of questions "how to write software"16:23
replaceafilli'll go sit next to you next time you profile something ;)16:24
yvlDozer helps, call graphs help16:24
replaceafill"how do you abstract from all the zope crap in the profile output?"16:24
yvlyou think about it and eventually say "ah-hah!"16:25
replaceafill"should i care about all those calls to connection"16:25
replaceafill"wow, persistent is called a lot..."16:25
replaceafilli even enabled directory/paths information16:26
replaceafilland filtered everything but schooltool!16:26
replaceafillbecause the rest seemed like noise to me16:26
yvlit's good to look at that noise16:26
yvlgives hints16:26
replaceafillok, i'll profile the **** out of the grading view ;)16:27
replaceafillthanks yvl16:27
yvland the rest of the app please16:27
replaceafillreally helps16:27
yvlalso try with cold cache16:27
yvli.e. on startup without visiting things16:27
yvlcompare with warmed up16:27
replaceafillwarmed up is skipping a few tries, right?16:28
yvldoes it get better with usage?16:28
replaceafillgot it16:28
yvldoes visiting some parts speeds up other parts?16:28
yvlare these the parts that will be visited often?16:28
yvlfor example, does looking at persons search in School tab speed up gradebooks?16:29
yvlso if updates to skills slow down everything we'll optimize16:31
th1aIn the evolve script I see "Sections get copies of skils instead of proxies."  How did the proxies work before?16:31
yvlsorry, can't remember16:32
yvlI only remember that the behavior was not desirable and I had to change that16:33
th1aDo you remember at all replaceafill?16:33
replaceafillnot really16:34
th1aSo... yvl, are you through with this task then?16:36
yvlI guess so16:36
yvlunless it's still not good enough and needs to use less RAM16:37
yvlthen I can work on it more :)16:38
th1aEh... well, I don't want to spend more time on the evolution aspect.16:38
th1aNobody else is going to need to *evolve* something this crazy.  Well... B2T? replaceafill?16:38
th1aEven so, it is a one time hit.16:38
th1aI think replaceafill has to follow up on trying to profile this in general.16:39
replaceafillb2t has *common* skills layout16:39
replaceafillsocial skills is special16:39
replaceafillin the way we set it up16:39
th1aBecause they effectively have 4x the number of skills they normally would?16:39
th1aAnd they did 4x multi-term sections...16:40
yvlso if this works, great16:40
th1aShould you get this running for them and then look at optimizing overall performance replaceafill?16:40
yvlif not, we'll add per-student per-skillset calls to both connection.cacheMinimize() and gc.collect()16:41
replaceafillth1a, i'd like to profile their custom grading view first16:41
replaceafilland understand why it takes so long *for some cases*16:41
replaceafillbtw, i moved all their custom code to:16:43
replaceafilli guess we could create a schooltool.socialskills project in LP16:43
replaceafillif we're going to support this in the long term :)16:44
th1aWait until they write us a check.  ;-)16:44
* replaceafill done16:45
th1aThanks replaceafill.16:46
th1aCan you set up an empty social skills for me to work on tonight?16:46
th1aWell, I need to add these new skills from VA DOE.16:47
replaceafilldo they use the same 1:1, G, S, etc stuff16:47
replaceafillcouldn't find a better word than "stuff" ;)16:48
replaceafillcontext! i guess that's what they're called16:48
th1aYeah, pretty much.16:48
th1aDo they have a special import sheet or anything?16:49
th1aCan you give me a copy of that?16:49
replaceafilland we had this script that set up the sheets16:49
th1aDo you want to meet in five hours?16:50
replaceafilllet me count right this time16:50
replaceafillsure, six is ok16:50
replaceafillit's 9pm for you, correct?16:50
th1aAnd yvl is back to his regularly scheduled task?16:52
yvlI guess yes?16:54
th1aOK.  ;-)16:55
menesissorry I haven't read everything16:56
th1aI've switched to working more in the evening, which seems to be working a little better for me.16:56
menesisdo you want me to include the cando evolve1 fix?16:56
replaceafillth1a, just fwd you the XLS we used for social skills16:56
th1aThanks replaceafill.16:56
replaceafilland the script to convert them is here:16:56
replaceafillalthought it may need to be updated16:56
replaceafillto spit the latest cando format16:57
replaceafilli'll look into that16:57
th1amenesis:  I think the answer is "yes" since it is just savepoints.16:57
th1aOK, I have to go pick up Vivian and Julia.16:57
th1aThanks guys!16:58
th1aHave a good week/end!16:58
th1aLet's wrap up the release, menesis.16:58
* th1a drops the bag of gravel.16:58
replaceafillthanks everybody16:58
yvlthanks guys16:59
yvlbest of luck with profiling replaceafill16:59
yvllet me know if I can help somehow :)16:59
replaceafillyvl, will do!17:00
*** replaceafill has quit IRC17:56
*** mibofra has quit IRC18:36
*** mibofra has joined #schooltool18:37
*** mibofra has quit IRC18:37
*** mibofra has joined #schooltool18:37
*** mibofra has joined #schooltool18:38
*** replaceafill has joined #schooltool22:30
th1abe with you in a few replaceafill...23:01
th1areplaceafill: Do we have a running instance?23:11
replaceafillth1a, not yet23:11
replaceafillwhere do you want it?23:11
replaceafilli already updated our tektonic vps23:11
replaceafillso we can use that23:12
th1aDid I send you the info?23:12
replaceafillin january ;)23:12
replaceafillare you going to use a domain name for the demo?23:13
replaceafillif so, i have to set up apache23:13
th1aI don't need it.23:14
replaceafilli'll use the tektonic vps then23:14
replaceafillth1a, it's downloading lots of uninstalled packages....23:27
replaceafillwill be ready in a moment23:27
th1aOK.  I'm still looking at spreadsheets.23:27
th1aWith some copy/paste to follow...23:27
th1aActually I only have three skillsets here from them.23:29
th1aSo that won't take long.23:29
th1aMy window manager crashed so I'm going to need to update myself and reboot...23:31
th1aLots of updates for me too...23:38
th1aReloading AppArmor profiles...23:39
replaceafillDefault Manager!23:39
replaceafillbye bye SchoolTool Administrator23:39
th1aNow I need to change the docs!23:39
replaceafillso, i tested the converter script for the autism skills and it works23:41
replaceafilli realized that the socialskills package doesn't need to overwrite cando's default scoresystem anymore23:41
th1aAh, yes.23:41
replaceafillsince you can set any scoresystem now at the skill level23:41
th1aOh... do we need to add that to the script?23:41
replaceafilli mean23:42
replaceafillit works for your demo23:42
replaceafillbut it's definitely something we want to clean23:42
*** th1a has quit IRC23:42
*** th1a has joined #schooltool23:46
th1areplaceafill: each of these skills are turned into four in the importer, right?23:55
replaceafillby the script23:56
th1aOK, now that I'm trying to do it I see how it is different...23:57
th1aThey've got skills/skillsets kind of inverted compared to how we did Autism social skills.23:59

Generated by 2.15.1 by Marius Gedminas - find it at!