IRC log of #schooltool for Monday, 2013-10-07

*** ignas has joined #schooltool00:16
*** ignas has quit IRC00:20
*** ignas has joined #schooltool01:16
*** ignas has quit IRC01:21
*** ignas has joined #schooltool02:17
*** ignas has quit IRC02:22
*** ignas has joined #schooltool02:28
*** ignas has quit IRC02:32
*** menesis has quit IRC03:04
*** yvl has joined #schooltool09:20
*** yvl has quit IRC11:10
*** menesis has joined #schooltool11:11
*** yvl has joined #schooltool11:23
*** yvl has quit IRC11:28
*** yvl has joined #schooltool11:33
*** yvl has quit IRC11:38
*** th1a has joined #schooltool11:50
*** menesis has quit IRC12:44
*** menesis has joined #schooltool13:12
*** menesis has quit IRC15:03
th1ajust a sec guys...15:30
*** yvl has joined #schooltool15:30
th1ahi yvl...15:32
yvlhi th1a15:33
th1aHow are section statuses coming?15:34
yvlhaving different status groups, like section statuses with completion15:35
yvlprobably students with graduation15:35
yvland so on15:35
yvladds extra plumbing15:36
th1aI'm sure.15:36
yvlbut it's moving forward15:36
yvlwhat else...15:37
yvloh, I fixed a small bug with timetables when they have only one day15:37
th1aI saw you marked one won't fix?15:38
yvlby accident15:38
yvlit's now marked fixed commited15:39
th1aOh, right.15:39
yvlso that's it15:40
yvlfinish custom schemes for statuses15:40
yvland move on to evolution15:40
th1amenesis was here...15:42
yvlcan't reach him on mobile15:45
* th1a rolls his eyes.15:45
yvlseems to be netsplit15:47
*** menesis has joined #schooltool15:47
th1aAh, hi menesis.15:47
yvlth1a, this:
menesisyes I thought I'm connected but no one was talking15:48
th1aOK, go ahead menesis.15:51
menesisfirst, I have not yet finalized the release15:51
menesisI was doing two things15:51
menesisfixed sorting by first name and last name in gradebook, cando and journal15:52
menesisjournal does not support reverse sort, but it did not before15:52
menesisalso I fixed many selenium tests15:52
menesisincluding autocompletion popup getting in the way, although that still fails sometimes, needs to wait for it to disappear15:53
menesisalso a bug was reported, a crash in courses CSV import15:54
menesisturns out many places crash, if schoolyear has non-ascii chars in it15:54
menesison submit/cancel15:54
th1aYeah, there are probably lots of little similar bugs in CSV.15:55
th1aWe should probably just get rid of it next year.15:55
menesisit's not related to CSV15:55
th1aOK then.15:55
menesisrather, the ?schoolyear_id=âr query string15:55
menesisso I traversed all School tab15:56
menesisand made sure that all Cancel, Done or Submit buttons return to the correct school year15:56
menesisthe one that was selected before15:56
menesisDouglas has done a lot of this a few weeks ago, but there were issues or missed places15:57
menesisso navigation inside another school year now works as expected15:57
th1aOK, that's a good cleanup.15:58
menesisalso, I renamed to "Default Manager" and fixed tests15:58
menesisthere are still tests of all kinds that fail15:58
menesisbut much less than before15:59
menesisso, today I am merging translations and making the release15:59
th1aThis'll be 12.04, 13.04 PPA and 13.10 Universe (just to repeat that for the 1000th time).16:00
menesisthe same archives that already have 2.5.316:02
menesisoh, there is one error in journal evolution that I still have to fix16:02
menesisbut I already got the exception needed to resolve it16:03
th1aI tried to downgrade out of the trunk PPA to Saucy Universe but I still have 2.5.4dev.16:03
th1aWhat do I have to do to make that switch?16:04
menesissudo ppa-purge ppa:schooltool-owners/trunk16:05
menesiswill remove the ppa and downgrade everything that came from it16:05
* menesis done16:06
th1aapt-get purge?16:07
menesissudo apt-get install ppa-purge16:07
th1aThanks menesis.  Let me know when you push the releases.16:07
th1aOK I do have some news.16:08
menesisa script that „will remove the PPA repository, removes the packages you installed from that PPA and install the packages from the official Ubuntu repositories.“16:08
th1aI had a meeting on Friday with a woman from the Virginia DOE about using CanDo statewide for special education students.16:08
th1aBasically they need to be able to document specific competencies for graduation.16:09
th1aAnd they seem to be grasping the idea that the CanDo model works for the state providing optional open source solutions to state requirements.16:09
th1aIf you're going to push more bookkeeping onto districts, it is nice to have a tool to offer them.16:10
th1aSo I'm going to do a little demo at a meeting on Friday.16:10
th1aThe big thing they'd want is what's usually called a curriculum map.16:11
th1aSo you have a sequence of skills.16:11
th1aIn particular their skills will -- while MUCH simpler than VA CTE -- have to span a pretty long continuum, starting with the most severely handicapped students, (e.g., can signal with finger, etc).16:12
th1aMost of which will be irrelevant to each particular student.16:12
th1aSo there will need to be views that focus in on the appropriate range for the individual student and not swamping the teacher.16:13
th1aBut it shouldn't be very complicated.  Computers are good at if/then.16:13
th1aAnd lists.16:14
th1aThings like that.16:14
th1aI explained it would be pretty easy if the requirements are clear and stable.  IF!16:14
th1aHopefully I'll have an example of their work so far today that I can try loading into a CanDo.16:14
yvlif you can get that in, please share exported xls16:15
th1aI will.16:15
th1aThis would be the first real SIELibre work.16:15
*** replaceafill has joined #schooltool16:16
th1aSpeaking of which.16:16
replaceafillgood afternoon16:16
th1aHi replaceafill.16:16
replaceafillsorry i'm late16:16
replaceafilli didn't hear the alarm today :(16:16
yvlgood morning16:16
th1aYou could have blamed the internet.16:17
th1aOr the electricity.  ;-)16:17
th1aYou can read the logs... otherwise it is your turn.16:19
replaceafillah ok16:19
replaceafillso, i started working on the social skills changes16:19
replaceafillit became really really painful16:20
replaceafillevolving the current database was really complicated16:20
replaceafillfirst, i had no enough RAM16:20
replaceafilland no enough swap space either16:20
replaceafillso i had to resize my partitions!16:20
replaceafilladding up to 10 GB for swap16:21
replaceafillthen evolution takes 2 hours +- 10 minutes to finish16:21
replaceafilland the last painful part is that the Data.fs file grows from16:22
replaceafill500 MB to 3.6 GB!16:22
replaceafilli think there's no way in hell their server is going to be able to handle that16:22
th1aI can't believe it can't be done much more efficiently.16:23
replaceafillth1a, i tried to "improved" the first two evolve scripts but wasn't able too16:23
replaceafillthe docstring in the second one is kind of scary:16:24
replaceafillFix the mess made by evolution 1.16:24
replaceafillthat's the docstring ^16:24
th1aWho wrote these?16:24
replaceafillok, but here's the worst part in my opinion16:25
replaceafillthe server takes around 5 minutes to start each time16:25
replaceafillso if i'm working on something and i want to test a change16:25
replaceafillit takes a while just to get it running16:25
*** ignas has joined #schooltool16:26
replaceafillfrom the data point of view, we're also kind of f***ed16:26
th1aSo who did the original evoluitons?16:26
menesishave you tried to pack the db?16:26
replaceafillmenesis, yes16:27
replaceafillit reduces to 3.1 GB only16:27
th1aThis is related to the skills being copied into sections?16:27
replaceafillth1a, correct16:27
replaceafillthey have around 2k sections iirc16:27
th1aOh... ok.16:27
th1aIs this for the whole state or something?16:27
replaceafillso, the evolve scripts tried to detect the date of the changes16:28
replaceafillto set history properly16:28
replaceafillth1a, no idea really16:28
replaceafilland since the sections are set up for 4 terms16:28
replaceafillwe have some empty scores in some gradebooks16:29
replaceafillbut i checked the dates and they kind of make sense16:29
th1aHow many students are there?16:29
replaceafillbut i think it'll be confused for some users16:29
replaceafilllet me check16:30
replaceafillthankfully i started the server when i began my report ;)16:30
replaceafill6583 persons in the students group16:30
replaceafill2151 ones in teachers16:31
replaceafillthere are only 6 courses16:31
replaceafilland 2092 sections16:31
th1a4 terms?16:31
th1aI guess it performs ok in general?16:31
replaceafillone thing i noticed is16:32
th1aSo about 500 linked sections?16:32
replaceafillterms are used for multiyear stuff16:32
replaceafillmp1 goes from september 2012 to november 201216:32
replaceafillthen mp2 goes from nov 2012 - jan 201316:32
replaceafillbut mp4 goes from april 2013 to june *2014*16:33
replaceafillwhich seems like a mistake to me16:33
replaceafillso, i worked on merging their view code16:34
replaceafillto the schooltool.socialskills package16:34
th1aI'm not trying to blame anyone here, but it would be useful to know who wrote the evolution originally.16:34
replaceafillah sorry16:34
replaceafilllet me check the log16:35
* yvl did16:35
yvlstill looking and trying to remember details16:36
th1aSo... I can see how this might be slow but why does it eat memory?16:36
yvlno savepoints16:37
replaceafillyvl, i tried inserting some16:37
replaceafillbut it took even more time16:37
replaceafill(i inserted prints with datetimes to check)16:37
th1aIt doesn't matter if it is slow if it doesn't crash.16:38
replaceafillbut i checked the RAM used even with the savepoints16:38
replaceafilland it's the same16:39
replaceafillincreases slowly16:39
replaceafilluntil it starts swapping16:39
th1aDoes it go down when the evolution is done?16:39
replaceafilldown as in "killing the process"?16:40
replaceafillthat happened when i had only 2 GB of swap16:40
replaceafillth1a, or if you mean RAM getting free, no16:40
th1aIf it ever finishes?16:40
replaceafillstays the same16:41
th1aIt seems to me that the first assumption should be to treat this as a bug in the evolution.16:41
replaceafillRAM seems to be "returned" until you CTRL + C the service16:41
replaceafilltheir current server has only 3 GB of RAM and 3 GB of swap16:43
replaceafill(according to the free command)16:43
replaceafillso again, i'm not sure it'll handle this16:43
th1aWell... to a degree they're just trying to do too much with one server.16:44
th1aBut on the other hand, this shouldn't eat so much RAM, and we don't know why or what is eating RAM at this point, right?16:44
yvlso the evolution there is quite tricky16:45
th1aBut the database is probably just getting big because there is a fair amount of text we're copying thousands of times.16:45
yvlit has to be optimised16:45
yvlsavepoints added16:45
yvlcaches flushed16:45
yvlprobably the database packed afterwards16:45
replaceafillhow do you flush the cache?16:45
yvla good question16:46
yvlas in - I have never done that16:46
replaceafillpacking the 3.6GB to 3.1GB helps only when starting the server16:47
replaceafillit goes from 5 minutes to 1.5 minutes to start16:47
replaceafilli'm still wondering about that term error though16:48
replaceafillsuppose we finally improve something16:48
replaceafillwhen they try to add this year16:48
yvlso we only have to optimize evolution16:48
yvlit's not going to be trivial16:48
replaceafillthey'll go start from oct-201316:48
th1areplaceafill:  Don't worry about that rightnow.16:48
replaceafillah ok16:48
replaceafilli tried improving stuff like not using .values()16:49
yvlso we have two choices here16:49
replaceafillor removing unnecessary lines:16:49
yvlassuming we only have one big deployment of CanDo16:49
yvl1) evolve everything ourselves16:50
yvl2) spend some time optimizing16:50
th1aThis is the autism database that's causing the problem.16:50
th1aIt is somewhat idiosyncratic because the sections are small and there are a ton of them.16:52
th1aI'm not sure what 1) and 2) mean here yvl.16:52
yvl1: download their database, run evolution, upload it to their server again16:52
yvl2: download their database, spend few days profiling and optimizing, release fix, see evolution run on their servers16:53
th1aI think it is probably worth learning wtf is going on memory-wise with the evolution.16:53
yvlI would expect it to eat a load of RAM16:54
yvlit just touches too many records16:55
yvland remakes them16:55
yvlit's ZODB hell :)16:55
yvlthen again, it's definitely possible to work around that16:55
th1aI can see that but it seems like it should be able to do a few and then clear the memory.16:55
yvlI just don't know how long it will take16:55
yvlprobably maybe yes16:56
yvlI would go that route16:56
th1aWe have to learn some stupid ZODB tricks if we're going to help people use this in real life.16:56
th1aWe also should say "OK, actually you're hitting the practical limits for number of sections here."16:57
replaceafillsections and skillsets i'd say16:57
th1aWe are at the point where we can say "Start this update before you go home and it should be running in the morning."16:57
yvland evaluations16:57
yvland they also have score history16:57
replaceafilleach course has around 90-120 skillsets associated16:57
th1aIf it takes six hours but doesn't crash, ok.16:57
yvlallright then16:58
yvlif you'd get me the database I could profile16:58
replaceafillmy pleasure16:58
replaceafilli'll move it to the VPS and send you the link16:58
replaceafillironically, the original 500 MB (their current db) only takes 36 MB as .tar.bz2 :D16:59
yvloh the magic of compression!17:00
replaceafillin the meantime, i can continue working on the views with the huge 3.1 GB one ;)17:00
th1aOK, I'm going to have to go get the girls.17:00
replaceafillth1a, i'll need to ask you later about jelkner request on the summary worksheet17:01
replaceafilli'll probably skip french today17:01
th1aWell, it isn't a big priority.17:01
th1aMaybe we can squeeze it in, but I can tell him that.17:02
th1aDefinitely get autism straightened out first.17:02
replaceafillah and dwelsh wants me to help with his custom lists too17:02
* replaceafill done17:02
th1aThere's definitely an infinite stream of things which could usefully be added to gradebook.17:03
replaceafillyvl, scp'ing the db17:03
th1aYou have to run his script on the custom skills or whatever?17:03
replaceafillyvl, just PM you the link17:03
yvlgot it17:04
th1aThanks guys.  Gotta go.17:04
replaceafillyvl, please let me know how it behaves in your hardware :)17:04
replaceafillthanks everybody17:04
* th1a drops the bag of gravel.17:04
yvlwill keep you posted17:04
*** yvl has quit IRC17:12
*** khildin has joined #schooltool17:15
*** replaceafill has quit IRC17:16
*** replaceafill has joined #schooltool17:19
*** menesis has quit IRC20:50
*** khildin has quit IRC21:37
*** khildin has joined #schooltool21:40
*** th1a has quit IRC22:12
*** th1a has joined #schooltool22:27
*** menesis has joined #schooltool22:29
*** ignas has quit IRC22:34
*** khildin has quit IRC22:36
*** ignas has joined #schooltool22:39

Generated by 2.15.1 by Marius Gedminas - find it at!