| *** ignas has joined #schooltool | 00:16 | |
| *** ignas has quit IRC | 00:20 | |
| *** ignas has joined #schooltool | 01:16 | |
| *** ignas has quit IRC | 01:21 | |
| *** ignas has joined #schooltool | 02:17 | |
| *** ignas has quit IRC | 02:22 | |
| *** ignas has joined #schooltool | 02:28 | |
| *** ignas has quit IRC | 02:32 | |
| *** menesis has quit IRC | 03:04 | |
| *** yvl has joined #schooltool | 09:20 | |
| *** yvl has quit IRC | 11:10 | |
| *** menesis has joined #schooltool | 11:11 | |
| *** yvl has joined #schooltool | 11:23 | |
| *** yvl has quit IRC | 11:28 | |
| *** yvl has joined #schooltool | 11:33 | |
| *** yvl has quit IRC | 11:38 | |
| *** th1a has joined #schooltool | 11:50 | |
| *** menesis has quit IRC | 12:44 | |
| *** menesis has joined #schooltool | 13:12 | |
| *** menesis has quit IRC | 15:03 | |
| th1a | just a sec guys... | 15:30 |
|---|---|---|
| *** yvl has joined #schooltool | 15:30 | |
| th1a | hi yvl... | 15:32 |
| yvl | hi th1a | 15:33 |
| th1a | How are section statuses coming? | 15:34 |
| yvl | steadily | 15:35 |
| yvl | having different status groups, like section statuses with completion | 15:35 |
| yvl | probably students with graduation | 15:35 |
| yvl | and so on | 15:35 |
| yvl | adds extra plumbing | 15:36 |
| th1a | I'm sure. | 15:36 |
| yvl | but it's moving forward | 15:36 |
| yvl | what else... | 15:37 |
| yvl | oh, I fixed a small bug with timetables when they have only one day | 15:37 |
| th1a | I saw you marked one won't fix? | 15:38 |
| yvl | by accident | 15:38 |
| yvl | it's now marked fixed commited | 15:39 |
| th1a | Oh, right. | 15:39 |
| th1a | OK. | 15:39 |
| yvl | so that's it | 15:40 |
| yvl | finish custom schemes for statuses | 15:40 |
| yvl | and move on to evolution | 15:40 |
| th1a | OK... | 15:41 |
| th1a | menesis was here... | 15:42 |
| yvl | pinging... | 15:43 |
| yvl | ... | 15:44 |
| yvl | can't reach him on mobile | 15:45 |
| * th1a rolls his eyes. | 15:45 | |
| yvl | oh | 15:47 |
| yvl | seems to be netsplit | 15:47 |
| *** menesis has joined #schooltool | 15:47 | |
| th1a | netsplit? | 15:47 |
| th1a | Ah, hi menesis. | 15:47 |
| menesis | hi | 15:47 |
| yvl | th1a, this: http://en.wikipedia.org/wiki/Netsplit | 15:48 |
| menesis | yes I thought I'm connected but no one was talking | 15:48 |
| th1a | Huh. | 15:49 |
| th1a | OK, go ahead menesis. | 15:51 |
| menesis | first, I have not yet finalized the release | 15:51 |
| menesis | I was doing two things | 15:51 |
| menesis | fixed sorting by first name and last name in gradebook, cando and journal | 15:52 |
| menesis | journal does not support reverse sort, but it did not before | 15:52 |
| menesis | also I fixed many selenium tests | 15:52 |
| menesis | including autocompletion popup getting in the way, although that still fails sometimes, needs to wait for it to disappear | 15:53 |
| menesis | also a bug was reported, a crash in courses CSV import | 15:54 |
| menesis | turns out many places crash, if schoolyear has non-ascii chars in it | 15:54 |
| menesis | on submit/cancel | 15:54 |
| th1a | Yeah, there are probably lots of little similar bugs in CSV. | 15:55 |
| th1a | We should probably just get rid of it next year. | 15:55 |
| menesis | it's not related to CSV | 15:55 |
| th1a | Oh! | 15:55 |
| th1a | OK then. | 15:55 |
| menesis | rather, the ?schoolyear_id=âr query string | 15:55 |
| menesis | so I traversed all School tab | 15:56 |
| menesis | and made sure that all Cancel, Done or Submit buttons return to the correct school year | 15:56 |
| menesis | the one that was selected before | 15:56 |
| menesis | Douglas has done a lot of this a few weeks ago, but there were issues or missed places | 15:57 |
| menesis | so navigation inside another school year now works as expected | 15:57 |
| th1a | OK, that's a good cleanup. | 15:58 |
| menesis | also, I renamed to "Default Manager" and fixed tests | 15:58 |
| menesis | there are still tests of all kinds that fail | 15:58 |
| menesis | but much less than before | 15:59 |
| menesis | so, today I am merging translations and making the release | 15:59 |
| th1a | This'll be 12.04, 13.04 PPA and 13.10 Universe (just to repeat that for the 1000th time). | 16:00 |
| menesis | yes | 16:01 |
| menesis | the same archives that already have 2.5.3 | 16:02 |
| menesis | oh, there is one error in journal evolution that I still have to fix | 16:02 |
| th1a | OK. | 16:03 |
| menesis | but I already got the exception needed to resolve it | 16:03 |
| th1a | I tried to downgrade out of the trunk PPA to Saucy Universe but I still have 2.5.4dev. | 16:03 |
| th1a | What do I have to do to make that switch? | 16:04 |
| menesis | sudo ppa-purge ppa:schooltool-owners/trunk | 16:05 |
| menesis | will remove the ppa and downgrade everything that came from it | 16:05 |
| * menesis done | 16:06 | |
| th1a | apt-get purge? | 16:07 |
| menesis | no | 16:07 |
| menesis | sudo apt-get install ppa-purge | 16:07 |
| th1a | Thanks menesis. Let me know when you push the releases. | 16:07 |
| th1a | OK. | 16:07 |
| th1a | OK I do have some news. | 16:08 |
| menesis | a script that „will remove the PPA repository, removes the packages you installed from that PPA and install the packages from the official Ubuntu repositories.“ | 16:08 |
| th1a | I had a meeting on Friday with a woman from the Virginia DOE about using CanDo statewide for special education students. | 16:08 |
| th1a | Basically they need to be able to document specific competencies for graduation. | 16:09 |
| th1a | And they seem to be grasping the idea that the CanDo model works for the state providing optional open source solutions to state requirements. | 16:09 |
| th1a | If you're going to push more bookkeeping onto districts, it is nice to have a tool to offer them. | 16:10 |
| th1a | So I'm going to do a little demo at a meeting on Friday. | 16:10 |
| th1a | The big thing they'd want is what's usually called a curriculum map. | 16:11 |
| th1a | So you have a sequence of skills. | 16:11 |
| th1a | In particular their skills will -- while MUCH simpler than VA CTE -- have to span a pretty long continuum, starting with the most severely handicapped students, (e.g., can signal with finger, etc). | 16:12 |
| th1a | Most of which will be irrelevant to each particular student. | 16:12 |
| th1a | So there will need to be views that focus in on the appropriate range for the individual student and not swamping the teacher. | 16:13 |
| th1a | But it shouldn't be very complicated. Computers are good at if/then. | 16:13 |
| th1a | And lists. | 16:14 |
| th1a | Things like that. | 16:14 |
| th1a | I explained it would be pretty easy if the requirements are clear and stable. IF! | 16:14 |
| th1a | Hopefully I'll have an example of their work so far today that I can try loading into a CanDo. | 16:14 |
| yvl | if you can get that in, please share exported xls | 16:15 |
| th1a | I will. | 16:15 |
| th1a | This would be the first real SIELibre work. | 16:15 |
| *** replaceafill has joined #schooltool | 16:16 | |
| th1a | Speaking of which. | 16:16 |
| replaceafill | good afternoon | 16:16 |
| th1a | Hi replaceafill. | 16:16 |
| replaceafill | sorry i'm late | 16:16 |
| replaceafill | i didn't hear the alarm today :( | 16:16 |
| yvl | good morning | 16:16 |
| th1a | You could have blamed the internet. | 16:17 |
| th1a | Or the electricity. ;-) | 16:17 |
| replaceafill | :D | 16:17 |
| th1a | You can read the logs... otherwise it is your turn. | 16:19 |
| replaceafill | ah ok | 16:19 |
| replaceafill | so, i started working on the social skills changes | 16:19 |
| replaceafill | it became really really painful | 16:20 |
| replaceafill | evolving the current database was really complicated | 16:20 |
| replaceafill | first, i had no enough RAM | 16:20 |
| replaceafill | and no enough swap space either | 16:20 |
| replaceafill | so i had to resize my partitions! | 16:20 |
| replaceafill | adding up to 10 GB for swap | 16:21 |
| replaceafill | then evolution takes 2 hours +- 10 minutes to finish | 16:21 |
| replaceafill | and the last painful part is that the Data.fs file grows from | 16:22 |
| replaceafill | 500 MB to 3.6 GB! | 16:22 |
| replaceafill | i think there's no way in hell their server is going to be able to handle that | 16:22 |
| th1a | I can't believe it can't be done much more efficiently. | 16:23 |
| replaceafill | th1a, i tried to "improved" the first two evolve scripts but wasn't able too | 16:23 |
| replaceafill | the docstring in the second one is kind of scary: | 16:24 |
| replaceafill | Fix the mess made by evolution 1. | 16:24 |
| replaceafill | that's the docstring ^ | 16:24 |
| th1a | Yes... | 16:24 |
| th1a | Who wrote these? | 16:24 |
| replaceafill | ok, but here's the worst part in my opinion | 16:25 |
| replaceafill | the server takes around 5 minutes to start each time | 16:25 |
| replaceafill | so if i'm working on something and i want to test a change | 16:25 |
| replaceafill | it takes a while just to get it running | 16:25 |
| replaceafill | anyway | 16:26 |
| *** ignas has joined #schooltool | 16:26 | |
| replaceafill | from the data point of view, we're also kind of f***ed | 16:26 |
| th1a | So who did the original evoluitons? | 16:26 |
| menesis | have you tried to pack the db? | 16:26 |
| replaceafill | menesis, yes | 16:27 |
| replaceafill | it reduces to 3.1 GB only | 16:27 |
| th1a | This is related to the skills being copied into sections? | 16:27 |
| replaceafill | th1a, correct | 16:27 |
| replaceafill | they have around 2k sections iirc | 16:27 |
| th1a | Oh... ok. | 16:27 |
| th1a | Why? | 16:27 |
| th1a | Is this for the whole state or something? | 16:27 |
| replaceafill | so, the evolve scripts tried to detect the date of the changes | 16:28 |
| replaceafill | to set history properly | 16:28 |
| replaceafill | th1a, no idea really | 16:28 |
| replaceafill | and since the sections are set up for 4 terms | 16:28 |
| replaceafill | we have some empty scores in some gradebooks | 16:29 |
| replaceafill | but i checked the dates and they kind of make sense | 16:29 |
| th1a | How many students are there? | 16:29 |
| replaceafill | but i think it'll be confused for some users | 16:29 |
| replaceafill | let me check | 16:30 |
| replaceafill | thankfully i started the server when i began my report ;) | 16:30 |
| replaceafill | 6583 persons in the students group | 16:30 |
| replaceafill | 2151 ones in teachers | 16:31 |
| th1a | Hrm. | 16:31 |
| replaceafill | there are only 6 courses | 16:31 |
| replaceafill | and 2092 sections | 16:31 |
| th1a | 4 terms? | 16:31 |
| replaceafill | yes | 16:31 |
| replaceafill | mp1...mp4 | 16:31 |
| th1a | I guess it performs ok in general? | 16:31 |
| replaceafill | one thing i noticed is | 16:32 |
| th1a | So about 500 linked sections? | 16:32 |
| replaceafill | terms are used for multiyear stuff | 16:32 |
| replaceafill | mp1 goes from september 2012 to november 2012 | 16:32 |
| replaceafill | then mp2 goes from nov 2012 - jan 2013 | 16:32 |
| replaceafill | but mp4 goes from april 2013 to june *2014* | 16:33 |
| replaceafill | which seems like a mistake to me | 16:33 |
| th1a | Probably. | 16:34 |
| replaceafill | so, i worked on merging their view code | 16:34 |
| replaceafill | to the schooltool.socialskills package | 16:34 |
| th1a | I'm not trying to blame anyone here, but it would be useful to know who wrote the evolution originally. | 16:34 |
| replaceafill | ah sorry | 16:34 |
| replaceafill | let me check the log | 16:35 |
| yvl | oh | 16:35 |
| * yvl did | 16:35 | |
| replaceafill | yes | 16:35 |
| th1a | OK. | 16:35 |
| yvl | still looking and trying to remember details | 16:36 |
| th1a | So... I can see how this might be slow but why does it eat memory? | 16:36 |
| yvl | no savepoints | 16:37 |
| replaceafill | yvl, i tried inserting some | 16:37 |
| replaceafill | but it took even more time | 16:37 |
| replaceafill | (i inserted prints with datetimes to check) | 16:37 |
| yvl | sure | 16:38 |
| th1a | It doesn't matter if it is slow if it doesn't crash. | 16:38 |
| replaceafill | but i checked the RAM used even with the savepoints | 16:38 |
| replaceafill | and it's the same | 16:39 |
| replaceafill | increases slowly | 16:39 |
| replaceafill | until it starts swapping | 16:39 |
| th1a | Does it go down when the evolution is done? | 16:39 |
| replaceafill | down as in "killing the process"? | 16:40 |
| replaceafill | that happened when i had only 2 GB of swap | 16:40 |
| replaceafill | th1a, or if you mean RAM getting free, no | 16:40 |
| th1a | If it ever finishes? | 16:40 |
| replaceafill | stays the same | 16:41 |
| th1a | It seems to me that the first assumption should be to treat this as a bug in the evolution. | 16:41 |
| replaceafill | RAM seems to be "returned" until you CTRL + C the service | 16:41 |
| replaceafill | their current server has only 3 GB of RAM and 3 GB of swap | 16:43 |
| replaceafill | (according to the free command) | 16:43 |
| replaceafill | so again, i'm not sure it'll handle this | 16:43 |
| th1a | Well... to a degree they're just trying to do too much with one server. | 16:44 |
| th1a | But on the other hand, this shouldn't eat so much RAM, and we don't know why or what is eating RAM at this point, right? | 16:44 |
| replaceafill | right | 16:44 |
| yvl | ok | 16:44 |
| yvl | so the evolution there is quite tricky | 16:45 |
| th1a | But the database is probably just getting big because there is a fair amount of text we're copying thousands of times. | 16:45 |
| yvl | exactly | 16:45 |
| yvl | it has to be optimised | 16:45 |
| yvl | savepoints added | 16:45 |
| yvl | caches flushed | 16:45 |
| yvl | probably the database packed afterwards | 16:45 |
| replaceafill | how do you flush the cache? | 16:45 |
| yvl | a good question | 16:46 |
| yvl | as in - I have never done that | 16:46 |
| replaceafill | packing the 3.6GB to 3.1GB helps only when starting the server | 16:47 |
| yvl | yes | 16:47 |
| replaceafill | it goes from 5 minutes to 1.5 minutes to start | 16:47 |
| yvl | good | 16:47 |
| replaceafill | i'm still wondering about that term error though | 16:48 |
| replaceafill | suppose we finally improve something | 16:48 |
| replaceafill | when they try to add this year | 16:48 |
| yvl | so we only have to optimize evolution | 16:48 |
| yvl | it's not going to be trivial | 16:48 |
| replaceafill | they'll go start from oct-2013 | 16:48 |
| th1a | replaceafill: Don't worry about that rightnow. | 16:48 |
| replaceafill | ah ok | 16:48 |
| replaceafill | i tried improving stuff like not using .values() | 16:49 |
| yvl | so we have two choices here | 16:49 |
| replaceafill | or removing unnecessary lines: | 16:49 |
| yvl | assuming we only have one big deployment of CanDo | 16:49 |
| replaceafill | http://bazaar.launchpad.net/~schooltool-owners/schooltool.cando/trunk/view/head:/src/schooltool/cando/generations/evolve1.py#L112 | 16:49 |
| yvl | 1) evolve everything ourselves | 16:50 |
| yvl | 2) spend some time optimizing | 16:50 |
| th1a | This is the autism database that's causing the problem. | 16:50 |
| th1a | It is somewhat idiosyncratic because the sections are small and there are a ton of them. | 16:52 |
| th1a | I'm not sure what 1) and 2) mean here yvl. | 16:52 |
| yvl | 1: download their database, run evolution, upload it to their server again | 16:52 |
| yvl | 2: download their database, spend few days profiling and optimizing, release fix, see evolution run on their servers | 16:53 |
| th1a | I think it is probably worth learning wtf is going on memory-wise with the evolution. | 16:53 |
| yvl | well | 16:54 |
| yvl | I would expect it to eat a load of RAM | 16:54 |
| yvl | it just touches too many records | 16:55 |
| yvl | and remakes them | 16:55 |
| yvl | twice | 16:55 |
| yvl | it's ZODB hell :) | 16:55 |
| yvl | then again, it's definitely possible to work around that | 16:55 |
| th1a | I can see that but it seems like it should be able to do a few and then clear the memory. | 16:55 |
| yvl | I just don't know how long it will take | 16:55 |
| yvl | probably maybe yes | 16:56 |
| yvl | I would go that route | 16:56 |
| th1a | We have to learn some stupid ZODB tricks if we're going to help people use this in real life. | 16:56 |
| th1a | We also should say "OK, actually you're hitting the practical limits for number of sections here." | 16:57 |
| replaceafill | sections and skillsets i'd say | 16:57 |
| th1a | We are at the point where we can say "Start this update before you go home and it should be running in the morning." | 16:57 |
| yvl | and evaluations | 16:57 |
| yvl | and they also have score history | 16:57 |
| replaceafill | each course has around 90-120 skillsets associated | 16:57 |
| th1a | If it takes six hours but doesn't crash, ok. | 16:57 |
| yvl | allright then | 16:58 |
| yvl | if you'd get me the database I could profile | 16:58 |
| replaceafill | my pleasure | 16:58 |
| th1a | lol | 16:58 |
| replaceafill | i'll move it to the VPS and send you the link | 16:58 |
| yvl | thanks! | 16:59 |
| replaceafill | ironically, the original 500 MB (their current db) only takes 36 MB as .tar.bz2 :D | 16:59 |
| yvl | oh the magic of compression! | 17:00 |
| replaceafill | in the meantime, i can continue working on the views with the huge 3.1 GB one ;) | 17:00 |
| th1a | OK, I'm going to have to go get the girls. | 17:00 |
| replaceafill | th1a, i'll need to ask you later about jelkner request on the summary worksheet | 17:01 |
| replaceafill | i'll probably skip french today | 17:01 |
| th1a | Well, it isn't a big priority. | 17:01 |
| replaceafill | ok | 17:01 |
| th1a | Maybe we can squeeze it in, but I can tell him that. | 17:02 |
| replaceafill | please | 17:02 |
| th1a | Definitely get autism straightened out first. | 17:02 |
| replaceafill | sure | 17:02 |
| replaceafill | ah and dwelsh wants me to help with his custom lists too | 17:02 |
| * replaceafill done | 17:02 | |
| th1a | There's definitely an infinite stream of things which could usefully be added to gradebook. | 17:03 |
| replaceafill | yvl, scp'ing the db | 17:03 |
| yvl | kk | 17:03 |
| th1a | You have to run his script on the custom skills or whatever? | 17:03 |
| replaceafill | yvl, just PM you the link | 17:03 |
| replaceafill | yes | 17:04 |
| yvl | got it | 17:04 |
| th1a | OK. | 17:04 |
| th1a | Thanks guys. Gotta go. | 17:04 |
| replaceafill | yvl, please let me know how it behaves in your hardware :) | 17:04 |
| replaceafill | thanks everybody | 17:04 |
| * th1a drops the bag of gravel. | 17:04 | |
| yvl | sure | 17:04 |
| yvl | thanks! | 17:04 |
| yvl | will keep you posted | 17:04 |
| replaceafill | thanks! | 17:04 |
| *** yvl has quit IRC | 17:12 | |
| *** khildin has joined #schooltool | 17:15 | |
| *** replaceafill has quit IRC | 17:16 | |
| *** replaceafill has joined #schooltool | 17:19 | |
| *** menesis has quit IRC | 20:50 | |
| *** khildin has quit IRC | 21:37 | |
| *** khildin has joined #schooltool | 21:40 | |
| *** th1a has quit IRC | 22:12 | |
| *** th1a has joined #schooltool | 22:27 | |
| *** menesis has joined #schooltool | 22:29 | |
| *** ignas has quit IRC | 22:34 | |
| *** khildin has quit IRC | 22:36 | |
| *** ignas has joined #schooltool | 22:39 | |
Generated by irclog2html.py 2.15.1 by Marius Gedminas - find it at mg.pov.lt!