I woke up this morning with a message in the forum from Arto saying that his news users wasn’t showing anywhere.
After digging for a while I could find out that actual server has not enough memory to import the files. This now explains me why laptop is taking 30 minutes to import them and server only 5 minutes. True is that it was only importing data for already existing users and when doing calculations that include create new users interface ran out of memory and crashed.
First thing to fix was set the webservice to use laptop database instead. Then I was planning to out and buy more memory. If you read earlier posts you might remember that current server is a Core2 Duo E8400, this means memory DDR2. And this fuckers are hard to find, especially in 4GB pieces.
I could not find in any local store, so I created a swap file to add more swap memory. This fixed temporarily, but increased quite a lot importation time.
So I went to internet and bought 2x 4GB DDR2-800. Quite expensive for its age, but that’s what we have now. Till now customs have not released the server Mr. Ed donated, but this is another story!
You might have already noticed that everything from http://kakaostats.alexbueno.me/ now redirects to http://www.kakaostats.net/ .
This will be the definitive domain, but since all over the world the known address is the subdomain to my own domain, I’ll left this redirect for like… forever!
Same will happen with this blog and Forum.
At 21/09/2016 roundly 00:00 UTC time restore was finished!
Took extra time for two reasons:
- Old Atom D2500 motherboard was screwed and it was showing some kernel errors from time to time that froze the OS. One friend gave me an Intel 775 motherboard, it was still in the box never used, so I bought a cheap Core2Duo E8400 plus 4GiB of memory in some kind of eBay we have in Brazil. It’s currently overclocked at 3.6GHz getting close to 90°C because it came without cooler and is relying on a Celeron 430 cooler. I bought a new one already, but did not receive yet.
- Because sometimes there’s no user or data file from Stanford, restore script always got stuck on that and stopped. Of course first Murphy’s law is unmerciful and the first time it happened was during weekend while I was on the beach without internet access. Almost tree days without restore running. Aaaand I sleep from time to time, so happened few times during that either.
I was checking yesterday the source code of KakaoStats everyone knew, and I guess I know how to get it running again. I’ll travel for training today and I’ll have some spare time in hotel today and tomorrow evening so I already started up a virtual machine to play on it.
What do you think? Should I do it? Which version you liked more? Should I get both running? Let me know…!
You might have noticed that since 13/09/2016 @ 15:20 UTC there’s no updates.
At first I was assuming this was just another time without new files. It happens from time to time. Then I connected the server to check and somehow it’s downloading the files and importing correctly without any errors.
Since I was already working on a script to restore the database with backup files in the coming soon Dell PowerEdge, I started doing this in a computer I have here. Pretty simple, it’s an Atom D2500 with 4GB of RAM, but works (or seems so). This one is serving new domain already.
While it started restoring I got back to current server and begin to investigate WTH no new data in the data base. No clue so far!
So I gave up for now, because I have other stuffs to do (like work) and decided to run the restore script in the old server. No data will be lost since even with the script running new files are being downloaded.
I just registered a new domain to KakaoStats, soon I’ll migrate from http://kakaostats.alexbueno.me/ to http://www.kakaostats.net/ . Forum and blog will also be migrated!
Unfortunately someone already registered the .com domain and it’s for sale for something like $10k. No way!
Also I detected some problems in the database after this long period without new files. The interface that download new files wasn’t downloading nothing because it detected that it was still the same file, but it was importing this file hourly.
This seems to create a little mess in the database, some donor names are showing wrong. I’ll begin to implement this checking in the interface to avoid problem, and will restore the database to fix it. Restore script is also needed.
Keep you posted…
What do you want first? Good news or bad news?
Let’s go with chronological order… Good news first!
Last Friday (02/08/2016) I got an email from Ed and he offered a whole new server to host KakaoStats. I wasn’t really expecting that! Really!! When I read the email with a link for the proposed server I was expecting something like a PC (what I was planning to replace the actual laptop), but then I opened the link and became speechless… A Dell PowerEdge R710 with 3x 1TiB HDD (hot swap), 96GiB of RAM and 2x Xeon 6 cores (don’t remember the model), plus everything else that comes in this kind of server. Not even in my hottest dreams I ever imagined a server like this to run KakaoStats!!
Don’t even need to talk about its reliability, but the power of this server opens a whole new world to KakaoStats. I already had the plan to re-write KakaoStats codes from front and backend, and rebuild its DataBase. But now it can be done in a new level, and that’s the plan. Stay tuned for news!!
Aaand the bad news…
Later in the same day someone broke my car to steal the spare tire and also stole both my laptops! Some documents were also lost, some were found, but still no clue of laptops and even my little daughter clothes that I bought couple of hours before that. My HTPC was also there and they didn’t steal, my RayBan glasses, phone charger, 5 liters of 2 stroke oil to my go-kart wasn’t steal also. I’m really sad because, not only I lost my brand new laptop (other one belong to the company I work for), but the contents inside. Most stuff was saved in OneDrive (
piece of shit, don’t use it) but 2 external HDDs with 1TiB was stole and my MP3 folder from 20 years of downloads (some was still downloaded with Napster) was in secondary HDD and I’m not sure if I have backup.
But anyways… Life still goes on!
Just to keep you posted, since 04.09.2016 10:20 there’s no valid team/donor files from Stanford. That’s why Kakao has no new data since then.
So there I was, drinking a beer with a friend of mine and we started talking about KakaoStats. He liked the system itself and bla bla bla, so I told him it’s running on an old laptop and I’m saving some money to buy a proper PC to run it.
After a couple more beers he gave me the idea to ask for donations. I’ve never actually considered that. I decide to run KakaoStats because I like to work with servers and I’ve been missing that lately. (I used to be network/system administrator until I started working with automation) So, it’s kind of a hobby for me…
But unlikely my go-kart hobby, I’m not the one who enjoys it more. Of course I do enjoy it quite a lot, even in the background or checking my points.
Anyway… I took some time, re-activated my Paypal account (and another Brazilian-only payment gateway) and created the donate buttons! Feel free to donate the amount you feel you should donate, small or big values will be only used to improve KakaoStats. All donations are done in BRL, so check the currency!
My first plan is to buy a new hardware. I already have a rack case, PSU and two 2TiB 2.5″ HDD. Plus I found this HDD case yesterday and I’m really mad because I just ordered a new one from China. When arrive I’ll try to sell it and save the money for other hardware.
While I don’t have the money, I’m checking some good cost-benefit CPUs and I found out a bottleneck in the current hardware. The 2TiB HDD I’m using runs only at 5.400 rpm, I have room but I don’t have performance. When importing data and it needs the HDD the CPU usage goes low and it takes time, but the worst part is that navigating in KakaoStats become slower. Indexed part still normal, but sometimes it’s needed to go to disk to fetch some data, and it also is bad for importing task. So I added to hardware list also two SSD drives (will be mirrored) to run the OS and the database and then HDD will be used only for backups.
Please don’t feel obligated to donate money. Kakao will still exists and I’ll also buy some things by myself for it.
There was a change/fix in “Last Update” column.
More information here.
Since I was with “dirty hands” installing this WordPress to keep you posted about Kakao, I also installed a forum so we could interact.