PANIC !!! D:! (i smell a rollback)

phyxius · Post by **phyxius** » April 26th, 2006, 6:54 am

http://q-ro.net/forum/index.php?showtopic=19240&st=0

seems like ppl is losing items if they trading, change, log on....omg, this is annoying..i'm on my work and i can't check about it...!!!!!!!

i smell a rollback soon..cuz ppl is complaining about a lot of stuff...i haven't checked..but...if i log on and lose my stuff, some1 will pay on this server..-.-"

is official now....

http://q-ro.net/forum/index.php?showtopic=19254

so...Don't Log in!!!!

Post by **Yoon** » April 26th, 2006, 9:22 am

That really sucks. >:( I wonder what could be causing the problem in the qRO item database? In any case, a rollback will probably happen. I just hope they made a recent backup...

phyxius · Post by **phyxius** » April 26th, 2006, 10:51 am

17:31] <~frostycpu> k
[17:31] <~frostycpu> this is what happened
[17:31] <~frostycpu> apparently, an error came up with the emu
[17:31] <~frostycpu> SQL server freaked out
[17:31] <~frostycpu> started deleting "errors" (your items)
[17:31] <~frostycpu> I have some backups
[17:31] <~frostycpu> but
[17:31] <~frostycpu> the only usable one might be only from april 5th
[17:32] <~frostycpu> meaning you'd lose ~20 days of play
[17:32] <~frostycpu> or what I can do is
[17:32] <~frostycpu> try to repair the current one
[17:32] <~frostycpu> but you'd lose what you lose since this morning

omg...
20 days...
GOOD BYE SIN!!! ><"

Post by **Yoon** » April 26th, 2006, 11:05 am

20 days!?

That is a while back...

<GM>Mirr has put up a poll here to see how many people are for the rollback or repairing current database (people that lost their items lose it permanently). I personally vote for the later option, seeing that I have not lost any of my crucial items. Either way, there are going to be a lot of angry people.

For those that have not signed on to qRO recently, DO NOT SIGN ON TO qRO LR until this problem is resolved, especially if the admins go with the option to repair the database.

NEW UPDATE:

Well, there might be a better solution to the current problem, according to <GM>HalfDemon:

We're currently discussing a option inside the staff if it all suceeds all lvls and your current data will stay and most of your equips will be returned.

We will inform you all once it's decided and confirmed in this section.

Any new threads connected to questions and such about this issue will be closed.

Source: Missing Equips

FINAL UPDATE:

The qRO LR admin, frostycpu, just made an announcement on the qRO forums:

Alright, so some of you have noticed this morning a problem with the server. Some pieces of equips were disappearing, big problem.

What caused this to happen?

To understand how this happened, you need to know one thing about the SQL database. All the information is contained in different parts callled "tables". Each table has a structure that tells the mySQL server how to insert each piece of information it recieves.

The table in question today was the "inventory" table, the table that manages all the equips that wear and carry around. In it's structure, the table insert item creation IDs with a limit of 10 numbers.

Ok, but what does that have to do with qRO?

qRO has been online for quite a while, almost 3 years. During this time, we've never had to do a complete wipe (still proud of that!). In that sense, it means that since the first item that was created on day 1 and today, a LOT of items have been created.

How many have been created you ask? 4294972426 items.

But wait, notice something about this number? Yes, it already has 10 numbers total. Normally that should be enough since the limit number for the current structure is 10. But what you don't see, is that the server will sometimes create numbers that are higher than the highest number.

That means that at some point today, the server created a number that had 11 numbers or more.

So what happened then?

What happens when too high numbers are used is somewhat simple. It tried new, lower numbers. But the thing is, it won't care if that number is already used or not. If it is used, it's detected as a "duplicate entry" and deleted immediately. That's basically what happened this morning. New IDs were being created/changed, duplicating some which made the mySQL server delete them.

Is this fixed now, will it happen again?

Yes it is fixed. What was done is that we increased the possible amount of numbers allowed from 10 to 20. That means there is a now a lot more space for IDs to be created without creating problems.

Awesome, but what about my items? You have backups, don't you?

qRO does weekly backups of it's database since the last incident with the high rate server. But, misfortune has struck down again.

The program we were using to create these backups apparently had an unknown and *undocumented* defect in it. If the database file created by this program is larger than 1 Gigabyte, it becomes corrupted. This is obviously something we could never have forseen. You'd think the program would output some sort of error if the backups are not usable, but it didn't.

Aside from those corrupted weekly backups, I had created a manual backup a while ago while testing some modifications. But that backup is from April 5th, which is way too old.

Some staff members proposed a partial rollback; to rollback only the items to what they were on the April 5th backup. Unfortunately, that could create other problems.

Inside a mysql database, each table contains indexes. These indexes are like bookmarks, they help the mysql server to look for information more quickly, thus reducing the char-server lag. But, a lot of tables in the database SHARE indexes that are synchronized togheter with the other tables. Only copying the item databases would desynchronize those indexes, since some would share the same indexes, but with different values.

So if I understand correctly, it's either we live with this or do a total restore to April 5th?

That's right. But since a total restore would mean losing 20 DAYS of work, for 133241 characters, the best solution in the interest of the server as a whole would be to keep the current database as it is. We were lucky that it happened at a time where very few people are on. Imagine if this had happened on WoE hours, we'd probably be looking at a total wipe.

*sob* I lost x millions in items sad.gif

You have no idea how much it pains me to be living this. This is basically a server admin's worst nightmare. I do think we were lucky in our misfortune that it happened at a time where the userbase is low, but it still ticks me off that some people are gonna lose their items and that I can't do anything about it.

I can't say it enough. I'm sorry this happened, even if I couldn't have stopped it if I wanted to. I will be reporting this table structure issue to the devs to make sure other servers don't have to deal with it. We may not have been lucky as to someone else finding it out before us to fix t, but at least we can help some other servers that are getting big so they don't have their database corrupted.

Thank you for understanding.

Source: What Happened With The Database Today., The why, the how, and the what now.

In short, the problem has been fixed, so it is safe to log on now. Also, there WILL BE NO ROLLBACK, so for those who have lost items because of this server glitch, they are pretty much screwed.

KillerSmo · Post by **KillerSmo** » April 26th, 2006, 2:45 pm

D@mn... I'm glad I'm at work and can't sign in - wow.....

Ghost B3N · Post by **Ghost B3N** » April 26th, 2006, 3:20 pm

they fixed it, at least.. kind of
http://q-ro.net/forum/index.php?showtopic=19292
there you can read what the problem was.

they now fixed the problem you can play again, and wont lose anny items. but the last update was 20 days ago, so there wont be a role back. for most of us that is a good thing. but for the ppl who did lose items.. they have a bad day, the items are gone. nothing can be done about that. (but i think 20 day roleback is far worse!!!)

so we can play again!!

phyxius · Post by **phyxius** » April 26th, 2006, 4:08 pm

BS.....rollback made...i lost my Sin and more than 300 m...-.-" (suix stuff+Angeling card+Excalibur+money+angels wings)

Post by **Yoon** » April 26th, 2006, 6:25 pm

This is retarded... they did roll back. :/

And frostycpu said that there wouldn't be a rollback. The main forums are down too. What the @#$# is going on?

UPDATE:

frostycpu just made an announcement on the server. He stated that experience and drop rates are now doubled for a week. He also stated that there was more damage to the database than initially thought, so he had no choice but to rollback.

Looks like we will have to re-recruit members again, re-establish our alliances with AD, KoV, Thug, and level up/hunt.

phyxius · Post by **phyxius** » April 26th, 2006, 10:35 pm

they think that ppl will be happy with just that? HA....they can kiss my @$$, i'm lvl 99..i don't care the lvls...i want my stuff back!!! is impossible..but double exp/rate suck

Slayer · Post by **Slayer** » April 27th, 2006, 10:25 am

My bro has quit for good since he lost his LVL 99 LK AGAIN

ghalleon · Post by **ghalleon** » April 27th, 2006, 12:22 pm

I consider us very lucky that they happen to have a backup saved from 20 days ago. Ever since HR had that complete wipe a few month ago they have been saving everything once a week just in case something bad happens and they need to roll back one week. And now QRO has gotten so big that they hit the 10 item limit and items are randomly dissapearing. So now they acctually have need of the weakly save for a one weak roll back, and they find out that they cant use it because it is bugged. Which means if they didnt just happen to have this save from 20 days ago we either would have had to have a complete wipe on LR, or just keep playing with the item bug and let items constently dissapear randomly, which they would not keep QRO going with a bug so horrible. So i am glad they happen to have a save from 20 days ago because if they didnt then QRO LR would have to start all over, complete wipe. Yes a lot of people are mad at lost item/exp/zeny but i think it is better to back up 20 days instead of backing up 3 years and starting QRO over again. ( My opinion ). But yes i do feel bad that people lost so much but i say it is much better than the alternative.

Ghost B3N · Post by **Ghost B3N** » April 28th, 2006, 11:27 am

i re-established our alliances with Astral Disorder.

so thats 1 thing less to worry about