FFXIAH.com - HOW?

[hongman]

I have always wondered this.

How the hell does FFXIAH.com get all it's information??

[Mhurron]

Re: FFXIAH.com - HOW?

Magic.

FFXIAH has forums where their devs post, maybe create an account and ask there.

[Aeolus]

Re: FFXIAH.com - HOW?

The guy who runs it has a character on every server that sits by jeuno AH all day writing down all transactions. Thats why it takes about 2hours to update.

[Mhurron]

Re: FFXIAH.com - HOW?

[Lambeaus]

Re: FFXIAH.com - HOW?

Does it matter?

FFXIAH > all

[ItazuraNhomango]

Re: FFXIAH.com - HOW?

Since they claim they do not add to S-E servers' load, they must be passively capturing the AH info on the client side, and transferring the captured info from clients to FFXIAH's server.

They've also asked for volunteers in the past, so that means they probably get information primarily from ordinary FFXI players who has FFXIAH's special software installed.

I can think of two ways their software can get the AH information; memory read (via hooking, like FFXI Windower), or network packet intercept ("sniffer"). Seems like a sniffer would be the cleaner implementation, so I'd guess that is what FFXIAH's client software does.

[Feba]

Re: FFXIAH.com - HOW?

Basically, they have a character on every server, and this character is basically a bot that goes through the auction history and checks every item. I believe they do this through memory reading/text log reading, but I'm not totally sure. When I asked them about it awhile ago, they said it took about 24 hours to run through a full cycle, which is why items that are frequently sold often have names missing from the site.

If people were actually writing things down, FFXIAH would've shut down long ago for lack of interest.

[Aeolus]

Re: FFXIAH.com - HOW?

Does it matter?

Obviously or he wouldnt have asked.

[Lunaryn]

Re: FFXIAH.com - HOW?

I don't believe it's a sniffer, I've heard of at least one project to reverse-engineer FFXI and produce a compatible server program, and it basically got stopped up on trying to break the encryption on the protocol. Haven't heard anything more on it in years.

Hooking memory a la Windower sounds much more likely.

[ItazuraNhomango]

Re: FFXIAH.com - HOW?

Basically, they have a character on every server, and this character is basically a bot that goes through the auction history and checks every item. I believe they do this through memory reading/text log reading, but I'm not totally sure. When I asked them about it awhile ago, they said it took about 24 hours to run through a full cycle, which is why items that are frequently sold often have names missing from the site.

If you have the actual text of the response, I may be able to make some more sense of it.

The "full cycle" probably referred to dumping raw data into the database and do some queries to pre-compute often used/displayed data. (There was an entry about 5 GB data backlog a few months back, as well this this explanation from Cliff.) They've made some substantial hardware and software upgrades to improve data stuffing, I think, but medians are computed once a day (per Scragg). Also, some of Cliff's post seem to indicate a "full [AH] scan" every hour or so, but that may not be part of the "full cycle" being referred to.

As to whether they read memory or packets, I don't know. I just think it'd be cleaner as a packet sniffer, but obviously the information does get into the POL/FFXI's memory space.

Whether they actually have a mule on every server dedicated to it or not presently, they did used to have very uneven coverage, varying from server to server. And, I do remember they asked for volunteers to help out with data collection, but didn't say the helper couldn't use his/her character for anything else, which implied to me their software doesn't prevent volunteers from using their characters normally.

(I tried to find as many references as I can, but it's been a while since I came across some of the info. )

* * *

• Example of someone trying to volunteer to help out with data collection, and Scragg thanking an Ifrit volunteer. That implies strongly at least at some point in the past, FFXIAH didn't have independent way of gathering data on every sever such as the FFXIAH owned bots Feba suggested.
• On the other hand, Cliff did ask for a Ragnarok worldpass, which may indicate that eventually the devs did put up a bot on every world, or maybe just every world where they didn't have good coverage.
• I was wrong about passively scanning; Cliff indicates he does a full scan of AH. This means he does add to the load of S-E's servers, but maybe not by very much, depending on exactly what he does.

* * *

Does that sufficiently answer the OP's question?

[hongman]

Re: FFXIAH.com - HOW?

Um... yeah... thank you...

Ill read it tomorrow LOL

[Feba]

Re: FFXIAH.com - HOW?

If you have the actual text of the response, I may be able to make some more sense of it.

Like I said, this was a long time ago.

I don't think Cliff was involved in this when I talked to them, it was still only on one or two servers, and much less updated than it is now. The character was tied to the AH though, afaik, they never moved from the Port Jeuno branch. I've heard that SE has a way of detecting packet sniffers (a friend warned me not to use Wireshark with FFXI because of this, I don't know how correct he is though), so memory reading would be a lot easier, considering they already are good at that, and the information they collect on AH items seems to all be in chat logs (click a transaction in the item's history)

Of course, I do remember someone on the windower team (probably Cliff) mentioning being able to read the FFXI AH info directly from the server, without even logging in, and because of the way the login/AH server is, I wouldn't be surprised if he was correct and got it working.

I don't really see why the specifics are of much use to anyone though. They collect info and put it on their site, simple enough?

[ItazuraNhomango]

Re: FFXIAH.com - HOW?

I've heard that SE has a way of detecting packet sniffers (a friend warned me not to use Wireshark with FFXI because of this, I don't know how correct he is though)

Sounds like a wild rumor. Any source for this?

so memory reading would be a lot easier, considering they already are good at that

Memory read isn't necessarily "easy" in the sense of knowing when the information changed and need to be looked at again. This is a synchronization problem which goes away automatically when talking about network packets --the OS network stack toss out an interrupt to notify the sniffer to grab data.

Provided that you can decode the packets, of course, the packet sniffer implementation seems much cleaner to me.

and the information they collect on AH items seems to all be in chat logs (click a transaction in the item's history)

Hmm. I don't think "previous transaction" list or number of items in stock gets into the chat log at all. I can test that when I next log on to FFXI.

Of course, I do remember someone on the windower team (probably Cliff) mentioning being able to read the FFXI AH info directly from the server, without even logging in, and because of the way the login/AH server is, I wouldn't be surprised if he was correct and got it working.

I remember something along this line as well.

I don't really see why the specifics are of much use to anyone though. They collect info and put it on their site, simple enough?

Useful? No. But, it's an interesting topic for me.

[Feba]

Re: FFXIAH.com - HOW?

Sounds like a wild rumor. Any source for this?

Nope, but that friend is an admin on a premier site, and is almost always right about tech questions I have for him, so I wouldn't be surprised if he were correct.

I didn't mean memory, I meant chatlogs, sorry. They even mention parsing in one of the above links, so this is most likely in my opinion, although this is a rather pointless debate, when we could simply ask one of them

Hmm. I don't think "previous transaction" list or number of items in stock gets into the chat log at all. I can test that when I next log on to FFXI.

I don't know if you can get number of items in, but I do know you can press enter on any item in the price history for a text output of whatever it is. This seems the most likely thing to me. They scroll through the lists with a bot, and parse the text results. Items in stock would be something they'd have to get from memory though.

[LyonheartLakshmi]

Re: FFXIAH.com - HOW?

The only way I could SE detecting packet sniffers is if POL was made to query your PC and looked for some well known packet sniffing utilities.

If you were using another machine on your LAN to capture and analyze FFXI traffic, I don't see how SE could possibly detect that case.