Corrected title (sorry, it got truncated): Xbpartitioner 1.1 *does Not* Format With 128k Clusters With Drives > 1 TB

XBPartitioner version 1.1 is not coded to correctly handle clusters. I just tried this with a 1.5TB drive installed (using a SATA-IDE adapter) and all it would do is 64k clusters, which means something is wrong with the code and we need a new release. I think everyone assumed it would work but no one could test it since they didn't have a drive larger than 1 TB to test it with. Here is the problem code:



With this code, the program will format partitions incorrectly like so:

0GB - 512GB ====> 16k clusters
512GB - 1TB ====> 32k clusters
1TB - 2TB ====> 64k clusters
2TB - 4TB ====> 128k clusters
...and so on

Instead we would like partitions formatted like so:

0GB - 256GB ====> 16k clusters
256GB - 512GB ====> 32k clusters
512GB - 1TB ====> 64k clusters
1TB - 2TB ====> 128k clusters
...and so on

The fix is quite simple. Here is the code once again, but this time with the fix. I have bolded the fix for clarity:



Notice that "compare" is a hexadecimal number that represents the kilobyte upper limit on each cluster size. If we examine the problem code, we see that "compare" is initially set to 0x20000000. Convert this to decimal and we get 536870912 kilobytes, or 512 gigabytes. So the code is saying that as long as partition is less than or equal to 512 gigabytes, set the cluster size to "g_iClusterSize", which is initially set to 16. This is wrong.

Now, if we change the initialization of compare to 0x10000000 (268435456 kilobytes, or 256 gigabytes), everything works out as it should. Does this make sense to anyone, or am I missing something?

I would love to compile this new code myself to test but I don't have the XDK, and without compiling and debugging it, I can't be 100% certain...

Can someone please recompile this so we can get some bigger drives working in the XBox? Thanks!!

I could never work out exactly what "compare" stood for, but my best bet was that it dealt with the number of entries in the partition's cluster index table. Cluster based corruption is caused by having too many, but the larger the things are the less you need to fill the drive.

Nevertheless I'm pretty sure the code is already correct in terms of the initial starting value I have there. But according to your statements, you find that the cluster size does not change from 16kb to 32kb when you cross the 256gb mark - I find that a little problematic to rationalise, considering that it worked fine in my tests, and people with larger drives then I own have confirmed it correctly jumps to 64kb clusters when you break the 512gb mark.

Can you confirm that XBP 1.1 uses the incorrect cluster size for all partition combinations with your drive? If so, provide some details on your BIOS, drive make, and console version.

If it only stuffs up when crossing the 1TB mark then I reckon there's gotta be an overflow in there somewhere, but I'm not sure where... Heck, I don't even know how many bytes a ULONG takes up these days. Is it still four? Hope so, should be an easy fix if that's the case.

Mind you, I still recommend a split partition setup. To my mind 128kb per cluster is wasteful to the extreme. A 32kb and 64kb cluster split would be vastly more efficient.

There's also the possibility even if XBP does decide that 128kb is the cluster size it's gonna format with, that doesn't mean 128kb clusters will work. You won't know for sure until you get 1tb of data on there.

Furthermore, another guy with a 1.5tb drive had trouble after the format was done (even though he went with a split setup). Though maybe I should get him to take a look at this, perhaps it'll offer some insight on his case as well.

Yep, looks like an overflow to me. A ULONG is indeed 4 bytes, and for partitions larger than 1 TB we'd be using 4294967296 (0x100000000). The range of a ULONG is 4294967295. Doh!

Bottom line - I think it should be using a ULONGLONG...

So a ULONG is four bytes... thanks for the heads up. I assume a ULONGLONG is something like eight?

But, the thing that gets me is this:

If "compare" is overflowing, then the "while" condition should always be true and the program would enter an infinite loop (that is, the X-Box'll crash if you try to define a 1tb+ partition).

If the LBASize function wants to return a ULONG, then there should be no infinite loop but if you try to define a partition 1tb+ the cluster size should drop down to 16kb.

Neither of these things are happening... But yeah, there's certainly room for an overflow error there.

Ok after I did some more intense testing, it looks I was wrong about wrong cluster sizes. It is indeed an overflow. Using xbpartitioner 1.1 I brought the F partition down to 0 GB and start working my way up, the whole time watching in the corner to see what cluster size it would use. In the beginning it said it was going to use 16 KB clusters, and as expected, when I got 256 GB, it switched to 32 KB clusters. When I got to 512 GB, it switched to 64 KB. Finally, when I got 1024 GB, the xbox froze. There is definitely an overflow. What now?

From what wildonrio just mentioned, the xbox does freeze, which confirms the infinite loop, I believe.


Is the LBASize a ULONG? What is its defined type? If it is a ULONG, then we have another problem, because the LBASize itself could be overflowing...

Unlikely. "compare" can only increment so long as it's less then LBASize. If LBASize is hitting an overflow, it becomes impossible for "compare" to do the same; hence no infinite loop, and no crash.

Which is all another way of saying "I can't find where LBASize is defined, I think that function is part of the XDK itself and it'd take me too long to hunt it down".

Ah, yes, that makes sense.

I'm interested in knowing if the overflow fix allows 128k cluster sizes. Let me know!

Ok I used ULONGLONG instead. Good news is, it actually formatted with 128k clusters! Bad news is, the partition is labeled on the dashboard (XBMC) as "Unavailable". As far as FTP, the F partition shows up but nothing will transfer to it without an error. Getting closer! What next?

Wow, could be one of several things. Maybe the partition code is not designed to handle this cluster size. Where is the auto-calculated cluster size being used? I tend to think this is the case, if FTP cannot perform a simple file transfer.

If someone can post the code snippet of where the auto-calculated cluster size is used, I will take a look. Hopefully it's not an XDK function, otherwise we'd have to modify the XDK source!

I don't think I can do much more for it then that... Back when the initial testing for 1.1 was being done, we had the same issue (as already mentioned).

The formatting command goes along these lines:



Where "status" is an int (the XBP packs on X-Bins include full source, by the way). Don't think the bitshift is causing an issue here 'cause if it was 64kb clusters wouldn't be working either.

I'm pretty sure that function is bundled somewhere in the XDK library files, but I've no idea as to how to extract them. Plus there's a good possibility that the issue isn't with the code, but rather another limitation of FATX (such as the one that warrants the use of XBP in the first place).

Seems far easier to me to prevent users from defining partitions 1tb+ and call 2tb the final X-Box drive limit.

LBASize is defined in partition.h

Now that's interesting. So I wonder where LBASize is being used, and I wonder why LBASize didn't mess up the cluster detection loop, since I would think there would be an overflow problem.

Changing LBASize to a ULONGLONG may have a positive effect...

For what it is worth I am positive I used 64k on a 1.5tb split between f&g and data corruption occurred after passing 512 gigs.When I first tried the app it froze everytime until I formatted once with an older build on a new install then the new app seemed to work fine.

I am 100% positive I built it correctly so I am positive there is a problem with the 64k partitions.I was going to try 1 big partition with 128k and see if the results were different but you all are saying that you all are having problems there to.

It confuses me too, but if XBP says it's gonna use 128kb clusters and then proceeds throughout the formatting process, then the size detection code has done it's job correctly - nothing more to mess with there.


While it's certainly a problem, I'm not thinking this is related anymore...

My current theory is that XBP 1.1 never formatted your drive. You first formatted with the older build (which used 32kb clusters), then with 1.1 (which froze the first time... and simply didn't work the second time).

Beats me as to why. I suppose the first test would be to try a few different layouts with 1.1, see if the drive layout changes correctly with each consecutive format.

I know 64kb partitions work. Heck, wildonrio here can vouch for that.

I agree, unless LBASize is used elsewhere in the formatting process. Is it only used for the cluster size detection?

Dunno, don't care. Take another look at that formatting line I quoted. As it isn't mentioned there (and nor is it's parent object(s)) it simply isn't relevant.

Edit:

Actually looking in a bit further detail I see that anything's possible. It could well even be the problem (though I'm still somewhat doubting it). In fact changing that to a ULONGLONG could well break the program, but...

... Well, one way to find out for sure. I'll compile a new build tomorrow later in the day and send it to wildonrio for the test. Too early in the morning for me to do it now.

Thanks for all your hard work

I can confirm that the xbox crashes when it sees the 128k partition.I went ahead and split mine again with f of 504gigs at 32k and g at 900gigs with 64k.It shows as more in xbmc and that worries me a little.xbmc f=516gig g=906gig The only way to find out is to fill it completely or leave 4gigs worth or slack in the partition and forget about it.

I did think however this setup would be the best way to save space making 1 partition to the 512g 32k limit.It will most likely be some time before the 2tb hdds hit the market and alot longer than that before anything bigger hits the market so we really got some time before the 128k will ever be needed.128k will be what is needed for drives from 2tb to 4tb.

Ok, I changed a heck of a lot of values to ULONGLONGs. We'll see what that does.


Hmm, that ain't so good. If I were you I'd format once more making F a little lower - You really want it to display at less then 512gb to be on the safe side.

Scratching my head as to why it'd do that. Actually, might be well worth checking what a few other dashboards report before formatting again.


Quite correct.

It will be a couple weeks before I will be able to test this. I'll keep everyone posted.

Hi guys,

Just bought a 1.5 TB drive and the XBP 1.1 insists on formatting the the large partition with only 32kb clusters. anyone that got a patched XBpartitioner that will format using 128 kb clusters?

/Mill

I wonder if the FATX structured in the same way as the FAT32...

Reading the information at the following link:

Windows XP Professional File Systems Overview

...gives me the information that the FAT32 (FATX?) uses the following cluster size convention:


FAT32 file system cluster sizes
===============================
Partition Size ----------------- Cluster Size

0M to less than 260MB ----- 0.5k (512 bytes)
260MB through 8GB -------- 4k (4,096 bytes
8GB through 16GB ---------- 8k (8,192 bytes)
16GB through 32GB -------- 16k (16,384 bytes)
32GB through 2TB ---------- 32k (32,768 bytes)
===============================

If there isn't a cluster redesign in the FATX, a 32k cluster size would therefore be able to handle upp to 2 TB disks. But why are partitions over 512 GB, using 32k cluster size, getting corrupted then?

FATX uses three bytes to address each cluster. Three bytes can store two to power of twenty four (2^24) different values, so that's 16,777,215 clusters tops.

Hence the maximum partition size is tied to your cluster size. The default of 16kb (2^14 bytes) multiplies out to 274,877,906,944 bytes (2^38).

Divide that by a gigabyte (2^30) and the final figure is 256 (2^8) gb.

Now, there's nothing stopping you from attempting to create a partition larger then 256gb using 16kb clusters. Catch is, as soon as you actually attempt to put more then that amount of data on there, you run out of addressing space, the file system overflows, and whoops! The file index table just got overwritten...

I dunno if this bug also exists in FAT32, but I've never heard it mentioned. So why do they bother to change the cluster size for larger drives in that case?

Well, the file index table keeps track of clusters, but the smaller your clusters are, the more of the things you have - and hence the bigger your index has to be. Attempting to index 2tb worth of data using half kilobyte sized clusters would require 4gb, while a 32kb cluster size reduces that figure to just 64mb.

I think NTFS likes to use 4kb clusters regardless, which is fair enough. There comes a point where drive efficiency will be taking a bigger hit due to huge clusters then it'll be gaining from having a small index.

While I hate resurrecting old threads, I haven't seen any further progress. Have you finished discussing this issue elsewhere? Did wildonrio ever test the new build?

I just threw in a 2TB drive in my V1.0. I would really like to assist if this was never completed.


Jason

I'm afraid not much has changed.

We'll probably never know whether wildonrio did that last test.

Any attempts to do a format with 128kb clusters result in a corrupt partition.

128kb clusters are still wasteful unless you want to install a drive larger then 2tb.

Alright. Thanks for the reply. I guess I'll just divide my extra space evenly between my F and G partitions.

What is the largest HD capacity the XBox can use, and I take it it's still IDE, and not SATA? And if I get a large HD (mine is currently 300GB or so), can I just use Slayers boot disc to format the drive OK?

Thanks for any answers.