Breaking The 32k/64k Cluster Size Limit In Linux Fatx Options

[ldotsfan]

I'm trying to develop a patch against fatx.c which is used in xboxdumper. xboxdumper contains mkfs.fatx which is used in all xbox based linux distributions (including xboxhdm) to create the FATX partitions. I'm trying to break the 32k/64k clustersize limit in this tool. Please refer to http://xbox-linux.cvs.sourceforge.net/xbox...amp;view=markup

For Line 347

CODE

partition->clusterSize = 0x4000;

I have patched it to:

CODE

    if ((partition->partitionSize >= 0x10000000) && !nCreate){
       if (partition->partitionSize >= 0x20000000){ //more than 512GB
        *(u_int64_t *)&partitionInfo[0x0008] = 0x00000080; // Clustersize in 512 bytes
            partition->clusterSize = 0x10000; // 64k clustersize
       }
       else { // between 256GB and 512GB
        *(u_int64_t *)&partitionInfo[0x0008] = 0x00000040; // Clustersize in 512 bytes
            partition->clusterSize = 0x8000; // 32k clustersize
       }
    }
    else
      partition->clusterSize = 0x4000; // 16k clustersize

Can somebody verify my work? Also I've searched the xbox 2.4 kernel source and can't find any other references to the cluster size that needs to be changed. Can anybody else confirm this?

[ldotsfan]

Revisiting this topic.

More code:

CODE

FATXPartition* createPartition(char *szFileName, u_int64_t partitionOffset,
        u_int64_t partitionSize, int nCreate) {

    FATXPartition* partition;
    unsigned char partitionInfo[FATX_PARTITION_HEADERSIZE];
    unsigned char *clusterData;
    char szBuffer[1024];
    long lRet;
    u_int64_t i;
    u_int32_t eocMarker;
    u_int32_t rootFatMarker;

    memset(szBuffer,0,1024);
    
    partition = (FATXPartition*) malloc(sizeof(FATXPartition));
    
    if (partition == NULL) {
        printf("createPartition -> Out of memory\n");
        return NULL;
    }

    memset(partitionInfo,0,FATX_PARTITION_HEADERSIZE);
    *(u_int64_t *)partitionInfo = FATX_PARTITION_MAGIC;
    *(u_int64_t *)&partitionInfo[0x0004] = (u_int64_t) time(NULL); // Volume id
    if(!nCreate) {
        *(u_int64_t *)&partitionInfo[0x0008] = 0x00000020; // Clustersize in 512 bytes
    } else {
        *(u_int64_t *)&partitionInfo[0x0008] = 0x00000004; // Clustersize in 512 bytes
    }
    *(u_int16_t *)&partitionInfo[0x000C] = 0x0001; //Number of active FATs (always 1) (?)
    *(u_int32_t *)&partitionInfo[0x000E] = 0x00000000; //Unknown (always set to 0)
    memset(&partitionInfo[0x0012],0xff,0xfee); //Unknown (usually set to 0xff, or 0). Probably padding.

    memset(partition,0,sizeof(FATXPartition));

    if(!nCreate) {
        partition->sourceFd = fopen(szFileName,"r+");
    } else {
        partition->sourceFd = fopen(szFileName,"w+");
        partition->partitionSize = partitionSize * 1024 *1024;
    }

    if(partition->sourceFd == NULL) {
        printf("createPartition -> Error creating File %s\n",szFileName);
        return NULL;
    }
    
    if(!nCreate && (partitionSize == 0)) {
        fseeko(partition->sourceFd,0L,SEEK_END);
        partition->partitionSize = ftello(partition->sourceFd);
        fseeko(partition->sourceFd,0L,SEEK_SET);
    } else if(!nCreate && (partitionSize != 0)) {
        partition->partitionSize = partitionSize;
    }

    partition->partitionStart = partitionOffset;
    if ((partition->partitionSize >= 0x10000000) && !nCreate){
               if (partition->partitionSize >= 0x20000000){ //more than 512GB
                *(u_int64_t *)&partitionInfo[0x0008] = 0x00000080; // Clustersize in 512 bytes
                    partition->clusterSize = 0x10000; // 64k clustersize
               }
               else
                   { // between 256GB and 512GB
                *(u_int64_t *)&partitionInfo[0x0008] = 0x00000040; // Clustersize in 512 bytes
                    partition->clusterSize = 0x8000; // 32k clustersize
                   }
        }
        else
              partition->clusterSize = 0x4000; // 16k clustersize
    
    partition->clusterCount = partition->partitionSize / partition->clusterSize;
    if (partition->clusterCount >= 0xfffffff4)
        partition->chainMapEntrySize = 8;
    else if (partition->clusterCount >= 0xfff4)
        partition->chainMapEntrySize = 4;
    else
        partition->chainMapEntrySize = 2;
    
    if(nCreate) {
        memset(szBuffer,0x00,1024);
        for(i = 0; i < (partitionSize * 1024);i++) {
            lRet = fwrite(szBuffer, 1024, 1, partition->sourceFd);
        }
    }
    
    fseeko(partition->sourceFd,partition->partitionStart,SEEK_SET);
    

    // Write the header
    fwrite(partitionInfo, FATX_PARTITION_HEADERSIZE, 1, partition->sourceFd);
    
    partition->chainTableSize = partition->clusterCount * partition->chainMapEntrySize;
    if (partition->chainTableSize % FATX_CHAINTABLE_BLOCKSIZE) {
    // round up to nearest FATX_CHAINTABLE_BLOCKSIZE bytes
        partition->chainTableSize = ((partition->chainTableSize / FATX_CHAINTABLE_BLOCKSIZE) + 1)
                        * FATX_CHAINTABLE_BLOCKSIZE;
    }
    
    // Create empty chain map table
    partition->clusterChainMap.words = (u_int16_t*) malloc(partition->chainTableSize);
    if (partition->clusterChainMap.words == NULL) {
        error("Out of memory");
    }
    
    
    memset(partition->clusterChainMap.words,0x00,partition->chainTableSize);
        
    if (partition->chainMapEntrySize == 2) {
        rootFatMarker = 0xfff8;
        eocMarker = 0xffff;
        partition->clusterChainMap.words[0] = rootFatMarker;
        partition->clusterChainMap.words[1] = eocMarker;
    } else if (partition->chainMapEntrySize == 4) {
        rootFatMarker = 0xfffffff8;
        eocMarker = 0xffffffff;
        partition->clusterChainMap.dwords[0] = rootFatMarker;
        partition->clusterChainMap.dwords[1] = eocMarker;
        }
    else {
            rootFatMarker = 0xfffffffffffffff8;
            eocMarker = 0xffffffffffffffffff;
            partition->clusterChainMap.dwords[0] = rootFatMarker;
            partition->clusterChainMap.dwords[1] = eocMarker;
    }
    
    
    // writing the new chaintable
    writeChainMap(partition);
    
    // Address of the first cluster
    partition->cluster1Address = partitionOffset + FATX_PARTITION_HEADERSIZE + partition->chainTableSize;
    fseeko(partition->sourceFd, partition->cluster1Address, SEEK_SET);
    
    // clearing the first cluster
    clusterData = (unsigned char *)malloc(partition->clusterSize);
    
    memset(clusterData,0xFF,partition->clusterSize);
    
    writeCluster(partition, 1, clusterData);
    // first direcory or file
    
    printf("createPartition : Filename : %s\n",szFileName);
    printf("createPartition : clusters       %ld\n",(unsigned long)partition->clusterCount);
    printf("createPartition : start          %lld\n",partition->partitionStart);
    printf("createPartition : size           %lld\n",partition->partitionSize);
    printf("createPartition : chainMapSize   %ld\n",(unsigned long)partition->chainMapEntrySize);
        printf("createPartition : chainTableSize %lld\n",partition->chainTableSize);
            
    return partition;
}

The next step is to compare xbpartitioner output vs this chunk of xboxdumper code.


This post has been edited by ldotsfan: Apr 6 2010, 02:18 PM

[ldotsfan]

There's a bug in the original xboxdumper code:

CODE

if(!nCreate) {
        *(u_int64_t *)&partitionInfo[0x0008] = 0x00000020; // Clustersize in 512 bytes
    } else {
        *(u_int64_t *)&partitionInfo[0x0008] = 0x00000004; // Clustersize in 512 bytes
    }
    *(u_int16_t *)&partitionInfo[0x000C] = 0x0001; //Number of active FATs (always 1) (?)

u_int64_t occupies too many bytes and partitionInfo[0x0008] will override the next value at partitionInfo[0x000C].

It's fixed at http://sourceforge.net/projects/xboxhdm2/f...ar.bz2/download.

CODE

FATXPartition* createPartition(char *szFileName, u_int64_t partitionOffset,
        u_int64_t partitionSize, int nCreate) {

    FATXPartition* partition;
    unsigned char partitionInfo[FATX_PARTITION_HEADERSIZE];
    unsigned char *clusterData;
    char szBuffer[1024];
    long lRet;
    u_int64_t i;
    u_int32_t eocMarker;
    u_int32_t rootFatMarker;

    memset(szBuffer,0,1024);
    
    partition = (FATXPartition*) malloc(sizeof(FATXPartition));
    
    if (partition == NULL) {
        printf("createPartition -> Out of memory\n");
        return NULL;
    }

    memset(partitionInfo,0,FATX_PARTITION_HEADERSIZE);
    *(u_int64_t *)partitionInfo = FATX_PARTITION_MAGIC;
    *(u_int64_t *)&partitionInfo[0x0004] = (u_int64_t) time(NULL); // Volume id
    if(!nCreate) {
        *(u_int64_t *)&partitionInfo[0x0008] = 0x00000020; // Clustersize in 512 bytes
    } else {
        *(u_int64_t *)&partitionInfo[0x0008] = 0x00000004; // Clustersize in 512 bytes
    }
    *(u_int16_t *)&partitionInfo[0x000C] = 0x0001; //Number of active FATs (always 1) (?)
    *(u_int32_t *)&partitionInfo[0x000E] = 0x00000000; //Unknown (always set to 0)
    memset(&partitionInfo[0x0012],0xff,0xfee); //Unknown (usually set to 0xff, or 0). Probably padding.

    memset(partition,0,sizeof(FATXPartition));

    if(!nCreate) {
        partition->sourceFd = fopen(szFileName,"r+");
    } else {
        partition->sourceFd = fopen(szFileName,"w+");
        partition->partitionSize = partitionSize * 1024 *1024;
    }

    if(partition->sourceFd == NULL) {
        printf("createPartition -> Error creating File %s\n",szFileName);
        return NULL;
    }
    
    if(!nCreate && (partitionSize == 0)) {
        fseeko(partition->sourceFd,0L,SEEK_END);
        partition->partitionSize = ftello(partition->sourceFd);
        fseeko(partition->sourceFd,0L,SEEK_SET);
    } else if(!nCreate && (partitionSize != 0)) {
        partition->partitionSize = partitionSize;
    }

    partition->partitionStart = partitionOffset;
    //partition->clusterSize = 0x4000;
    if ((partition->partitionSize >= 256ULL*1024ULL*1024ULL*1024ULL) && !nCreate){
               if (partition->partitionSize >= 512ULL*1024ULL*1024ULL*1024ULL){ //more than 512GB
                *(u_int64_t *)&partitionInfo[0x0008] = 0x00000080; // Clustersize in 512 bytes
                    partition->clusterSize = 0x10000; // 64k clustersize
               }
               else
                   { // between 256GB and 512GB
                *(u_int64_t *)&partitionInfo[0x0008] = 0x00000040; // Clustersize in 512 bytes
                    partition->clusterSize = 0x8000; // 32k clustersize
                   }
        }
        else
              partition->clusterSize = 0x4000; // 16k clustersize

    partition->clusterCount = partition->partitionSize / partition->clusterSize;
    partition->chainMapEntrySize = (partition->clusterCount >= 0xfff4) ? 4 : 2;
    /*if (partition->clusterCount >= 0xfffffff4)
            partition->chainMapEntrySize = 8;
        else if (partition->clusterCount >= 0xfff4)
            partition->chainMapEntrySize = 4;
        else
            partition->chainMapEntrySize = 2;
*/
    if(nCreate) {
        memset(szBuffer,0x00,1024);
        for(i = 0; i < (partitionSize * 1024);i++) {
            lRet = fwrite(szBuffer, 1024, 1, partition->sourceFd);
        }
    }
    
    fseeko(partition->sourceFd,partition->partitionStart,SEEK_SET);
    
    *(u_int16_t *)&partitionInfo[0x000C] = 0x0001; //Number of active FATs (always 1) (?)

    // Write the header
    fwrite(partitionInfo, FATX_PARTITION_HEADERSIZE, 1, partition->sourceFd);
    
    partition->chainTableSize = partition->clusterCount * partition->chainMapEntrySize;
    if (partition->chainTableSize % FATX_CHAINTABLE_BLOCKSIZE) {
    // round up to nearest FATX_CHAINTABLE_BLOCKSIZE bytes
        partition->chainTableSize = ((partition->chainTableSize / FATX_CHAINTABLE_BLOCKSIZE) + 1)
                        * FATX_CHAINTABLE_BLOCKSIZE;
    }
    
    // Create empty chain map table
    partition->clusterChainMap.words = (u_int16_t*) malloc(partition->chainTableSize);
    if (partition->clusterChainMap.words == NULL) {
        error("Out of memory");
    }
    
    
    memset(partition->clusterChainMap.words,0x00,partition->chainTableSize);
        
    if (partition->chainMapEntrySize == 2) {
        rootFatMarker = 0xfff8;
        eocMarker = 0xffff;
        partition->clusterChainMap.words[0] = rootFatMarker;
        partition->clusterChainMap.words[1] = eocMarker;
    } else {
        rootFatMarker = 0xfffffff8;
        eocMarker = 0xffffffff;
        partition->clusterChainMap.dwords[0] = rootFatMarker;
        partition->clusterChainMap.dwords[1] = eocMarker;
    }
    
    // writing the new chaintable
    writeChainMap(partition);
    
    // Address of the first cluster
    partition->cluster1Address = partitionOffset + FATX_PARTITION_HEADERSIZE + partition->chainTableSize;
    fseeko(partition->sourceFd, partition->cluster1Address, SEEK_SET);
    
    // clearing the first cluster
    clusterData = (unsigned char *)malloc(partition->clusterSize);
    
    memset(clusterData,0xFF,partition->clusterSize);
    
    writeCluster(partition, 1, clusterData);
    // first direcory or file
    
    printf("createPartition : Filename : %s\n",szFileName);
    printf("createPartition : clusters       %ld\n",(unsigned long)partition->clusterCount);
    printf("createPartition : start          %lld\n",partition->partitionStart);
    printf("createPartition : size           %lld\n",partition->partitionSize);
    printf("createPartition : chainMapSize   %ld\n",(unsigned long)partition->chainMapEntrySize);
        printf("createPartition : clusterSize %ld\n",partition->clusterSize);
            
    return partition;
}

[jigglywiggly]

Glad to see people working on this...

[run088]

How would one go about getting this code into one of the xdsl or xebian kernels?

[ldotsfan]

The code doesn't belong to the kernel. It is part of xboxdumper and mkfs.fatx - which comes with any xbox derivative linux - like xdsl or xebian.

The kernel code is mentioned in this post.

To explain the relationships between this code and the kernel, let's use something more familiar. A LBA48 bios (including nkpatcher bios patcher used in softmods) allow the xbox to overcome the LBA24 size boundaries and use a large sized hdd. xbpartitioner creates an on-disk partition table that tracks the starting locations of each partition and the size. In addition, xbpartitioner also formats the FATX partition with the right cluster sizes with XapiFormatFATVolumeEx, ie 16k , 32k or 64k. The bios AFAIK doesn't seem to care though about the change of the cluster size from 16k for these larger partitions as Paul's LBA48 patches don't handle those. Any homebrew dash (ie UnleashX, XBMC , EvoX) seems happy with bigger sizes clusters.

Now let's switch gears to Linux. Linux is already LBA48 compliant so we don't need to handle that. However it needs to understand the xbpartitioner styled partition table - to know where each partition starts and the size and it needs to handle 32k/64k cluster sizes. Torne Wuff (referenced in the linked post) wrote the kernel code for that. Next I did two other things. I modified the kernel code (inode.c in fatx fs driver) and removed hard coded references to 16k cluster sizes and used the value read from the on-disk FATX superblock. This seemed to make Linux happy with bigger size clusters and I was able to operate on files on F partition which was formatted to 32k clusters.

The code 3 posts above is the final piece of the puzzle which allows Linux to format a 32k/64k compliant FATX partition. In most cases, this is not needed as most people used xbpartitioner to do that. I was working on xboxhdm3 which is supposed to handle the equivalent operation hence it is needed.

I briefly tested the kernel (with Torne's changes and my FATX changes) from xebian - frugal install (binaries can be found here). You have to replace vmlinuz with bzImage and/or vmlinuz.with.squashfs and edit linuxboot.cfg if needed. The kernel tree I used is available at the same location. I didn't test this extensively so it might break other things on the xbox like sound etc. xebian uses squashfs which is already patched in my kernel tree.

xdsl is more involved as it used cloop module to mount KNOPPIX and I have not be able to load that module successfully - missing kernel symbols. I also tried to modify minirt24.gz to use squashfs and loop instead of cloop but those attempts did not work out.

I welcome feedback and ideas on how to modify xdsl so that this will eventually work.

This post has been edited by ldotsfan: May 15 2010, 05:43 AM