Scenyx Entertainment Community > Tcp/ip Stack For Openxdk

openxdkman

Aug 27 2006, 09:03 AM

This morning I've finally finished porting wattcp 16bits v1.2 to xbox (and ps2, but who cares...)
It compiles well with current version of OpenXDK (and ps2dev, on emotion engine, but... once again...)
An FTP sample (put, get, dir) and an HTTP sample (get) are provided.
DHCP works fine (means you can let your consoles connect to internet through your windows XP with ICS, or directly through DHCP compatible xDSL or cable modems)

The problem is the license. It's an odd old license saying you need written permission to publish modified sources, but you can do whatever you want with the binary result... As shown on wattcp.com (created after the release of watt-16 1.2) a newer version exists and has a GPL license... So it's a bit confusing... Also watt-32 exists and has probably more features.

I've sent the result of this port to all the people who have write accesses to CVS, and I guess it will be up to Craig (owner of openxdk.org) to take a decision...

So don't PM me or post to ask for it, I won't give any answer. I don't want to bypass Craig and the other nice guys who worked so hard on OpenXDK. I guess they will probably aim for another tcp/ip stack, but with this port done, they won't have any technical obstacle anymore...

This message was just to tell you what's going on behind the scene... You can be optimistic, at least, now.

Sirmatto

Aug 27 2006, 06:10 PM

Excellent, networking is one more step for the OpenXDK becoming a viable alternative to that nasty XDK. Hopefully this, or something similar, gets included into the OpenXDK.

PedrosPad

Aug 29 2006, 11:08 AM

QUOTE(openxdkman @ Aug 27 2006, 09:10 AM)

This morning I've finally finished porting wattcp 16bits v1.2 to xbox (and ps2, but who cares...)
It compiles well with current version of OpenXDK (and ps2dev, on emotion engine, but... once again...)
An FTP sample (put, get, dir) and an HTTP sample (get) are provided.
DHCP works fine (means you can let your consoles connect to internet through your windows XP with ICS, or directly through DHCP compatible xDSL or cable modems)

Haven't seen it (I'm not in the select set

), but sounds like excellent news.

An FTP server could be grafted onto OpenDash. One more closedXDK feature matched in OpenXDK.

d0wnlab

Aug 29 2006, 07:29 PM

QUOTE(PedrosPad @ Aug 29 2006, 06:15 AM)

An FTP server could be grafted onto OpenDash.

That's the plan, once time allows

openxdkman

Sep 21 2006, 01:48 PM

I realize there is no good reason for me to not give away the packet driver source.

I've built upon it my wattcp port, but while spreading an unofficial stack may be confusing and unwise (since several tcp/ip stack implementations exist and may have huge impact on what programs will be ported or not), I think spreading the little element that was the only technical obstacle for networking on xbox (the very small driver that allows to just receive and send ethernet packets) is not unwise.

So here it is, have fun!
Supplied with a small sample that detects incoming pings (but doesn't reply to them. only replies to arp request)

That way, if you can't wait you will be able to port the stack of your choice. But try to offer it to owner of openxdk.org before spreading it.

http://minilgos.perso.sfr.fr/xbox_pktdrv.zip
(forcedeth driver from linux source helped a lots of course...)

I'm now working on the nvidia agp miniport driver for xbox, in order to be able to send prepared pushbuffers to GPU (to reach maximal limit of numbers of triangles displayed per frame, probably more than 30000). I will release it as soon as I have a small working sample. d0wnlab said a "mesa" port could be made with that, but I'm not expert in Linux graphics libraries, so I can't tell more.

The Zep Man

Sep 22 2006, 07:05 AM

Now THAT is making progress. This development should get to the front page of Xbox-Scene. Much better compared to all the news about the 1000 different USB/SATA kits for the 360.

Good job on writing the driver. Can't wait to see OpenXDK-based applications using it, like Opendash.

[edit]
I will PM Xantium and ask if this can be posted as news.

openxdkman

Oct 18 2006, 08:56 AM

Since we are waiting for an official stack, here is a little demo binary to play with.
It reads first lines of first quote found in xbox-scene home page...

http://minilgos.perso.sfr.fr/xb1_wattcp_http_get_demo.zip

Mi©®os∞ft

Feb 22 2008, 12:36 AM

QUOTE(openxdkman @ Oct 18 2006, 03:32 AM)

Since we are waiting for an official stack, here is a little demo binary to play with.
It reads first lines of first quote found in xbox-scene home page...

http://home.tele2.fr/~fr-51785/xb1_wattcp_http_get_demo.zip

Good demo.

Any news on Official stack or SDL Sockets port?

--

Also, I am unable to find the ORIGINAL Wat tcp 16 source. I tried googling but only thing I can find it wat-tcp 32.

I woud appreciate if you provide the link

openxdkman

Feb 22 2008, 06:36 PM

I don't think you should follow my path (I started studying Lynx, then Bobcat. Bobcat07.zip was available on internet at that time and this text only internet browser was using a standard packet driver and wattcp 1.2).

I really needed a stack that compiles under msdos and under bare bios in real mode, but it's not your case.
I suffered a bit porting it to 32 bits for xbox1 and 64 bits for ps2. You should start with a 32 bits stack.

Maturion is hosting a mirrored version of original OpenXDK home page where Craig (its creator) wrote a todo list. In this list he talks about LWIP stack and gives an url :
http://www.sics.se/~adam/lwip/
http://openxdk.maturion.de/todo.html

If wattcp is your thing, just grab wattcp32 (or watt-32) and bend it so it can accept my packet driver.
http://www.erickengelke.com/wattcp/

lwip and wattcp or any other tcp/ip stack have necessarily a bottom that needs to be interfaced with a packet driver. And packet drivers are not over complex. The difficulty to write them is just the difficulty to find or guess the network adapter chipset registers documentation... They do always the same, send packet, receive packets... and packets are always around the same size, i.e 1500 bytes. So what requires lwip is probably not far from what requires wattcp or wattcp32... (mainly ring buffers and interrupts).

But the top of stacks show different high level apis... And I was happy to find plenty of samples for wattcp (http, ftp, pop, etc...).

Dunno for lwip, never tried it.

To help you, here is the function which gave me the most trouble...
A function that gives a correct 'pc ticks' value on xbox1 (for wattcp 16 bits)

QUOTE

unsigned long times(void *);

static longword pc_ticks_counter(void) //longword is unsigned 32 bits
{
static longword clk=0;
longword out;

out=times(0); //1000=1s
out=((out*1193L)>>16); //18.203735=1s (overflows after 1 hour)
//Try to guess if we got a clock overflow (0000ffff->00000000)
//Works well if we are called often (more often than every 15 mn)
if (((clk&0xffff)>=0xC000)&&(out<0x4000))
clk=(clk&0xffff0000)+out+0x10000; //18.203735=1s (overflows after 7 yrs)
else
clk=(clk&0xffff0000)+out; //18.203735=1s (overflows after 7 yrs)
return clk; //18.203735=1s (overflows after 7 yrs)
}

But since then, in Demo04 (main.c) I've found a much better way to detect exact processor clock :

QUOTE

unsigned int cpu_ticks(void)
{
unsigned int v;
__asm__ __volatile__ ("rdtsc":"=a" (v));
//edx:eax holds cpu ticks count after 'rdtsc'
return v;
}

However, typical algorithm in tcp/ip stacks deal with seconds and minutes rather than with microseconds for timeouts, and more important for the Van Jacobson algorithm (when you don't get replies for your sent packets, you DO NOT HAVE THE RIGHT to madly resend packets over internet, so a protocol exists that tells you to slow down the frequency of your resending) .
So, 'pc ticks' are just fine for the job and avoid too fast 32 bits counters overflows.

My notes about Van Jacobson algorithm (which was badly implemented in wattcp 1.2 16 bits, so I re-implemented it) :

QUOTE

Implementation of following algorithm is NECESSARY
(avoids congestion, improves performance)

Van Jacobson algorithm (from his book) :

Constants : alpha=1/8 beta=1/4 gamma=4 (for TCP)
RTT : Round Trip Time (time between sending and reply)
SRTT : Smoothed Round Trip Time (predicted RTT)
SRTT[K]=(1-alpha)*SRTT[K-1]+alpha*RTT[K]
ERR : Difference between prediction and reality
ERR[K]=RTT[K]-SRTT[K-1]
RTTVAR : Round Trip Time Variation (predicted error)
or SDEV : Smoothed Deviation
SDEV[K]=(1-beta)*SDEV[K-1]+beta*abs(ERR[K])
RTO : Retransmission Timeout
RTO[K]=SRTT[K]+gamma*SDEV[K]

Exponential smoothing algorithm (from rfc2988) :

1) G : timer granularity (<=100ms is better)
heartbeat : 2.5<2.5+G<3 seconds

2) First RTT measurement : R
SRTT=R
RTTVAR=R/2
RTO=SRTT+max(G,gamma*RTTVAR)

3) Next RTT measurements : R2
RTTVAR=(1-beta)*RTTVAR+beta*abs(SRTT-R2)
SRTT=(1-alpha)*SRTT+alpha*R2
(order is important)

4) if RTO<1s RTO=1s

5) if RTO>60s RTO=60s

Suggested sequence (from rfc2988) :

1) Start timeout detection with RTO
each time a packet containing data is sent

2) When all is done suspend timout detection

3) When ACK packet is received reset to RTO

4) If RTO timeout is detected, resend last packet,
double RTO value and detect next timout : RTO=RTO*2 (max 60s)
If happens too often, restart with first measurement R

Beside these timing problems, the stack was working good. No need to improve it.

Mi©®os∞ft

Feb 23 2008, 04:28 AM

QUOTE(openxdkman @ Feb 22 2008, 01:12 PM)

I don't think you should follow my path (I started studying Lynx, then Bobcat. Bobcat07.zip was available on internet at that time and this text only internet browser was using a standard packet driver and wattcp 1.2).

I really needed a stack that compiles under msdos and under bare bios in real mode, but it's not your case.
I suffered a bit porting it to 32 bits for xbox1 and 64 bits for ps2. You should start with a 32 bits stack.

Maturion is hosting a mirrored version of original OpenXDK home page where Craig (its creator) wrote a todo list. In this list he talks about LWIP stack and gives an url :
http://www.sics.se/~adam/lwip/
http://openxdk.maturion.de/todo.html

If wattcp is your thing, just grab wattcp32 (or watt-32) and bend it so it can accept my packet driver.
http://www.erickengelke.com/wattcp/

lwip and wattcp or any other tcp/ip stack have necessarily a bottom that needs to be interfaced with a packet driver. And packet drivers are not over complex. The difficulty to write them is just the difficulty to find or guess the network adapter chipset registers documentation... They do always the same, send packet, receive packets... and packets are always around the same size, i.e 1500 bytes. So what requires lwip is probably not far from what requires wattcp or wattcp32... (mainly ring buffers and interrupts).

But the top of stacks show different high level apis... And I was happy to find plenty of samples for wattcp (http, ftp, pop, etc...).

Dunno for lwip, never tried it.

To help you, here is the function which gave me the most trouble...
A function that gives a correct 'pc ticks' value on xbox1 (for wattcp 16 bits)
But since then, in Demo04 (main.c) I've found a much better way to detect exact processor clock :
However, typical algorithm in tcp/ip stacks deal with seconds and minutes rather than with microseconds for timeouts, and more important for the Van Jacobson algorithm (when you don't get replies for your sent packets, you DO NOT HAVE THE RIGHT to madly resend packets over internet, so a protocol exists that tells you to slow down the frequency of your resending) .
So, 'pc ticks' are just fine for the job and avoid too fast 32 bits counters overflows.

My notes about Van Jacobson algorithm (which was badly implemented in wattcp 1.2 16 bits, so I re-implemented it) :
Beside these timing problems, the stack was working good. No need to improve it.

Thanks, will sure be of help.

Guess, most packages like these {lwip, wattcp} purposefully contain large number of files just to intimidate the end users + use obscure file names

Hope my luck kicks in soon.

Mi©®os∞ft

Apr 9 2008, 10:02 PM

QUOTE

unsigned int cpu_ticks(void)
{
unsigned int v;
__asm__ __volatile__ ("rdtsc":"=a" (v));
//edx:eax holds cpu ticks count after 'rdtsc'
return v;
}

How would I obtain seconds? 18.203735 = 1s for older method. But what's the one for the newer one above?
The older method works but it overflows after 1 hr as stated.

openxdkman

Apr 10 2008, 08:34 AM

Normally pc_ticks_counter overflows after 7 years, if you call the function enough often (i.e you don't wait more than 15mn between two calls).

For PPPOE for example, you can have an internet service provider active or passive. If active, it will send you every minute an echo request you have to reply to (or you lose connection). If passive, it will expect an echo request from you before 2mn (or you lose connection).

So there are many reasons to call such function often (to detect connection loss or to prevent it). So I never got hit by the 1 hour overflow. The tcp/ip stack source will want to call such function all the time to detect different timeouts.

The cpu_ticks function I give you returns 32 bits which is too low. It increments so fast, that you overflow 32 bits too quickly. It was made to know where you are inside a frame (1/60Hz=166ms) while optimizing 3D rendering. So it's not well adapted unless you know how to get 64 bits and know precisely the CPU frequency. I give it out because it's good to know that such mnemonic exists.
(check Demo 04 to see how I convert it into percentage of the frame time -the yellow bar-)

Mi©®os∞ft

Apr 10 2008, 09:35 PM

Few questions:
--------------

Does this implementation look good?

QUOTE

void XboxStartup
{
initialticks = ; //obvious
for (;;)
{
curr_seconds =(int)((cpu_ticks() - initialticks)/18.203735);
}
}

long int set_timeout(int seconds)
{
return curr_seconds*18.203735+seconds*18.203735;
}

long int set_ttimeout(int ticks)
{
return curr_seconds*18.203735+ticks;
}

int chk_timeout(long int sectimeout)
{
if (sectimeout<= (long int)curr_seconds*18.203735) return 0;
else
return 1;
}

----------

QUOTE

int min(int a, int B )
{
return a<b ? a : b;
}

Is this what min function does like the one below, for example

tmpdiff=min(s->maxrdatalen-s->rdatalen-dst,tmpdiff);

Or does min do something else?

---------

Also, When I do debugPrint or pb_print and do XSleep(sometime), the text disappers for sleep time.. What about debug messages?

My first successful compile NO linking errors or errors.

Mi©®os∞ft

Apr 12 2008, 04:59 PM

Small yet Crucial fix:

CODE

int chk_timeout(long int sectimeout)
{
if (sectimeout > = (long int)curr_seconds*18.203735) return 0;
else
return 1;
}

Also are there any -Easy to fall into- Pitfalls that I might need to avoid while working with wattcp16, the ones you faced already?

This seems to take more than half of the time of my actual project

Also, Which file contains Van Jacobson Algorithm pctcp.c? Which parts specifically?

openxdkman

Apr 13 2008, 08:23 AM

1) Van Jacobson :

Look for "karn_count" inside function
static void tcp_handler(in_Header *ip)
of tcp.c file

I no longer have time (got a job for a year now, I did pbkit and network stuff while unemployed for years) to go back inside source and fully re-understand it.
So here is the part of my modified code (shouldn't break the license that prevents publishing wattcp16 source code) :

CODE

    /* update our retransmission stuff */
    /* new algorithms */
/*
    Implementation of following algorithm is NECESSARY
    (avoids congestion, improves performance)

    Van Jacobson algorithm (from his book) :

    Constants : alpha=1/8 beta=1/4 gamma=4 (for TCP)
    RTT : Round Trip Time (time between sending and reply)
    SRTT : Smoothed Round Trip Time (predicted RTT)
    SRTT[K]=(1-alpha)*SRTT[K-1]+alpha*RTT[K]
    ERR : Difference between prediction and reality
    ERR[K]=RTT[K]-SRTT[K-1]
    RTTVAR : Round Trip Time Variation (predicted error)
    or SDEV : Smoothed Deviation
    SDEV[K]=(1-beta)*SDEV[K-1]+beta*abs(ERR[K])
    RTO : Retransmission Timeout
    RTO[K]=SRTT[K]+gamma*SDEV[K]

    Exponential smoothing algorithm (from rfc2988) :

    1) G : timer granularity (<=100ms is better)
    heartbeat : 2.5<2.5+G<3 seconds

    2) First RTT measurement : R
    SRTT=R
    RTTVAR=R/2
    RTO=SRTT+max(G,gamma*RTTVAR)

    3) Next RTT measurements : R2
    RTTVAR=(1-beta)*RTTVAR+beta*abs(SRTT-R2)
    SRTT=(1-alpha)*SRTT+alpha*R2
    (order is important)

    4) if RTO<1s RTO=1s

    5) if RTO>60s RTO=60s

    Suggested sequence (from rfc2988) :

    1) Start timeout detection with RTO
    each time a packet containing data is sent

    2) When all is done suspend timout detection

    3) When ACK packet is received reset to RTO

    4) If RTO timeout is detected, resend last packet,
    double RTO value and detect next timout : RTO=RTO*2 (max 60s)
    If happens too often, restart with first measurement R
*/

    if (s->karn_count == 2) {
    s->karn_count = 0;
#ifdef DEBUG
    if (debug_on > 1 ) printf("finally got it safely zapped from %u to ????\n\r",s->unacked);
#endif
    } else {
    if ( unal2l(s->vj_last) ) {
        // unnecessary to use unhappy || s->datalen )
        if ((diffticks = set_ttimeout( 0 ) - unal2l(s->vj_last)) >= 0 ) {
#ifdef TRACECALLS
setlastcall("tcp_handler 6");
#endif
        // we ignore the overnight case
        diffticks -= (longword)( s->vj_sa >> 3 );
        s->vj_sa += (int)diffticks;
        if (diffticks < 0)
            diffticks = - diffticks;
        diffticks -= (s->vj_sd >> 2);
                s->vj_sd += (int)diffticks;
        if (s->vj_sa > MAXVJSA) s->vj_sa = MAXVJSA;
        if (s->vj_sd > MAXVJSD) s->vj_sd = MAXVJSD;
#ifdef TRACECALLS
setlastcall("tcp_handler 7");
#endif

        }
        // only recompute rtt hence rto after success
        s->rto = 1 + (((s->vj_sa >> 2) + (s->vj_sd)) >> 1);
#ifdef DEBUG
        if (debug_on > 1 ) printf("rto  %u  sa  %u  sd  %u   cwindow %u  wwindow %u  unacked %u\n",
        s->rto, s->vj_sa, s->vj_sd, s->cwindow, s->wwindow, s->unacked );
#endif
    }
        s->karn_count = 0;
        if ( s->wwindow != 255 ) {
            if ( s->wwindow++ >= s->cwindow ) {
                if ( s->cwindow != 255 )
                    s->cwindow ++;
            }
            s->wwindow = 0;
        }
    }
    // all new
    scheduleto = set_ttimeout( s->rto + 2 );
    if ( unal2l(s->rtt_time) < scheduleto )
    {
        movl2unal(scheduleto,s->rtt_time);
    }
#ifdef TRACECALLS
setlastcall("tcp_handler 8");
#endif

    switch ( s->state ) {

2) Debugging

Basically when xbox1 hangs, I reboot and retry
(It's much cleaner on PS2, so sometimes, thx to a unique source that recompiles on pc0, xb1 & ps2, I could debug on PS2 where I can catch the faulty line in source and so get it fixed for other platforms. So I never tried to improve things on xbox1. When I studied it I instantly gave up because you need to play with stack structure to recover from bad crash and I don't know it well enough on xbox1...)

The setlastcall allows on ps2 to have the name of the last executed section before crash.

3) XSleep erases things...

A hang in xbox1 may often erase everything on screen for sure. For XSleep I don't know.

If pbkit is activated it can do screen swaps at next VBL unless you ask to show the debug screen.

If there is no hang, then you should be able to control what is displayed by using correctly pbkit functions. The top of debug screen may be overlapped by normal screens (so if they get cleaned you lose your debugging text). Try to display text on the half top of the screen then display your debugging text.

You can always ignore pbkit and not activate it and have the standard openxdk behaviour regarding text.

4) Takes a lots of time...

Of course it does!!!
But in my case (several months) it was worthy because I got instantly a stack under control running on SEVERAL platforms with same source to maintain.