I make a call from SIP client (CSipSimple or CSS) to PhonerLite (PL).
1. Call was answered by PL
2. In a couple of seconds PL sends its own INVITE message back with the same Call-ID, but with reversed From/To fields (as call made from PL back to CSS)... And that breaks the audio channel.

Details. SIP client is CSipSimple (CSS) softphone, running on Android. It supports only one codec at a time. The way how it works now is:
1. CSS sends INVITE with SDP, containing set of available codecs
2. Another party (PL, in this case) answers the call, sending OK/INVITE message with its own set of supported codecs
3. CSS sends second INVITE message, now containing only one codec (supported by both CSS and PL)
4. PL replies with OK/INVITE message with SDP containing that one codec
At this point the call begins successfully. Audio comes and goes in both directions just as it should.

Then, in a couple of seconds, PL suddenly sends new INVITE with the same Call-ID, but now it has From/To reverted and SDP contains full set if supported codecs again. That message breaks audio coming from PL to CSS.

Case is 100% repeatable. 

Now, if I make call in opposite direction (from PL to CSS) there are 2 cases:
1. I pick up CSS within next 2 sec of ringing (time is approximate here), the audio between two parties is always OK
2. I pick up CSS after that time, there is no audio from PL to CSS. The audio from CSS to PL is always OK though.

It's weird, but it's PhonerLite specific. If I use CSS with other SIP clients (calling to them or from them), I don't have any such problems at all.

PhonerLite v1.97 running on WXP.

All calls go via FreeSWITCH SIP server. I can monitor its logs.

Smartphone (running CSS), PC (running PhonerLite) and server, running FreeSWITCH use separate networks. All they have different WAN IP's. SIP server is configured to provide direct media mode for all calls between CSS and PL.

What is your problem with "switch" from and to? That is normal! Example:
1) incoming INVITE received by CSS with from is "ABC" and to is "XYZ"
2) after call establishment CSS disconnects a call with BYE: from is "XYZ" and to is "ABC"

PhonerLite sends a re-INVITE only of no data (RTP) is received at all after 5 seconds. I asume that is the problem in your situation.
I installed CSipSimple from the market. I used my PC running PhonerLite as registrar. That built in registrar in PhonerLite is just for testing purposes.
Initiating calls from CSS to PhonerLite is exactly as you described. CSS sends some codecs and PhonerLite answers with all codecs that are supported by both of them - that is normal behavior. After that CSS sends a re-INVITE with only one codec. I have a call with media in both directions - no problems at all here.

I don't have a Freeswitch between.
So I need a Wireshark trace from you on PhonerLite side to see all messages and media flows.

BTW, I forgot to mention, that the author of CSipSimple always suggests to use latest developers version (not one, that is published on the Market, wich is quite obsolete). He provides detailed instructions how to do it on this page:
http://code.google.com/p/csipsimple/wiki/HowToInstallDevVersion

After you install the developer's version, it's very easy to check/get latest updates. Just go to its "Help" and click on "Update nightly build" and everything will be done automatically, no any manual download is required.

Currently I'm running the latest r1330 revision.

I'm running the latest 2.00 version now. And at this time the audio problem is reversed. When I call from CSS to PL after I answer the call in PL there is no audio at all for about first 4 sec. Then audio recovers in both directions.

What I've noticed in PL's debug window is:
When call is answered, PL sends its OK INVITE message. SDP contains 5062 port (m=audio 5062 RTP/AVP 0 101). At this point there is no media stream in both directions. Approximately in 4 sec PL sends its re-INVITE message, but at this time it refers a completely different port - 5135 (m=audio 5135 RTP/AVP 0 97 8 2 3 110 111 9 101) and after common re-negotiation with CSS for using just one codec, the audio becomes functional well in both directions.

What is the port 5062? Why it's replaced with port 5135 in PL's INVITE that it sends in 4 sec? Why PL doesn't send port 5135 in first place?

It looks to me that with 1st port 5062 the audio is not working, while with the second port 5135 it does. Could it be the source of the problem?

BTW, when I call from PL to CSS and answer immediately, the audio is OK in both directions right from the beginning. If I delay with answer for more then 3 sec (approximately), there will be no audio from PL to CSS, while form CSS to PL will be just fine... It's all 100% repeatable and quite weird.

Do you have that problems only with activated ZRTP or any other encryption enabled? If so, please check with completely disabled encryption (SRTP, ZRTP, TLS).
I don't have a Freeswitch environment here. So can you please do a Wireshark trace of such calls and send the trace by e-mail?
I can't reproduce that problem here, so I need that trace to analyze the problem.

Debug in PhonerLite only shows SIP message, but not the RTP data.
If you are using NAT, do you have configured a STUN server?
If another softphone (X-Lite, NinjaLite, Zoiper, Jitsi, 3CXphone, ...) works without problems, but latest PhonerLite version not - then only a Wireshark trace on the PC can compare the differences.

As long I can't test for my own, I can't do anything. Sorry. Do you have a test account?