PS3 Fault finding YLOD with the SYSCON - First steps and Error reporting

[sandungas]

Btw, i mentioned "USB adapter" because this kind of USB devices are used a lot with raspberry, arduino, or even with routers
Actually, the one i have was announced as "USB to TTL adapter for arduino"

 

[nightmareflash]

Hello.
Can someone answer two questions
1. To see the err log do you need to be in internal mode. I see that the ERR LOG comand is in the external mode too.
2. Do we need to reasemble whole PS3 to run the script?

 

[Arin]

The serial ports of the old desktop PC motherboards cant be used for this... just incase you was thinking in it, dont do it because works at 12v, or you will fry the PS3 motherboard

The only way is to connect with serial UART TTL (3.3v)
And the easyest way to do it is if you buy one of the USB adapters we was mentioning in the first pages of the thread
There are tenths of manufacturers building them and are very cheap (the cheapest we found was 1$ free shipping)

thank you for that info do you connect the motherboard tx to the usb uart rx or just matched tx to tx and rx to rx?

 

[nyislander]

Can someone answer two questions
1. To see the err log do you need to be in internal mode. I see that the ERR LOG comand is in the external mode too.

Yes, you must be in internal mode.

2. Do we need to reasemble whole PS3 to run the script?

No, just put the motherboard in the metal shell, connect the power supply, fan, eject/power board and support it in the bottom half of the console.

thank you for that info do you connect the motherboard tx to the usb uart rx or just matched tx to tx and rx to rx?

TX to TX & RX to RX.

 

[sandungas]

thank you for that info do you connect the motherboard tx to the usb uart rx or just matched tx to tx and rx to rx?

No, thats how it was made years ago with the legacy RS232 chips... the problem is the voltage in the serial ports of a PC motherboard is 12v and it was required to be converted it to a different voltage (usually in a dirty way with some resistors)
For syscon we need a very low voltage and well stabilised... this is achieved with the new "USB to TTL" chips that was intended as a replacement for the legacy RS232

The way it works... is the USB device "emulates" a #COM port
In the device manager of your operative system (windows, or linux) you are going to see a new device that looks like a legacy #COM port (like in the old PC motherboards that had serial and printer ports)... but is virtual
I mean... is the USB device, not a real #COM port... but it works exactly like a real #COM port

------------------------
The Rx and Tx pins are in the USB device... they needs to be reversed (Tx in the USB connects to Rx in the PS3... and the other way around)
The problem is many of this devices are labeled in a bit confusing way
So is very frequent to connect the 2 Rx and Tx wires "flipped"... but dont worry, incase this happens is not dangerous... the only thing that is going to happen is the output texts from syscon are going to be displayed as "garbage" in the #COM terminal window in PC
But this is actually a good signal, because uses to happen the first time you connect.... and it means everything is working fine (except the flipped wires... so the only thing left to do is to flip the wires)

------------------------
I suggest to keep the heatsink in is place with some thermal paste, is not needed to use a good thermal paste, this is just for testing, so just recycle and spread again the old thermal paste
Also, keep the fan in his place too and the wire connected

Or alternativelly... use a heatsink from PC (are smaller) and a external fan (kind of fan you could have at home running at 220v for summer) aiming to the heatsink

This is not really needed, because most of the things you are going to do in syscon works while the PS3 is in standby... but eventually it could be handy to boot the PS3 (to populate the error log with new errors)... or it could happen that you could boot the PS3 by mistake

Just to show you an example of this... one of the first things you need to do when entering inside syscon is to display the error log
Make a good backup of that error log, im not sure how long is it and how many errors contains, but it could be very confusing because there could cointain errors from months ago not related with the problem you are trying to solve now, eventually you could return to that error log for a better forensic analisys... but initially you dont need that error log, so.... the next thing you can do is to delete the error log
After that is when you can try to boot the PS3 normally, at that point the PS3 is going to generate an error, then you poweroff the PS3, and the errorlog will contain 1 error only... so you are completly sure you need to focus your attention in that one

 

[shawly]

I have a DYN-001 which shuts down randomly (green led > delayed and quick yellow led > 3 beeps > red blinking) it can do immediately or last an hour or two, I've already tried other power supplies, so I want to see the syscon error log before I go into the NEC-TOKIN replacement endeavor.

Can nobody tell me how this guy found the necessary pads on a DYN-001 board?

 

[Arin]

So I started this thread to have a deep discussion about the YLOD on the PS3 (mostly FAT versions) and how to use the SYSCON to understand further what can be done to repair the ps3.

My main reason that drives me to do this is that I've seen alot of miss informed threads on how to fix YLOD ps3, there is alot of valid pointers but the basics have to be done first.

DISCLAIMER:

Before attempting this procedure, PLEASE PLEASE make sure you are confident in basic soldering and diagnostics, I will not be held responsible for you breaking your ps3!!!

NOT all PS3 boards have the known serial connection or capability to connect!!
So far - COK-001,002, SEM-001 and DIA-001,002 can be connected

READ THE GUIDE AND ASK QUESTIONS IN THIS FORUM, THANK YOU!

Ok, thats the disclaimer over and done with:

So in case everyone has missed a thread about the syscon discovery - https://www.psx-place.com/threads/s...oxao-what-does-it-mean.26148/page-10#comments

This thread will be more of a deep dive in using the syscon.

What I hope from this thread is that people contribute their error codes they get from their syscon and we can create a database of error code and whats issues are related to that error codes that come out of it.

So to get everyone started, I will share my first draft syscon connection guide:

This guide covers what tools to use, software to use and versions of PS3 boards (known to access syscon) with identification of pins to connect to.

As this thread grows overtime we should have a good grasp of the errors and what needs fixing to keep these old PS3 consoles alive.

SYSCON Guide

Further changes are in the git repo

https://github.com/db260179/ps3syscon

Please contribute on this git repo in the issues section
New error messages, python script improvements
NOT PERSONAL SUPPORT ISSUES!

In this guide we will be:

1. Identfy the motherboard and its pins - RX, TX, GND and DIAG pin
2. Solder the jumper cables to the correct pins - RX, TX and GND go to the USB uart serial cable and DIAG get shorted to GND once the eeprom has set its mode to allow diag mode
3. Run python syscon script to interact with the syscon - 'auth' command is used to access the high level commands on the syscon, and internal and external commands to run actions.
4. External commands - used mostly to do basic stuff - activate internal commands
5. Internal commands - Spend most of our time running - 'errlog', 'bringup' and other common used commands to help identify YLOD issues.
6. Identify the YLOD issue based on the error code - and investigate with basic multimeter - voltage checks, resistance etc
7. Tweaked fantables - use supplied tweaks from the my gitrepo for each motherboard

Big thanks and shout out to Mina for his help and patience!

Recorded errors (errlog) in the syscon shell:

POWER ERRORS:

0003001 POW_FAIL

A0093004 RSX_POW_FAIL

A0201B02 RSX VRAM FAIL - Faulty vrams (core would read a 0.2 ohm reading)

A0093003 CELL_POW_FAIL

BE ERRORS:

A0213013 BE_SPI DI/DO ERROR - CELL not communicating to syscon via SPI (1.2V MC2_VDDIO and 1.2V BE_VCS no output) = Possible shorts on the line, check C4001 and trailing caps. Possible CELL dead?

A0213011 BE_SPI CS ERROR

A0203010 BE_INIT OR BE_POWGOOD OR CLOCK ERRORS

A0801200 CELL overheating - poor thermal paste or no heatsink attached

RSX ERRORS:

A0404002 RSX_SPI DI/DO ERROR

A0404411 - ERROR ON RSX SPI?

A0A02031 - Thermal monitor DI/DO not communicating to RSX (possible dead Diodes in RSX)

A0403034, A0404402,A0404411 - Poor BGA solder connections for RSX ( you will see errors like - [POWERSEQ] Error : BitTraining RSX:RRAC:RX0:GLOBAL1:RX_STATUS )

A0232102 - IC6301 faulty (1.5v RSX_VDDIO) or in that area

SB ERRORS:

A0302203 SB_SPI DI/DO ERROR

A0313032 SB_CLOCK OR INIT ERROR

A0902203 SB GLOD issues, system update to repair nand/nor hashes


OTHERS:

A0022110 MK I2C ERROR (OR OTHER CLOCK's ERRORS)

A0401001 - BE VRAM Power Fail. It can be NEC Tokins

A0401002 - RSX VRAM Power Fail. It can be NEC Tokins

A0402120 - HDMI Error (IC2502)

A0401301 - BE PLL Unlock

Please submit your full error messages and what you did to resolve that error!

Look at the git repo - 'syscon error log codes.pdf'

im sorry but I'm new at this I have the board connected to the UART but I don't know how to pull error codes in python. Do you need more to connect to it? Is there a step by step guide?

 

[squeept]

I've got about 10 systems on the way right now, so I figured this was a good time to actually start logging my repairs again and share all of the results in a nifty spreadsheet. Since I've got everything to make a comprehensive diagnosis, this should really help nail down the usual suspects. I also dug around in my drawers and found I already had the USB adapter sitting here, so I'm mad at myself for not doing it earlier. Anyway, I started dicking around with this syscon stuff so I can add the error codes to the log.

Is there any reason I can't just manually do "errorlog get (the first few numbers)" from the external mode so I don't have to worry about screwing up anything in internal mode? And it would speed the process up a little for me which would be great since I'll be doing this a billion times.

Is there any other relevant syscon information I should be recording?

 

[M4j0r]

Is there any reason I can't just manually do "errorlog get (the first few numbers)" from the external mode so I don't have to worry about screwing up anything in internal mode? And it would speed the process up a little for me which would be great since I'll be doing this a billion times.

Well, the internal mode can provide more information than the errlog but if you only want the errlog it's fine to use the external mode.

Is there any other relevant syscon information I should be recording?

In case something goes wrong it's always good idea to have a copy of the perconsole stuff of the eeprom. But that requires desoldering the chip or applying a patch which patches the access checks of the eeprom commands.

 

[squeept]

Here's what I whipped up so far. Let me know if there's anything else anyone thinks should be on there and I'll jam it in to my workflow if it seems relevant or necessary. I'll print a template of this to keep with every board as I work on it, then input it into a spreadsheet. I have a general idea of the physical layout of the rows and columns that should make it fairly concise visually. Should be starting one tonight, so hurry up!

Spoiler

Start Date: (the date, dumbass...)
Model: (the model #...)
Serial #: (the serial number...)
Warranty Seal: (is original Sony seal present?)
Evidence of Drops?: (...was it dropped?)
Fault(s): (lights, beeps, time before shutdown, etc)
Fan Ramp: (does the fan spin up?)
Inrush Current: (kill-a-watt max reading)
Run Current: (kill-a-watt reading after inrush)
Safe Mode: (does it beep correctly for safe mode even if no video?)
Visual Inspection: (anything look funny? Heatsink fully caked with dust? Burned areas? Corrossion?)
Previous Rework?: (yes/no)
SYSCON Error(s): (list of recent codes)
Pressure Test CPU: (changes with 2kg weight on heatsink)
SYSCON Error(s) Change: (no/yes with list)
Pressure Test GPU: (changes with 2kg weight on heatsink)
SYSCON Error(s) Change: (no/yes with list)
NEC/TOKIN Waveform CPU: (pass/fail or description on other)
NEC/TOKIN Waveform GPU: (pass/fail or description on other)
Ohm Test CPU: (reading at NEC/TOKIN)
Ohm Test GPU: (reading at NEC/TOKIN)

***Next 4 will almost never be filled in***
Visibly Oxidized CPU Pads: (pass/fail with count)
Visibly Damaged CPU Pads: (pass/fail)
Ohm Test CPU: (reading direct from chip)
Ohm Test CPU missing: (reading across NEC/TOKIN missing CPU)
***Next 4 will be filled in on every BC console, I will not sell BC without reballing since there is almost always a visible BGA fault on GPU***
Visibly Oxidized GPU Pads: (pass/fail with count)
Visibly Damaged GPU Pads: (pass/fail)
Ohm Test GPU: (reading direct from chip)
Ohm Test GPU missing: (reading across NEC/TOKIN missing GPU)

***restart diagnosis with reballed GPU if not fixed***
Fault(s): (lights, beeps, time before shutdown, etc)
Fan Ramp: (does the fan spin up?)
Inrush Current: (kill-a-watt max reading)
Run Current: (kill-a-watt reading after inrush)
SYSCON Error(s) Change:
NEC/TOKIN Waveform CPU: (pass/fail or description on other)
NEC/TOKIN Waveform GPU: (pass/fail or description on other)

***back to all consoles***
Troubleshooting: (description of misc. failed components, odd voltages, shorts, etc that needed dealt with)
Success?: (simple yes/no in own column for easy identification)
Failure?: (description of unresolved issues, best guesses, other notes)
Stress Test: (pass/fail of 10 full heat cycles and 24 hours of burn-in)

***when they die during stress testing, it's getting chopped for parts, I'll probably just grab the last SYSCON error and call it***

***post sale bookkeeping***
Date of Sale: (the date...)
Warranty Length: (the length...)
Returned?: (simple yes/no in own column for easy identification)
Date of Return: (the date...)
Reason for Return: (Actual failure or did the post office throw it off a cliff?)
Troubleshooting: (quick description of post-mortem with codes, etc)

 

[sandungas]

@squeept for the research related with fan settings it could be handy if you tell the checksum of the "thermal config" area of the syscon eeprom. The command "eepcsum" displays it, is explained in post #2 of this thread
https://www.psx-place.com/threads/syscon-fan-settings-coordinate-graphs.31188/

Also, that checksum is super handy when talking about the factory thermal settings, because is an tiny unique identifyer (only 2 bytes) of all the thermal settings (hundreds of values)
I guess you are going to find checksums that matches the image graphs of the other thread... but just incase, if you find something different please advise me (and make a dump of that "thermal config" area and upload it for me to take a look at it, and eventually to make a new graph for it)

 

[squeept]

@squeept for the research related with fan settings

Is this going to help in any way with fault finding or do you just want more data points? I'm afraid of dyslexic-ing the stuff for the internal commands and borking a repairable console. I can check my pile of dead boards for you though if you just want more data, I probably have 15-20 shoved in a box waiting to get sent in for scrap.

 

[squeept]

First one started. 3 second YLOD, A0403034, all other checks normal. In the drying oven now, will reball RSX tomorrow. I have the form ready and printed, but I want to make a really nice spreadsheet, so that might be a little while.

I'll throw the xls on my empty website when it's ready. After that, it will take some time to fill. It takes me about 5 days start to finish for each system between all the work, drying, cleaning, and stress testing.

 

[sandungas]

Is this going to help in any way with fault finding or do you just want more data points? I'm afraid of dyslexic-ing the stuff for the internal commands and borking a repairable console. I can check my pile of dead boards for you though if you just want more data, I probably have 15-20 shoved in a box waiting to get sent in for scrap.

Is mostly for manteinance, if we where talking about classics cars it would be like connecting some probes to the carburator and exhaust pipe to see if the carburator is well adjusted

There are a lot of syscon commands that are "read only"... his purpose is to output info, but doesnt writes anything in syscon, that ones are completly safe to use. The 3 commands i mentioned in the other thread (required to check the "thermal config" area) are
eepcsum> displays 5 lines
r 3300 200> this is a "read" command, it makes a dump of the "thermal config" area
fantbl get 0> this is a "get" command, it displays the fan settings related with thermal sensor 0

What im asking you do do is to run the "eepcsum" command, and check if the value displayed in the second line (where it tells "Addr:0x000034fe") matches with the checksums of my images (either 0x7115, 0x86D6, or 0x985B)
If the value is different please run the other commands to make a dump, is a tiny amount of data, only takes few seconds to dump it
Otherway... if the value matches with my images everything is normal, and there is no need for you to run the dump command or to send me any info

Im asking you to do this because by now we dont have much samples of that "thermal config" areas... it looks ilke all the PS3 motherboards of an specific model are going to have exactly the same "thermal config" data (in other words, same checksum when you run the "eeepcsum" command)... but this is not so clear, are needed more samples to confirm that
You are about to check a lot of motherboards, so your info could be really handy

-----------------
Btw, if at some point we learn how to modify all that "thermal config" settings with some noob-friendly tool and using nice thermal profiles... this is the kind of service you could do for most/all your clients
Maybe is just a personal oppinion, but i can tell you if i was going to buy a PS3 from you i would like if you change them because i know the factory settings are wrong, the best we can do with them is to modify them

Another btw... take a read at some of my old posts in this thread, i was mentioning a method i did learn from a forum dedicated to hacking routers, there is many people that built some kind of "tripod" with a needle to touch the motherboard pads with the needle in the Tx and Rx points (and the PS3 requires another for "diag" pad)
This allows you to connect with syscon very fast and without any soldering job, it could be very handy for someone like you that is going to do this frequently

The only better alternative to the tripod and needles is if you builld some kind of clamp with "pogo pins" matching the positions of the pads located in the "service connector", this way you just need to "clamp" the motherboard with it... is even faster than the tripod with needles
The idea of building something like this is not so crazy, it can be made with a piece of wood/plastic with tiny drills and some pogo-pins inserted in the drills, is a bit tricky to build because requires lot of precission, but are just 3 pogo-pins for Rx, Tx, and DIAG (and for the fourth wire for ground use a alligator clip)

 

[squeept]

First one in the books minus stress testing.

http://www.squeept.com/junk/tokins_rule.xls

Let me know if that format looks... nonsensical. There is a lot of information to try to cram in to a small space in a way that actually gives useful information. I also decided that it would be best to break it down in to two sheets. One with all the detailed analysis, and one "at a glance" sheet that gives just the meat. Also, if you notice serial numbers change retroactively, that's because I put a new case on it to sell it. Nothing to get suspicious about.

Remember, updates will trickle in. 5 days between each PS3 when I'm at full speed.

edit: I left off the current measurements because my kill-a-watt appears incapable of accurately catching the inrush current. I was also bothered by the lower ohm values after rework, and I thought it might be because the board was still warm. So I went and shoveled the driveway and came back in and measured, now they are cell 1.7 rsx 1.9. Go figure. Anyway, I want room temperature for each measurement, so I'll update that next time I upload the spreadsheet. Also going to use "n/a" on fan ramp and safe mode when status changes to "all good" just for a little more ease of reading results at a glance.

 

[squeept]

eep

It's the eep set when patching that I don't want to fat finger and screw up. I'll see if I can remember how to code and I'll just add it to the python script as an intercepted command. (or someone that didn't have headaches just from installing python and getting it to work could do it for me...) Otherwise, I'll dick around with some dead boards for you later. I'll have tons of data for you either way.

I've got a JBC, my tip is at 350 before I could open the drawer to grab some clamps, no need to reinvent the wheel here.

 

[squeept]

Second one finished and in the books, lol. It was a heatgunned massacre and I trashed it for parts, but I think it still gave us some useful verification on A0202120.

Realized I forget to try to capture tokin waveforms on it since I gave up, but the board is still on the bench. I'll fill it in after dinner.

I likely would have attempted to reball this one just to see what happens if I didn't have the syscon error log at my disposal to paint a clear picture of the sequence of events. Already saving me time. w00t.

I need to put this in my signature: http://www.squeept.com/junk/tokins_rule.xls

 

[squeept]

CECHE01 with 3 second YLOD, A0403034, A0404422 now has RSX reballed to no video and A0801701.

Before I decide to waste my time reballing the CELL, can anyone tell me whether 1701 (BE Attention) means "BE needs attention because it is fucked" or "BE is waiting at attention because other things are fucked"?

From the timeline and analysis on this console, I'm pretty well inclined to assume RSX die or bump failure. I would need someone to authoritatively talk me in to reballing CELL...

 

[vyktormvmpay25]

What resistance do you see on cpu. Usually for me is simple to reball cpu. Desolder and see resistance when it cool outside board. 3 ohms is always fine.

 

[Pacorretaco]

CECHE01 with 3 second YLOD, A0403034, A0404422 now has RSX reballed to no video and A0801701.

Before I decide to waste my time reballing the CELL, can anyone tell me whether 1701 (BE Attention) means "BE needs attention because it is fucked" or "BE is waiting at attention because other things are fucked"?

From the timeline and analysis on this console, I'm pretty well inclined to assume RSX die or bump failure. I would need someone to authoritatively talk me in to reballing CELL...

Hmm, So your classic YLOD (~3 sec long, 3034) turned into a GLOD after RSX reballing.
Hmm, sadly I think I'm with you thinking it's still the RSX causing problems. And since the reballing was a "success", it's fair enough to say it's a bad chip. Maybe candidate for a frankenstein 65nm RSX swap attempt? Not a Cecha after all, maybe less intimidating.
I can't convince you to do work on the CELL. If it were the cause of the problem we'd be looking at both rsx and cell failure at the same time. Hard to believe. Also, you say the console beeps and such, shouldn't that point to a working CPU? ~3 ohms reading also, though I never give much weight to that.
CPU problems in general tend to mean full freeze, no responses.
You might as well try simply heating the RSX again a little, see if behavior changes again, which it might.

Also it's news to me that the GLOD can be diagnosed at all with the syscon.
I ordered the dongle without YLODs nearby, mostly for curiosity. But maybe it can help with the 3 GLOD C models I have. Though 2 of them I saw they had artifacting as well. I always assumed the three were victims of the Ugly, failed RSXs after all, without that much troubleshooting.

Oh, and it crossed my mind since you asked, maybe you'd want to check on the consoles that you bring back to life, on webMan it can tell you how much use they had.
Maybe it can give us some ballpark time before the common failures start developing. Though it's mostly curiosity, on many I saw them hover the 150 day mark.

Cheers