BFL board project COINTAMINATION - EU facility - ORDERS OPENED for Chili

[GrapeApe]

I'll be honest this is the first I'm hearing of low volt resets. I was under the impression that the Lucko boards were reseting on the high side. At this point I wish I hadn't posted that at the top of this page....

[Swimmer63]

Very little has gone right with this group buy. Most have been patient despite losing most of our investment. We understand it's a risky endeavor.
But at the very least we need a working board out of this. We paid above and beyond for a proven, 2nd choice board Lucko. Time to make that happen please.
And one that is properly packed when shipped.
Sorry if this is overly harsh but patience is gone.

[asjfdlksfd]

Hello Mr. Teal,

how did you control the output Vcore at chip level? Are there a sense lines added to the bucket controller? Is U12 the related bucket controller? I'm still guessing that the Vcore measured on chips could be higher than on your own produced boards maybe Lucko used a board with better/thicker copper so the potential difference between FETs and BFL chips is less than on your boards. This could be brings the FETs earlier in the limit.
(This is the reason why I want to have a Z-Command to limit the voltage floatable and not fixed in the FW  )

I want to understand why my low voltage resetters are working stable if I preheat the FET with my soldering iron gun. How is the temperature measured which is shown in bfgminer from the M-menu:


 BFL 0 : 51.0C | 28.70/28.46/27.23Gh/s | A: 5909 R:33+1(.56%) HW:14077/3.6%
  BitFORCE SHA256 SC from MrTeal and ChipGeek
Serial: FTX33YKB
Kernel: bulk queue
Temperatures: 49.0C 51.0C
Voltages: 3.290 / 1.032 / 11.890

I wonder me I cannot see an increasement of this temperatures if I use my heating gun focussed on U25 which I guessed for measure temperature near by the Buckets FET.

Remember, this board has to been reset below 1.0 V if I do a cold start. At the moment it is only hashing with 27 GH/s, but it's better than nothing (having sparrow in hands instead pigeons on roof

Also why is this board now working with > 1.0 V after preheating the pcb at bfgminer startup, if it is resets ~1.0 V before I made cold starts? The board is now running stable for 36 hours.
Another board with the same issue is working after preheating with 29 GH/s at 1.007 which has been reset at cold start ~ 0.95 V before.
I have no scope here to check the ripple and voltage near to the bfl chips with ones which have not all these issues. But in fact, more or less all 7 boards which I have now assambled have the same issue, but 5 of them are starting without active cooling of the FETs without preheating mostly.

Cheers and sorry for my mixed sentences...

[Lucko]

I told you that some need more restarts. I manage to get 7 boards going and they were not the best but the worst. But I do use 2 rail 600W PSU. It might be that every board has it own rail. It takes me about 1 hour to make them all run but once they do run they run stable...

I see this same massage:
BFL 0: Received unexpected queue result response:
BFL 0: Error: Get temp returned empty string/timed out

But after a minute it is gone and it reconnect to board... It doesn't start hashing until selftests are done(blinking 7 and 8 ). And if ambient temperature is too high 7 will not stop blinking since board is too hot.

As bx8389 told you(and I before making them). Board needs restarts and after that it runs OK... It only needs to go over critical voltage... Once over it runs...

EDIT: asjfdlksfd made a good point running boards couple of times heat it up so that might be what solves the problem.

EDIT2: MrTeal what about firemwere that will run board for 5 minutes at 0,9V and then go up? It might be solution for many boards if preheating is all that is needed...

[Mudbankkeith]

I told you that some need more restarts. I manage to get 7 boards going and they were not the best but the worst. But I do use 2 rail 600W PSU. It might be that every board has it own rail. It takes me about 1 hour to make them all run but once they do run they run stable...

I see this same massage:
BFL 0: Received unexpected queue result response:
BFL 0: Error: Get temp returned empty string/timed out

But after a minute it is gone and it reconnect to board... It doesn't start hashing until self tests are done(blinking 7 and . And if ambient temperature is too high 7 will not stop blinking since board is too hot.

As bx8389 told you(and I before making them). Board needs restarts and after that it runs OK... It only needs to go over critical voltage... Once over it runs...

EDIT: asjfdlksfd made a good point running boards couple of times heat it up so that might be what solves the problem.

EDIT2: MrTeal what about firmware that will run board for 5 minutes at 0,9V and then go up? It might be solution for many boards if preheating is all that is needed...

I asked on december 29th, if the "gentle preheat" could be done from the firmware.

  Question:-
   Maybe the (firmware) needs a bigger time delay between the voltage steps, to give the chips and temperature more time to stabilize, before the next step up.
   This may also help with the low temperature start up problems (sub zero ambient) some users are reporting. (gentle pre-heat)

  Reply:-
   The issue I am seeing doesn't really have anything to do with the voltage steps. What is happening is that PWM for the first phase is causing the second phase to turn on prematurely, which is causing a
   whole host of issues.

I will also ask again, Will this help?

[Lucko]

Yes but MrTeal has one of the worst boards where symptoms are biggest... That one can't be run at more then 27GH. Some of the rest can even get to 42GH... So just slow preheat might solve 90% of the problems... I do need to remove power supply heetsinks and stop the CPU fans to avoid one board crashing at 27(power supply) and 41(fan) GH... I know that 41 crash can be solved with firmware but I just didn't get to it... And that would also mean that it will not hash at 39 but 36GH... So not sure but I think that the problem is only at a some point in voltage increase and only if right conditions are present...

[HellDiverUK]

I'm thoroughly confused - is it still possible to buy one of these things with chips installed?  

Edit: After having read the thread completely, I think I'll pass.  I have enough stress keeping the Cube running reliably. 

[Mudbankkeith]

Yes but MrTeal has one of the worst boards where symptoms are biggest... That one can't be run at more then 27GH. Some of the rest can even get to 42GH... So just slow preheat might solve 90% of the problems... I do need to remove power supply heetsinks and stop the CPU fans to avoid one board crashing at 27(power supply) and 41(fan) GH... I know that 41 crash can be solved with firmware but I just didn't get to it... And that would also mean that it will not hash at 39 but 36GH... So not sure but I think that the problem is only at a some point in voltage increase and only if right conditions are present...

The 2 I have assembled both crash at:-

40c  120/140w at the wall  27Gh at 5 sec    5/10Gh at the pool   HW10%/15%

when they re-set the log page scrolls very quickly:-

BFL 0: Received unexpected queue result response:
BFL 0: Error: Get temp returned empty string/timed out

EDIT:-
3rd unit almost exactly the same:-
42c    120/140w       26Gh at 5 sec      5/10Gh at pool        HW 30%

EDIT
4th unit SUCCESS HASHING SPOT ON 

69c   power ?  w     37.5Gh at 5 sec     36.5Gh at pool     HW 3%

I WILL POST SOME DIAGNOSTICS HERE LATER TODAY

[MrTeal]

Hello Mr. Teal,

how did you control the output Vcore at chip level? Are there a sense lines added to the bucket controller? Is U12 the related bucket controller? I'm still guessing that the Vcore measured on chips could be higher than on your own produced boards maybe Lucko used a board with better/thicker copper so the potential difference between FETs and BFL chips is less than on your boards. This could be brings the FETs earlier in the limit.
(This is the reason why I want to have a Z-Command to limit the voltage floatable and not fixed in the FW  )

I want to understand why my low voltage resetters are working stable if I preheat the FET with my soldering iron gun. How is the temperature measured which is shown in bfgminer from the M-menu:


 BFL 0 : 51.0C | 28.70/28.46/27.23Gh/s | A: 5909 R:33+1(.56%) HW:14077/3.6%
  BitFORCE SHA256 SC from MrTeal and ChipGeek
Serial: FTX33YKB
Kernel: bulk queue
Temperatures: 49.0C 51.0C
Voltages: 3.290 / 1.032 / 11.890

I wonder me I cannot see an increasement of this temperatures if I use my heating gun focussed on U25 which I guessed for measure temperature near by the Buckets FET.

Remember, this board has to been reset below 1.0 V if I do a cold start. At the moment it is only hashing with 27 GH/s, but it's better than nothing (having sparrow in hands instead pigeons on roof

Also why is this board now working with > 1.0 V after preheating the pcb at bfgminer startup, if it is resets ~1.0 V before I made cold starts? The board is now running stable for 36 hours.
Another board with the same issue is working after preheating with 29 GH/s at 1.007 which has been reset at cold start ~ 0.95 V before.
I have no scope here to check the ripple and voltage near to the bfl chips with ones which have not all these issues. But in fact, more or less all 7 boards which I have now assambled have the same issue, but 5 of them are starting without active cooling of the FETs without preheating mostly.

Cheers and sorry for my mixed sentences...

There are dedicated sense lines near the ASIC for measuring the voltage. The boards I produced use a 2Oz copper top and bottom layer, as do some of the boards Lucko made (the others are 1.5Oz, which shouldn't cause a huge difference).

The reason you don't see an increase in the temperature in bfgminer when you heat the temperature sensor is that bfgminer is reporting the hottest of the ASIC temperatures. I am not sure what is going on with that other board that runs at 29GH/s and 1.007V. Something is obviously still wrong with it as it should be working better than that. How many cores are enabled on the chips?



For everyone, I am waiting for some information from Lucko on what changes might have been made to the gerbers by the board house and whether the other boards with low voltage problems are causing similar issues to what I was seeing. I really do want you all to get boards the function properly and at full capacity, as I know there are a lot of guys out there who have been waiting a really long time to get a product. Those who have been through the wringer like dwdoc and the rest of you are the reason why I've been coming in to investigate this on the weekends, which is over and above the original agreement to just provide the board files and programming information. I do not appreciate nasty PMs and emails, and it will not help you get your boards any faster.

[Lucko]

Yes but MrTeal has one of the worst boards where symptoms are biggest... That one can't be run at more then 27GH. Some of the rest can even get to 42GH... So just slow preheat might solve 90% of the problems... I do need to remove power supply heetsinks and stop the CPU fans to avoid one board crashing at 27(power supply) and 41(fan) GH... I know that 41 crash can be solved with firmware but I just didn't get to it... And that would also mean that it will not hash at 39 but 36GH... So not sure but I think that the problem is only at a some point in voltage increase and only if right conditions are present...

The 2 I have assembled both crash at:-

40c  120/140w at the wall  27Gh at 5 sec    5/10Gh at the pool   HW10%/15%

when they re-set the log page scrolls very quickly:-

BFL 0: Received unexpected queue result response:
BFL 0: Error: Get temp returned empty string/timed out

EDIT:-
3rd unit almost exactly the same:-
42c    120/140w       26Gh at 5 sec      5/10Gh at pool        HW 30%

EDIT
4th unit SUCCESS HASHING SPOT ON 

69c   power ?  w     37.5Gh at 5 sec     36.5Gh at pool     HW 3%

I WILL POST SOME DIAGNOSTICS HERE LATER TODAY

Not sure what is going on. Every board I did send out I manage to get it working. Test were done alone on a power supply so that might be a factor (or even power supply might be a factor as asjfdlksfd figure out) but all did run in less then 10 restarts. Ambient temperature might be a factor... Did you add any fans on a power supply? I notice that if I do that I have problems to. Need to add them after startup. How many restarts did you attempted? What I do is start all boards but connect then two by two to computer. Make two going and go to the next... But it takes time to make them run... But when they do they don't have a problem unless they get to close to 40GH to 42GH... All I did was testing ways to get it run until it run. You might use heat from one to preheat another. Not sure what order you have them since it took all boards asked my girlfriend to move them around and then put them we pack them but I would guess that top are worst hashing and at the bottom you will get better hashing units... But depends how good job of mixing did she made...

[Mudbankkeith]

Yes but MrTeal has one of the worst boards where symptoms are biggest... That one can't be run at more then 27GH. Some of the rest can even get to 42GH... So just slow preheat might solve 90% of the problems... I do need to remove power supply heetsinks and stop the CPU fans to avoid one board crashing at 27(power supply) and 41(fan) GH... I know that 41 crash can be solved with firmware but I just didn't get to it... And that would also mean that it will not hash at 39 but 36GH... So not sure but I think that the problem is only at a some point in voltage increase and only if right conditions are present...

The 2 I have assembled both crash at:-

40c  120/140w at the wall  27Gh at 5 sec    5/10Gh at the pool   HW10%/15%

when they re-set the log page scrolls very quickly:-

BFL 0: Received unexpected queue result response:
BFL 0: Error: Get temp returned empty string/timed out

EDIT:-
3rd unit almost exactly the same:-
42c    120/140w       26Gh at 5 sec      5/10Gh at pool        HW 30%

EDIT
4th unit SUCCESS HASHING SPOT ON  

69c   power ?  w     37.5Gh at 5 sec     36.5Gh at pool     HW 3%

I WILL POST SOME DIAGNOSTICS HERE LATER TODAY

Not sure what is going on. Every board I did send out I manage to get it working. Test were done alone on a power supply so that might be a factor (or even power supply might be a factor as asjfdlksfd figure out) but all did run in less then 10 restarts. Ambient temperature might be a factor... Did you add any fans on a power supply? I notice that if I do that I have problems to. Need to add them after startup. How many restarts did you attempted? What I do is start all boards but connect then two by two to computer. Make two going and go to the next... But it takes time to make them run... But when they do they don't have a problem unless they get to close to 40GH to 42GH... All I did was testing ways to get it run until it run. You might use heat from one to preheat another. Not sure what order you have them since it took all boards asked my girlfriend to move them around and then put them we pack them but I would guess that top are worst hashing and at the bottom you will get better hashing units... But depends how good job of mixing did she made...

I remixed them when checking the bent pins, so my assembly is now random.
The first 2 boards were in parallel for about 2 hours, re-flashed them to the 1v1 and messed about for about another 8 hours.(between other jobs).
The 3rd board was this mornings excersize and was just the same, then I re flashed all 3 back to the standard software. The 3 boards were then on and off all morning.
Including swapping power supplies.
some good news in my next post.

[Mudbankkeith]

The first 2 Chili's I have assembled both crash at:-

40c  120/140w at the wall  27Gh at 5 sec    5/10Gh at the pool   HW10%/15%   (restart and run for about 2 mins then crash again)

when they start to re-set the log page scrolls very quickly:-

BFL 0: Received unexpected queue result response:
BFL 0: Error: Get temp returned empty string/timed out

The 3rd unit almost exactly the same:-
42c    120/140w       26Gh at 5 sec      5/10Gh at pool        HW 30%  (restart and run for about 2 mins then crash again)


The 4th unit SUCCESS HASHING SPOT ON
69c   217w     37.5Gh at 5 sec     36.5Gh at pool     HW 3%  (started up, ran up to about 55c  30Gh, I added some gentle cooling to mosfets, then, up to 70c and 37.5Gh)

The 5th unit SUCCESS HASHING SPOT ON
69c   170w     35.3Gh at 5 sec     34.1Gh at pool     HW 1.7%  (started up, ran up to about 70c  30Gh, then dropped back to 61c 33Gh, I added some gentle cooling to mosfets, then, up to 70c and 35.3Gh)

The 6th unit SUCCESS HASHING SPOT ON
70c   192w     37.5Gh at 5 sec     34.1Gh at pool     HW 3.9%  (started up, ran up to about 70c  39Gh, then dropped back to 60c 35Gh, I added some gentle cooling to mosfets, then, up to 70c and 37.5Gh)

The 7th unit fails.
 On power up, the leds 5,6,7,8, come on and stay on, then leds 1,2,3,4, flash once.
The com port is not found. nothing else happens.

The 8th unit SUCCESS HASHING SPOT ON
67c   200w     36.1Gh at 5 sec     33.1Gh at pool     HW 5.5%  (started up, ran up to about 65c  36.8Gh, then dropped back to 53c 33Gh, I added some gentle cooling to mosfets, then, up to 65c and 36.1Gh)

The 9th unit SUCCESS HASHING SPOT ON
70c   182w     36.3Gh at 5 sec     32.8Gh at pool     HW 5.6%  (started up, ran up to about 71c  37Gh, then dropped back to 63c 33Gh, I added some gentle cooling to mosfets, then, up to 70c and 36.3Gh)

[Lucko]

It depends when you look at frequency. It starts low and go up... So if you look at it at startup it will be low but if you have mined with it it will go up...

[asjfdlksfd]

Hi,

I assembled now #9 and have exactly same problem as described. It resets allready at ~ 1.0 V at ~ 29 GH/s.
So I used my soldering heat gun to preheat the FETs also. It reaches 48 °C faster than voltage reaches 1.0 V so now it is stable running after a while with 32 GH/s at 50 °C max on BFL Asics. I guess if temperature shown in bfgminer Manage device section is not the regulator ones, how can I see the regulators temperature or the temp of U25 near by the regulators?

Here the menu of my #8 bfgminer results after running 36 Minutes:

 BFL 1 : 50.0C | 33.88/32.95/31.93Gh/s | A:263 R:3+3(1.2%) HW: 492/2.9%
  BitFORCE SHA256 SC from MrTeal and ChipGeek
Serial: FTX3113X
Kernel: bulk queue
Temperatures: 42.0C 50.0C
Voltages: 3.290 / 1.038 / 11.656

Is 42 °C the vrm temperature? 50° C is looks like the ones from the hottest bfl chips, I understood.

I think tomorrow I can assemble my last to boards for further tests.

[asjfdlksfd]

Hi,

one of my boards which has been runned since last day now after power cycle whole the time LED7 is blinking. I understood its overtemp failure of FETs?
It's hard to mine stable if the FETs must cool down because after power switch I must preheat again the board before it will run again. So the chips must be preheat again before the board will not further reset.
I will cool down now the board to see more tomorrow.
Cheers....

[asjfdlksfd]

There are dedicated sense lines near the ASIC for measuring the voltage. The boards I produced use a 2Oz copper top and bottom layer, as do some of the boards Lucko made (the others are 1.5Oz, which shouldn't cause a huge difference).

Ok, with sense lines the voltage at the asics should been stable and not different from yours.

The reason you don't see an increase in the temperature in bfgminer when you heat the temperature sensor is that bfgminer is reporting the hottest of the ASIC temperatures. I am not sure what is going on with that other board that runs at 29GH/s and 1.007V. Something is obviously still wrong with it as it should be working better than that. How many cores are enabled on the chips?

I meant the other voltage from the Temperature line which is on bfl jally the temp near of the FETs for Vcore. I thought it comes also from the FETS (I mean the bolded 49° not the 51° C)

 BFL 0 : 51.0C | 28.70/28.46/27.23Gh/s | A: 5909 R:33+1(.56%) HW:14077/3.6%
  BitFORCE SHA256 SC from MrTeal and ChipGeek
Serial: FTX33YKB
Kernel: bulk queue
Temperatures: 49.0C 51.0C
Voltages: 3.290 / 1.032 / 11.890

EDIT2: MrTeal what about firmware that will run board for 5 minutes at 0,9V and then go up? It might be solution for many boards if preheating is all that is needed...

I asked on december 29th, if the "gentle preheat" could be done from the firmware.

  Question:-
   Maybe the (firmware) needs a bigger time delay between the voltage steps, to give the chips and temperature more time to stabilize, before the next step up.
   This may also help with the low temperature start up problems (sub zero ambient) some users are reporting. (gentle pre-heat)

  Reply:-
   The issue I am seeing doesn't really have anything to do with the voltage steps. What is happening is that PWM for the first phase is causing the second phase to turn on prematurely, which is causing a
   whole host of issues.

Could this help? If I've assembled all my boards I will try to reflash the ones first with the 1.1 V patch. Some boards have only problems at 1.16 V so this could help for these ones. For all others maybe a patch as suggested above could help, also why not try this?

[Mudbankkeith]

It depends when you look at frequency. It starts low and go up... So if you look at it at startup it will be low but if you have mined with it it will go up...

I was looking at the individual chip frequency shown by the Chili Flash utility.

[Mudbankkeith]

Hi,

one of my boards which has been runned since last day now after power cycle whole the time LED7 is blinking. I understood its overtemp failure of FETs?
It's hard to mine stable if the FETs must cool down because after power switch I must preheat again the board before it will run again. So the chips must be preheat again before the board will not further reset.
I will cool down now the board to see more tomorrow.
Cheers....

No hot air soldering gun here, but when the good lady goes out tomorrow the hairdryer is going to be tested. its worth a try.

The Irony of this is, when its up and running, the mosfets may need cooling again   

[Keefe]

I don't have any boards from Lucko, only one from MrTeal from months ago. But I thought I'd share something I've been using to monitor chip speeds, temps, and number of active engines per chip, while mining. I'm running bfgminer 3.8.1 on linux. This is not a good way to do it, and sometimes it interferes with bfgminer, so YMMV. If someone knows how to patch bfgminer to include this command, either on demand by keystroke, or preferably periodically for logging, I'd appreciate it.

Code:

~$ chiliport=/dev/ttyUSB1 ; echo -n "ZcX" > $chiliport ; timeout 0.01s cat < $chiliport
DEVICE: Chili SC
MANUFACTURER: MrTeal and ChipGeek
FIRMWARE: 1.2.14e
CHIP PARALLELIZATION: NO
QUEUE DEPTH:40
PROCESSOR 0: 12 engines @ 286 MHz -- MAP: F74F 70 C
PROCESSOR 1: 15 engines @ 309 MHz -- MAP: EFFF 63 C
PROCESSOR 2: 16 engines @ 313 MHz -- MAP: FFFF 61 C
PROCESSOR 3: 15 engines @ 316 MHz -- MAP: FFEF 68 C
PROCESSOR 4: 16 engines @ 278 MHz -- MAP: FFFF 63 C
PROCESSOR 5: 15 engines @ 270 MHz -- MAP: FFF7 69 C
PROCESSOR 6: 15 engines @ 305 MHz -- MAP: FFFD 77 C
PROCESSOR 7: 16 engines @ 297 MHz -- MAP: FFFF 76 C
THEORETICAL MAX: 35.63 GH/s
ENGINES: 120
FREQUENCY: 296 MHz
CRITICAL TEMPERATURE: 0
TOTAL THERMAL CYCLES: 0
XLINK MODE: MASTER
XLINK PRESENT: NO
OK

Here's what bfgminer reports at the same time:

Code:

 BFL 0: 67.0C | 35.51/36.34/32.21Gh/s | A:6471 R:3+0(.05%) HW:49474/ 11%
--------------------------------------------------------------------------------

Select processor to manage using up/down arrow keys
 BFL 0 : 67.0C | 35.56/36.34/32.21Gh/s | A:6471 R:3+0(.05%) HW:49464/ 11%
  BitFORCE SHA256 SC from MrTeal and ChipGeek
Serial: FTWWY6BD
Kernel: bulk queue
Temperatures: 33.0C 67.0C
Voltages: 3.290 / 1.129 / 11.437

I keep hearing that the reported temp is that of the hottest chip, but the ZcX response seems to disagree. It seems more like bfgminer is reporting the average. If you do ZlX (lower-case L), it gives just the temps but with more precision, rounded to 1/4 degree. ZlX seems to work only when mining, not when idle.

[asjfdlksfd]

I'm running bfgminer 3.8.1 on linux. This is not a good way to do it, and sometimes it interferes with bfgminer, so YMMV. If someone knows how to patch bfgminer to include this command, either on demand by keystroke, or preferably periodically for logging, I'd appreciate it.

I asked in Luke-Jr forum https://asktom.cf/index.php?topic=168174.1880 allready for extension of the code. It would be nice to see them in "Manage devices" section to get the ZcX command.
In my opinion the Information menu is possible to used for. The related file looks like for me is driver-bitforce.c and bfgminer-driver-bitforece.o.
I didn't find an answer yet. So it would be nice that others ask them for extension, too

Cheers...