Questionable cause of HWS hard drive failure message

[tennisnut]

Saw a hard drive failure error message (970?) this evening when I turned on the TV. It's the one that says "Your timers have been saved but any recordings have been lost..." and telling me to select OK to reformat the hard drive. It wasn't exactly a surprise, since the box acted flaky a couple of times this week, and two shows failed to copy to the EHD. However, the CSR I spoke with claimed that a likely cause of the failure was operating with the DVR hard drive too full.

Since practically every word out of her mouth was from a script, she couldn't give me any plausible reason why a relatively full HD would cause such a problem or tell me where such a claim is made in writing. Granted, we've had the drive up around 91-92% at times, but I had cleaned some things out earlier in the week, and the HD was about 76% full yesterday night. Does anybody know whether she is right? Technically sound reasons preferred, but SWAGs could be entertaining.

BTW, reformatting is a joke; it took less than 5 minutes, so there's no way bad physical sectors on the HD could all be detected and flagged from further use. These hard drives get a workout, but DISH should be buying higher quality hardware and/or using spindles in a RAID array. Regardless, they're sending another HWS, probably rebuilt. It's a good thing the contract expires in September...

 

[Jim S.]

I don't know about the too-full part because I've never got there, but otherwise I fully agree with you that their filesystem handling is highly suspect! You can search my posting history for the reasons why, I don't have time to type it all again.

 

[rglore]

There's been many reports of false hard drive error messages. Try doing a red button or power cord reset to get rid of the message and return to normal operation.

 

[Skytrooper]

My original bedroom Hopper had the hard drive failure and I lost all my recordings. I got a replacement Hopper and it occasionally gets the hard drive failure message. The red reset button does not work when this happens. Power Cord Reset must be used. I don't even use the red reset button anymore.

I sometimes question Dish's replacement Hoppers. I wonder if they are sending you someone else's problem Hopper that was not properly repaired.

Why I say this is that my Family Room Hopper has never had a problem. Lucky that at least I got one that is dependable.

 

[KAB]

I am sure this will draw some flack, but here goes...

Been a couple of years since some many of us warned about keeping units well ventilated. Some heeded, some did not. Since then, seems like a lot of new reports of failure have come from new members to the forum. So, my two cents...keep them out in the open, if not, keep a fan of some sorts moving air. If not, you will be calling for replacements...and blaming Dish for it.

 

[Tampa8]

Even in the open in the Summer even with air conditioning it may get warm enough that a small computer fan or "room to room fan" blowing across the unit will make a difference. Since keeping my 612's ventilated it has been years with no real problems.

 

[JSheridan]

Ventilate, ventilate, ventilate...

 

[JeffN9]

Found this thread after doing a search. My 2 1/2 year od H2K started flaking out last night. I noticed when trying to skip back the red, can't record symbol was appearing on the screen. When I opened the DVR all recordings were gone. Tried doing the red button reset twice with no help. Then I pulled the power cord. When it re-booted I had Error message 760, hard drive failure. It also said timers are saved but all recordings have been lost. It also wanted me to re-format the drive which I didn't do fearing that it really would erase everything. I then unplugged it over night but had the same result this morning. Arrgh! Needless to say the wife was pretty upset that almost an entire season of Mad Men was lost plus lot's of other stuff. I have two external drives attached to it. All of those recordings are still there but can't be played. When I try to play one it immediately says that the recording ended. Even with all the stuff we had recorded on the internal drive it was only about 43% full. I keep the Hopper in the lower level of the house which is always cool. It also has plenty of space around it for ventilation.

Is there anything I can try to get the recordings back or are they lost forever? I remember the hard drive on a 922 I had started dying but I was able to get it back long enough to transfer everything to a EHD.

 

[KAB]

Is there anything I can try to get the recordings back or are they lost forever? I remember the hard drive on a 922 I had started dying but I was able to get it back long enough to transfer everything to a EHD.

If a power reset did not bring them back, they are gone.

 

[Jim S.]

But the point is that they shouldn't be! If the drive hasn't suffered a complete failure, which it hasn't because it can be reformatted, then Dish's software should try to salvage what it can, NOT just reformat it!

 

[tennisnut]

Is there anything I can try to get the recordings back or are they lost forever?

Returning to my original post, I'm still convinced that my HD failure had nothing to do with how full it was!

I can only speculate on an answer to your question, but you could try something like the following. In general (PC or DVR), hard drive failures occur gradually. It's not unusual for a PC HD to have bad physical sectors that are never detected (because they aren't accessed or don't always fail to read) or that present themselves as file reading errors (which don't prevent accessing other files). If critical system files (e.g. master boot record, physical sector allocation tables, directories) are bad, that's another story. However, a "bad" physical sector isn't necessarily unrecoverable. Sometimes the failures are "soft" failures, which only occur occasionally. Utilities such as SpinRite can often recover such data by doing multiple low level reads of the sector(s) and analyzing the data.and error codes read.

I've never tried to read or analyze a Hopper HD, but from reading other posts, I suspect that it uses Linux-based External File System (EXT) partitions. You could pull the HD from your DVR and try to read it with a Linux system. If my assumption is correct, you could then look for utilities capable of recovering physical sectors in the manner described above. SpinRite is capable of working with EXT files, but it's also possible that ES uses another disk organization system or even their own proprietary system. However, even then there are other utilities out there capable of recovering sectors and copying a "bad" HD to a new HD, regardless of disk formats or file systems. There are also services that will attempt to recover data, but I have no experience with them. Even after trying the above, there's no assurance that you will recover your recordings.

But do you really want to go through all of that pain, and possibly violate contractual terms, when you could purchase many of the things you recorded? I don't.

I don't know for sure, but I suspect that that the Hopper uses a very primitive, but fast, disk/file format with errors only detected when a recording or critical system file has been corrupted or become unreadable. I wish we could get DISH to employ commonly available tools that monitor the condition of a HD during idle times, attempt to repair or move bad sectors, and warn us to back up (not just move) recordings to an external HD before the internal HD fails, so that they can be restored to a new Hopper. Alternatively, they could provide the capability of continuously mirroring (a la RAID 1) the Hopper HD on an external HD, so that it could be used as a backup to restore the recordings. However, when they don't even fix the bugs that still exist, can we expect them to address reliability? Sorry I'm such a pessimist...

 

[JeffN9]

After trying another hard re-boot last night (which failed to do anything) I decided to go ahead and click ok to re-format. It seemed to take longer than 5 minutes and when it was done wanted me to wait another 5 minutes or so for the dvr to update and the guide data to load. It almost acted like it was reloading the entire operating system which made me wonder if the last software update screwed something up?? Anyway everything seems back to normal now except that all the internal drive recordings are gone. Of course I don't trust it to remain stable and will be requesting a replacement. This time I will want a HWS.

 

[KAB]

After trying another hard re-boot last night (which failed to do anything) I decided to go ahead and click ok to re-format. It seemed to take longer than 5 minutes and when it was done wanted me to wait another 5 minutes or so for the dvr to update and the guide data to load. It almost acted like it was reloading the entire operating system which made me wonder if the last software update screwed something up?? Anyway everything seems back to normal now except that all the internal drive recordings are gone. Of course I don't trust it to remain stable and will be requesting a replacement. This time I will want a HWS.

"Replacements" will be of like kind. You will need to upgrade to HWS. Contact Dirt.

 

[tennisnut]

This time I will want a HWS.

I hope you get a new one. I just activated the second replacement HWS since the original post on 5/15. The first replacement was a remanufactured unit with a Seagate HD. It worked OK for 16 days and began failing with a different HD failure (error 311). This one didn't require "reformatting" (what a joke!), but all recordings since the failure date were painful to watch - stalls, skips, pixelation, etc. This second replacement, also remanufactured, has a Western Digital model WD20EURX HD, which has received good reports from others in the forum. So far so good.

Good luck!

 

[DishSubLA]

Well, usually HDD's do Wear-leveling, meaning the HDD's tries not to use the same sectors or portions of the disk all time or too often, which can cause a failure far sooner if the drive just used the same available portion of the HDD and will instead use the other parts of the HDD that have not been used as much or at all to help the HDD wear more evenly and last longer. However, when the drive gets close to full, as in a DVR, then it becomes very difficult if almost impossible to perform Wear-leveling. I'm not saying this is what happened in your case (we will never know), but what the CSR said from her/his script is correct in the sense of how we know HDD's work and how they can die. I have no doubt there are more than a few instances of the DVR's HDD failing because the HDD was forced to keep using such a small portion of it again and again, and could not perform wear-leveling.

My last Dish HDD failure did occur when I was nearly full. I do get nervous when I am in the 90% because I know the small portion left is gonna bet POUNDED by my FREQUENT DVR recordings and playback. I usually move stuff off the DVR HDD to the external HDD so that more of the DVR HDD can be used and allow wear-leveling. This last time, I did NOT move the recordings over in time, and I KNEW the HDD was gonna fail if I kept putting off the chore. Then it started doing things it does when HDD failure is imminent. I lost quite a few recordings, but I really do blame myself because I knew better.

Yes, you are correct, no good inspection of the HDD can occur in such a short time. But I think Dish is afraid customers will NOT tolerate a good long, full re-format or disk check nor the time-consuming, but thorough, checking (and re-checking and re-checking) like SpinRite performs. Dish HDD re-formats or most likely something like a "Quick Format" and not a proper one that can either mark as "do not use" or fix the problem sectors that could also result in the recovery of the recordings on the HDD, but that takes far more time than the average pay-TV subscriber will tolerate.

I really would like Dish to license SpinRite because it is so good and more than a few TiVo subscribers use it for their TiVo HDD's. SpinRite will work on ANY HDD and SSD. I would rather deal with a day or two (it can sometimes take less or more time than that depending on what level of inspection) of a SpinRite operation that will mark bad sectors, fix others, and have a good chance of recovering the recordings so that I can transfer them instead of losing all those recordings and having to start all over with a replacement DVR. But that's me, not the average Joe.

Again, your DVR HDD failure may not have been related to your HDD being so full, but it true that HDD's will fail far sooner if they can't perform wear-leveling.

 

[Foxbat]

Flash-based Solid State Disks (SSD) use "wear-leveling" as the act of writing to a block of flash memory actually is a destructive process. Wear-leveling allows the SSD to make best use of all its flash memory blocks since there are no moving parts, thus no "seek" time when accessing data.

Magnetic rotational media (a.k.a. HDD) can write to the same location over and over with no damage to the media. HDDs may use a "surface Scan" in which the drive uses idle time to verify the tracks of the disk are still usable and mark any bad blocks so they won't be used, but most HDDs don't try to randomize the location of data on the platters as that would have an adverse effect on access times. HDDs work best when the data are stored contiguously to minimize head movement. Bad-Block replacement happens when the drive determines the error rate on a particular block has exceeded a threshold set by the manufacturer and the drive relocates the block to a reserved area of the disk. Unfortunately, the data now takes longer to access as the reserved area is rarely located close to where the rest of the data resides, meaning the head has to skip back and forth to read the desired contents.

We've used SpinRite a few times at work to save user's data, I agree it's a good tool for taking a "flaky" HDD and making it usable again. My problem is that knowing that the drive flaked out would destroy my trust in the device, so I tend to use SpinRite to prepare the source drive for a full backup to a replacement drive.

 

[tennisnut]

At this point, I would be happy if they would just monitor the S.M.A.R.T. parameters and warn us when the drive shows signs of failure. One week into replacement unit #2 and counting!

 

[DishSubLA]

Flash-based Solid State Disks (SSD) use "wear-leveling" as the act of writing to a block of flash memory actually is a destructive process. Wear-leveling allows the SSD to make best use of all its flash memory blocks since there are no moving parts, thus no "seek" time when accessing data.

Magnetic rotational media (a.k.a. HDD) can write to the same location over and over with no damage to the media. HDDs may use a "surface Scan" in which the drive uses idle time to verify the tracks of the disk are still usable and mark any bad blocks so they won't be used, but most HDDs don't try to randomize the location of data on the platters as that would have an adverse effect on access times. HDDs work best when the data are stored contiguously to minimize head movement. Bad-Block replacement happens when the drive determines the error rate on a particular block has exceeded a threshold set by the manufacturer and the drive relocates the block to a reserved area of the disk. Unfortunately, the data now takes longer to access as the reserved area is rarely located close to where the rest of the data resides, meaning the head has to skip back and forth to read the desired contents.

We've used SpinRite a few times at work to save user's data, I agree it's a good tool for taking a "flaky" HDD and making it usable again. My problem is that knowing that the drive flaked out would destroy my trust in the device, so I tend to use SpinRite to prepare the source drive for a full backup to a replacement drive.

I would also add that TiVo has a similar pattern: the two TiVo's I got directly from TiVo had each of their HDD's fail at startup (It would not boot-up) and the other just a few weeks after purchasing it. In each instance TiVo replaced the boxes only to have the same thing (failed HDD) happen again. With one STILL refusing to boot-up right out of the box and the other failing a few DAYS later. I was fed up with them and lost trust in their quality control of HDD's (I am assuming they probably re-use HDD's if software can "repair" or otherwise make the HDD usable) and had Weeknees do the HDD or replaced it myself. The score is Weaknees and me ALWAYS successful, never TiVo.

I agree that such early failure makes me doubt the device (HDD), as well. Considering, that both Dish and TiVo use high quality HDD's in their DVR's, it makes me think that if they get a "flaky" HDD as part of an RMA, they probably try some solution to revive it to a--presumed--sufficiently reliable HDD than can be put in another DVR box instead of just dumping those HDD's because we all know that after reviving it, it may still give us grief. That probably saves them a lot of money. It is the only reason I can think of for what seems like such a high failure rate of replacement DVR's HDD that are high quality HDD's to begin with.

I really don't know what's going on, but what do you think? It is a CLEAR pattern for both Dish and TiVo. Does DirecTV have the same DVR replacement failure pattern for its HDD? The thing is I and Weaknees use BRAND NEW HDD's and we just never have problems.

 

[Foxbat]

I have been out of the new consumer disk hardware arena for awhile, now I see Seagate and WU (and I'm sure the other manufacturers) have "Video"-class drives that specialize in constant writes and reading from different areas, something that would occur in a DVP or Surveillance application. I'm not sure how different the drives are from "normal" personal computer drives these Video drives are, but it sure sounds good on paper.

Since they cost more than regular drives, I'm willing to guess they wouldn't be what Echostar or TiVo is using, but they should.

 

[Tony S]

I have been out of the new consumer disk hardware arena for awhile, now I see Seagate and WU (and I'm sure the other manufacturers) have "Video"-class drives that specialize in constant writes and reading from different areas, something that would occur in a DVP or Surveillance application. I'm not sure how different the drives are from "normal" personal computer drives these Video drives are, but it sure sounds good on paper.

Since they cost more than regular drives, I'm willing to guess they wouldn't be what Echostar or TiVo is using, but they should.

You may not be right about that. About a month ago the Seagate drive in my HWS went bad. Dish replaced my HWS with a refurbished one. The replacement receiver had a WD WD20EURX drive in it. The manufacture date of the WD drive was 6 March 2015. That WD drive was brand new, and was specifically made for DVR use. So Dish does use the type of drive that 'you wished they would use'. If you look up the specs for the WD20EURX drive, you will find that Dish chose a very suitable drive with great specs.