OTA EPG grabber

[nzsjb]

I am the author of a popular, free, open source OTA EPG grabber for Windows named EPG Collector. I currently have around 1500 users in Europe, Australia, New Zealand and a few other places.

Someone recently posted on the forum on Sourceforge asking if I intended to develop it for North America. Unfortunately, even though I replied that I was he didn't reply back.

I am very interested in implementing it for North America which I assume means implementing PSIP. I have the documentation and it doesn't look that difficult but I will need a transport stream dump (around 60 seconds) and someone willing to do some testing once I have it working.

Anyone interested in helping?

Cheers

Steve

 

[DRCars]

Is there some way to use TSReader Pro to get the TS that you want? I could give it a try. I don't know much about TS analysis but with some coaching I would be willing to try. However, I hope someone with more experience will respond.

Edit: See post below for info about what I have tried so far using TSReader and DVBViewer. I am willing to "learn" to help.

DRCars

 

[SatelliteAV]

Do you need TS on Terrestrial ATSC or DVBS/S2?

 

[DRCars]

Since the OP mentioned "OTA" and "North America" I assumed Over-the-Air (Terrestrial) ATSC. This thread probably belongs in the Digital Over the Air forum section. That would have eliminated any confusion.

What would you use to get a raw dump of the TS?

I tried TSReader and it ignores the EIT (EPG) data when doing a mux record. The only way I could see the EIT (EPG) data was by recording the PIDs for it. However, that is not a raw dump and is not an image of the original trransport stream. It is a small file about 1 MB but is only the EIT data and is not mixed in with video, audio or other data.

If I do a "PID Dump" and select all PIDs with DVBViewer it does include the EIT (EPG) data. I can't swear it is exactly in the same order or an exact image of the original TS though but it appears to be since the EPG data is distributed all throughout the data file. 1 Minute is 136 MB. It is a mux with 4 channels 3 of which have EIT (EPG) data.

DRCars

 

[nzsjb]

I thought he meant ATSC as well. That's what I have the documentation for.

Can you upload that file to a file sharer somewhere and let me have the link and I'll take a look. I have my own analysis software which can show me the contents of any PID.

Thanks for your help on this.

Cheers

Steve

 

[DRCars]

Which file? The one that has video in it too (huge) , or the one with only EIT data (small)?

Edit: Never mind. I put both files and a log file into a zip and did a PM to you with the link to it.

DRCars

 

[nzsjb]

Which file? The one that has video in it too (huge) , or the one with only EIT data (small)?

Edit: Never mind. I put both files and a log file into a zip and did a PM to you with the link to it.

DRCars

Hi,

I've done a quick analysis of the dumps and it looks like all the PSIP pid's/tables that I need are there so that's a great start.

All I need to do now is write the code that analyses them!!!!.

I should think it will take me 3-4 days. I'll be back in touch once I can process the dumps and produce an xmltv file.

Thanks for your help on this.

Cheers

Steve

 

[DRCars]

You're welcome.

I'd like to give it a try when you get it going. I've yet to run across something that actually works right. DVBViewer doesn't handle ATSC EPG right (puts last subchannel data in as the primary channel and doesn't display any data for the subchannels). Mytheatre has no support for ATSC EPG at all and I've never been able to get any ATSC EPG data with Mediaportal either.

Keep us informed please.

DRCars

 

[nzsjb]

Hi,

What tuner and version of Windows do you use?

Steve

 

[DRCars]

I am using a Pinnacle PCTV 800e USB tuner which has a BDA driver and have found it to work with TSReader, DVBViewer, MyTheatre, MCE 2005, and MediaPortal. I am running XP SP3 MCE 2005.

DRCars

 

[nzsjb]

An update for you.

I have processed the dumps and can produce an xmltv file containg EPG for 4 stations (KETS-1 thru KETS-4). KETS-4 only has one entry which for some reason is dated last December. The other 3 stations have 12 hours of EPG for the 13th January. There is audio and parental rating info about each program but no genre/category.

I just need to do the ATSC tuning code and I can let you have a version to try. There is very little info around on ATSC tuning so maybe you could tell me what parameters you normally set to receive broadcasts. As far as I can tell it is just frequency, physical channel, major channel and minor channel.

Cheers

Steve

 

[Anole]

The frequency = the secret channel
The indicated channel is from a table, somewhere.
So, channel 2 (CBS in LA), might actually be broadcast on the frequency of 43.
edit: It appears the terminology "physical channel" means "2" in this case; the one the user sees.

Subchannels would be 2.1, 2.2, 2.3
Generally the x.1 is the main channel. (There is no 2.0)
x.2, x.3 etc, are often news, weather, advertizing, but...
In some instances there will be 10 subchannels, all with some different sort of programming.
At least in the Los Angeles, California, USA area.

 

[DRCars]

An update for you.

I have processed the dumps and can produce an xmltv file containg EPG for 4 stations (KETS-1 thru KETS-4). KETS-4 only has one entry which for some reason is dated last December. The other 3 stations have 12 hours of EPG for the 13th January. There is audio and parental rating info about each program but no genre/category.

I just need to do the ATSC tuning code and I can let you have a version to try. There is very little info around on ATSC tuning so maybe you could tell me what parameters you normally set to receive broadcasts. As far as I can tell it is just frequency, physical channel, major channel and minor channel.

Cheers

Steve

Thanks for the update Steve.

The data you described is just as I had expected. My TV EPG showed no data for KETS-4. It wouldn't have shown the bit from December.

Programs that I have used appear to use the actual RF frequency (relates to actual channel number as shown at first link below) to tune the channel then get the virtual channel number using the Terrestrial Virtual Channel Table (TVCT). The TVCT also provides linkage to the EIT data.

Here is a link to some information on broadcast television in North America (including a list of channels and their frequencies). Each channel is 6 MHz wide (Symbol rate = 6000 KHz) so for center tuning you want to use "Lower Edge" + 3. For example Channel 7 is at center frequency 174+3 = 177. Also, note that TV channels now stop at 59, so no need to check for channels over 59.

North American broadcast television frequencies - Wikipedia, the free encyclopedia

I'm a little out of my range of knowledge here but, if you don't already have the info, here is a link to what appears to describe the VCT as well as other stuff:

Terrestrial Virtual Channel Table (TVCT) [ATSC Technology]

Here is a link to technical data for the KETS-DT stream that I generated using TSReader and sent to rabbitears.info:

SI Parsing by TSReader 2.7.45h

The example Anole cited is appropriate since, coincidentally, the 4 virtual channels (KETS) in the stream are 2.1, 2.2, 2.3, 2.4 and are on actual (RF) channel 7 (177 MHz, 177000 KHz).

DRCars

 

[nzsjb]

Hi,

Thanks for the links. I have all the protocol specs as published by ATSC and they are pretty easy to follow especially as far as PSIP is concerned. I have also found the Wiki cable broadcast frequencies.

How do you want me to deliver the software to you? It's packaged up as an msi so its a simple install. No drivers or registry changes and it's around 5Mb.

I've done the tuning code and simulated it as far as I can but it's going to be suck it and see basically. There are comprehensive log files created and plenty of debugging options built in (found to be necessary from the other protocols I have developed without having a live feed).

Steve

 

[Babadem]

This is a very interesting development.

 

[DRCars]

Hi,

...snip...
How do you want me to deliver the software to you? It's packaged up as an msi so its a simple install. No drivers or registry changes and it's around 5Mb.
...snip...

Steve

I thought maybe you would put it on the EPG Collector sourceforge page with some sort of ATSC beta description about it but, if not, a RapidShare link would be okay.

DRCars

 

[DRCars]

I downloaded the EPG Collector tryout. I'll give it a whirl and see what happens.

DRCars

 

[DRCars]

I installed EPG Collector and set it up as you described (I think). I used a tuning frequency of 177000 (same channel as I sent data for) and chose to use the Pinnacle 800e USB BDA tuner (tuner 2). I was uncertain whether to use PSIP or EIT collection so I tried both. Both quit with an error code 6 as you can see in the attached logs. I have also attached the EPG Collector INI files.

One time, after I installed EPG Collector, I tried to immediately tune a ATSC channel with TSReader (did this to verify proper tuner operation). I got a "Can not run graph!" error. This happened even before I ran EPG Centre. I had to uninstall EPG Collector and reboot to get TSReader to work again. A simple reboot did not suffice and I had to remove EPG Collector to get TSReader to tune the channel again.

I mention this because TSReader fails to tune just like EPG Collector after EPG Collector is installed. I have also tried NOT running TSReader after installing EPG Collector to verify that running TSReader is not effecting EPG Collector. Those logs are the ones that are attached. I can't say exactly why but this seems to be significant to me.

BTW, when it fails it does so almost immediately. There doesn't seem to be a timeout wait of any sort.

DRCars

 

[nzsjb]

Thanks for the logs, I'll get onto them straight away today.

The issues with TSReader are very strange. As I said before the install doesn't put anything on a machine other than what you see in the install directory. It doesn't change any registry settings, BDA settings, drivers etc so the reaction of TSReader to the install is weird. I test on Win XP SP3 by the way. I have TSReader Lite installed along with other DVB software such as DVBStreamExplorer and DVBViewer.

Steve

 

[DRCars]

Here's what happened (as Monk would say). When I was about to install EPG Collector the first time I wanted to have few system services running to keep things simpler. I had killed Mediaportal's TVService. It turns out anytime I start or stop Mediaportal's TVService it disrupts TSReader until the ATSC tuner is reset (e.g. unplug re-plug USB tuner). Not a problem at all with EPG Collector. Sorry for the red herring.

I have tried the latest incarnation and we got further this time but had error code 5. You will find the results attached. Appears to have something to do with that old black magic you know so well...DirectShow.

BTW, I see in the log FEC: Rate3_4. Does this mean an FEC of 3/4? I thought ATSC FEC was 2/3. I am pretty sure I read or deduced it somewhere in the ATSC documentation, but I could be wrong now. I'll look into it.

Changing Rate3_4 to Rate2_3 in ATSC\Terrestrial.xml did not help and the FEC: Rate3_4 still appeared in the log.

Can I install a new version over an old or should I uninstall first?

Edit:

Here is where I deduced that FEC is 2/3:

From: http://www.atsc.org/cms/standards/a53/a_53-Part-2-2007.pdf

6.1 Overview of Main Service

.... snip ....

Incoming data is randomized and then processed for forward error correction (FEC) in the form of Reed-Solomon (RS) coding (20 RS parity bytes are added to each MPEG-2 packet), one-sixth data field interleaving and two-thirds rate trellis coding.

DRCars