Loading...

ivtv-users@ivtvdriver.org

[Prev] Thread [Next]  |  [Prev] Date [Next]

Re: [ivtv-users] Help With Troubleshooting Jeffrey Preston Mon Feb 13 11:00:07 2012

Andy,

Thank you for all the help.  I will use whatever resources I can in order to 
isolate and resolve this issue.  To further the research a little, I decided to 
try the Mythdora distro, and found some very interesting results.  Loading up 
and trying version 12.23 got me a 100% working system right away, without any 
modifications.  I then performed updates, and it all went to hell.  Didn't get 
the jumping distorted pictures, but it would lock up at a blank screen anytime 
I tried live TV.  I am going to go back and check the version of the backend / 
frontends, where it was working and not, but then this could be an entirely new 
issue, and not even related to the issue I am hunting down. 

I will post the information here when I get a chance.  Maybe it will help...

-----Original Message-----
From: Andy Walls [mailto:[EMAIL PROTECTED] 
Sent: Saturday, February 11, 2012 11:02 AM
To: Jeffrey Preston
Cc: [EMAIL PROTECTED]
Subject: RE: [ivtv-users] Help With Troubleshooting

On Thu, 2012-02-09 at 11:22 -0600, Jeffrey Preston wrote:
> Andy,
> 
> I have tried the following things:
> 
> 1 - I disabled Hyperthreading on my P4 3.2Ghz CPU, and for several 
> minutes, this seemed to resolve the issue, but then it started 
> happening again, and the picture went distorted, with lots of 
> skipping, and jumping.  MythTV would sometimes exit with "Error 
> opening jump program file", or "Video frame buffering failed too many 
> times".
> 
> 2 - I changed grub and rebooted the machine without the GUI loaded.
> So this accomplished two things....no nVidia driver loaded and no 
> Window managers running, so a fairly clean system as far as processes 
> running.  I started mythbackend, but then it exited at the end (maybe 
> it is supposed to do that).  I then started a remote frontend and the 
> first station it tuned to gave me the same trouble.  Re-started the 
> system with graphics again.
> 
> 3 - I ran the command 'v4l2-ctl --log-status' several times, but could 
> see nothing in the output that would give me any idea of the signal 
> quality or strength.  Maybe I am not looking at the output correctly.

You just want to see that a signal is present consistently.  I suspect it was.


> 4 - I issued the command "modprobe ivtv debug=0x4f", but could not 
> find any logs in the folder you specified.  I did look at the output 
> from "dmesg" and found what I believe to be the results of the debug 
> output from the ivtv module.  However, there is only so much space, it 
> seems, in the output from dmesg, and I could not find anything that 
> was there that would give me a clue as to what is going on.  Is there 
> a way to get the debug mode for the ivtv module to redirect its output 
> to a log file, so I can comb through it without worrying about the 
> buffer space in dmesg?

That's a syslog daemon configuration.  The ivtv driver uses KERN_INFO and 
KERN_DEBUG (and KERN_ERROR and KERN_WARN) logging facilities.

As root, in /etc/rsyslod.conf or /etc/syslogd.conf you would want to direct 
"kern.*" logging to /var/log/kernelmsgs or whatever.  Then send syslod or 
rsyslogd a SIGHUP to re-read the configuration:

        # kill -HUP (process ID of system log daemon)


> 5 - I looked to see if I could load IRQbalance as you suggested, and 
> found that it was already running under Mythbuntu 11.04.  I then 
> unloaded it, rebooted and tried again - no difference.

OK.

> I tried ftrace, but I believe it is beyond my abilities to configure 
> and run at the moment.  I do not know how to mount the debug file 
> system or enable the option on the Ubuntu kernel.

I agree, it is complex.  I really don't expect normal users to be able to use 
it as an effective troubleshooting tool.

>  I have seen the MythTV bug reports, but they do not seem to apply 
> exactly to my situation, however, I cannot help but think that they 
> are at least linked to what is going on with my system.  I do not 
> believe it is a latency issue, as adjusting the latency of the cards 
> and other things on my system made no difference in the behavior of 
> the analog tuner (PVR-150's) at all.

The PCI latency timer is just so that no one card can hog and hang the PCI bus 
(remember the days of ISA cards?).

The PCI bus has a Round-Robin arbiter for bus access, and the latency timer is 
the maximu amount of time that device can master the bus before having to yield 
the bus back to the next bus grantee by the arbiter.

You can only set PCI latency timer values in multiples of 8 BTW.  The PCI clock 
runs at 33 MHz which makes 8 cycles to be approximately 250 nanoseconds (242 ns 
actually).

The ivtv driver tries to set the card's latency timer to allow its internal DMA 
engine to have the PCI bus for 64 cycles (~2 microseconds) before the card has 
to yield the PCI bus to another waiting PCI device.

You might want to check the latency timers of any PCI bridges upstream of the 
PVR-150.  If their latency timer is shorter than the PVR-150s, then the latency 
timer of the PVR-150 doesn't really matter.

>   I also do not think it has anything to do with changing channels, as 
> I have had distorted video come up when first bring up my 
> frontend...any frontend.

> I know I should not let this get to me, but I had a working system 
> under Mythbuntu 8.10, and it doesn't work under 11.04.  I found that 
> the issue actually began, or has something to do with the changes 
> between Mythbuntu 8.10 and 9.10.  I don't know where the changes are 
> located that would affect analog tuners, but I will guarantee you the 
> source of the issue is there.

To find the root cause blindly but deterministically, with no guess-work, one 
would have to

- be able to reproduce your problem with a known bad kernel
- have a known working kernel version
- do a git bisect on the kernel source between those two versions,
- rebuild the kernel and install it and test it
- iterate the bisection process until the bad kernel change is isolated

Each iteration takes ~30 minutes for me.  Unfortunately that's time I really 
don't have available.  :( 


>   The problem still existed in Mythbuntu 10.10, as well.  
> 
> Perhaps someone that is familiar with the design architectures of 
> Linux and Mythbuntu could point us all in the right direction.
> 
> Andy, I want to thank you for your help so far.  I would not have 
> gotten this far without your help.  Thanks for taking pity on an aging 
> engineer.

You're welcome.  Sorry I don't have a lot of time available lately.

Aging engineers have to watch out for each other. ;)

Regards,
Andy


> -----Original Message-----
> From: Andy Walls [mailto:[EMAIL PROTECTED]
> Sent: Saturday, January 28, 2012 8:25 AM
> To: [EMAIL PROTECTED]
> Cc: [EMAIL PROTECTED]
> Subject: Re: [ivtv-users] Help With Troubleshooting
> 
> On Thu, 2012-01-26 at 18:39 -0600, [EMAIL PROTECTED] wrote:
> > ==============Original message text=============== On Wed, 25 Jan 
> > 2012
> > 9:40:11 pm CST Andy Walls wrote:
> > [EMAIL PROTECTED] wrote:
> > >>Hello,
> > >>I have two servers running Mythbuntu 11.04 and MythTV 0.24.1.
> > >>I have
> > >>three PVR-150 tuners in the server configured as a slave backend, 
> > >>and some digital tuners in the master backend.  The problem I have 
> > >>is that whenever the analog tuners are called for by any frontend 
> > >>(even a local one), I get distorted video, with stuttering and 
> > >>image freezes.  There is currently a bug report in with MythTV on 
> > >>a  Analog Channel Change  issue.
> > >>I have tried the work-a-round (tune a digital channel before using 
> > >>one of the analog tuners), but it does not work for me.  I finally 
> > >>saw some troubleshooting steps for IVTV drivers, and followed some 
> > >>of those steps.
> > >>The one that caught my attention, is the  cat /dev/video1 > 
> > >>test.mpg  test.  I looked at the file that was produced while the 
> > >>command was active, and VLC would only play a few seconds of the 
> > >>file and then skip to the end, and then terminate the viewing 
> > >>window.  The video jumped and was distorted, just the same as it 
> > >>was in MythTV.  So the issue, evidently, is not MythTV, but 
> > >>perhaps with IVTV or some related item.
> > >>On a related note, the same hardware works without issues, when 
> > >>running MythTV 0.21 under Mythbuntu 8.10.
> > >>What would you do next and where would you look for a>>resolution?
> > >>Thanks,
> > >>Jeff
> 
> > >What are the kernel versions shown with uname -a?
> 
> > Linux MYTHBE2 2.6.38-13-generic #54-Ubuntu SMP Tue Jan 3 13:44:52 
> > UTC
> > 2012 i686 i686 i386 GNU/Linux
> 
> OK, fairly modern; that's good.  Do you happen to know the kernel 
> version under Mythbuntu 8.10? (so I check the differences between the 
> two kernels)
> 
> > >What are the ivtv-driver versions shown with v4l2-ctl -d 
> > >/dev/video1 --log-status?
> 
> > ivtv1: Version: 1.4.2 Card: Hauppauge WinTV PVR-150
> 
> OK, good.
> 
> > >What other devices are shown sharing the interrupt lines with ivtv 
> > >using cat /proc/interrupts?
> 
> >           CPU0       CPU1
> >   0:        580          0   IO-APIC-edge      timer  
> >   1:       3649          0   IO-APIC-edge      i8042
> >   6:          3          0   IO-APIC-edge      floppy
> >   7:          0          0   IO-APIC-edge      parport0
> >   8:          1          0   IO-APIC-edge      rtc0
> >   9:          0          0   IO-APIC-fasteoi   acpi
> >  14:      63335          0   IO-APIC-edge      ata_piix
> >  15:    1294927          0   IO-APIC-edge      ata_piix
> >  16:    7153671    2827260   IO-APIC-fasteoi   uhci_hcd:usb2, 
> > uhci_hcd:usb5, nvidia
> >  17:      12238       7179   IO-APIC-fasteoi   ivtv1
> >  18:    5690498          0   IO-APIC-fasteoi   ata_piix, 
> > uhci_hcd:usb4,eth0, Ensoniq AudioPCI
> >  19:      74042          0   IO-APIC-fasteoi   uhci_hcd:usb3, ivtv2
> >  22:       1829          0   IO-APIC-fasteoi   ivtv0
> >  23:          3          0   IO-APIC-fasteoi   ehci_hcd:usb1
> > NMI:          0          0   Non-maskable interrupts
> > LOC:   14667959   15620982   Local timer interrupts
> > SPU:          0          0   Spurious interrupts
> > PMI:          0          0   Performance monitoring interrupts
> > IWI:          0          0   IRQ work interrupts
> > RES:    2407721    2344290   Rescheduling interrupts
> > CAL:        789       2664   Function call interrupts
> > TLB:      13663      20105   TLB shootdowns
> > TRM:          0          0   Thermal event interrupts
> > THR:          0          0   Threshold APIC interrupts
> > MCE:          0          0   Machine check exceptions
> > MCP:        540        540   Machine check polls
> > ERR:          0
> > MIS:          0
> 
> This looks pretty imbalanced: CPU0 handles a lot more of the interrupt 
> service than CPU1.  This imbalance of CPU0 handling most of the hardware 
> interrupts might be a source of problems, *if* CPU0 is actually busy with 
> higher priority interrupts (ata_piix, nvidia) when the ivtv* card interrupts 
> come in. 
> 
> The disk drive (ata_piix) and network card (eth0) interrupts have only been 
> handled by CPU0.
> 
> Is "irqbalance" running on your system?
> 
> For a back end system, I have to wonder what the nvidia card is doing for 
> you, assuming it it the only device generating interrupts on IRQ16.
> It has genearted 9.98M interrupts in the same time of 30.3M local timer 
> interrupts (1 nvidia card interrupt per every 3 timer interrupts).
> 
> 
> 
> > Looks like one of the USB ports shares its IRQ with one of the 
> > cards, but my issue happens with all three cards, so that doesn't 
> > seem to be the likely cause of the issue.
> 
> Right, especially if there are no USB devices connected to the port 
> that
> IRQ19 serves.
> 
> > >What is the cpu loading shown with top during video capture?  Are 
> > >there high numbers for user or io wait percentages?
> > Tasks: 146 total,   1 running, 145 sleeping,   0 stopped,   0 zombie
> > Cpu(s):  7.2%us,  4.0%sy,  0.0%ni, 88.8%id,  0.0%wa,  0.0%hi,  0.0%si,  
> > 0.0%st
> > Mem:   2059632k total,  1143540k used,   916092k free,   112388k buffers
> > Swap:  2095100k total,        0k used,  2095100k free,   688688k cached
> >   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND      
> >  2799 jpreston  20   0  214m  49m  24m S    2  2.5  36:37.44 knotify4     
> >  7959 jpreston  20   0 72220  12m 9.8m S    2  0.6   0:04.54 xfce4-terminal 
> >     
> >  1263 root      20   0 88408  51m  14m S    1  2.6   1:35.49 Xorg         
> >  8028 jpreston  20   0  3360  504  440 S    1  0.0   0:00.35 cat          
> >  8082 jpreston  20   0  2632 1136  848 R    1  0.1   0:00.08 top          
> >  1345 jpreston  20   0  6464 2516 1788 S    0  0.1   0:06.88 xscreensaver 
> >     1 root      20   0  3028 1792 1216 S    0  0.1   0:00.90 init         
> >     2 root      20   0     0    0    0 S    0  0.0   0:00.04 kthreadd     
> >     3 root      20   0     0    0    0 S    0  0.0   0:01.95 ksoftirqd/0  
> >     5 root      20   0     0    0    0 S    0  0.0   0:03.60 kworker/u:0  
> >     6 root      RT   0     0    0    0 S    0  0.0   0:00.00 migration/0  
> >     7 root      RT   0     0    0    0 S    0  0.0   0:00.00 migration/1  
> >     9 root      20   0     0    0    0 S    0  0.0   0:00.71 ksoftirqd/1  
> >    11 root       0 -20     0    0    0 S    0  0.0   0:00.00 cpuset       
> >    12 root       0 -20     0    0    0 S    0  0.0   0:00.00 khelper      
> >    13 root       0 -20     0    0    0 S    0  0.0   0:00.00 netns        
> >    15 root      20   0     0    0    0 S    0  0.0   0:00.27 sync_supers
> 
> >  This was captured during a cat /dev/video1 > test.mpg, and the CPU 
> > usage never went above a few percent for any process.  The file was 
> > still distorted and flawed.
> 
> OK.  Plenty of memory (894 MB RAM still free), plenty of CPU (88.8% idle), 
> nothing waiting on I/O (0.0% wa).
> 
> 'cat' consuming only 1% of CPU may be a little low, since the ivtv driver and 
> cat collectively do some buffer copies in the read() call that cat makes into 
> the ivtv driver.  (May indicate not many filled video buffers being provided 
> by the ivtv driver).

>  
> 
> Given this, the best guesses I have are:
> 1. an interrupt handling latency problem.
> 2. a PCI bus/chipset problem: DMA transfers aborting or IO 
> transactions to registers and memory on the CX23416 chip are being 
> aborted 3. an ivtv driver bug/race in setting up the CX23416 decoder, 
> that causes problems in the decoder actually running 4. a cx25840 
> driver bug in setting up the CX25843 chip to lock on to the video 
> signal and digitize it properly 5. an analog tuner driver bug that 
> causes tuning to fail
> 
> > Where else can I look for this?  Do you have any other ideas?
> 
> For #4 and #5 above, the output of several runs of 'v4l2-ctl --log-status' 
> should let you know if the CX25843 has a good video signal.
> 
> For #2 and #3, setting the debug=... option to the ivtv driver in 
> /etc/modprobe.conf or on the modprobe ivtv command line will turn on various 
> levels of logging in the ivtv driver (with messages emitted to 
> /var/log/messages normally).
> 
> $ modinfo ivtv
> [...]
> parm:           debug:Debug level (bitmask). Default: 0
>                          1/0x0001: warning
>                          2/0x0002: info
>                          4/0x0004: mailbox
>                          8/0x0008: ioctl
>                         16/0x0010: file
>                         32/0x0020: dma
>                         64/0x0040: irq
>                        128/0x0080: decoder
>                        256/0x0100: yuv
>                        512/0x0200: i2c
>                       1024/0x0400: high volume
> [...]
> 
> So maybe, as root,
> 
> # modprobe -r ivtv
> # modprobe ivtv debug=0x4f  (or =0x44f)
> 
> And then do a 'cat /dev/videoN > foo.mpg' and examine all the debug spew in 
> /var/log/messages to see what is wrong.
> 
> 
> For #1, I'd experiment with how does video capture behave with
> 
> a. having irqbalance running (or not running) b. booting into only 
> run-level 3 (text console, no GUI/X) and unloading the nvidia driver 
> module
> 
> 
> I would also look at IRQ handling latency with ftrace and other kernel 
> tracing mechanisms:
> 
> http://lwn.net/Articles/322666/
> http://lwn.net/Articles/322731/
> http://lwn.net/Articles/366796/
> http://lwn.net/Articles/365835/
> http://www.spinics.net/lists/linux-media/msg15762.html
> 
> Regards,
> Andy
> 
> > Thank you much for your assistance so far.
> 
> > Jeff
> > 
> 
> 
> 
> 
> -----
> No virus found in this message.
> Checked by AVG - www.avg.com
> Version: 2012.0.1901 / Virus Database: 2109/4772 - Release Date: 
> 01/28/12
> 




-----
No virus found in this message.
Checked by AVG - www.avg.com
Version: 2012.0.1913 / Virus Database: 2112/4804 - Release Date: 02/11/12


_______________________________________________
ivtv-users mailing list
[EMAIL PROTECTED]
http://ivtvdriver.org/mailman/listinfo/ivtv-users