Thursday, November 24, 2011

Download any Flash video you can view, using Linux.

Some time ago, it was possible to grab any flash video you were watching under Linux very easily. The flash player buffers the data, we've all noticed the progress bar at the bottom of the video, filling in ahead of the marker or play head showing where we are in the time line. Where is this data stored?

Up until a year or so ago, that place was /tmp. The systems official temporary directory. While the video was playing, and the buffering indicator had reached the end, indicating the entire video had been downloaded, the Flash video file could simply be copied out of the /tmp folder to your desktop or other destination. This could even be done in the GUI, no need for the terminal.

Then something changed. I don't know if it was an addition to the Flash player, or just Flash developers getting smarter about hiding their temporary data to prevent anyone from obtaining a local copy of the video. Regardless, the simple fact remains that the /tmp folder no longer appears to contain the downloaded video while watching. The key word in that sentence is “appears”.

The file is actually still there, but a bit or flag is set, marking it as a deleted file! Even though the Flash player is actively using the file. Since it is marked 'deleted', the filesystem denies access to it and nothing can see it. Here's where we get tricky with Linux tools...

To illustrate this, I'll open my browser to a video file on CollegeHumor.com. If you want to follow along, here is the link.

http://www.collegehumor.com/video/6653193/occupy-wall-street-vs-the-iphone-line

Once the video starts, you can pause it or watch, that doesn't matter. What does is the grey filling in of the time line indicating the file is downloading. Once it has filled in completely, open a terminal and type:

lsof | grep Flash

lsof is a command that lists all open files on your system, including pipes. We're sending it's long output via the pipe symbol | into the grep command which will filter the output, only passing lines that meet our criteria. In this case, we only want to see lines that contain the text, “Flash”. The output I get is:

npviewer. 12550 loughkb 11u REG 8,1 22889528 13631698 /tmp/FlashXXdNKpIi (deleted)

The first string is the name of the process that owns the file, the next number is important, it is the process ID number. Following is my username and then the fourth column contains the other important bit of info, in this case, “11U”. The number 11 will be the name of a link we will find shortly

Now, we'll CD down into the proc folder, a folder that contains live information about everything going on in your system. One could write a book about the contents of /proc, and someone probably has. Without getting technical, the process number we obtained above, will be represented as a directory within /proc. We'll cd down into that folder and a sub folder called “fd”.

cd /proc/12550/fd

Now, we'll pull a file listing with details.

ls -l

This gives us a file listing, and here we find our Flash video data within a long list of data. My example gives:

lrwx------ 1 loughkb loughkb 64 2011-11-24 20:10 11 -> /tmp/FlashXXdNKpIi (deleted)

The lower case letter l at the beginning tells us this is a link to a file named 11 that is linked to FlashXXdNKpli in /tmp. You can see the file is flagged as deleted, even though the Flash player is currently using it.

All we need to do now is copy this linked file out somewhere to get a real copy of the file. I'll copy and rename it to a file on my desktop:

cp ./11 ~/Desktop/flashvid.flv

This copies the linked (deleted) file to my desktop as flashvid.flv. Now I have the Flash video file and I can watch it with vlc or any media viewer that plays flash. I can convert it to other formats with Winff, Open and edit it directly in OpenShot, etc.


Isn't Linux great!



15 comments:

  1. Neat trick, thanks for sharing! Would it not eventually be overwritten by the system since it's deleted?

    ReplyDelete
  2. Simon: Yes, the original file in /tmp will disappear as soon as the currently running Flash player exits. That's why we copy it out somewhere else.

    ReplyDelete
  3. Yep, Linux is great. Here the scripted version of your blog post :)


    #!/bin/bash
    if [ $# -ne 2 ]; then echo "Usage: $0 "; exit; fi
    url=$1
    dest=$2
    browser=chromium
    echo "Opening $browser with URL $url"
    echo "Notice that only the first instance of Flash will be recognized!"
    echo "Please pause the video and wait, until it's completely loaded, then press [ENTER] to go on..."
    ( $browser $url 1>/dev/null 2>&1 )
    read
    pid=$(lsof | grep -m 1 Flash | awk '{print $2}')
    linked_file=$(ls -l /proc/$pid/fd 2>/dev/null | grep -m 1 Flash | awk '{print "/proc/'$pid'/fd/"$9}')
    if [ "$linked_file" != "" ] && [ -f $linked_file ]; then
    cp $linked_file $dest
    if [ $? -eq 0 ]; then echo "Your backup is available under $(readlink -f $dest)"; else echo "Could not backup Flash content..."; fi
    else
    echo "Could not recognize Flash process..."
    fi

    ReplyDelete
  4. You can also ....
    ffmpeg2theora 11 -o ~/Desktop/flashvid.ogv

    ReplyDelete
  5. Thanks to TeoBigusGeekus at
    http://ubuntuforums.org/showthread.php?t=1688948&page=4

    $ lsof | grep Flash

    Then update process id to the 2nd number, and choose file descriptor number according to the video with big file size...
    see below example shows process id is 2451 and file descriptor number is 17 (not 17u)

    $ cp /proc/2451/fd/17 /mnt/nth_partition/vid1

    ReplyDelete
  6. tekdude: Actually, using sed, or other tools, you could take it down to one single line. My point in taking it through step by step, was to provide an educational experience for those less fluent in the shell environment.

    ReplyDelete
  7. Nothing like a quick and easy way to do it, huh? Get real. Just use video downloadhelper in firefox. (addon) Simple and easy.

    ReplyDelete
  8. Excellent pieces. Keep posting such kind of information on your blog. I really impressed by your blog.
    Vee Eee Technologies

    ReplyDelete
  9. johnstanton: Part of the point of this is to be a tutorial to the shell for newbies to a posix OS like Linux. Sure, there are add-ons to the major browsers, most of which only work on youtube. I haven't run across any that work on all sites with flash video though.

    ReplyDelete
  10. This comment has been removed by the author.

    ReplyDelete
  11. This comment has been removed by the author.

    ReplyDelete
  12. This comment has been removed by the author.

    ReplyDelete
  13. I just made a script to automate this. It looks for /tmp/Flash in the filename instead of just Flash, which can bring up false positives like loaded libraries. The legitimate Flash can show up under multiple PIDs due to multiple browsers and private sessions. This script goes through all of them, and will copy multiple open videos from each. Files are named using a combination of SCRIPTPID, FLASHPID and FDNUM to avoid name collisions and to keep videos organized by the "batch" they were captured in.

    Shell script to save (deleted) Flash video on Linux

    ReplyDelete
  14. @lilfluffy i got this after posting lsof code:- chromium- 2395 prady4 mem REG 7,0 19216944 1057793 /usr/lib/flashplugin-installer/libflashplayer.so

    rather than deleted flag file you mentioned above. what to do?

    ReplyDelete
  15. I am now longer getting the line ending with (deleted) after entering ls -l

    Where are they hiding the flash now?

    ReplyDelete