The Innovation Lover

Recently I were looking for youtube downloaders over the web. I found many websites filled with lot of ads. Few websites where there which gave some neat interface to download videos. I got interested and decided to write my own shell script to parse and download videos. By looking into HTTP requests, I came through how downloaders work.

The logic is pretty simple. By passing the video_id which is received as v=videoid from youtube video URLs, to? http://www.youtube.com/get_video_info?video_id=videoid, we obtain a metadata file which contain metadata about the video we need to download. We extract a parameter called tokenid. Again pass the video_id and token to the same URL to obtain the video. We can also specify formats in which it is to be downloaded such as mp4,flv or 3gp in different video qualities. fmt=id parameter is passed to specify file format. By carefully watching the HTTP requests from youtube page, I collected variety of fmt arguments for different formats and quality. I have compiled all these info to write a youtube video downloader shell script. Download it and have fun.

#!/bin/bash
#Description: Youtube video downloader script
#Author: Sarath Lakshman
#url: http://sarathlakshman.info

if [ $# -ne 3 ];
then
    echo -e "Usage: $0 URL -format FORMAT\nFormats of different video qualities:\n1080 (mp4) - highest\n720  (mp4) - higher\n360  (flv) - high\n480  (flv) - low\n240  (flv) - lower\n3gp  (3gp) - least\n\nEg: $0 http://www.youtube.com/watch?v=yZPSx2r3TiY -format 1080\n"
    exit 0
fi
url=$1

declare -A formats;
declare -A extension;
formats[1080]=37;
formats[720]=22;
formats[480]=35;
formats[360]=34;
formats[240]=5;
formats[3gp]=13;

extension[1080]=mp4;
extension[720]=mp4;
extension[480]=flv;
extension[360]=flv;
extension[240]=flv;
extension[3gp]=3gp;

vid=`echo $1 | cut -d= -f2`

wget -O /tmp/meta.data "http://www.youtube.com/get_video_info?video_id=$vid&el=vevo" &> /dev/null

token=`sed 's/.*token=\([^&=]\+\).*/\1/g' /tmp/meta.data`
title=`sed 's/.*title=\([a-zA-Z0-9%+-]\+\).*/\1/g; s/-//g; s/[%+0-9]\+/_/g' /tmp/meta.data`

echo "Downloading..."

wget -o /tmp/log.$$ -O "$title.${extension[$3]}" "http://www.youtube.com/get_video?video_id=$vid&t=$token&fmt=${formats[$3]}"

if grep -q "Not Found" /tmp/log.$$ ; then
    echo "Unsupported format. Please try again with lower video quality format"
    rm $title.${extension[$3]} ;
else
echo Download complete. Play $title.${extension[$3]} and enjoy \).
fi

Download the scrpt from here : youtube_dl.sh

slynux@slynux-laptop:~/scripts$ ./youtube_dl.sh
Usage: ./youtube_dl.sh URL -format FORMAT
Formats for different video qualities:
1080 (mp4) - highest
720  (mp4) - higher
360  (flv) - high
480  (flv) - low
240  (flv) - lower
3gp  (3gp) - least

Eg: ./youtube_dl.sh http://www.youtube.com/watch?v=yZPSx2r3TiY -format 1080

Hey,

I have been working on GSOC project on Pardus Live installer. I work on a separate svn branch for GSOC. When working across many files, we many need to generate diff between numerous files recursively working directory and some other branch directory. svn diff command helps only if the directories are fork of same svn repo and with change in revisions or so.

I needed a generic tool to make diff out of two directories. But googling a few minutes didn’t give me pleasing results and hence I am here with my own script.

Try out and comment.

#!/bin/bash
#Filename: diffdir.sh
#Description: A utility to take diff of files recursively between two directories
#Author: Sarath Lakshman
#e-mail: sarathlakshman@slynux.com
#Homepage: http://www.sarathlakshman.info

olddir=$1
newdir=$2

if [ $# -ne 2 ];
then
	echo -e "\nUsage: $0 [DIR OLD] [DIR NEW] > data.diff\n\n"
	exit 0
fi

(cd $olddir; find . -type f ! -regex ".*\.svn.*" ) > /tmp/files.$$
(cd $newdir; find . -type f ! -regex ".*\.svn.*" ) >> /tmp/files.$$

sort /tmp/files.$$ -u -o /tmp/reqfiles.$$

cut -c3- /tmp/reqfiles.$$ > /tmp/uniqfiles.$$

for file in `cat /tmp/uniqfiles.$$`;
do

	[ -e "$olddir/$file" ] && [ -e "$newdir/$file" ]

	if [ $? -ne 0 ];
	then

		echo [NEW] $file\:
		echo ===================================================================
		[ -e "$olddir/$file" ] && cat "$olddir/$file"
		[ -e "$newdir/$file" ] && cat "$newdir/$file" 

	else

		diffed_data=`diff -u "$olddir/$file" "$newdir/$file"` 

		if [[ -n $diffed_data ]];
		then

			echo $file\:
			echo ===================================================================
			echo "$diffed_data"
			echo
		fi
	fi
done

Download the script : diffdir.sh

UPDATE : diff -Naur olddir newdir will do the stuff :p

Recently I participated in India’s first Capture the Flag style hacking challenge, InCTF. 160 teams from all over India participated in the CTF contest. Each team consisted of at most 5 members.

The finals of the event was really fun. We had real reach machines on the network connected through a VPN. The scores are based on attacking other machines, defending, giving ethical advisories. Scores are based on Flags submission. Flags can be captured from vulnerable services on machines of other teams.

Each team is given a vulnerable virtual image of a GNU/Linux system with four services. We have to patch it to prevent others attacking and capturing flag from our host. We have to exploit other’s services and capture the flag.

The most exciting part of the game was that, we exploited a service called therasus and got a root shell.

We did rm -rf / on many machines of oponents and had fun watching their systems getting down

My team r00tkit had only two members, my friend Syamlal and I. It was the first time we were participating for a realtime CTF event.

We reached second position in the hacking challenge.

Good work Team BIOS, CTF organizers.

For more see, http://inctf.amrita.ac.in/

Sometimes we get shocked when something unexpected occurs. We get stuck and feels like everything is over. Today I also had such a situation.

I was writing an important document over the last couple of days. I hadn’t kept a backup or draft, I had only single Open office text document file. While I was writing, my laptop got switch off suddenly due to some power button issue. When I powered it on and looked at the file, I got stuck. It shows 0 byte as size.
I ran file filename.odt, it shows empty data. OMG. It was around 12 Pages of text and WTH !

I lost my hope. But I decided to go for an experiment. Installed TestDisk, which the GNU/Linux recovery suite.
Ran the Photorec command. I selected the disk drive where the data is located, then chose scan unallocated space. I received 1000s of files of different format.

Ran,

 find . -type f -name "*.odt" | wc -l

gave me around 200 files.
I thought of greping the files to find the right file. But I understood that grep over binary files won’t work.
So I moved odt files to a directory.

mkdir odts
find . -type f -name "*.odt" -exec mv {} odts \;

Then I manually opened some of the files. Then I got earlier versions of the file which contains initial text only. So my breath became normal. I looked at the file size and noticed that it is of the size 27KB.
So again I filtered out files using size as a contraint.

mkdir filtered_odts
find . -type f -name "*.odt" -size +27k \
-exec mv {} filtered_odts \;

So I got a list of fewer number of files. I manually found out highest size which in the order of 27K, which was the file with all text I wrote. Voila, thanks to find and testdisk.