Sining's profileSining的共享空间PhotosBlogListsMore Tools Help

Blog


    October 26

    Automate Sequential Data Processing

    When running a long-term program, how to define the next program to run?

    1) Whenever you run a long-term program (let us call it "prog.exe"), please do not run it directly.  You can create a batch file named "RunProg.bat" with following content:
    ----------------------
    prog.exe
    call next.bat
    ----------------------

    2) Then, run the program with "RunProg.bat"

    3) Now, you can define the next program to run freely and write them in "next.bat"
    October 17

    Enumerate Date (Perl)

    Check Date String
    use HTTP::Date;
    my $daystr="20091026";

    if( str2time($daystr) )
    {
       print "Yes";
    }else
    {
       print "No";
    }

    Today's String
    my ($sec,$min,$hour,$mday,$mon,$year,$wday);
    ($sec,$min,$hour,$mday,$mon,$year,$wday,undef,undef)=localtime(time()); $year+=1900;$mon++;
     
    $mon = "0".$mon unless $mon >= 10;
    $mday = "0".$mday unless $mday >= 10;
     
    my $today = "$year$mon$mday";

    Iterate Days
    for( my $str="20091001"; $str < "20101010"; $str++)
    {
       if( str2time($str) )
       {
        #valid date string
       }
    }


    October 11

    Excel Notes

    NameManager代表离散地选择的几个列

     

    =Sheet1!$B$1:$B$3,Sheet1!$J$1:$J$3,Sheet1!$A$1:$A$3

     

     

     

    选择另一个表格的最后一行

    行号'Newest Line'!$B$12 =  COUNT(Sheet1!A:A)

    第一个值=OFFSET(Sheet1!$A$1,'Newest Line'!$B$12,COLUMN()-1)

    其它值通过自动填充,自动添加。

    September 07

    Managing Your Computer

    Add Local Administrators
    net localgroup Administrators "group\name" /add

    Check Running Process
    tasklist /m ie*
    September 03

    Stored String Function (SQL Server)

    Creating

    SET ANSI_NULLS ON
    GO
    SET QUOTED_IDENTIFIER ON
    GO

    CREATE FUNCTION GetDateString ( @val varchar(200) )
    RETURNS varchar(20)
    AS
    BEGIN
        DECLARE @left integer;
        DECLARE @right integer;
        DECLARE @datestring varchar(20);
        DECLARE @ret varchar(20);

        set @left = PATINDEX('%200%', @val);
        set @datestring = SUBSTRING(@val, @left, 8);
        set @right = CHARINDEX('\', @datestring);
       
        set @ret = SUBSTRING(@datestring,0,@right);
        RETURN @ret;
    END
    GO

    Alter

    SET ANSI_NULLS ON
    GO
    SET QUOTED_IDENTIFIER ON
    GO

    ALTER FUNCTION [dbo].[GetDateString] ( @val varchar(200) )
    RETURNS varchar(20)
    AS
    BEGIN
        DECLARE @left integer;
        DECLARE @right integer;
        DECLARE @datestring varchar(20);
        DECLARE @ret varchar(20);

        set @left = PATINDEX('%200%', @val);
        set @datestring = SUBSTRING(@val, @left, 12);
        set @right = CHARINDEX('\', @datestring);
       
        set @ret = SUBSTRING(@datestring,0,@right);
        RETURN @ret;
    END

    Test
    select dbo.GetDateString('as\2009_6_22\33')

    Reference

    "DECLARE" http://doc.ddart.net/mssql/sql70/de-dz_2.htm
    "SUBSTRING, PATINDEX, CHARINDEX" http://doc.ddart.net/mssql/sql70/fa-fz_23.htm
    "CREATE FUNCTION" http://www.umachandar.com/technical/SQL2000Scripts/Main12.htm
    "PATINDEX's pattern" http://www.cndw.com/tech/data/200602157855.asp
    "IF" http://www.databasejournal.com/features/mssql/article.php/3087431/T-SQL-Programming-Part-1---Defining-Variables-and-IFELSE-logic.htm
    "CAST" http://stackoverflow.com/questions/951320/how-to-concatenate-numbers-and-strings-to-format-numbers-in-t-sql


    August 18

    Using Exit Code in DOS Batch

    1) How to set Exit Code in .NET Console Application?
    Use Environment.Exit method

    2) How to get Exit Code in dos batch?
    program.exe
    If Errorlevel
    EQU 2 Goto C2
    If Errorlevel EQU 1 Goto C1

    C2:
    echo "C2"
    goto END

    C1:
    echo "C1"
    goto END

    END:
    echo "END"

    EQU - 等于

    NEQ - 不等于

    LSS - 小于

    LEQ - 小于或等于

    GTR - 大于

    GEQ - 大于或等于

    3) How to set Exit Code in dos batch?
    SET Errorlevel=1


    Reference:
    http://www.allenware.com/icsw/icswidx.htm
    http://hi.baidu.com/lhkyzh/blog/item/692a30ada50fdf0d4a36d6a7.html


    July 31

    Sort TSV/CSV File

    Perl Module: File::Sort
    Setup:

    1. Start a command line prompt of Visual Studio
    2. Run cpan in perl\bin
    3. Type "install File::Sort"

    Usage:
    Assuming we are sorting a.tsv as below:
    1       2
    3       3
    2       1
    1       2
    2       3
    3       1

      use File::Sort qw(sort_file);
      sort_file({
        I => 'a.txt',
        o => 'a2.txt', k => '2n', -t => '\t'
      });

      sort_file({
        I => 'a.txt',
        o => 'a21.txt', k => ['2n','1n'], -t => '\t'
      });

    Unix Sort
    Examples:
    sort -t, +2n
    sort -t, +2nr
    sort +4 -5 +1 -2 infile

    sort -c
    sort -t, -k2,2n -k1,1n
    sort -t, -k2,2n -k1,1n -r

    sort -t "`echo -e "\t"`"

    Explanation:
    "-t:" -> use ":" as field separator
    "+2n" -> sort from the field 2,3,4... (0,1 are ignored, numbered from 0)
    "+2nr" -> same as above, but reverse order
    "+4 -5 +1 -2" -> sort by field [4,5), [1,2)

    "-c" -> check order
    "-k2n,2n -k1n,1n" -> sort by field 2,1 (others are ignored, numbered from 1)
    "-r" -> reverse order

    References
    Perl Module
    http://search.cpan.org/~cnandor/File-Sort-1.01/Sort.pm

    Unix Sort
    http://www.softpanorama.org/Tools/sort.shtml


    BTW:
    Isn't this something awesome?
    #ls -l | sort +4n | awk '{print $5 "\t" $9}'

    Keep The Command Line Window After Running Program

    cmd /k set xx=yy&set aa=bb&c:\xxxx\xxxEnv.bat

    Float Constants

    float "1.0F"
    double "1.0"
    long double "1.0L"

    July 28

    Get Current Time

        FILETIME now;
        GetSystemTimeAsFileTime(&now);

        ULONG uTime = now.dwHighDateTime;

    NTFS Command Line Compress Tool

    Expand files recursive
    compact /U /S /I


    July 18

    Perl Debugger Notes

    Start

    perl -d abc.pl in.txt out.txt

     

    View Source

    "l 1-10"

    "."

    "v 10"

     

    Step

    step: "s"

    step over: "n"

    step out: "r"

     

    Print Var

    "p $name"

     

    Breakpoint

    set breakpoints: "b DBI::install_driver"

    list breakpoints: "L"

    one time breakpoint: "c 19"

    delete breakpoints "B 19" "B *"

    condition breakpoints "b 20 $a eq 'abc'"

     

    Halt When Change

    "w $count"

    Delete watch: "W $count"

     

    List Breakpoints and Watch

    "L"

     

    Stack Trace

    "T"





    April 24

    Useful Tool for Unerase Files

    Tool's name DiskGenius.
    April 21

    Using Printf Notes

        unsigned __int64 u64 = 1;
        u64 = u64 << 63;
        cout << u64 << endl;
        printf("%I64u", u64 ); //right
        printf("%I64d", u64 ); //wrong

        DWORD dwVal = GetLastError();
        printf("%u", dwVal);

    April 04

    STL File Stream Limitation

    Scenario:
    You make a data file which contains variable length records, and use another file to index from records' key hash to position in data file for random access.

    Problem:
    When the data file's size is larger than 2GB, ofstream.tellp() and ifstream.seekg() works wrong.
    Detail (in 64-bit version):
    When the file's size is between 2^31 to 2^32 , ofstream.tellp() give a negative number.
    When the file's size is larger than 2^32, ofstream.tellp() give only lower 32(or may be 31) bits number.
    When the file's size is larger than 2^32, ifstream.seekg() causes the ifstream invalid.

    Solution:
    Directly use WINAPI: CreateFile, WriteFile, ReadFile.

    Active Debug

    Scenario:
    We have to use a binary executable tool, but it crashes on some input files.
    We can:
    1. Remove the input file which causes the tool crash
    2. Remove the input files which are processed
    3. Restart the tool

    Problem:
    When crashes the tool pop up a Just-In-Time Debugger window.  We have to manually close the window and find the problematic input file in its console output.

    Analysis:
    A program without debugger will forward the unhandled exception to JIT Debugger.  We can make a special "debugger" which terminates the tool program when it sees a second chance exception.
    As to the file name in console output, we can use command line redirection to save the file name.  And make a special input file remover program to remove the files with names listed in the tool program's console output.

    Notes for Implementation:
    EnumProcesses: list processes PID
    OpenProcess: PID->Process Handle
    EnumProcessModules: get process' .exe name
    DebugActiveProcess: as a debugger, attach to another process
    Exception + FirstChance: return DBG_EXCEPTION_NOT_HANDLED
    Exception + SecondChance: TerminateProcess



    March 26

    What Is A Thread?

    What Is A Thread?

    1. When the thread encounter exception/interruption, its spirit is separated from its flesh.  Flesh includes stack & context data
    2.
    IDT will may use another spirit to utilize the flesh
    3. User Debugger only identifies the flesh (not the spirit)

    March 09

    Tool: FollowProcess

    FollowProcess <processName/processId> <next command>

    For e.g.:
    FollowProcess.pl excel.exe notepad.exe

    Code:
    my $have = 1;
    while(1)
    {
        open $task,"tasklist.exe|";
        $have=0;
        while(<$task>)
        {
            if( $_ =~ /$ARGV[0]/i )
            {
                $have=1;   
            }
        }
        close($task);
        if( not $have )
        {
            last;
        }
        sleep(10);
    }

    exec($ARGV[1]);
    March 01

    Matching in Files with Perl

    use File::Find;
    if( scalar(@ARGV) < 2 )
    {
        print "usage: MakeSingle.pl <folder> <output file>";
        exit;
    }

    open $output,">$ARGV[1]";
    find(\&wanted, $ARGV[0]);
       


    sub wanted
    {
        my $file = $File::Find::name;
        if( $file =~ /.*.html$/)
        {
            print $file,"\n";
            open $fh, "<$file" or print "open failed!\n";
            my $file;   
            while(<$fh>)
            {
                $file = $file.$_;
            }
            print $file;
            $file =~ /<body>(.*)<\/body>/ims;
            print $output $1;
            close($file)
        }
    }
    close($output);

    Alternative for big whole file reading.

            my $fileContent;
            {
              local( $/, *fh);
              open fh, "<$file" or print "open failed!\n";
              $fileContent = <fh>;
            }

    Another way for file reading.
           open $fh,"<$ARGV[0]";
           @lines=<$fh>;
           close($fh);
    February 26

    TSV Toolset

    1. GetColumn <in.tsv> <colnums 0,1,2> [remove_first_line, rfl] [use_csv, csv]
    2. MergeFiles <file list> [colnames, "Name,Tel"] [use_csv, csv]
    3. AppendColumn <from.tsv> <to.tsv> [use_csv, csv] [append_first_line, afl] [colname]