"[[0] * n] * m" is not equal to "[[0 for i in range(n)] for j in range(m)]"

	# When generate list of list, be careful of following methods, they produce different results
	>>> x = [[0] * 2] * 3
	>>> x
	[[0, 0], [0, 0], [0, 0]]
	>>> y = [[0 for i in range(2)] for j in range(3)]
	>>> y
	[[0, 0], [0, 0], [0, 0]]
	# although they look the same and are equal to each other
	>>> x == y
	True
	# but... the operation can generate different results
	>>> x[0][0] = 1
	>>> x
	[[1, 0], [1, 0], [1, 0]] # Surprise!! every list is changed
	>>> y[0][0] = 1
	>>> y
	[[1, 0], [0, 0], [0, 0]] # this is what we expected

view raw List.py hosted with ❤ by GitHub

Use Jazzy to generate Swift documentation.

Format:

• /// description
• - parameter param: description
• - returns: xxxxxxx

run Jazzy

$ jazzy --min-acl internal --swift-version 2.1

Embed images to latex

\graphicspath{ {pic/} } % The folder containing images.
\includegraphics[scale=0.5]{imageName} % Embed a image.

CS229 Machine Learning - 3 (Linear regression, Probabilistic interpretation, Logistic regression, Newton's method)

Outline:
- Linear regression
- Probabilistic interpretation
- Logistic regression
- Newton's method

CS229 Machine Learning - 2 (Linear regression, Gradient descent, Normal equation)

Outline:
- Linear regression
- Gradient descent
- Normal equation

[Swift] Model View Controller

source: https://www.youtube.com/watch?v=QLJtT7eSykg&list=PLZYLTko_GQdGk39r8OTL-9lC92kd4AJy3&index=2

Elements of Reinforcement Learning

Elements of reinforcement learning:

• a policy: is a mapping from perceived states of the environment to actions to be taken when in those states.
• a reward function: defines the goal in a reinforcement learning problem. iIt maps each perceived state (or state-action pair) of the environment to a single number, a reward, indicating the intrinsic desirability of that state.
• a value function: specifies what is good in the long run. The value of a state is the total amount of reward an agent can expect to accumulate over the future, starting from that state.
• a model (optional)

reference: Reinforcement Learning: An Introduction, Richard S. Sutton and Andrew G. Barto

Install Maven on OSX causes Error

If you get error "Exception in thread "main" java.lang.UnsupportedClassVersionError: org/apache/maven/cli/MavenCli : Unsupported major.minor version 51.0" when installing Maven on mac, try the following command (make sure you modify the version of jdk on your computer):

export JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk1.8.0_40.jdk/Contents/Home 

Reference: http://stackoverflow.com/questions/29255495/maven-installation-osx-error-unsupported-major-minor-version-51-0

Great Latex syntax query website

You can just draw the sign you want for Latex.
 

http://detexify.kirelabs.org/classify.html

Output data from a sqlite database to a csv file

	sqlite> .mode csv
	sqlite> .output filename.csv
	sqlite> select * from tablename;
	sqlite> .output stdout

view raw outputCSV.sqlite hosted with ❤ by GitHub

Old style vs. new style class in Python

http://stackoverflow.com/questions/54867/what-is-the-difference-between-old-style-and-new-style-classes-in-python

Use Sphinx to automatically create Python documents

Creating program documents is a tedious task. However, with the assistance of automatic tools for creating program documents, we can produce the documents in a very easy and quick way. (We still have to write the comments of program functions.)

Here are the steps of that I used Sphinx to create documents for my python project.

1. Install Sphinx

$ pip install sphinx 

2. Create a directory (folder) for documents
Create a directory named "docs", and the structure of the "docs" and "src" directories is shown below:

Python Project/
  |--------docs/
  |--------src/(source code)


3. Create a Sphinx project
In the docs/ directory, perform the following command:

$ sphinx-quickstart docs

During the process, you will be asked several questions. The following are my answers to some of the questions (empty answer means using the default setting):

Separate source and build directories (y/n) [n]: y

autodoc: automatically insert docstrings from modules (y/n) [n]: y

intersphinx: link between Sphinx documentation of different projects (y/n) [n]: y

ifconfig: conditional inclusion of content based on config values (y/n) [n]: y

viewcode: include links to the source code of documented Python objects (y/n) [n]: y

After step 3 is finished, the structure of the Sphinx project will be that as follows:

Python Project/

  |--------docs/
  |             |----- build
  |             |----- source
  |--------src/(source code)


4. Modify docs/source/conf.py
In the conf.py file created in step 3, find the following comments.

# If extensions (or modules to document with autodoc) are in another directory,
# add these directories to sys.path here. If the directory is relative to the
# documentation root, use os.path.abspath to make it absolute, like shown here.

Add a setting under the above comments so that Sphinx can find the source code.

sys.path.insert(0, os.path.abspath('../../src/'))

5. Use sphinx-apidoc to create module_name.rst files

$ sphinx-apidoc -o docs/source src

6. Create html files
In the docs directory, perform the following command to create the documents' html files.

$ make html

If the documents are successfully created, you can open the docs/build/html/index.html to read it.


Reference:
http://chimerhapsody.blogspot.com/2014/07/python.html
http://sphinx-doc.org/invocation.html#invocation-of-sphinx-apidoc

Be careful of implementing "binary search"

When using a binary search, we usually will calculate the middle of the search space by
mid = (start + end) / 2

However, if implement this calculation on a computer program, it might cause an overflow problem if the start and end are already too large. Therefore, instead of using (start + end) / 2, we could use
mid = start + (end - start) / 2


See more detail: http://googleresearch.blogspot.com/2006/06/extra-extra-read-all-about-it-nearly.html

The running time of a program in Java

$ time java yourProgram

sample output

real 0m0.127s

user 0m0.111s

sys 0m0.030s

Char array to string in Java

	char[] array = {'a', 'b', 'c', 'c'};
	System.out.println(String.valueOf(array));//print "abcc"

view raw charToString.java hosted with ❤ by GitHub

Static and Dynamic Binding in Java

Static Binding or Early Binding

The binding which can be resolved at compile time by compiler is known as static or early binding. All the static, private and final methods have always been bonded at compile-time . Why binding of Static, final and private methods is always a static binding? You would understand it better after reading dynamic binding. Still let me explain this – Compiler knows that all such methods cannot be overridden and will always be accessed by object of local class. Hence compiler doesn’t have any difficulty to determine object of class (local class for sure). That’s the reason binding for such methods is static.

Dynamic Binding or Late Binding

When compiler is not able to resolve the call/binding at compile time, such binding is known as Dynamic or late Binding.

Reference:
http://beginnersbook.com/2013/04/java-static-dynamic-binding/

The "finally" block in Java

The "finally" block will get executed if we insert a return statement inside the try block of a try-catch-finally. Even when we attempt to exit within the try block (via a return statement, a continue statement, a break statement or any exception), the finally block will still be executed.

Note that there are some cases in which the finally block will not get executed, such as the following:

• If the virtual machine exits during try/catch block execution.
• If the thread which is executing the try/catch block gets killed.

"final" in Java

"final" doesn't guarantee immutable. It only protect the reference, not the object it references to.

ex:
private final int[] x; //x cannot change, but x[0] can change.

Access control in Java inheritance and package

• the variable or method in a interface is actually public even if we left no modifier before it.

Bayes' theorem

• Bayes' theorem was used to convert a prior probability into a posterior probability by incorporating the evidence provided by the observed data.
• P(W | D) = P(D | W)P(W)/P(D)     ==> posterior = likelihood x prior
• P(W | D): posterior probability
• P(D | W): likelihood function
• P(W): prior probability
• P(D): normalization constant, ensures that the posterior distribution on the left-hand side is a valid probability density and integrates to one.
• maximum likelihood: w is set to the value that maximizes the likelihood function p(D | W). This corresponds to choosing the value of W for which the probability of the observed data set is maximized.

Machine learning applications

Machine learning:

• Supervised learning
• finite number of discrete categories --> classification problem
• continuous variables --> regression problem
• Unsupervised learning
• discover groups of similar examples --> clustering problem
• determine the distribution of data within the input space --> density estimation problem
• project the data form a high-dimensional space down to two or three dimensions --> visualization purpose

[CS61B-2015] Generics, Conversion, Promotion

An example of unboxing and autoboxing.

Import a module from your package in Python

If I have two modules in two packages in the same folder described below:
myfolder/
        package1/
                __init__.py  (need to add an empty file)            
                module1.py (has a function test())
        package2/
                __init__.py
                module2.py

I can use the following code in module2 to import module1

	import sys
	import os
	sys.path.append(os.path.join(os.path.dirname(__file__), '..'))
	from package1.module1 import test

view raw importModule.py hosted with ❤ by GitHub

3 ways to prevent a class from being subclassed in Java

1. Access control: Even though a class can't be marked private, a class can be non-public (by not declaring it as public). A non-public class can be subclassed only by classes in the same package as the class.
2. final keyword: A final class means that it's the end of the inheritance line.
3. If a class has only private constructors. Ex. like built-in Math class in Java.

MIN_VALUE in Java

It is no difference between declaring a MIN_VALUE with a "-" notation or without a "-" notation in Java. Both of the variables will be -2147483648.

	int min1 = -Integer.MIN_VALUE;
	int min2 = Integer.MIN_VALUE;
	//min1 = -2147483648
	//min2 = -2147483648

view raw MIN_VALUE.java hosted with ❤ by GitHub

Bias vs Variance in machine learning

High bias -> underfitting

High variance -> overfitting

Reference: https://www.coursera.org/learn/machine-learning/home/welcome

Default initial value of variables in Java

For instance variables, Java has default initial value for those variables. But, the wrapper class of primitive variable, like Integer, Float..., will not have default initial value even they are declared as instance variables.

For variable within a method, Java has NO initial value for those variables.

Java Visualizer

"Java Visualizer" is a very helpful tool that can visualize the relationships between references and objects. See the following example.

Link: http://cscircles.cemc.uwaterloo.ca/java_visualize/

[CS61B-2015] Notes for class 5-Binary and Bits

8 primitive types of Java:

• The 5 integer types
• Exact representations of integers.
• byte, short, int, long, char.
• The boolean type
• The 2 floating point types
• Approximate representation of reals.
• float, double.

Integer:

bit operations:

[CS61B-2015] Notes for class 4-Testing

https://berkeley-cs61b.github.io/public_html/materials/lab/lab2/lab2.html

import org.junit.Test;
import static org.junit.Assert;
assertEquals(expected, actual);

To make testing output more concise
1. Annotate each test with @org.junit.Test
2. Make methods non-static
3. jh61b.junit.textui.runClasses

example:
https://github.com/Berkeley-CS61B/lectureCode/blob/master/lec4/live1/TestSort.java

Using the "final" keyword in Java

Using the "final" keyword in Java has different meaning depending on what it applied to:

• Variable: the value cannot be changed once initialized.
• Method: the method cannot be overridden by a subclass.
• Class: the class cannot be subclassed.

Notes for "Head first Java"

• A boolean and an integer are not compatible types in Java
• int x = 1;
• While(x)...
• Objects are stored in heap (Garbage-collectible heap).
• If you're running low on memory, the Garbage Collector will run, throw out the unreachable objects (no reference points to them).
• There is no "global variable" in Java. Use public static instead. Any code, in any class of your application, can access a public static method/variable.
• If the end-user doesn't have a JVM, then you'll also need to include that with your application's classes.

[CS61B-2015] Notes for class 3

A variable name in Java is a name for a simple container.
The 9 things that can go in simple containers:

• primitive values(byte, short, int, long, float, double, boolean, char
• references to Objects (an arrow), references may be null

Java is "Pass by value" (in some case, the "value" is a reference?)

[CS61B-2015] Notes for class 2-Defining and using classes

Static vs. Non-static
A class may have a mix of static and non-static members.

• A variable or method defined in a class is also called a member of that class.
• Static members are accessed using class name, e.g. ClassName.StaticMember.
• Non-static members cannot be invoked using class name: Dog.makeNoise().
• Static methods must access instance variables via a specific instance, e.g. an object.
• Static members are shared in a class.

Good Git tutorials

Tutorial
https://berkeley-cs61b.github.io/public_html/materials/lab/lab1/lab1.html#c-using-git

Documentation
http://git-scm.com/documentation

Practice
https://try.github.io/levels/1/challenges/1

[CS61B] Notes for class 5-Linked lists

Array-based lists
Advantage: very fast access of each item
2 disadvantages:
    (1) Insert item at beginning or middle takes time proportional to length of array.
    (2) Array has a fixed length.                        

Linked lists (a recursively data type)
Made up of nodes. Each node has:
(1) an item
(2) a reference to next node in list

Linked lists vs. Array lists
Advantages of linked lists:
(1) Inserting item into middle of linked lists takes constant time, if you have a ref to previous node.
(2) List can keep growing until memory runs out.

Disadvantage of linked lists: Finding n-th item of a linked list takes time proportional to n. (Start at head, walk n-1 nodes.)

The "public" and "private" keywords
Private method or field: invisible and inaccessible to other classes.
Why?
(1) To prevent data form being corrupted by other classes.
(2) You can improve the implementation without causing other classes to fail.


The interface of a class: prototypes for public methods, plus description of their behaviours.

Abstract Data Type (ADT): A class with well-defined interface, but implementation details are hidden from other classes.

Invariant: A fact about a data structure that is always true. "A Date object always stores a valid date." Enforced by allowing access only through method calls.
If we don't have any restriction of a field, than we could declare this filed to "public"; Otherwise, we should declare it as "private."

Not all classes are ADTs! Some classes are data storage units, no invariants; field can be public.

Double-Linked Lists
Insert and delete items at both ends in constant running time.

Sentinel: A special node that does not represent an item. It points to the first and last nodes.

[CS61B] Notes for class 6-Stack frames

The heap stores all objects, including all arrays and all class variables.

The stack stores all local variables, including parameters.

When a method is called, Java creates a stack frame (aka activation record); stores the parameters and local variables.

Stack of tack frame.

Garbage collection only performs on heap.

When a method finishes, its stack frame is erased.

Method "Thread.dumpStack()": Prints a stack trace of the current thread to the standard error stream. This method is used only for debugging.

Parameter Passing
Java passes all parameters by value: copied.
When parameter is a reference, the reference is copied, but the object is shared.

[CS61B] Notes for class 4-Iteration and array

Declare arraies:
int[] a, b[]; //a is 1D array, b is 2D array

Constant:
"final" keyword: value that can never be changed.
BAD: if (month == 2)
GOOD: public final static int FEBRUARY = 2;
              if (month == FEBRUARY)

Output data to csv files

There is a little difference between the code implemented in Python 2 and 3.

	#Python 3
	with open(filename, 'w', newline='') as output:
	writer = csv.writer(output)
	writer.writerows(dataSet) #dataSet is a list
	output.close()
	#Python 2
	with open(filename, "w") as output:
	writer = csv.writer(output, lineterminator='\n')
	writer.writerow(dataSet) #dataSet is a list
	output.close()

view raw csv.py hosted with ❤ by GitHub

[CS61B] Notes for class 3-types, conditionals, iteration and arrays

Primitive types: (don't have to create an object to store it)
byte : 8-bit integer    => -128~127
short:16-bit integer   => -32,768~32,767
int    : 32-bit integer  => -2,147,483,648 ~ 2,147483647
long : 64-bit
float : 32-bit
double: a 64-bit floating-point number
boolean: "true" or "false" ("True" or "False" in Python)
char: a character, 2byte, non-signed

e.g.
long x = 43L;

double & float values must have a decimal point:
double x = 10.0;
float x = 43.9f;

                               Object types                            Primitive types
-------------------------------------------------------------------------------------------
Contains a              reference                                   value
How defined?        class definition                          built into Java
How created?         "new"                                        "6", "3.4", "true", etc.
How initialized?    constructor                                default (usually zero)
How used?            method                                      operators("+", "*",...)


Function: methods declared to return a non-void type.

char c = new char[4];
c[4] = 'x'; //Run-Time error

The size of an array cannot be changed once it has been created.

[ML] Precision / Recall

For classification problem

Precision:
    True positives                            True positives
------------------------   =   ---------------------------------------
#predicted positives         True positives + False positives

Recall:
    True positives                            True positives
------------------------   =   ---------------------------------------
  #actual positives            True positives + False negative


There is a tradeoff between precision and recall.
One way to compute the precision and recall is using F_1 score
                           precision * recall
F_1 score = 2 * -----------------------
                           precision + recall

Reference: https://www.coursera.org/learn/machine-learning/lecture/tKMWX/error-metrics-for-skewed-classes

[CS61B] Notes for class 2-Defining classes

Java gives a default constructor that takes no parameter, does no initializing. However, if we write a constructor, the default constructor goes away.

Static field: a single variable shared by a whole class of objects. Also called class variables.
e.g. System.in & System.out are static fields.

Static method: Doesn't implicitly pass an object as a parameter.
main() is always static.
IMPORTANT: In a static method, THERE IS NO "this".

[CS61B] Notes for class 1-Using objects

Object: A repository of data.

Class: Type of object.

Method: Procedure or function that operates on an object.
e.g. addItem: adds an item to any ShoppingList object

Inheritance: A class may inherit properties from a more general class.
e.g. ShoppingList inherits from List the property of storing a sequence of items.

Polymorphism: One method works on several classes; even if the classes need different implementations.
e.g. "addItem" method on every kind of List, though adding item to a ShoppingList is different from a ShoppingCart.

Object-Oriented: Each object knows its own class and methods.
e.g. Each ShoppingList and ShoppingCart knows which implementation applies to it.


Java
Variables: You must declare them and their type.
Variables also used to reference objects.

e.g.
String myString; 
//Only declare a String type reference variable that can point to a String object.
//It doesn't create an object.

String myString = new String();
//"new String()" is a constructor call. It creates a String object.
//assignment "=" causes myString to reference the String object.


Objects & Constructors
If an object is not referenced by a variable, it will eventually be deleted (garbage collection).
Constructors always have same name as their class, except "stuffinquotes".

3 ways to construct a string object:
1. new String()
2. "test"
3. new String("test")

Strings are immutable.

Some object reference examples:
e.g.
String s1 = new String();
s1 = "test";
String s2 = s1; //s1 and s2 reference to the same object
s2 = new String(s1); //s1 and s2 reference to different objects

e.g.
String s1; //null pointer (reference to nothing)
String s2 = s1; //s2 is also a null pointer
s1 = "test"; //while s2 still is a null pointer

e.g.
String s1;
String s2 = s1;
s1 = "test";
s2 = "test"; //Java will make s2 point to the same object that s1 pointed because their value is the same.
s2 = new String(s1); //force Java to create a copy of that String object.
s2 = s1;
s1 = "another test"; //now s2 = "test" while s1 = "another test"

[Python] Don't use "<<" to double a number

As we know, we can use "a << b" to represent "a*(2^b)." But this method is much slower than just using addition, performing "a+=a" for b times. In other words, "<<" operation is much more expensive than "+" operation.

I have performed some simple experiments. The results are shown below.

	import datetime
	"""test << operation"""
	# Test A
	x = 3
	print datetime.datetime.now() #2015-05-28 10:00:34.886600
	x <<=1
	print datetime.datetime.now() #2015-05-28 10:00:34.887079
	# Test B
	x = 3
	print datetime.datetime.now() #2015-05-28 10:02:51.338504
	x <<=10
	print datetime.datetime.now() #2015-05-28 10:02:51.339138
	"""test + operation"""
	#Test C
	x = 3
	print datetime.datetime.now() #2015-05-28 10:00:34.887085
	x += x
	print datetime.datetime.now() #2015-05-28 10:00:34.887089
	# Test D
	x = 3
	print datetime.datetime.now() #2015-05-28 10:02:51.339146
	x += x
	x += x
	x += x
	x += x
	x += x
	x += x
	x += x
	x += x
	x += x
	x += x
	print datetime.datetime.now() #2015-05-28 10:02:51.339152

view raw CostofDouble.py hosted with ❤ by GitHub

As we can see, the running time for "<<" operations are much slower than that of "+" operation. It takes more than several hundred times for "<<" to produce the same result as "+" does. Therefore, using "+" operation instead of using "<<" for the same purpose. (Here I am under the restriction of not using multiplication.)

Compiling C program

-std=c11 -Wall -Wextra -Wpedantic -O2

Lexicographic order in Java

The following notes are from Udacity-Intro to Java Programming

The term lexicographic order is not defined in the videos. It means the order the computer sorts the strings in. In the computer world, numbers are considered to be smaller than letters of any kind. So they come first. Uppercase letters are smaller than lowercase numbers letters. so "X" comes before "x"

This is another interesting thing about alphabetizing Strings containing numbers in computers. Here is how numbers would be ordered. 1, 11, 12, 13, 14 ... 2, 21, 22, 23 ... Anything that starts with 1 comes before anything that starts with 2. so even 1000 would come before 2.

Thus, numbers (1 < 11 < 12 < ... < 2 < 21 ...) < Uppercase letters < Lowercase letters
Ex:
123 < John < john
lesson1 < lesson10 < lesson2

We can use "compareTo" function to compare the lexicographic order between two strings.

ex: String1.compareTo(string2)

the return value:

0    : if two strings are identical

< 0 : if String1 is lexicographically less than String2

> 0 : if String1 is lexicographically greater than String2

See more detail on 

http://docs.oracle.com/javase/7/docs/api/java/lang/String.html#compareTo%28java.lang.String%29

[Python] Conversion between int and bit

	"""int to bit (string)"""
	#a method of converting int into bit
	>>> bit = bin(43261596)
	>>> bit
	'0b10100101000001111010011100' #bit is a string
	# another method of converting int into bit
	# '{0:0ab}'.format(n)
	# the first 0 means taking the variable at argument position 0
	# the second 0 means zero-padded on the left
	# a: the total number of bits
	# b: means binary
	# n: the int number;
	>>> bitStr = '{0:032b}'.format(43261596)
	>>> bitStr
	'00000010100101000001111010011100' #bitStr is a string with 0 peddled on the left, total 32 bits
	"""bit into int"""
	>>> int('00000010100101000001111010011100', 2)
	43261596

view raw int_bit.py hosted with ❤ by GitHub

Install OpenCV for Python on Mac OSX (Yosemite)

The tutorial from the following website works for me. I followed the steps and installed OpenCV v2.4.11_1 on Mac OSX (Yosemite).

https://jjyap.wordpress.com/2014/05/24/installing-opencv-2-4-9-on-mac-osx-with-python-support/

[Python] get all file names in a directory

	from os import walk
	def getFilename(directory):
	"""Get all the file names in the directory. Return a filename list."""
	f = []
	for (dirpath, dirnames, filenames) in walk(directory):
	f.extend(filenames)
	break
	#f contains all the file names in the directory
	return f

view raw getFilenames.py hosted with ❤ by GitHub

Call by assignment in Python

When passing a mutable object (list, dictionary, etc.) to a function, it works like "call by reference" that the object will be shared.

When passing a immutable object (int, tuple) to a function, it works like "call by value" that it pass a copy of the original object to a function.

See more detail:
[1] https://docs.python.org/3/faq/programming.html#how-do-i-write-a-function-with-output-parameters-call-by-reference
[2] http://stackoverflow.com/questions/986006/how-do-i-pass-a-variable-by-reference

[Python] Find the key with most items in a dictionary

	#suppose we have the following x dictionary
	>>> x
	defaultdict(None, {1: [2], 2: [1], 6: [9, 5], 7: [9], 8: [9], 9: [6, 7, 8]})
	#we want to find the key with most items
	#Python 2
	>>> sorted(x.viewitems(), key=lambda k: len(k[1]), reverse=True)
	[(9, [6, 7, 8]), (6, [9, 5]), (1, [2]), (2, [1]), (7, [9]), (8, [9])]
	>>> sorted(x.viewitems(), key=lambda k: len(k[1]), reverse=True)[0][0]
	9
	#Python3
	>>> sorted(x.items(), key=lambda k: len(k[1]), reverse=True)[0][0]
	9

view raw most_item.py hosted with ❤ by GitHub

Reference: http://stackoverflow.com/questions/9363670/sort-dictionary-by-number-of-values-under-each-key

[Python] A simple way to sort a dictionary

	>>> import operator
	>>> x #x is a dictionary
	{'cde': 456, 'abc': 789, 'efg': 123}
	# only return keys
	>>> sorted(x)
	['abc', 'cde', 'efg']
	# sort by key, "reverse=True" means from large to small
	>>> sorted(x.items(), key=operator.itemgetter(0), reverse=True)
	[('efg', 123), ('cde', 456), ('abc', 789)]
	# sort by value, just change the parameter in the itemgetter()
	>>> sorted(x.items(), key=operator.itemgetter(1), reverse=True)
	[('abc', 789), ('cde', 456), ('efg', 123)]

view raw sortDict.py hosted with ❤ by GitHub

[Python] An example of scraping date from a website and writing to a csv file

In order to scrape data from a website, I used the "BeautifulSoup" module for Python. The data I want to get from the website (http://www.charitynavigator.org/index.cfm?bay=topten.detail&listid=24#.VTfpxa3BzGc) is the "10 Super-Sized Charities."

A sample code is shown below:

	from urllib.request import urlopen
	from bs4 import BeautifulSoup
	import csv
	def outputCSV(dataSet, filename):
	print(dataSet)
	with open(filename, 'w', newline='') as csvfile:
	csvW = csv.writer(csvfile)
	csvW.writerows(dataSet)
	csvfile.close()
	def main():
	html = urlopen('http://www.charitynavigator.org/index.cfm?bay=topten.detail&listid=24#.VTfpxa3BzGc')
	soup = BeautifulSoup(html.read())
	dataSet = []
	data = []
	#get column name
	for row in soup('table')[0].findAll('tr')[0].findAll('th'):
	data.append(row.contents[0])
	dataSet.append(data)
	#get data
	for i in range(1, len(soup('table')[0].findAll('tr'))):
	data = []
	for row in soup('table')[0].findAll('tr')[i].findAll('td'):
	if row.a == None:
	tmpString = row.contents
	else:
	#remove hyperlink
	tmpString = row.a.contents
	if tmpString != []:
	data.append(tmpString[0].strip())
	if data != []:
	#print(data)
	dataSet.append(data)
	for row in dataSet:
	print(row)
	outputCSV(dataSet, 'web_scrape.csv')
	if __name__ == '__main__':
	main()

view raw scrape.py hosted with ❤ by GitHub

And the result is

[Python] defaultdict will not always generate default value

If you didn't assign default data type, then the defaultdict will not generate a default value for a missing key. For this case, we can use get(key) to find the key value and it will not cause error if the key is missing. See the following example.

	#defaultdict will not always generate default value
	#if you didn't assign default data type to it, using defaultdict[key] to find the key might get an error
	#defaultdict has default data type
	>>> x = defaultdict(int)
	>>> x
	defaultdict(<type 'int'>, {})
	>>> x["abc"]
	0
	>>> x
	defaultdict(<type 'int'>, {'abc': 0})
	#defaultdict has no default data type
	>>> x = defaultdict()
	>>> x
	defaultdict(None, {})
	>>> x["abc"]
	Traceback (most recent call last):
	File "<stdin>", line 1, in <module>
	KeyError: 'abc'
	#for safety, instead, we can use get() to find the key
	>>> x.get("abc") #will return nothing, but will not cause error
	>>>

view raw deafultdict.py hosted with ❤ by GitHub

Using the Stanford parse tree through NLTK

This is a note for using the Stanford parse tree through NLTK.

Environment setup:
http://stackoverflow.com/questions/13883277/stanford-parser-and-nltk

Stanford parse tree module:
http://nlp.stanford.edu/software/lex-parser.shtml#Download

Remember to update the JRE and JDK to the version you need.

Manual for commands:
http://www.nltk.org/api/nltk.parse.html#module-nltk.parse.stanford
http://www.nltk.org/_modules/nltk/tree.html

int is not an object in Java

Type int in Java is not an object. So when we assign a previously created int variable to another newly created int variable, they are not sharing the same memory (they are mutually independent).

	int nu1 = 13;
	int nu2 = nu1;
	nu2 = 12;
	//now nu1 is still 13, and nu2 is 12

view raw int.java hosted with ❤ by GitHub

Unlike type int, String is an object in Java. When we assign a previously created string assigned to another newly created string, they share the same object.
PS. String is not mutable.

	String str1 = "Hello World";
	String str2 = str1;
	str2.replace("H", "X");//replace is an accessor, not a mutator
	System.out.println(str2);
	//print "Hello World"
	System.out.println(str1);
	//print "Hello World"
	String str3 = str1.replace("H", "X");
	System.out.println(str3);
	//print "Xello World"

view raw string.java hosted with ❤ by GitHub

[Python] Counter

The function of "Counter" make it very convenient to count the number of repeated items in two list. See the following example.

	>>> from collections import Counter
	>>> x = [1, 1, 2, 3, 3, 4, 5, 6, 6]
	>>> y = [1, 2, 2, 2, 3, 3, 4, 6, 6, 6]
	>>> x_counter = Counter(x)
	>>> y_counter = Counter(y)
	>>> x_counter
	Counter({1: 2, 3: 2, 6: 2, 2: 1, 4: 1, 5: 1})
	>>> y_counter
	Counter({2: 3, 6: 3, 3: 2, 1: 1, 4: 1})
	>>> x_counter & y_counter #find the overlapping numbers
	Counter({3: 2, 6: 2, 1: 1, 2: 1, 4: 1})
	>>> sum((x_counter & y_counter).values()) #summation of the total number of item overlapped
	7

view raw Counter.py hosted with ❤ by GitHub

[Python] some operations on list

When using a list in Python, we can see the indicator of the items as follows:
x = [0, 1, 2, 3, 4, 5, 6, 7, 8]
x[0],   x[1],   x[2],   x[3],   x[4],   x[5],   x[6],   x[7],   x[8]
x[-0],  x[-8], x[-7],  x[-6],  x[-5], x[-4],  x[-3],  x[-2], x[-1]   <= reverse direction
x[-9]
some usages of list in Python are shown below:

	>>> x = [0,1,2,3,4,5,6,7,8]
	>>> x[1:5]
	[1, 2, 3, 4] #print items from x[1] to x[4] (excluding x[5])
	>>> x[1:]
	[1, 2, 3, 4, 5, 6, 7, 8] #print the items start from x[1] to the end
	>>> x[:5]
	[0, 1, 2, 3, 4] #print the first 5 items
	>>> x[1::3]
	[1, 4, 7] #print items..x[1 + 0*3], x[1 + 1*3], x[1 + 2*3] ....
	>>> x[-1]
	8 #print the last item in x
	>>> x[:-1]
	[0, 1, 2, 3, 4, 5, 6, 7] #print items start from the head, x[0], to the one before x[-1] (excluding x[-1])
	#note that if you want to copy a list x to a new list y, using y = x doesn't perform the copy task, it is a reference
	#example of reference operation
	>>> x = [1, 2, 3, 4, 5]
	>>> y = x
	>>> y[1] = 7
	>>> x
	[1, 7, 3, 4, 5]
	#example of copy operation
	>>> y = x[:]
	>>> y[0] = 7
	>>> x
	[1, 7, 3, 4, 5] #not affected by the operation on y

view raw list.py hosted with ❤ by GitHub

[Python] strip vs. replace

Here are some examples showing the differences between the function of "strip()" and "replace()" in Python.

	#EX.1
	>>> x = "citt""
	>>> x.strip(""")
	'ci'
	>>> x.replace(""","")
	'citt'
	#EX.2
	>>> x = ""city" is"
	>>> x.strip(""")
	'city" is'
	>>> x.replace(""","")
	'city is'

view raw strip_replace.py hosted with ❤ by GitHub

Be careful! strip() will delete any character in the beginning or end of the word that matches "any" character in the word we put in the strip function. For example, we put "& quot;" as a parameter into strip(), then any character matches ('&', 'q', 'u', 'o', 't', ';') will be deleted. It doesn't have to be exactly the same "string."

Install CMake command line tools on MacOS

The following video is a good demonstration.

[Python] Find part of speech (POS) by using NLTK

	>>> import nltk
	#remember to use list
	>>> x = nltk.pos_tag(['New'])
	>>> print(x)
	[('New', 'NNP')]
	>>> print(x[0][1])
	NNP
	#if you don't use list, then the pos_tag function will find the pos for each character
	>>> x = nltk.pos_tag('New')
	>>> print(x)
	[('N', 'NNP'), ('e', 'VBP'), ('w', 'NN')]

view raw pos.py hosted with ❤ by GitHub

[Python] Performance enhance by not using range() for comparison

When I was trying to enhance the performance of my Python code, the frist thing I tried is reducing the usage of functions. After I substituted the range() function, the performance had improved a lot, from 404ms to 106ms. The following is the difference between the two versions of the "ZigZag Conversion" code.

404ms

	class Solution:
	# @return a string
	def convert(self, s, nRows):
	row_len = 2 * nRows - 2
	string = ["" for _ in range(nRows)]
	if nRows <= 2:
	for i in range(len(s)):
	string[i % nRows] += s[i]
	else:
	for i in range(len(s)):
	res = i % row_len
	if res in range(nRows, row_len):
	string[row_len - res] += s[i]
	else:
	string[res] += s[i]
	output = ""
	for i in range(nRows):
	output += string[i]
	return output

view raw ZigZag_404.py hosted with ❤ by GitHub

106ms

	class Solution:
	# @return a string
	def convert(self, s, nRows):
	row_len = 2 * nRows - 2
	string = ["" for _ in range(nRows)]
	if nRows <= 2:
	for i in range(len(s)):
	string[i % nRows] += s[i]
	else:
	for i in range(len(s)):
	res = i % row_len
	if res >= nRows and res < row_len: # I just replaced the range() function here
	string[row_len - res] += s[i]
	else:
	string[res] += s[i]
	output = ""
	for i in range(nRows):
	output += string[i]
	return output

view raw ZigZag_106.py hosted with ❤ by GitHub

Although the performance result depends on the test data, this substitution really improves the entire performance.

[Python] Find synonyms by using WordNet, NLTK

WordNet is a great reference when you want to find synonyms of a word. For more information, please see the website: http://wordnet.princeton.edu/

The following Python code is a simple example.

	#Python 3
	#Find synonyms of a word
	>>> from nltk.corpus import wordnet as wn
	>>> syn = wn.synsets("car")
	>>> syn
	[Synset('car.n.01'), Synset('car.n.02'), Synset('car.n.03'), Synset('car.n.04'), Synset('cable_car.n.01')]
	>>> syn[0].lemma_names()
	['car', 'auto', 'automobile', 'machine', 'motorcar']
	#Note that synonyms are always in singular form.
	#If you want to compare words, need to deal with the words that are in plural form.

view raw WordNet.py hosted with ❤ by GitHub

WordNet Interface: http://www.nltk.org/howto/wordnet.html

[Python] try... except

	#python
	try: #prevent unexpected crash
	ofile = open(filename)
	except:
	print "file cannot be openned:", filename
	exit()
	#read one line at a time
	for line in ofile:
	line = line.rstrip() #delete "/n" from original file line
	print line
	#read "whole" file at a time
	wholeFile = ofile.read()

view raw read_file.py hosted with ❤ by GitHub

Apply BFS to special MST problems

Given a undirected and connected graph, and the edges have weights of either 1 or 2 (actually it is also applicable for weight larger than 2). How can we find a MST of the graph by using BFS?

Ans (I guess):
First, we can view an edge with weight of 2 as two edges with weight of 1 connected together. Then we can simply apply BFS to solve this problem. After finish the BFS algorithm, we check all the edges originally with weight of 2. If both parts of an edge are not both chosen as a path of the MST, then we delete this path. Finally, the result is a MST.

Use pdb to debug Python code

use pdb to debug python code:

$ python -m pdb filename.py

view raw pdb hosted with ❤ by GitHub

some (not all) commands of pdb:
h: see the entire help list

l: l(ist) [first [,last]]

List source code for the current file.

Without arguments, list 11 lines around the current line or continue the previous listing.

With one argument, list 11 lines starting at that line.

With two arguments, list the given range; if the second argument is less than the first, it is a count. 

s:

Execute the current line, stop at the first possible occasion

(either in a function that is called or in the current function).

c: 
Continue execution, only stop when a breakpoint is encountered.

b: b(reak) ([file:]lineno | function) [, condition]

p: p expression

Print the value of the expression.

for more information: https://docs.python.org/2/library/pdb.html

The definitions of "Implication" and "Axiom"

Def: An "implication" p => q is true if p is false or q is true.

Truth table
p      q    p => q    q => p    p <=> q
-------------------------------------------
T     T       T            T               T
T     F       F            T               F
F     T       T            F               F
F     F       T            T               T

False implies everything is true
ex: pig flies => I'm a kind   True!


Think about the following statement:
"This statement is false"

If the statement is false, then the statement is true (which is a contradiction!)



Def: an "axiom" is a proposition that is assumed to be true.
ex: if a = b and b = c, then a = c

New life in Baltimore

My new life has started! 

This is the view from my apt.  

台北，咱們後會有期！

13多年前，我一個人到台北念書，畢業後就留在台北工作，

現在離開台北，真的非常不捨，許多的回憶、朋友都在這裡，我想有機會還是會回來的。

台北，咱們後會有期！

Stay tuned

英文書信中如果想要對方等候進一步的消息，可用"stay tuned"

Ex:

Thank you for your reply.

We will pass your image to our client for final approval. Once approved, we would like you to provide the original image to us.

For your information, the usage of the image will be valid online globally for 2 years, mainly on AAA’s website and Facebook. The starting period will start from January 20XX.

Please stay tuned!


Best Regards,
XXX

淺談雲端運算

2012年時，為了想讓人更容易了解什麼是雲端運算，於是就把當時我對雲端運算的了解整理並寫成文章 -「淺談雲端運算」，沒想到今天一看，國內居然有10篇論文引用它，真是欣慰，沒辦法對國內證券產業雲端服務應用盡到什麼心力，但還是能幫助國內學生快速了解雲端運算，也算是一點小小的貢獻吧。