---

layout: post
title: "MongoDB The Definitive Guide(1)"
category: Study Notes

tags: ["MongoDB", "读书"]

{% include JB/setup %}

Chapter 2 Getting Started

  1. A document is the basic unit of data for MongoDB, roughly equivalent to a row in a relational database management system.
  2. A collection can be thought of as the schema-free equivalent of a table.
  3. MongoDB comes with a simple but powerful Javascript shell.
  4. Every document has a special key, "_id", that is unique across the document's collection.

Document

  1. A document: an ordered set of keys with associated values.

Example:

    {"greeting" : "Hello world!"}
Several Important concepts
  • Key/ value pairs in documents are ordered - the following two documents are distinct from each other, example:

    {"greeting" : "Hello world!", "foo" : 3}
    {"foo" : 3, "greeting" : "Hello world!"}
    
  • Values in documents are not just "blobs". They can be one of several different data types.

Key naming exceptions
  • Keys must not contain the character \0 (the null character)
  • The . and $ characters have some special properties and should be used only in certain circumstances. In general, they should be considered reserved, and drivers will complain if they are used inappropriately.
  • Keys starting with _ should be considered reserved; although this is not strictly enforced.

These documents are distince:

    {"foo" : 3}
    {"foo" : "3"}

As are as these:

    {"foo" : 3}
    {"Foo" : 3}

MongoDB cannot contain duplicate keys.

Collections

  1. A collection is a group of documents. If a document is the MongoDB analog of a row in a relational database, then a collection can be thought of as the analog to a table.

  2. Collection are schema-free. This means that the documents within a single collection can have any number of different "shapes".

  3. Any document can be put into any collection.

  4. Why should we use more than one collection? There are several good reasons:

    • Keeping different kinds of documents in the same collection can be a nightmare for developers and admins.
    • It is much faster to get a list of collections than to extract a list of the types in a collection.
    • Grouping documents of the same kind together in the same collection allows for data locality. Getting several blog posts from a collection containing only posts will likely require fewer disk seeks than getting the same posts from a collection containing posts and author data.
    • By putting only documents of a single type into the same collection, we can index our collections more efficiently.
Collection naming exceptions

A collection is identified by its name. Collection names can be any UTF-8 string, with a few restrictions:

  • The empty string ("") is not a valid collection name.
  • Collection names may not contain the character \0 (the null character) because this delineates the end of a collection name.
  • You should not create any collections that start with 'system.', a prefix reserved for system collections.
  • User-created collections should not contain the reserved character \( in the name. You should not use \) in a name unless you are accessing one of these collections.

One convention for organizing collections is to use namespaced subcollections separeted by the . character.

Databases

  1. In addition to grouping documents by collection, MongoDB groups collections into databases.
  2. A single instance of MongoDB can host several databases, each of which can be thought as completely independent.
  3. Each database is stored in separate files on disk.
  4. A good rule of thumb is to store all data for a single application in the same database. Separate databases are useful when storing the data for several application or users on the same MongoDB server.
Database naming exceptions
  • The empty string ("") is not a valid database name.
  • Database names should be all lowercase.
  • Database names are limited to a maximum of 64 bytes.

Starting MongoDB

  1. mongod will use the default data directory, /data/db/ and port 27017
  2. mongod also sets up a very basic HTTP server that listens on a port 1000 higher than the main port. in this case 28017.
  3. You can safely stop mongod by typing Ctrl+c in the shell that is running the server.

    --- select which database to use---
    > use foobar
    Switched to db foobar
    
    --- look at the db variable---
    > db
    foobar
    ---
    

Basic operations with MongoDB Shell

  1. Create

    > post = {"title" : "My Blog Post",
    ... "content" : "Here's my blog post.", 
    ... "date" : new Date()}
    {
        "title" : "My Blog Post",
        "content" : "Here's my blog post.",
        "date" : "Sat Dec 12 2009 11:23:21 GMT-0500 (EST)"
    }
    
    > db.blog.insert(post)
    
    ---We can see it by calling find on the collection:
    > db.blog.find()
    {
        "_id" : ObjectId("4b23c3ca7525f35f94b60a2d"), 
        "title" : "My Blog Post",
        "content" : "Here's my blog post.",
        "date" : "Sat Dec 12 2009 11:23:21 GMT-0500 (EST)"
    }
    
  2. Read
    find returns all of the documents in a collection. If we just want to see one document from a collection, we can use findone:

    > db.blog.findOne()
    {
        "_id" : ObjectId("4b23c3ca7525f35f94b60a2d"),
        "title" : "My Blog Post",
        "content" : "Here's my blog post.",
        "date" : "Sat Dec 12 2009 11:23:21 GMT-0500 (EST)"
    }
    
  3. Update
    Update takes (at least) two parameters: the first is the criteria to find which document to update, and the second is the new document.
    Example:

    ---We decide to enable comments on the blog post
    
    ---The first step is to modify the variable post and add a "comments" key:
    > post.comments = []
    []
    ---Then we perform the update, replacing the post titled "My Blog Post" with our new version of the document:
    > db.blog.update({title : "My Blog Post"}, post)
    
    > db.blog.find()
    {
        "_id" : ObjectId("4b23c3ca7525f35f94b60a2d"), 
        "title" : "My Blog Post",
        "content" : "Here's my blog post.",
        "date" : "Sat Dec 12 2009 11:23:21 GMT-0500 (EST)" 
        "comments" : [ ]
    }
    
  4. Delete
    remove deletes documents permanently from the database. Called with no parameters, it removes all documents from a collection. It can also take a document specifying criteria for removal.

    > db.blog.remove({title : "My Blog Post"})
    
  5. Help
    Help for database-level commands is provided by db.help(); and help at the collections can be accessed with db.foo.help().

A good way fo figuring out what a function is doing is to type it without the parentheses. This will print the JavaScript source code for the function.

    ---if you needed to perform some operation on every blog subcollection, you could iterate through them with something like this:
    var collections = ["posts", "comments", "authors"];
    for (i in collections) {
        doStuff(db.blog[collections[i]]);
     }

    ---instead of this:
    doStuff(db.blog.posts); 
    doStuff(db.blog.comments); 
    doStuff(db.blog.authors);

Basic Data Types

  1. null: null can be used to represent both a null value and a nonexistent field:

    {"x" : null}
    
  2. boolean: There is a boolean type, which will be used for the values 'true' and 'false':

    {"x" : true}
    
  3. 64-bit floating point number: all numbers in the shell will be of this type. Thus, this will be a floating-point number:

    {"x" : 3.14}
    ---As will this:
    {"x" : 3}
    
  4. string: Any string of UTF-8 characters can be represented using the string type:

    {"x" : "foobar"}
    
  5. symbol: This type is not supported by the shell. If the shell gets a symbol from the database, it will convert it into a string.

  6. object id: An object id is a unique 12-byte ID for documents.

    {"x" : ObjectId()}
    
  7. date: Dates are stored as milliseconds since the epoch. The timezone is not stored:

    {"x" : new Date()}
    
  8. regular expression: documents can contain regular expressions.

    {"x" : /foobar/i}
    
  9. code: Documents can also contain JavaScript code:

    {"x" : function(){/*...*/}}
    
  10. undefined: undefined can be used in documents as well

    {"x" : undefined}
    
  11. array: sets or lists of values can be represented as arrays:

    {"x" : ["a", "b", "c"]}
    
  12. embedded document: documents can contain entire documents, embedded as values in a parent document:

    {"x" : {"foo" : "bar"}}
    

Numbers

  1. By default, any number in the shell is treated as a double by MongoDB. This means that if you retrieve a 4-byte integer from the database, manipulate its document, and save it back to the database even without changing the integer. the integer will be resaved as a floating-point number. Thus, it is generally a good idea not to overwrite entire documents from the shell.

  2. Another problem with every number being represented by a double is that there are some 8-byte integers that cannot be accurately represented by 8-byte floats.

Arrays

  1. Arrays are values that can be interchangeably used for both ordered operations and unordered operations.
  2. Arrays can contain different data types as value. In fact. array values can be any of the supported values for normal key/ value pairs, even nested arrays.

Embedded Documents

  1. Embedded documents are entire MongoDB documents that are used as the value of a key in another document.

    {
        "name" : "John Doe", 
        "address" : {
        "street" : "123 Park Street", 
        "city" : "Anytown",
        "state" : "NY"
        }
    }
    

    The value for the "address" key is another document with its own values for "street", "city" and "state".

MongoDB "understands" the structure of embedded documents and is able to "reach" inside" of them to build indexes, perform queries, or make updates.

Suppose "address" were a seperate table in a relational database and we needed to fix a typo in an address. When we did a join with "people" and "addresses", we'd get the updated address for everyone who shares it. With MongoDB, we'd need to fix the typo in each person's document.

id and ObjectIds

  1. Every documents stored in MongoDB must have an "_id" key. The "_id" key's value can be any type, but it defaults to an ObjectId.

If you have two collections, each one could have a document where the value for "_id" was 123. However, neither collection could contain more than one document where "_id" was 123.

ObejectId use 12 bytes of storage.

  • The first four bytes of an ObjectId are a timestamp in seconds since the epoch. The actual value of the timestamp doesn't matter, only that it is often new and increasing.
  • The next three bytes of an ObjectId are an unique identifier of the machine on which it was generated. This is usually a hash of the machine's hostname.
  • The next two bytes are taken from the process identifier(PID) of the ObjectId-generating process.

These first nine bytes of an ObjectId guarantee its uniqueness across machines and processes for a single second. The last three bytes are simply an incrementing counter that is responsible for uniqueness within a second in a single process.

REFERENCE

\K. Chodorow, M. Dirolf, "MongoDB: The Definitive Guide", O'Reilly, 2010, US

2015/10/25

---

layout: post
title: "High Availability for the Hadoop Distributed File System (HDFS)"
category: Reading Notes

tags: ["读文章", "HDFS", "分布式系统"]

{% include JB/setup %}

  1. HDFS, the Hadoop Distributed File System, is the primary storage system of Hadoop, and is responsible for storing and serving all data stored in Hadoop. MapReduce is a distributed processing framework designed to operate on data stored in HDFS.

  2. Despite very high level of reliability, HDFS has always had a well-known single point of failure which impacts HDFS's availability: the system relies on a single Name Node to coordinate access to the file system data.

  3. In the case of HBase, used to directly serve customer requests in real time. Adding high availability(HA) to the HDFS Name Node became one of the top priorities for the HDFS community.

  4. The remainder of this post discuss the implementation of a new feature for HDFS, called the "HA Name Node".

  5. The goal of the HA Name Node project is to add support for deploying two Name Nodes in an active/passive configuration. This is a common configuration for highly-available distributed systems, and HDFS's architecture lends itself well to this design. Even in a non-HA configuration, HDFS already requires both a Name Node and another node with similar hardware specs which performs checkpointing operations for the Name Node.

  6. The HDFS Name Node is primarily responsible for serving two types of file system metadata: file system namespace information and block locations.

  7. All mutations to the file system namespace, such as file renames, permission changes, file creations, block allocation, etc, are written to a persistent write-ahead log by the Name Node before returning success to a client call.

  8. In addition to this edit log, periodic checkpoints of the file system, called the fsimage, are also created and stored on-disk on the Name Node.

  9. Block locations are stored only in memory.

  10. The locations of all blocks are received via "block reports" sent from the Data Nodes when the Name Node is started.

  11. The goal of the HA Name Node is to provide a hot standby Name Node that can take over serving the role of the active Name Node with no downtime.

  12. Empirically, starting a Name Node from cold state can take tens of minutes to load the namespace information (fsimage and edit log)from disk, and up to an hour to receive the necessary block reports from all Data Nodes in a large cluster.

  13. To address the issue of sharing state between the active and standby Name Nodes, the HA Name Node feature allows for the configuration of a special shared edits directory. This directory should be available via a network file system, and should be read/write accessible from both Name Nodes. This directory is treated as being required by the active Name Node, meaning that success will not be returned to a client call unless the file system change has been written to the edit log in this directory. The standby Name Node polls the shared edits directory frequently, looking for new edits written by the active Name Node, and reads these edits into its own in-memory view of the file system state.

  14. The design of the HA Name Node is such that the passive Name Node is capable of performing this checkpointing role, thus requiring no additional Hadoop server machines beyond what HDFS already requires.

  15. Requiring a single shared edits directory does not necessarily imply a new single point of failure.

  16. The other part of keeping the standby Name Node hot is making sure that it has up-to-date block location information.

  17. Since block locations aren't written to the Name Node edit log, reading from the shared edits directory is not sufficient to share this file system metadata between the two Name Nodes. To address this issue, when HA is enabled, all Data Nodes in the cluster are configured with the network addresses of both Name Nodes.

2015/10/25

---

layout: post
title: "分布式计算开源框架Hadoop介绍"
category: Reading Notes

tags: ["读文章", "Hadoop", "分布式系统"]

{% include JB/setup %}

Hadoop框架最核心的设计就是:MapReduce和HDFS。

HDFS是分布式计算的存储基石。分布式文件系统基本的几个特点:

  1. 对于整个集群有单一的命名空间。
  2. 数据一致性。适合一次写入多次读取的模型,客户端在文件没有被成功创建之前无法看到文件的存在。
  3. 文件会被分割成多个文件块,每个文件块被分配存储到数据节点上,而且根据配置会由复制文件块来保证数据的安全性。

整个HDFS有三个重要角色:NameNode、DataNode和Client。

  • NameNode可以看作是分布式文件系统中的管理者,主要负责管理文件系统的命名空间、集群配置信息和存储块的复制等。

    NameNode会将文件系统的metadata存储在内存中,这些信息主要包括了文件信息、每一个文件对应的文件块的信息和每一个文件块在DataNode的信息等。

  • DataNode是文件存储的基本单元,它将Block存储在本地文件系统中,保存了Block的metadata,同时周期性地将所有存在的Block信息发送给NameNode。

  • Client就是需要获取分布式文件系统文件的应用程序。

文件写入:

1、 client向NameNode发起文件写入的请求。

2、 NameNode根据文件大小和文件块配置情况,返回给Client它所管理部分DataNode的信息

3、 Client将文件划分成多个Block,根据DataNode的地址信息,按顺序写入到每一个DataNode块中。

文件读取:

  1. Client向NameNode发起文件读取的请求。

2、 NameNode返回文件存储的DataNode的信息。

3、 Client读取文件信息。

文件Block复制:

1、 NameNode发现部分文件的Block不符合最小复制数或者部分DataNode失效。

2、 通知DataNode相互复制Block。

3、 DataNode开始直接相互复制。

HDFS的几个设计特点:

  1. Block的放置:默认不配置。一个Block会有三分备份,一份放在NameNode指定的DataNode另一份放在与指定DataNode非同一Rack上的DataNode,最后一份放在与指定DataNode同一Rack上的DataNode上
  2. 心跳检测DataNode的健康状态,如果发现问题就采取数据备份的方式来保证数据的安全性。
  3. 数据复制:使用HDFS的balancer命令,可以配置一个Threshold来平衡每一个DataNode磁盘利用率。执行balancer命令的时候,首先统计所有DataNode的磁盘利用率的均值,然后判断如果某一个DataNode的磁盘利用率超过这个均值Threshold以上,那么将会把这个DataNode的block转移到磁盘利用率低的DataNode,这对于新节点的加入来说十分有用
  4. 数据交验:在文件Block写入的时候除了写入数据还会写入交验信息,在读取的时候需要交验后再读入。
  5. NameNode是单点:如果失败的话,任务处理信息将会记录在本地文件系统和远端的文件系统中。
  6. 数据管道性的写入:当客户端要写入文件到DataNode上,首先客户端读取一个Block然后写到第一个DataNode上,然后有第一个DataBode传递到备份的DataNode上,一直到所有需要这个Block的DataNode都成功写入,客户端才会继续开始写下一个Block
  7. 安全模式:在分布式文件系统启动的时候,开始的时候会有安全模式,当分布式文件系统处于安全模式的情况下,文件系统中的内容不允许修改也不允许删除,直到安全模式结束。安全模式主要是为了系统启动的时候检查各个DataNode上数据块的有效性,同时根据策略必要的复制或者删除部分数据块。运行期通过命令也可以进入安全模式。

在Hadoop的系统中,会有一台Master,主要负责NameNode的工作以及JobTracker的工作。JobTracker的主要职责就是启动、跟踪和调度各个Slave的任务执行。还会有多台Slave,每一台Slave通常具有DataNode的功能并负责TaskTracker的工作。TaskTracker根据应用要求来结合本地数据执行Map任务以及Reduce任务。

就是在分布式处理中,移动数据的代价总是高于转移计算的代价。简单来说就是分而治之的工作,需要将数据也分而存储,本地任务处理本地数据然后归总,这样才会保证分布式计算的高效性。

REFERENCE

岑文初,"分布式计算开源框架Hadoop介绍"

2015/10/25

---

layout: post
title: "浅析爆库和社工库扫描"
category: Reading Notes

tags: ["读文章", "杂项"]

{% include JB/setup %}

1. 目前网站中主流存放用户名和密码有三种:

* 明文存放。这种网络的用户数据特别危险,网络被黑客拿下,用户数据直接拿走。
* 可逆加密存放。密码被加密一次存放在网站的数据库中,可逆加密也是非常危险的。
* 不可逆加密。密码通过MD5等不可逆加密算法加密后存放在网络数据库中,比上述2中密码加密方式安全。(如果md5加密后的密码泄露,明文密码仍有通过查找表的方式反查出来的可能)

2. 黑客针对后台数据库的入侵手法:

爆库,指将网站的数据库下载到本地。防爆库不仅仅是防止别人拿到你的库,还要做到让别人拿去也没用。

密码明文存储的,一定是死路;可逆加密的,只要黑客用点心,基本也不安全;

不可逆的,类似md5加密(md5虽然被证明可逆,但是逆向的成本很高,基本无人使用)应该很多人认为比较安全,但是遇到碰撞也很无奈。类似于碰撞库,其规模已经非常巨大,常规的密码的破解几率大于95%。

3. 如何规避这样的行为呢?

一种是两次md5或者多次md5等加密保存方法,这种方法很好的避免了加密后的密码在碰撞库中出现。但是如果黑客拿到的数据库足够大时,有心的黑客会发现加密逻辑,还是有可能破解的。

Discuz就在使用一种低成本的安全密码保护策略,他们使用随机salt二次加密。这种方法会使黑客破解密码的成本大大提高。黑客如果想破解这样的密码,需要对每一个用户建立一个碰撞库,时间成本很高。对于大量用户的破解,黑客一般是选择放弃,投入产出比太低;但是对于单个明确目标的话,他们还是会乐于尝试的。

4. 社工库

简单来说,你可能会在很多网站使用同样的邮箱和密码,社工库扫描就是利用这点,知道你一个账户后,就可以得到其他更有价值的账号。哪怕密码不相同,这个密码作为暴力破解中的关键字,也能大大提高破解率。

对于社工库扫描,网站通常的做法就是验证码(防暴力破解也是这种方法)。但是简单的验证码是很容易

而BT的验证码却会伤害用户体验,目前黑客也有应对BT验证码的方法。他们会将验证码集中传回到一个集中的地方,在网上请廉价的网络打工者填写。虽然这种方法成本有所提高,但是对于收益还是很低廉的。

2015/10/25

---

layout: post
title: "有关京东商城采用.NET架构的社区讨论"
category: Reading Notes

tags: ["读文章", "杂项"]

{% include JB/setup %}

观点:选择.NET不明智

廖雪峰:用.NET,意味着你被捆绑在Windows平台上。不是.NET效率本身比Java,PHP差,语言其实差距很小,

差距在于:

  1. Windows Server授权费太贵,Linux免费,如果你有上千台服务器需要买上千台Windows授权。
  2. Windows不但贵,性能还远不如Linux,这里说的是服务器端性能,跟桌面没有关系。
  3. 许多开源、高端服务器组件只有Linux/Unix版本,移植到Windows上的基本上是半残品。
  4. 许多优化技术、高性能分布式缓存、数据库、NoSQL解决方案等等,仅针对Linux。
  5. 需要的一切组件和技术几乎都可以在Linux平台上找到免费、稳定而且高性能的东西,如果是Windows平台,就不一定能做到。
  6. 在虚拟化的今天,一台高性能服务器可以跑十几台虚拟机,用Linux,你得到的是免费、稳定的虚拟机,用windows,一台服务器的授权费将乘以N。
2015/10/25